Unable to load shared library nironcompress (Macbook Pro Intel, Azure Function) #207
Comments
|
Do you mind attaching a sample project and instructions how to run it? Not sure I'm good with Azure Functions. |
|
Thanks for the quick response. No problem, here you go! You only need to add a parquet file called Run the function (vscode might be best for this, you might need the vscode Azure Functions extension for this), then hit the http endpoint that is outputted in the console with the tool of your choice (curl, Postman etc.). The exception occurs here: |
|
@aloneguid if there is anything I can do to help, let me know. Might need some directions in the solution though... |
|
@josdeweger just looked at the attached zip, and inside |
|
@aloneguid That's unexpected indeed. I just removed the bin folder and built the project again on my macbook, with the same result:
|
|
@aloneguid had a bit of time to look at this tonight. When I copy |
|
tried again with my new Macbook Pro with M1 processor, same issue. Had to copy the dylib file manually to the local/lib folder to make it work |
|
@aloneguid I had a similar issue where I could correct create a parquet file when testing my Azure Function locally, but found it failed once I deployed the function application to run in Azure. I was able to resolve my issue but tweaking the configuration of the Azure Function application in Azure, changing the Platform from 32 bit to 64 bit.
|
|
You can run any bitness or platform if data is uncompressed, gzip or brotli. Other codecs use native interop and hence call to native compression lib. Unfortunately those are not available in .net and I couldn't find any good implementations. It should be improved soon though. Optionally we can have a compression callback if you have preferred library available on your platform. |
|
Regarding windows, native compression does not support 32bit. Only 64 and arm. |
|
Please try with v4.2.3. If the worst case it should give a better error message. |
|
Any findings on this one? Seeing the same thing |
|
@aloneguid I gave v4.2.3 a try, unfortunately the problem was not resolved, but indeed the error message is a bit more clear: |
|
@josdeweger do you mind trying in a console project with your set up? Just want to rule out it's not Azure Function related issue. |
|
@aloneguid I just gave it a try in a console app (removed the dylib file from local/lib), and it works without issues. So it might be Azure Function related |
|
Thanks, that's interesting! I suspect Azure Functions is using a custom hosting runtime. I'll get a proper setup to investigate. |
|
awesome, let me know if I can be of any further assistence! |
|
Yep I encountered this afree updating to new versions also. I use Parquet.Net exclusively in serverless background processes that run as Azure Functions. I’ll try checking the bitness of my Function App as noted by @rustyswayne and report back if that resolves it… @josdeweger Thanks for posting this issue and all the details in the thread! Saved me a lot of time trying to sort out what this error was all about! 🙏 |
|
@aloneguid is doing all the work here, so kudo's to him! |
|
Woot! 🙌 Yes I confirmed that updating my Azure Function (
For anyone else looking for how to update your Function App, the setting is located in:
Edited - 01/22/2023: Initially I thought this was Linux deployment but it's actually a Windows AzFunc deployment. |
|
I was able to overcome this by copying the missing dll to the bin folder. However I am getting a differnt errorr now: "An attempt was made to load a program with an incorrect format". this is solved only if setting the target platform of the project as x64 which isn't an option in my case. any idea? |
|
@amnontopel are you writing or reading files from the function? |
|
Reading
בתאריך יום ו׳, 20 בינו׳ 2023, 14:10, מאת Ivan G ***@***.***>:
… @amnontopel <https://github.com/amnontopel> are you writing or reading
files from the function?
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEHUALLERP6ZPKTZAHSINLWTJ6BXANCNFSM6AAAAAAS7YCEVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
As a workaround today you can set compression level to Gzip or Brotli, those are not using native calls so it works on any platform. Ik working on bringing the rest to more platforms, including web assembly. |
|
Can you explain how i set this?
…On Sun, Jan 22, 2023 at 12:20 PM Ivan G ***@***.***> wrote:
As a workaround today you can set compression level to Gzip or Brotli,
those are not using native calls so it works on any platform. Ik working on
bringing the rest to more platforms, including web assembly.
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEHUAMABAYZDFHVG3EGUWDWTUCXZANCNFSM6AAAAAAS7YCEVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
|
|
I see. So for reading files (specifically with RLE encoding) I still need
to wait for your future release
…On Mon, Jan 23, 2023 at 11:41 AM Ivan G ***@***.***> wrote:
ParquetWriter has CompressionMethod property.
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEHUANZQYYYBAQ4HJT466TWTZG5JANCNFSM6AAAAAAS7YCEVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
No, encoding and compression are unrelated. |
|
Ok, but reading fails on ironcompress. Btw I had no issues with version 3.7
בתאריך יום ב׳, 23 בינו׳ 2023, 15:44, מאת Ivan G ***@***.***>:
… No, encoding and compression are unrelated.
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEHUAOQWBRQM5ZOIUQ4GFDWT2DMNANCNFSM6AAAAAAS7YCEVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
try with 4.3.4 |
|
Reading works!! thanks you
…On Mon, Jan 23, 2023 at 4:49 PM Ivan G ***@***.***> wrote:
try with 4.3.4
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEHUAPK6J2OYX44TALNT4DWT2K55ANCNFSM6AAAAAAS7YCEVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I see a new problem with writing files now where it fails to encode
non-string columns. I am basically writing data from DataTable Object to
parquet. It fails to encode DateTime, DatetimeOffset and int32 columns.
works if I string it.
…On Mon, Jan 23, 2023 at 4:49 PM Ivan G ***@***.***> wrote:
try with 4.3.4
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEHUAPK6J2OYX44TALNT4DWT2K55ANCNFSM6AAAAAAS7YCEVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Code and error please |
|
error on red line below: "failed to encode data"
public static async void CreateParquetFromDataTable(DataTable dt, string
OutputFilePath)
{
try
{
// var dt = GenerateTestData();
var fields = GenerateSchema(dt);
var schema = new ParquetSchema(fields);
if (File.Exists(OutputFilePath))
File.Delete(OutputFilePath);
//Stream fileStream =
System.IO.File.OpenWrite(OutputFilePath);
// Open the output file for writing
using (var stream = File.Open(OutputFilePath,
FileMode.Create, FileAccess.Write))
{
using (var writer = await
ParquetWriter.CreateAsync(schema, stream))
{
writer.CompressionMethod = CompressionMethod.Gzip;
var startRow = 0;
// Keep on creating row groups until we run out of
data
while (startRow < dt.Rows.Count)
{
using (var rgw = writer.CreateRowGroup())
{
// Data is written to the row group column
by column
for (var i = 0; i < dt.Columns.Count; i++)
{
var columnIndex = i;
// Determine the target data type for
the column
var targetType =
dt.Columns[columnIndex].DataType;
if (targetType == typeof(DateTime))
targetType = typeof(string);
// Generate the value type, this is to
ensure it can handle null values
var valueType = targetType.IsClass
? targetType
:
typeof(Nullable<>).MakeGenericType(targetType);
// Create a list to hold values of the
required type for the column
var list = (IList)typeof(List<>)
.MakeGenericType(valueType)
.GetConstructor(Type.EmptyTypes)
.Invoke(null);
// Get the data to be written to the
parquet stream
foreach (var row in
dt.AsEnumerable().Skip(startRow).Take(RowGroupSize))
{
// Check if value is null, if so
then add a null value
if (row[columnIndex] == null ||
row[columnIndex] == DBNull.Value)
{
list.Add(null);
}
else
{
// list.Add(row[columnIndex]);
// Add the value to the list,
but if it's a DateTime then create it as a DateTimeOffset first
list.Add(dt.Columns[columnIndex].DataType == typeof(DateTime)
?
row[columnIndex].ToString()
: row[columnIndex]);
}
}
// Copy the list values to an array of
the same type as the WriteColumn method expects
// and Array
var valuesArray =
Array.CreateInstance(valueType, list.Count);
list.CopyTo(valuesArray, 0);
// Write the column
//rgw.WriteColumn(new
Parquet.Data.DataColumn(fields[i], valuesArray));
var column = new
Parquet.Data.DataColumn(fields[i], valuesArray);
* await rgw.WriteColumnAsync(column); }*
}
startRow += RowGroupSize;
}
}
stream.Close();
}
}
catch (Exception e)
{
throw new Exception(e.Message);
}
}
…On Mon, Jan 23, 2023 at 9:39 PM Ivan G ***@***.***> wrote:
Code and error please
—
Reply to this email directly, view it on GitHub
<#207 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEHUAPVMOHKWMFFXYMJBL3WT3M67ANCNFSM6AAAAAAS7YCEVQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I can't run above as it's hard to guess out of context what need to be done. The exception indicates that data type is not supported, so I'd check what exactly you are passing to the library. It does support non-string types for sure, including DateTime and int32. Exception message can be improved of course. |
Version: Parquet.Net v4.1.2
Runtime Version: .NET 6
OS: MacOSX (Intel)
Expected behavior
Expected Parquet.NET to load all necessary dependencies
Actual behavior
I get the following error when running the function locally:
Steps to reproduce the behavior
Thank you for any help on this!
The text was updated successfully, but these errors were encountered: