-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Go][Parquet] Panic when writing a ListOf(DeltaBinaryPacked field) with no data #39309
Comments
(In case this helps, I've found #37102 which is somewhat close but different, and that was fixed. Maybe the issue lies somewhere close) |
The issue is likely similar given the stack trace you're seeing.
That's absolutely the correct way to do it, with the exception that you're missing a I'll take a look and see if i can figure out the issue here. With some luck I'll have a fix before we do the code freeze for v15 |
@ewjoachim please have a look at the linked PR and verify that it fixes things for you. Thanks! |
If my understanding of Thanks a lot for your reactivity ! (At some point in the near future, I may submit other bugs I've found but I'll need to dive deeper and make sure the problem is real and comes from arrow. And even if we'll need to wait for the following release, it will not be that much of a problem) |
(as a follow-up, I cleared my stack of other issues and none of them lead to opening new issues here) |
I'll waiting for Matt to merge this patch |
…39347) ### Rationale for this change If using the DeltaBinaryPacked encoding, we end up with a nil pointer dereference if we end up with an empty column. ### What changes are included in this PR? Add a nil check in `EstimatedDataEncodedSize` for the base `deltaBitPackEncoder`. This should only ever occur if we have an empty column with this encoding when closing a row group. ### Are these changes tested? Yes a unit test was added to verify the fix. * Closes: #39309 Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…ked (apache#39347) ### Rationale for this change If using the DeltaBinaryPacked encoding, we end up with a nil pointer dereference if we end up with an empty column. ### What changes are included in this PR? Add a nil check in `EstimatedDataEncodedSize` for the base `deltaBitPackEncoder`. This should only ever occur if we have an empty column with this encoding when closing a row group. ### Are these changes tested? Yes a unit test was added to verify the fix. * Closes: apache#39309 Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…ked (apache#39347) ### Rationale for this change If using the DeltaBinaryPacked encoding, we end up with a nil pointer dereference if we end up with an empty column. ### What changes are included in this PR? Add a nil check in `EstimatedDataEncodedSize` for the base `deltaBitPackEncoder`. This should only ever occur if we have an empty column with this encoding when closing a row group. ### Are these changes tested? Yes a unit test was added to verify the fix. * Closes: apache#39309 Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
…ked (apache#39347) ### Rationale for this change If using the DeltaBinaryPacked encoding, we end up with a nil pointer dereference if we end up with an empty column. ### What changes are included in this PR? Add a nil check in `EstimatedDataEncodedSize` for the base `deltaBitPackEncoder`. This should only ever occur if we have an empty column with this encoding when closing a row group. ### Are these changes tested? Yes a unit test was added to verify the fix. * Closes: apache#39309 Authored-by: Matt Topol <zotthewizard@gmail.com> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Describe the bug, including details regarding any error messages, version, and platform.
It's been quite hard to understand how to write code for using arrow in go, but I believe I've managed to get it working. One of the thing that still stumbles me is the following scenario (I did my best to make it minimal, so it's a bit strange, but as long as you don't think I'm doing something that should be "forbidden" altogether, then consider it's the short form of my use case).
(I have no idea whether this is the right way to build a parquet with the lib, feel free to point me to better ways if there are)
So in human words: I'm creating an Arrow schema containing a list of uints. Then I write a single empty list. I indicate that parquet should write the uints as
DeltaBinaryPacked
. Finally, I write the parquet.Without the Encoding part, this works, but when I specify
DeltaBinaryPacked
, I get a panic.Failure is in BitWriter.Written, but the BitWriter pointer is nil.
Runs on
v14, supposedly on master, but I new enough in go that I'm not really sure it's really master and not the latest stable v14 tag,
go version go1.21.4 linux/amd64
,ubuntu 22.04 LTS
Component(s)
Go, Parquet
The text was updated successfully, but these errors were encountered: