Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Go] firstTimeBitmapWriter.Finish() panics with list of 8 structs #33600

Closed
asfimport opened this issue Dec 16, 2022 · 0 comments · Fixed by #14989
Closed

[Go] firstTimeBitmapWriter.Finish() panics with list of 8 structs #33600

asfimport opened this issue Dec 16, 2022 · 0 comments · Fixed by #14989

Comments

@asfimport
Copy link
Collaborator

Even after ARROW-17169 I still get a panic at the same location.

Below is a test case that panics:

func (ps *ParquetIOTestSuite) TestStructWithListOf8Structs() {
	bldr := array.NewStructBuilder(memory.DefaultAllocator, arrow.StructOf(
		arrow.Field{
			Name: "l",
			Type: arrow.ListOf(arrow.StructOf(
				arrow.Field{Name: "a", Type: arrow.BinaryTypes.String},
			)),
		},
	))
	defer bldr.Release()

	lBldr := bldr.FieldBuilder(0).(*array.ListBuilder)
	stBldr := lBldr.ValueBuilder().(*array.StructBuilder)
	aBldr := stBldr.FieldBuilder(0).(*array.StringBuilder)

	bldr.AppendNull()
	bldr.Append(true)
	lBldr.Append(true)
	for i := 0; i < 8; i++ {
		stBldr.Append(true)
		aBldr.Append(strconv.Itoa(i))
	}

	arr := bldr.NewArray()
	defer arr.Release()

	field := arrow.Field{Name: "x", Type: arr.DataType(), Nullable: true}
	expected := array.NewTable(arrow.NewSchema([]arrow.Field{field}, nil),
		[]arrow.Column{*arrow.NewColumn(field, arrow.NewChunked(field.Type, []arrow.Array{arr}))}, -1)
	defer expected.Release()

	ps.roundTripTable(expected, false)
}

I've tried to trim down the input data and this is as minimal as I could get it. And yes:

  • wrapping struct with initial null is required
  • the inner list needs to contain 8 structs (or any multiple of 8)

Reporter: Min-Young Wu / @minyoung
Assignee: Min-Young Wu / @minyoung

PRs and other links:

Note: This issue was originally created as ARROW-18438. Please see the migration documentation for further details.

zeroshade pushed a commit that referenced this issue Jan 24, 2023
There is still a panic in go's bitmap writer, but specifically for when the list contains a multiple of 8 structs (the outer struct starting with null is also required).

Not sure where the off by one actually is, I somewhat blindly based this PR on #14183
* Closes: #33600

Authored-by: Min-Young Wu <minyoung@wandb.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
@zeroshade zeroshade added this to the 12.0.0 milestone Jan 24, 2023
minyoung added a commit to minyoung/arrow that referenced this issue Jan 24, 2023
There is still a panic in go's bitmap writer, but specifically for when the list contains a multiple of 8 structs (the outer struct starting with null is also required).

Not sure where the off by one actually is, I somewhat blindly based this PR on apache#14183
* Closes: apache#33600

Authored-by: Min-Young Wu <minyoung@wandb.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
westonpace pushed a commit to westonpace/arrow that referenced this issue Jan 25, 2023
There is still a panic in go's bitmap writer, but specifically for when the list contains a multiple of 8 structs (the outer struct starting with null is also required).

Not sure where the off by one actually is, I somewhat blindly based this PR on apache#14183
* Closes: apache#33600

Authored-by: Min-Young Wu <minyoung@wandb.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
sjperkins pushed a commit to sjperkins/arrow that referenced this issue Feb 10, 2023
There is still a panic in go's bitmap writer, but specifically for when the list contains a multiple of 8 structs (the outer struct starting with null is also required).

Not sure where the off by one actually is, I somewhat blindly based this PR on apache#14183
* Closes: apache#33600

Authored-by: Min-Young Wu <minyoung@wandb.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
minyoung added a commit to minyoung/arrow that referenced this issue Apr 14, 2023
There is still a panic in go's bitmap writer, but specifically for when the list contains a multiple of 8 structs (the outer struct starting with null is also required).

Not sure where the off by one actually is, I somewhat blindly based this PR on apache#14183
* Closes: apache#33600

Authored-by: Min-Young Wu <minyoung@wandb.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants