Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Go] Marshal break on map data type with slice #14780

Closed
forsaken628 opened this issue Nov 30, 2022 · 3 comments · Fixed by #14793
Closed

[Go] Marshal break on map data type with slice #14780

forsaken628 opened this issue Nov 30, 2022 · 3 comments · Fixed by #14793
Assignees
Milestone

Comments

@forsaken628
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

func TestName2(t *testing.T) {
	mem := memory.DefaultAllocator
	dt := arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String)
	schema := arrow.NewSchema([]arrow.Field{{
		Name: "map",
		Type: dt,
	}}, nil)

	arr, _, err := array.FromJSON(mem, dt, bytes.NewReader(
		[]byte(`
[[{"key":"index1","value":"main2"}]
,[{"key":"index3","value":"main4"},{"key":"tag_int","value":""}]
,[{"key":"index5","value":"main6"},{"key":"tag_int","value":""}]
,[{"key":"index6","value":"main7"},{"key":"tag_int","value":""}]
,[{"key":"index7","value":"main8"},{"key":"tag_int","value":""}]
,[{"key":"index8","value":"main9"}]
]`),
	))
	if err != nil {
		panic(err)
	}

	rec := array.NewRecord(schema, []arrow.Array{arr}, int64(arr.Len()))
	// rec2 := rec // ok
	// rec2 := rec.NewSlice(0, 2) // ok
	rec2 := rec.NewSlice(1, 2)

	var buf bytes.Buffer
	w := ipc.NewWriter(&buf, ipc.WithSchema(rec.Schema()))
	err = w.Write(rec2)
	if err != nil {
		panic(err)
	}
	err = w.Close()
	if err != nil {
		panic(err)
	}

	r, err := ipc.NewReader(&buf)
	if err != nil {
		panic(err)
	}

	r.Next()
	fmt.Println(r.Record())
	// record:
	//  schema:
	//  fields: 1
	//    - map: type=map<utf8, utf8>
	//  rows: 1
	//  col[0][map]: %!v(PANIC=String method: arrow/array: index out of range)
}

Component(s)

Go

@pitrou
Copy link
Member

pitrou commented Nov 30, 2022

cc @zeroshade

@zeroshade
Copy link
Member

Taking a look into this, will comment back soon.

@zeroshade
Copy link
Member

i found the issue which was the handling of the offsets for slices when building the IPC values. The attached PR fixes the issue and adds tests for it.

zeroshade added a commit that referenced this issue Dec 1, 2022
…14793)

* Closes: #14780

Authored-by: Matt Topol <zotthewizard@gmail.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
@zeroshade zeroshade added this to the 11.0.0 milestone Dec 1, 2022
zeroshade added a commit to zeroshade/arrow that referenced this issue Dec 15, 2022
…rrays (apache#14793)

* Closes: apache#14780

Authored-by: Matt Topol <zotthewizard@gmail.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
(cherry picked from commit 02bbcc5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants