Extension type not preserved on reading from the serialized schema

[candiduslynx]

Describe the bug, including details regarding any error messages, version, and platform.

I have an extension type json that is defined as an extension over binary.
When I serialize the schema I see that the field is correctly serialized as Extension(json, BINARY).

When reading the serialized schema back the filed type for json column becomes BINARY.

I suppose it's because https://github.com/apache/arrow/blame/main/java/format/src/main/java/org/apache/arrow/flatbuf/Type.java doesn't support reading extension types properly.

Component(s)

Java

[danepitkin]

Hi @candiduslynx , do you have a code snippet to reproduce the issue by chance?

The flatbuffer generated file you linked to is expected not to have the ExtensionType defined there. Instead, it is defined here

arrow/java/vector/src/main/codegen/templates/ArrowType.java

Line 284 in b0e1f74

public abstract static class ExtensionType extends ComplexType {

[candiduslynx]

@danepitkin you can take extension from https://github.com/cloudquery/plugin-sdk-java/blob/main/lib/src/main/java/io/cloudquery/types/JSONType.java
& the schema from https://github.com/cloudquery/plugin-sdk-java/blob/main/lib/src/test/java/io/cloudquery/helper/ArrowHelperTest.java#L125

Specifically, the testRoundTripTableEncoding test was failing without the following patch:
https://github.com/cloudquery/plugin-sdk-java/blob/main/lib/src/main/java/io/cloudquery/helper/ArrowHelper.java#L273-L284

[danepitkin]

By chance, is the test case not registering the JSON extension type e.g. https://github.com/search?q=repo%3Acloudquery%2Fplugin-sdk-java%20ExtensionTypeRegistry&type=code ?

I haven't actually tested anything yet, just looked through the code a bit so far.

[candiduslynx]

By chance, is the test case not registering the JSON extension type e.g. https://github.com/search?q=repo%3Acloudquery%2Fplugin-sdk-java%20ExtensionTypeRegistry&type=code ?

I haven't actually tested anything yet, just looked through the code a bit so far.

Seems like it.
Thanks for looking into it!

[danepitkin]

No problem! Glad I could help.