FlatBuffer Issue

[asheeshgarg]

Describe the bug, including details regarding any error messages, version, and platform.

org.slf4j log4j-over-slf4j org.apache.iceberg iceberg-common 1.0.0 org.apache.iceberg iceberg-core 1.0.0 org.apache.iceberg iceberg-data 1.0.0 org.apache.iceberg iceberg-arrow 1.0.0 org.apache.iceberg iceberg-hive-runtime 1.0.0 org.apache.iceberg iceberg-hive-metastore 1.0.0 org.apache.hive hive-metastore 3.1.3 org.apache.hadoop hadoop-common 3.3.1 org.apache.hive hive-common 2.3.9 org.apache.hive hive-storage-api 2.8.1 org.apache.thrift libthrift 0.13.0 org.apache.hadoop hadoop-mapreduce-client-core 3.3.4 org.apache.hadoop hadoop-aws 3.3.1 com.google.flatbuffers flatbuffers-java 1.12.0 org.apache.arrow arrow-vector 9.0.0 org.apache.arrow arrow-memory-netty 9.0.0 org.apache.arrow flight-core 9.0.0 org.apache.arrow flight-sql 9.0.0 org.apache.arrow flight-grpc 9.0.0 com.google.protobuf protobuf-java 3.21.8 commons-cli commons-cli 1.4

java.lang.NoSuchMethodError: com.google.flatbuffers.FlatBufferBuilder.createString(Ljava/lang/CharSequence;)I
at org.apache.arrow.vector.types.pojo.Field.getField (Field.java:189)
at org.apache.arrow.vector.types.pojo.Schema.getSchema (Schema.java:208)
at org.apache.arrow.vector.ipc.message.MessageSerializer.serializeMetadata (MessageSerializer.java:187)
at org.apache.arrow.flight.ArrowMessage. (ArrowMessage.java:152)
at org.apache.arrow.flight.DictionaryUtils.generateSchemaMessages (DictionaryUtils.java:64)
at org.apache.arrow.flight.OutboundStreamListenerImpl.start (OutboundStreamListenerImpl.java:61)
at org.apache.arrow.flight.OutboundStreamListener.start (OutboundStreamListener.java:67)
at org.apache.arrow.flight.FlightClient.startPut (FlightClient.java:242)
at org.apache.arrow.flight.FlightClient.startPut (FlightClient.java:225)
at com.icebergread.CookbookProducer.main (CookbookProducer.java:190)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
at java.lang.Thread.run (Thread.java:834)

Component

Java

[asheeshgarg]

While using the Example
try(VectorSchemaRoot vectorSchemaRoot = VectorSchemaRoot.create(schema, allocator);
VarCharVector varCharVector = (VarCharVector) vectorSchemaRoot.getVector("name")) {
varCharVector.allocateNew(3);
varCharVector.setSafe(0,"Ronald".getBytes("utf-8"));
varCharVector.setSafe(1, "David".getBytes("utf-8"));
varCharVector.setSafe(2, "Francisco".getBytes("utf-8"));
vectorSchemaRoot.setRowCount(3);
FlightClient.ClientStreamListener listener = flightClient.startPut(
FlightDescriptor.path("profiles"),
vectorSchemaRoot, new AsyncPutListener());
listener.putNext();
varCharVector.setSafe(0, "Manuel".getBytes());
varCharVector.setSafe(1, "Felipe".getBytes());
varCharVector.setSafe(2, "JJ".getBytes());
vectorSchemaRoot.setRowCount(3);
listener.putNext();
listener.completed();
listener.getResult();
System.out.println("C2: Client (Populate Data): Wrote 2 batches with 3 rows each");
}

[asheeshgarg]

Its causing at the time of put
FlightClient.ClientStreamListener listener = flightClient.startPut(
FlightDescriptor.path("profiles"),
vectorSchemaRoot, new AsyncPutListener());

Any pointers to resolve this?

[assignUser]

Could you please use code blocks to format your logs etc. so they are more readable. Could you please also list your java version.

I am not versed in java but the method does exists in 1.12.0 which you seem to have installed.

cc @lwhite1

[asheeshgarg]

@assignUser @lwhite1
Java version is 17.0.4

[asheeshgarg]

Any update on this I tried upgrading to 10.0.0 jars as well but still hitting the same issue

[lwhite1]

@asheeshgarg Sorry. Thurs/Friday was a major holiday in the US. I'm not familiar with the Flight code but I will take a look.

FWIW, the most common source of this issue when the method seems to exist is a difference between the version found at compile time and the one found at runtime. It might be the case that another library you're using loads a different version of flatbuffers. Could you check that?

[lwhite1]

Also, please take a look at https://stackoverflow.com/questions/69236212/java-lang-nosuchmethoderror-com-google-flatbuffers-flatbufferbuilder-createstri

[asheeshgarg]

@lwhite1 thanks for the pointer I removed all the dependency and just run simple flight example it worked. Now need to figure out which Jar is adding the different flat buffer version. Any way to find it easily or force to use the version of the flatbuffer needed by flight?

[lwhite1]

see the answer here for details, but you might try adding the needed flat buffer dependency directly to your project pom.

https://stackoverflow.com/questions/24962607/multiple-versions-of-the-same-dependency-in-maven

[asheeshgarg]

@lwhite1

org.apache.arrow
arrow-vector
9.0.0

Figure out this jar is causing the issue
Arrow vector also has reference to flatbuffer

[lwhite1]

Arrow vector uses the flat buffer version 1.12.0 (granted, I'm looking at arrow version 11-SNAPSHOT), as defined in the arrow root pom

<dep.fbs.version>1.12.0</dep.fbs.version>

Do you have another dependency somewhere on flatbuffers that is pointing to a different flatbuffer version?

[lwhite1]

FWIW, I see the reference in the arrow-format pom, which has no version defined, but that will just pick up the version defined in the parent pom (arrow-root) which uses the same vision as arrow-vector. I think we're looking for another project with a dependency on flatbuffers that is different from 1.12.0. (There are several newer and older versions in maven so it wouldn't be too surprising if there was a conflict.)

edit:
Also, I verified that arrow has been on 1.12.0 of flatbuffers since at least version 8.0

[asheeshgarg]

@lwhite1 its working for me know.