This repository has been archived by the owner on Dec 20, 2018. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 310
In 1.8 generated fixed types do not extend GenericData.Fixed anymore #256
Closed
+2
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixed types generated with the 1.8 avro code generator do not extend from GenericData.Fixed anymore. This causes a ClassCastException when converting such a class to a GenericRow. This problem is solved by using the GenericFixed base type.
Codecov Report
@@ Coverage Diff @@
## branch-4.0 #256 +/- ##
===========================================
Coverage 90.71% 90.71%
===========================================
Files 5 5
Lines 334 334
Branches 50 50
===========================================
Hits 303 303
Misses 31 31 |
Hi @steven-aerts , is this patch compatible with AVRO 1.7? |
Yes it is, as GenericFixed is much older than 1.8. Btw spark-avro is still using avro 1.7.6 and everything still works after this patch. |
I take a quick try with py avro 1.8.2. And spark-avro can load the avro with fixed field. |
When you take the following schema: {
"type": "record",
"name": "Record",
"namespace": "org.example.bug",
"fields": [{"name": "macAddress", "type": {"type": "fixed", "size": 6, "name":"MacAddress" }}]
} And you generate the java code for it: java -jar /usr/localdisk/software/avro-tools/avro-tools-1.8.0.jar schema compile record.avsc src/main/java Then you can write a test like this: @Test
public void testBug256() {
Record record = Record.newBuilder().setMacAddress(new MacAddress(new byte[7])).build();
Schema schema = Record.getClassSchema();
StructType strucType = (StructType) SchemaConverters$.MODULE$.toSqlType(schema).dataType();
GenericRow test = (GenericRow) SchemaConverters$.MODULE$.createConverterToSQL(schema, strucType).apply(record);
} Which will fail with the stacktrace above. |
gengliangwang
pushed a commit
that referenced
this pull request
Nov 28, 2017
Fixed types generated with the 1.8 avro code generator do not extend from `GenericData.Fixed` anymore. This triggers a `ClassCastException` when converting an object from such a class to a `GenericRow`. ``` java.lang.ClassCastException: com.technicolor.avro.common.MacAddress cannot be cast to org.apache.avro.generic.GenericData$Fixed at com.databricks.spark.avro.SchemaConverters$$anonfun$com$databricks$spark$avro$SchemaConverters$$createConverter$1$3.apply(SchemaConverters.scala:160) at com.databricks.spark.avro.SchemaConverters$$anonfun$com$databricks$spark$avro$SchemaConverters$$createConverter$1$5.apply(SchemaConverters.scala:207) at com.databricks.spark.avro.SchemaConverters$$anonfun$com$databricks$spark$avro$SchemaConverters$$createConverter$1$6$$anonfun$apply$1.apply(SchemaConverters.scala:227) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at com.databricks.spark.avro.SchemaConverters$$anonfun$com$databricks$spark$avro$SchemaConverters$$createConverter$1$6.apply(SchemaConverters.scala:222) at com.databricks.spark.avro.SchemaConverters$$anonfun$com$databricks$spark$avro$SchemaConverters$$createConverter$1$5.apply(SchemaConverters.scala:207) at com.technicolor.doctor.hadoop.explore.SparkEventsEnvelope$AvroToRowConverter.call(SparkEventsEnvelope.java:249) ``` This problem is solved by using the GenericFixed base type. Author: Steven Aerts <steven.aerts@gmail.com> Closes #256 from steven-aerts/generic-fixed. (cherry picked from commit 55d4f08) Signed-off-by: Wang Gengliang <ltnwgl@gmail.com>
Thanks, merge to master/branch-4.0 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed types generated with the 1.8 avro code generator do not extend from
GenericData.Fixed
anymore.This triggers a
ClassCastException
when converting an object from such a class to aGenericRow
.This problem is solved by using the GenericFixed base type.