New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to Serialize Guava Multimap in Avro #103

Closed
wbautistaEXPEDIA opened this Issue Aug 9, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@wbautistaEXPEDIA

wbautistaEXPEDIA commented Aug 9, 2017

Using jackson version 2.9.0

One of our classes utilizes Guava's Multimap. It is a Multimap<String, String>. When attempting to write our object in Avro, the following exception is thrown:

com.fasterxml.jackson.databind.JsonMappingException: Not an array schema: "string" (through reference chain: com.company.www.app.domain.model.OurObject["ourMultimap"])

	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:391)
	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
	at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:727)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
	at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
	at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1120)
	at com.fasterxml.jackson.databind.ObjectWriter.writeValueAsBytes(ObjectWriter.java:1017)
	at com.company.www.app.domain.model.OurObjectMarshallerTest.testAvroSerialization(OurObjectMarshallerTest.java:277)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: org.apache.avro.AvroRuntimeException: Not an array schema: "string"
	at org.apache.avro.generic.GenericData$Array.<init>(GenericData.java:241)
	at com.fasterxml.jackson.dataformat.avro.ser.AvroWriteContext._createArray(AvroWriteContext.java:184)
	at com.fasterxml.jackson.dataformat.avro.ser.MapWriteContext.createChildArrayContext(MapWriteContext.java:40)
	at com.fasterxml.jackson.dataformat.avro.AvroGenerator.writeStartArray(AvroGenerator.java:364)
	at com.fasterxml.jackson.datatype.guava.ser.MultimapSerializer.serializeFields(MultimapSerializer.java:327)
	at com.fasterxml.jackson.datatype.guava.ser.MultimapSerializer.serialize(MultimapSerializer.java:279)
	at com.fasterxml.jackson.datatype.guava.ser.MultimapSerializer.serialize(MultimapSerializer.java:39)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	... 40 more

This is how we are are setting up the writer:

final AvroSchemaGenerator generator = new AvroSchemaGenerator();
final ObjectMapper avroMapper = new AvroMapper()
        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
        .registerModule(new GuavaModule())
        .registerModule(new AvroModule());
avroMapper.acceptJsonFormatVisitor(OurObject.class, generator);
final AvroSchema schema = generator.getGeneratedSchema();
avroWriter = avroMapper.writer(schema);
...
final byte[] avro = avroWriter.writeValueAsBytes(ourObject);

And the generated Avro schema looks something like:

{
  "type":"record",
  "name":"OurObject",
  "namespace":"com.company.www.app.domain.model",
  "fields":[
    {
      "name":"ourMultimap",
      "type":[
        "null",
        {
          "type":"map",
          "values":"string",
          "java-class":"com.google.common.collect.Multimap"
        }
      ]
    }
  ]
}

And the class would look something like:

public class OurObject {
    public Multimap<String, String> ourMultimap;

    public Multimap<String, String> getOurMultimap() {
        return ourMultimap;
    }

    public void setOurMultimap(Multimap<String, String> ourMultimap) {
        this.ourMultimap = ourMultimap;
    }

    public Collection<String> getOurMultimapValue(String ourMultimapKey) {
        return this.ourMultimap.get(ourMultimapKey);
    }
}

It would appear that these two libraries (jackson-dataformat-avro & jackson-datatype-guava), or more specifically, AvroSchemaGenerator & GuavaModule, don't play nice together. Unless, I'm totally messing things up.

@cowtowncoder

This comment has been minimized.

Show comment
Hide comment
@cowtowncoder

cowtowncoder Aug 9, 2017

Member

Thank you for reporting this. I suspect problem comes from Guava module declaring content model that differs from actual serialization: that is, it claims to serialize as simple string, but that can not be: it produces something like Map<String,String[]>.
If so it'd be Guava problem -- but I'll first look into this to see if that is true.

Member

cowtowncoder commented Aug 9, 2017

Thank you for reporting this. I suspect problem comes from Guava module declaring content model that differs from actual serialization: that is, it claims to serialize as simple string, but that can not be: it produces something like Map<String,String[]>.
If so it'd be Guava problem -- but I'll first look into this to see if that is true.

@cowtowncoder

This comment has been minimized.

Show comment
Hide comment
@cowtowncoder

cowtowncoder Aug 11, 2017

Member

Doh! At this thought serializer implemented discovery properly, but just realized that code SHOULD differ from standard MapSerializer to account for that intermediate Array node...

Member

cowtowncoder commented Aug 11, 2017

Doh! At this thought serializer implemented discovery properly, but just realized that code SHOULD differ from standard MapSerializer to account for that intermediate Array node...

@cowtowncoder

This comment has been minimized.

Show comment
Hide comment
@cowtowncoder

cowtowncoder Aug 11, 2017

Member

Reproducing the issue can be done easily enough (although due to deps, need to stash in jackson-compat-minor or such). But fixing gets bit involved because there's no good way to indicate wrapping as is: rather, and intermediate JsonSerializer (or its equivalent) is needed.

Member

cowtowncoder commented Aug 11, 2017

Reproducing the issue can be done easily enough (although due to deps, need to stash in jackson-compat-minor or such). But fixing gets bit involved because there's no good way to indicate wrapping as is: rather, and intermediate JsonSerializer (or its equivalent) is needed.

@cowtowncoder

This comment has been minimized.

Show comment
Hide comment
@cowtowncoder

cowtowncoder Aug 11, 2017

Member

Done -- this will be fixed via

FasterXML/jackson-datatypes-collections#19

which will be included in 2.8.10 and 2.9.1 once those are released.
In the meantime may want to use snapshot builds or local builds.

Member

cowtowncoder commented Aug 11, 2017

Done -- this will be fixed via

FasterXML/jackson-datatypes-collections#19

which will be included in 2.8.10 and 2.9.1 once those are released.
In the meantime may want to use snapshot builds or local builds.

@cowtowncoder cowtowncoder modified the milestones: 2.9.0, 2.9.1 Aug 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment