Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support nested collections of Java Bean classes in Spark #1021

Closed
jbaiera opened this issue Jul 19, 2017 · 0 comments
Closed

Support nested collections of Java Bean classes in Spark #1021

jbaiera opened this issue Jul 19, 2017 · 0 comments

Comments

@jbaiera
Copy link
Member

jbaiera commented Jul 19, 2017

Our Spark integration can handle JavaBean style classes. A corner case exists with that right now that keeps the serialization from succeeding:

  1. A java bean class at the root (call this BeanA.class)
  2. BeanA has a java.util.Map with a String key and a BeanB value type
  3. BeanB is a simple java bean

How this breaks:

  1. The ScalaValueWriter sees BeanA and converts it into a Scala Map. It passes this map to itself.
  2. The ScalaValueWriter sees the Map, and passes the java.util.Map to itself.
  3. The ScalaValueWriter cannot handle the java.util.Map and delegates that to the JavaValueWriter.
  4. The JavaValueWriter sees a map, and tries to serialize the first entry. It grabs the value from the current entry (a BeanB instance) and passes it to itself.
  5. The JavaValueWriter doesn't know how to handle a JavaBean object and throws an error

Instead how this should work is that at step 4, the JavaValueWriter should pass the bean instance to the ScalaValueWriter instead of itself. If at this point the ScalaValueWriter cannot handle a value it would then re-delegate to the JavaValueWriter after trying itself. This may mean taking a minor performance hit to guarantee all values supported by the Spark integration are accounted for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant