Nested List and Map in Scala module

[JamiePullar]

I was suggested to raise this as an issue.

I wish to convert lists of json objects, which in themselves may contain Json objects, or Lists of Json objects and would like the results to be Scala Maps or Lists as appropriate, however, I can only get the top level object to convert to a Map or a List. Is there a simple way to do this?

For example in REPL

objectMapper.readValue("{"test":"113123","myList":[{"test2":"321323"},{"test3":"11122"}]}", classOf[Map[String,Any]])
Will return

res: Map[String,Any] = Map(test -> 113123, myList -> [{test2=321323}, {test3=11122}])
Where I would like

res: Map[String,Any] = Map(test -> 113123, myList -> List(Map(test2 -> 321323), Map(test3 -> 111222)))

[christophercurrie]

Background

This is happening because Jackson doesn't have any information about what types you want. So it uses default representations to provide a best-effort mapping with the most functionality it can.

scala>  val map1 = objectMapper.readValue("""{"test":"113123","myList":[{"test2":"321323"},{"test3":"11122"}]}""", classOf[Map[String,Any]])
map1: Map[String,Any] = Map(test -> 113123, myList -> [{test2=321323}, {test3=11122}])

scala> map1("myList").getClass
res4: java.lang.Class[_] = class java.util.ArrayList

scala>  val list = map1("myList").asInstanceOf[java.util.ArrayList[Any]]
list: java.util.ArrayList[Any] = [{test2=321323}, {test3=11122}]

scala>  list.get(0).getClass
res6: java.lang.Class[_] = class java.util.LinkedHashMap

scala>  val map2 = list.get(0).asInstanceOf[java.util.LinkedHashMap[String, Any]]
map2: java.util.LinkedHashMap[String,Any] = {test2=321323}

Core Jackson is selecting the types here, not the Scala module, specifically through a class called UntypedObjectDeserializer.

Workarounds

Use JavaConverters

Without tweaking your Jackson configuration, you can always add another layer of conversion when you type match the list:

scala> import scala.collection.JavaConverters._
import scala.collection.JavaConverters._

scala>  val list = map1("myList") match { case list: java.util.ArrayList[_] => list.asScala }
list: scala.collection.mutable.Buffer[_] = Buffer({test2=321323}, {test3=11122})

scala>  val map2 = list(0) match { case map: java.util.Map[_, _] => map.asScala }
map2: scala.collection.mutable.Map[_, _] = Map(test2 -> 321323)

Set USE_JAVA_ARRAY_FOR_JSON_ARRAY

You can alter your deserialization configuration to tell Jackson to use Array[Object] instead of java.util.ArrayList[Object], which although it's not a List, would at least provide IndexedSeq semantics for the result.

scala>  import org.codehaus.jackson.map.DeserializationConfig
import org.codehaus.jackson.map.DeserializationConfig

scala>  objectMapper.configure(DeserializationConfig.Feature.USE_JAVA_ARRAY_FOR_JSON_ARRAY, true)
res8: org.codehaus.jackson.map.ObjectMapper = org.codehaus.jackson.map.ObjectMapper@54ede19e

scala>  val map1 = objectMapper.readValue("""{"test":"113123","myList":[{"test2":"321323"},{"test3":"11122"}]}""", classOf[Map[String,Any]])
map1: scala.collection.immutable.Map[String, Any] = Map(test -> 113123, myList -> Array({test2=321323}, {test3=11122}))

Declare a case class to use as a value mapping

The most robust solution in the current codebase is to declare a case class that maps your values to the types you want. This gives Jackson the opportunity to directly construct the types of your data.

scala>  case class Foo(test: String, myList: List[Map[String, String]])
defined class Foo

scala>  val foo = objectMapper.readValue("""{"test":"113123","myList":[{"test2":"321323"},{"test3":"11122"}]}""", classOf[Foo])
foo: Foo = Foo(113123,List(Map(test2 -> 321323), Map(test3 -> 11122)))

scala>  foo.test
res10: String = 113123

scala>  foo.myList
res11: List[Map[String,String]] = List(Map(test2 -> 321323), Map(test3 -> 11122))

Solutions

If you have a strong use case for why you'd want to change the default mapping used by UntypedObjectDeserializer, a future version of the Scala module could implement an AnyDeserializer that would take its place. This wouldn't be terribly hard to implement and would probably add value, even if it's just for querying with the REPL. I'd be interested in hearing any other specific examples for how this might help you.

[cowtowncoder]

I think the idea is exactly to use one of Scala List/Map types for contained JSON arrays, lists, but I only now realized that since use of Scala module does not necessarily imply all-Scala, it can't by default change mapping that is used by UntypedObjectDeserializer.

However, maybe I should think of ways to make behavior of UntypedObjectDeserializer configurable, so that it would be possible to force use of Scala types (or any other objects).

[cowtowncoder]

Chris, thinking about this bit more, perhaps there should just be an option to override UntypedObjectDeserializer to use Scala Lists and Maps? This is legal (they are Objects after all). And should be easy to configure as well; just need a setting in module, and it could register handler for Object.class.

[christophercurrie]

That's probably what I'll end up doing, and probably as the default. Most users are going to want to minimize their exposure to Java standard types anyway, and it looks like UntypedObjectDeserializer will be straightforward to derive from.

[cowtowncoder]

Makes sense to me.

[christophercurrie]

@JamiePullar, a follow-up question: is your desire to just see the returned types be Scala types, or do you want specific control over what the default types should be? In other words, for JSON arrays, must it be a List, or would any Seq be OK?

[JamiePullar]

In my particular case I needed a List, however I would think a seq could be made to work. Specific control does sound cool though..

[christophercurrie]

For efficiency reasons, I've implemented this using JavaConverters under the hood. Since the Jackson default is an ArrayList, the conversion selects mutable.Buffer as the closest match. This is in the recently released version 1.9.3 (see a369660).

The code also respects the USE_JAVA_ARRAY_FOR_JSON_ARRAY deserialization feature, giving you three different options (you can revert to the original ArrayList behavior by composing a custom Jackson module that omits the UntypedObjectDeserializerModule), which I think at this point should be sufficient. If you need more specific mappings then I would recommend using case classes.

[JamiePullar]

I have finally had the chance to implement these changes, I got a nearly
1/3 performance increase, and used a custom Deserializer:

private class NestedTypeObjectDeserializer extends
JacksonUntypedObjectDeserializer {

override def mapArray(jp: JsonParser, ctxt: DeserializationContext):
AnyRef =
super.mapArray(jp, ctxt).asInstanceOf[ArrayList[AnyRef]].asScala.toList

override def mapObject(jp: JsonParser, ctxt: DeserializationContext):
AnyRef =
super.mapObject(jp, ctxt).asInstanceOf[LinkedHashMap[String,
AnyRef]].asScala.toMap
}

Thank you very much for your time on this.

On Thu, Mar 8, 2012 at 4:09 PM, Christopher Currie <
reply@reply.github.com

wrote:

For efficiency reasons, I've implemented this using JavaConverters under
the hood. Since the Jackson default is an ArrayList, the conversion
selects mutable.Buffer as the closest match. This is in the recently
released version 1.9.3 (see a369660).

The code also respects the USE_JAVA_ARRAY_FOR_JSON_ARRAY deserialization
feature, giving you three different options (you can revert to the original
ArrayList behavior by composing a custom Jackson module that omits the
UntypedObjectDeserializerModule), which I think at this point should be
sufficient. If you need more specific mappings then I would recommend using
case classes.

---

Reply to this email directly or view it on GitHub:

#12 (comment)