Can't serialise documents that contain null values

[domrout]

Null value features in documents crash serialisation to JSON - this is a problem with any document that comes from ANNIE

Test added at gateplugin-python/java-test/gate/python/PythonPRTest.java

[domrout]

This issue also affects GATE core

[greenwoodma]

Well I guess it affects Format_Twitter rather than gate-core so I've opened an issue on that plugin

[ianroberts]

To be more precise, JSON does not allow null as a property name, so when you try build a JSON object to represent a Java Map that has null as one of its keys then the serialization fails. Null values are fine, it's null keys that are the problem, unfortunately one place where null commonly appears as a key is in the matches map document feature created by the orthomatcher.

An alternative JSON representation for feature maps could fix this, e.g. if we serialized a map as an array of key/value pairs instead of as a single JSON object, e.g.

[
  {"k":"key1", "v":"value1"},
  {"k":null, "v":"valueForNull"}
]

[greenwoodma]

I'm tempted to suggest that allowing null for a key is the real bug and we should change the orthomatcher (null is for the default set right) but I'm worried how many other things that might break

[johann-petrak]

I would agree that using null as key is the real bug. There is a reason why most default Java map implementations do not even support that. Null is already horrendous when used like it unfortunately must be used in Java, giving additional meaning to null is something we should get rid of rather than start working around even more.
But if we really need to work around this using standard json, why not represent a map as an array
with one or optionally two elements, the first being the actual map without the null key entry and the second the value of the null key, if any? That would be much less verbose and probably also a bit faster to read.

[greenwoodma]

turns out there is a bigger can of worms here than I imagined. I was wondering how much of an issue changing the orthomatcher to use the empty string instead of null would be -- easy as it turns out. What I discovered though is not only is null returned for the name of the default set, but null is also the name of any AnnotationSet that is a subset of another, so you can't even differentiate between, for example, a set containing all Token annotations and the default set. I think we should seriously consider finally giving the default set a name, even if that name is the empty string? As long as we still return the default set on a call to document.getAnnotations(null) I don't think this is actually too big a change, i.e. it shouldn't break too many things. We are not getting away from the original problem though so I might open another issue to discuss this further

[greenwoodma]

see GateNLP/gate-core#18 for the spun of issue

[ianroberts]

Alternative thought - we could use something like https://github.com/irmen/Pyrolite to use pickle (Python's native equivalent of Java object serialization) rather than JSON as the back and forth data format.

[johann-petrak]

Can somebody please provide more info when/how exactly we can get features with a null key and how to reproduce this?
I am closing this for now, please open a new issue with a concrete test case, if necessary.