Serialisation fails when handling a single ContextProvider

[amedranogil]

While creating serialisation examples for REST users we need to serialise ContextProviders (as well as ServiceProfiles, ContextEvents, and ServiceCalls; although these seem to be fine).

take this example:
https://github.com/City4AgeProject/pilot-madrid/blob/master/city4age.madrid.ont/src/test/java/org/universAAL/ontology/test/SerializationTest.java

when executing testContextEvent() the produced error is:
java.lang.NullPointerException: null at org.universAAL.middleware.serialization.turtle.TurtleWriter.writeValue(TurtleWriter.java:989) at org.universAAL.middleware.serialization.turtle.TurtleWriter.handleStatement(TurtleWriter.java:540) at org.universAAL.middleware.serialization.turtle.TurtleWriter.handleResourceProps(TurtleWriter.java:511) at org.universAAL.middleware.serialization.turtle.TurtleWriter.writeResource(TurtleWriter.java:836) at org.universAAL.middleware.serialization.turtle.TurtleWriter.writeValue(TurtleWriter.java:928) at org.universAAL.middleware.serialization.turtle.TurtleWriter.writeValue(TurtleWriter.java:894) at org.universAAL.middleware.serialization.turtle.TurtleWriter.handleStatement(TurtleWriter.java:540) at org.universAAL.middleware.serialization.turtle.TurtleWriter.handleResourceProps(TurtleWriter.java:511) at org.universAAL.middleware.serialization.turtle.TurtleWriter.writeResource(TurtleWriter.java:800) at org.universAAL.middleware.serialization.turtle.TurtleWriter.serialize(TurtleWriter.java:649) at org.universAAL.middleware.serialization.turtle.TurtleWriter.serialize(TurtleWriter.java:178) at org.universAAL.middleware.serialization.turtle.TurtleSerializer.serialize(TurtleSerializer.java:52) at org.universAAL.ontology.test.SerializationTest.testContextProvider(SerializationTest.java:68)

[Alfiva]

I have been debugging and it seems to be caused by, for some reason, the ContextEventPattern to be serialized as part of the provider has rdf:type null. I still don't know why or how a ContextEventPattern could be of type null. It may be a problem in the class, or the ontology definition, or the way the serializer works to specialize it.

I was wondering if it could be related to using an empty CEP, which is what I use in my test, but I see yours does contain something.

[cstockloew]

I think you discovered something I marked as "potential bug", see issue M8 (although I didn't think it would materialize into an Exception..).

Do you really need to serialize the ContextProvider and not the ContextEventPattern (as opposed to ServiceProfile)? Because the serialization of patterns works..

There is a quick workaround for this case: go to class ContextProvider and change the method getPropSerializationType to always return PROP_SERIALIZATION_FULL. This should fix the concrete issue, although it does not solve the general problem, i.e. the definition and handling of serialization type. Thus, this problem may occur again for a different class (other than ContextProvider). Help from Saied is needed to clarify, e.g., what exactly "reduced form" means..

very small things:

• typo: "andorid" -> "android"
• there is no need to create a TurtleSerializer, BusTestCase already does this and provides methods serialize and deserialize

[amedranogil]

Thanks for the effort and the tips. on the second one, my intention is to go with junit removing bus.junit, and if you test it, the second test (creating the context event) fails because :
java.lang.SecurityException: AccessControl can only be initialized once at org.universAAL.middleware.bus.permission.AccessControl.init(AccessControl.java:75) at org.universAAL.middleware.bus.model.AbstractBus.initBrokerage(AbstractBus.java:123) at org.universAAL.middleware.bus.junit.BusTestCase.setUp(BusTestCase.java:236) at org.universAAL.ontology.test.SerializationTest.setUp(SerializationTest.java:48)
(not relevant)

@saiedt : please look into this issue when possible.

[Alfiva]

I had forgotten about all that "serialization type" thing, since I almost always use FULL.
Serialization options FULL and NEVER are self-explanatory. IIRC, the OPTIONAL means that it is only serialized when communicated within a node - while when coming from a different node it would not be serialized. I don't kow the innards of that functionality, but I guess the way it is used in the test it should be serialized, hence this would be a bug. That said, I would recommend using always FULL (and NEVER in very specific circumstances).

[cstockloew]

@amedranogil : I just committed a fix for the bus test case to avoid the SecurityException (not the most elegant way, but sufficient for tests)

@Alfiva : there is no serialization if the communication is within the same node. The serializer has no notion about where a message goes to or where it comes from (and serialization can also occur in different contexts other than bus communication).

There are 4 options: undefined, optional, reduce, and full. For me, this is not clear.

• Javadoc states: "when serializing an instance of this class in a minimized way" - but what is a "minimized way"? Is it not always the same? Where can you change this or set this?
• undefined: "it is undefined whether a property can be ignored" - what is the default policy?
• Same applies to optional: what is the default and how can this be changed? Is it really a decision of the serializer?
• What is the actual difference between undefined and optional?
• What is the "reduced form"?
  ...

I can see the benefit of this method, i.e. some properties are normally not needed for the typical usage of the middleware. Thus, the ContextProvider normally does not need to be fully serialized and getPropSerializationType should not return full.

However, it should be possible to overwrite this somehow to allow for different UCs as it is now needed by @amedranogil . My idea (see M7) was to change the serializer interface by adding options (e.g. as Dictionary):
public Object deserialize(String serialized, Dictionary options);
public String serialize(Object messageContent, Dictionary options);
In the options, you can, e.g. provide a rootURI, a different serialization type, etc. (maybe even some serializer-specific options, if you know what you are doing).

But to make these mods, the meaning of serialization type must be clearly defined.

[amedranogil]

Just realized what Serialization_Optional is for! This is for information that is represented in the ontology itself, so for example there shouldn't be any need to serialize enumeration instances; as these would be recognised by the remote peer when it has the same ontology installed. This means that there should be some type of protocol to know whether the serialization is needed.
Maybe this is as simple as adding an additional parameter with the list of "remote" ontologies to the serializer so that it skips those optional parameters with dose prefixes. The default would be an empty list which would result in allways serialize optional.
This could be used in different ways, for example the buses will assume other peers will have the standard list of ontologies (those used by the minimal instance of universAAL), there by skimming a little of the serialization; where a tool for extracting turtle serializations whould assume no ontologies (empty remote list), and therefore serialize everything (solving this issue). In the future when we have the system of ontology sharing amongst peers in the same uSpace, this system could also be used to manage the list of "remote" ontologies a bit more precisely.

[amedranogil]

According to Resource.getPropSerializationType() javadoc:

Answers if the given property has to be considered when serializing this individual in a minimized way, and if not ignore-able, whether its value should be presented in its full form or can be reduced. The return value must be one of PROP_SERIALIZATION_OPTIONAL, PROP_SERIALIZATION_REDUCED, or PROP_SERIALIZATION_FULL. It can be assumed that the given property is one of those returned by getPropertyURIs().
Decision criterion should be if the value of this property is absolutely necessary when this resource is being sent to a remote node. If the subclass rates it as unlikely that the receiver side would need this info, the answer should be PROP_SERIALIZATION_OPTIONAL in favor of lower communication traffic and higher performance even at risk of a possible additional query on the receiver side for fetching this info. With the same rationale, if a property should be included in the process of serialization, it is preferable to include it in a reduced form; in this case the return value should be PROP_SERIALIZATION_REDUCED, otherwise PROP_SERIALIZATION_FULL can be returned. Subclasses should normally overwrite this method as this default implementation returns always PROP_SERIALIZATION_FULL.

Thus, PROP_SERIALIZATION_OPTIONAL seems like it is ignored by default, assuming serialisation is used to transmit information to another node. There should be another logic which changes whether the property is serialised or not; some examples above (a switch to serialise all optionals, or a list of known ontologies on the remote side).

On the other side, Resource.PROP_SERIALIZATION_OPTIONAL Java doc says:

PROP_SERIALIZATION_OPTIONAL says that, when serializing an instance of this class in a minimized way, a property can be ignored.

If "in a minimized way" is referring to PROP_SERIALIZATION_REDUCED, then this means that PROP_SERIALIZATION_OPTIONAL should only be considered in the context when serialising in reduced form (which means that all references to a resource have PROP_SERIALIZATION_REDUCED as PropSerializationType Or it is whithin a subtree of a reduced resource).

This behaviour, although more complex, seems smarter for default behaviour.

[saiedt]

From the discussion, I do not understand if there is need for any changes or only explanation is needed. Therefore, I try here to provide just an explanation of the idea behind Resource#getPropSerializationType(String). Depending on reactions to that, we may decide about the action point.

First on the need for it: in IoT systems, the expectation is that the network traffic will be very high with the threat of becoming a bottleneck. To mitigate this, I think that the contribution expected from components that intend to send some data over network is that they try to be very economical with regard to the amount of data to be put on the network and avoid unnecessary volumes of data. As in universAAL, most of the data put on the network are serialized RDF graphs, we had to provide some methodology for minimized graph serialization. One option was that we define a convention, asking the developers to create "clean copies" of their RDF graphs (in the sense of avoiding data irrelevant for the potential receivers) when using them within messages sent to the buses. But this would have meant that the platform itself is not going to provide any suport for this purpose. On the other side, we concluded that for any mechanism from the universAAL side, we would need some hints from the developers, because we didn't see the capacity in our team to provide a fully automatic decision making which parts of data could be skipped.

Secondly on the chosen mechanism: Finally, the decision was to force only the developers of ontologies to provide such hints and let all the other developers to benefit from this mechanism automatically when they create their data based on those ontologies. The decision to be made by the ontology developers was to override Resource#getPropSerializationType(String) to provide hints on which properties are mandatory to be serialized (PROP_SERIALIZATION_FULL), which ones may be skipped (PROP_SERIALIZATION_OPTIONAL), and which ones may be reduced (PROP_SERIALIZATION_REDUCED). PROP_SERIALIZATION_UNDEFINED was thought as the default value returned by the Resource class so that the ontology developers are forced to override it but because in the young days of universAAL, there was no real life implementations, the default was finally set to FULL to make the life for the early developers easier.

Finally, the effect of the chosen mechanism: by default, the serialization of the root element in a given RDF graph starts without any skipping or reduction because this mechanism decided about the serialization of the "object" part of RDF statements and not about the subject. The root element, however, does not appear as object in any RDF statement. In other words, if we assume that the serialization of resources always happens either in a FULL or a REDUCED context, the initial context would be FULL. This means that all properties of the root element will be serialized, but in the serialization of each (prop, value) pair, if the prop can be serialized in a reduced way and the value is a resource, it will change the serialization context in that "next level" from FULL to REDUCED. The effect would be that all UNDEFINED and OPTIONAL props will be skipped and all REDUCED props bound to a resource will serialize the resource only by URI without further description.

The following image summarizes the expected serialization behaviour in a table:

[saiedt]

Two additional points:

1. I'm not sure how precisely this behaviour has been implemented in the serialization bundle, especially the one case with "only URI".
2. I opt for changing the behaviour in the FULL-UNDEFINED case to have a behaviour like in the REDUCED-UNDEFINED case. This way, there will be a real differentiation between UNDEFINED and OPTIONAL. In addition, I would also opt for changing the default in the resource class to UNDEFINED so that (a) we really force the ontology developers to make their decitions on this point and overwrite the default, and (b) cause that all non-ontological properties possibly added to resources have a local scope irrelevant for ontological communication with others.