Deserializing Measurement fails when using custom data types

[xelahalo]

When calling DataStreamService with the following request

{
    "__type": "dk.cachet.carp.data.infrastructure.DataStreamServiceRequest.AppendToDataStreams",
    "apiVersion": "1.1",
    "studyDeploymentId": "9fddfa03-cd51-4435-8c43-ada33cd6b7bf",
    "batch": [
        {
            "dataStream": {
                "studyDeploymentId": "9fddfa03-cd51-4435-8c43-ada33cd6b7bf",
                "deviceRoleName": "Primary Phone",
                "dataType": "dk.cachet.carp.coverage"
            },
            "firstSequenceId": 47,
            "measurements": [
                {
                    "sensorStartTime": 1680084517488488,
                    "data": {
                        "__type": "dk.cachet.carp.coverage",
                        "frequency": 5
                    }
                }
            ],
            "triggerIds": [
                0
            ]
        }
    ]
}

We get the following error response
Invalid name representation: expected a single lowercase alpha-numeric (underscore included) word.

When trying to investigate the issue further I found that this is because the measurements surrogate has dk.cachet.carp.common.infrastructure.serialization.UnknownPolymorphicSerializer<Data> and this is what we are trying to use as the type of the measurement. It fails is UnknownPolymorphicSerializer<Data> is not a valid name.

[bardram]

See also

• https://github.com/cph-cachet/carp.webservices-docker/issues/89

[bardram]

dk.cachet.carp.coverage is not a known type for carp core -- from CAMS we are uploading a LOT of data that is not modeled in carp core. BUT - according to @Whathecode, this should be possible -- and probably the reason why he is using this UnknownPolymorphicSerializer thing.

[Whathecode]

Could you attach a call stack?

[xelahalo]

Could you attach a call stack?

Sorry, I just got around to work on this again, here is the call stack:

Invalid name representation: expected a single lowercase alpha-numeric (underscore included) word. java.lang.IllegalArgumentException: Invalid name representation: expected a single lowercase alpha-numeric (underscore included) word. at dk.cachet.carp.common.application.NamespacedId.<init>(NamespacedId.kt:37) at dk.cachet.carp.common.application.NamespacedId$Companion.fromString(NamespacedId.kt:62) at dk.cachet.carp.data.infrastructure.SerializerDerivedMethodsKt.getDataType(SerializerDerivedMethods.kt:27) at dk.cachet.carp.data.application.MeasurementSerializer.deserialize(Measurement.kt:77) at dk.cachet.carp.data.application.MeasurementSerializer.deserialize(Measurement.kt:60) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:70) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:162) at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:533) at kotlinx.serialization.internal.CollectionLikeSerializer.readElement(CollectionSerializers.kt:80) at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51) at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36) at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:70) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:162) at dk.cachet.carp.data.application.DataStreamSequenceSerializer$DataStreamSequenceSnapshot$$serializer.deserialize(DataStreamSequence.kt:150) at dk.cachet.carp.data.application.DataStreamSequenceSerializer$DataStreamSequenceSnapshot$$serializer.deserialize(DataStreamSequence.kt:150) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:70) at dk.cachet.carp.data.application.DataStreamSequenceSerializer.deserialize(DataStreamSequence.kt:180) at dk.cachet.carp.data.application.DataStreamSequenceSerializer.deserialize(DataStreamSequence.kt:148) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:70) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:162) at kotlinx.serialization.encoding.CompositeDecoder$DefaultImpls.decodeSerializableElement$default(Decoding.kt:533) at kotlinx.serialization.internal.CollectionLikeSerializer.readElement(CollectionSerializers.kt:80) at kotlinx.serialization.internal.AbstractCollectionSerializer.readElement$default(CollectionSerializers.kt:51) at kotlinx.serialization.internal.AbstractCollectionSerializer.merge(CollectionSerializers.kt:36) at kotlinx.serialization.internal.AbstractCollectionSerializer.deserialize(CollectionSerializers.kt:43) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:70) at dk.cachet.carp.data.application.DataStreamBatchSerializer.deserialize(DataStreamBatch.kt:147) at dk.cachet.carp.data.application.DataStreamBatchSerializer.deserialize(DataStreamBatch.kt:135) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:70) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableValue(AbstractDecoder.kt:43) at kotlinx.serialization.encoding.AbstractDecoder.decodeSerializableElement(AbstractDecoder.kt:70) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableElement(StreamingJsonDecoder.kt:162) at dk.cachet.carp.data.infrastructure.DataStreamServiceRequest$AppendToDataStreams$$serializer.deserialize(DataStreamServiceRequest.kt:34) at dk.cachet.carp.data.infrastructure.DataStreamServiceRequest$AppendToDataStreams$$serializer.deserialize(DataStreamServiceRequest.kt:34) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:86) at dk.cachet.carp.common.infrastructure.serialization.NotSerializableKt$ignoreTypeParameters$1.deserialize(NotSerializable.kt:43) at dk.cachet.carp.data.infrastructure.DataStreamServiceRequest$Serializer.deserialize(DataStreamServiceRequest.kt) at dk.cachet.carp.data.infrastructure.DataStreamServiceRequest$Serializer.deserialize(DataStreamServiceRequest.kt:24) at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:70) at kotlinx.serialization.json.Json.decodeFromString(Json.kt:95) at dk.cachet.carp.data.infrastructure.DataStreamServiceRequestsTest.tryDeserialize(DataStreamServiceRequestsTest.kt:42) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84) at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:212) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:208) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.stop(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193) at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71) at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69) at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

[xelahalo]

Okay, so I managed to understand why the issue occurs, the reason NamespaceId was dk.cachet.carp.common.infrastructure.serialization.UnknownPolymorphicSerializer<Data> is because for CustomData we use UnknownPolymorphicSerializer which has a burnt-in descriptor and thus a serialName. Now the doc of serialName says the following:

"Serial name of the descriptor that identifies pair of the associated serializer and target class.
For generated serializers, serial name is equal to the corresponding class's fully-qualified name or, if overridden, SerialName. Custom serializers should provide a unique serial name that identify both the serializable class and the serializer itself, ignoring type arguments, if they are present."

Since UnknownPolymorphicSerializer is not a generated serializer, it hase a burnt in descriptor with a serialName (which is dk.cachet.carp.common.infrastructure.serialization.UnknownPolymorphicSerializer<${baseClass.simpleName}>). That is okay, but when we are trying to get DataType of a class, we actually request this exact serialName (which I don't know the reason for). In the case of generated serializers/types annotated with @SerialName this works fine, but not for CustomData. If I change this name in UnknownPolymorphicSerializer to the wrapper's fully qualified name, then the problem is solved, however I don't think it should be the solution. I'd rather we checked whether the serializer is generated and if not see if the class has a fully qualified name. But that makes me wonder, what the reason is for using the serializer to get that name. Can't we just infer it from KClass?