CBOR Feature Drop: COSE #2412

JesusMcCloud · 2023-08-18T11:18:06Z

This PR obsoletes #2371 and #2359 as it contains the features of both PRs and many more.

Specifically, this PR contains all feature required to serialize and parse COSE-compliant CBOR (thanks to @nodh). While some canonicalization steps (such as sorting keys) still needs to be performed manually. It does get the job done quite well. Namely, we have successfully used the features introduced here to create and validate ISO/IEC 18013-5:2021-compliant mobile driving license data.

This PR introduces the following features to the CBOR format:

Serial Labels
Tagging of keys and values
Definite length encoding (this is the largest change, as it effectively makes the cbor encoder two-pass)
Globally preferring major type 2 for byte array encoding
Encodes null complex objects as empty map (to be COSE-compliant)

sandwwraith · 2023-08-21T12:54:59Z

Hi, thanks for your PR! Just as a quick heads-up so you know I've seen it — I'm going on vacation soon, so I'll be able to properly review it in September.

sandwwraith

Hi, I appreciate the amount of work you've done — especially for the tests. I didn't look into the implementation (yet), focusing on surface API for now.

Regarding failure on CI — it's connected to the fact that Long.toUnsignedString was added only in Android API 26. However, it is possible to use this declaration on Android with desugaring enabled, so there shouldn't be any problems. You have to crate a special annotation @SuppressAnimalSniffer (it already exists for core and json, but not for cbor, see e.g. here:

kotlinx.serialization/formats/json/commonMain/src/kotlinx/serialization/json/JsonElement.kt

Line 92 in d192d24

@SuppressAnimalSniffer // Long.toUnsignedString(long)

) and use it in configuration here:

kotlinx.serialization/build.gradle

Line 205 in a87b0f1

switch (name) {

formats/cbor/commonMain/src/kotlinx/serialization/cbor/ByteStringWrapper.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/SerialLabel.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborArray.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Tags.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

sandwwraith

I didn't look at the test thoroughly (to understand if there is a need for more test cases), but I highlighted all of the problematic moments from my perspective. Don't forget to press 'show hidden conversations' on Github :)

sandwwraith · 2023-12-11T16:20:51Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/ByteStringWrapper.kt

+ * Note that `equals()` and `hashCode()` only use `value`, not `serialized`.
+ */
+@Serializable(with = ByteStringWrapperSerializer::class)
+public class ByteStringWrapper<T>(


NB: it seems you need to update apiDump since this class is no longer data

sorry, done!

sandwwraith · 2023-12-11T16:30:45Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/ByteStringWrapper.kt

+@Serializable(with = ByteStringWrapperSerializer::class)
+public class ByteStringWrapper<T>(
+    public val value: T,
+    public val serialized: ByteArray = byteArrayOf()


I've skimmed through the diff and can't find any usages of this value. Is there any reason to be a val? In what cases will the user be interested in accessing it? Also, it is very easy to create inconsistent data with this approach (e.g. ByteStringWrapper(original.value, byteArrayOf(garbage))). If it is not necessary to have it, this class can be a value class. Or even no class would be needed, as this can be handled with yet another @SerialInfo annotation.

@nodh can you take care of this, please?

We'll use the CBOR serialization for ISO 18013-5 compliaent mobile driving licences. There verifiers need to check that (the digest of) the serialized byte values appear in another structure (signed by the issuer). So in that case its quite useful to have the bytes as they were serialized (and parsed) in the deserialized ByteStringWrapper object too.

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

sandwwraith · 2023-12-11T17:48:23Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborArray.kt

+@SerialInfo
+@Target(AnnotationTarget.CLASS)
+@ExperimentalSerializationApi
+public annotation class CborArray(@OptIn(ExperimentalUnsignedTypes::class) vararg val tag: ULong)


vararg val tag is still undocumented though

sorry! done!

sandwwraith · 2023-12-12T17:38:22Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

            readProperties++
-            descriptor.getElementIndexOrThrow(elemName)
+            descriptor.getElementIndexOrThrow(elemName).also { index ->
+                if (cbor.verifyKeyTags) {


This block of duplicated code can be extracted to function (perhaps local)

sandwwraith · 2023-12-12T17:40:32Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

                } else {
+                    if (cbor.verifyKeyTags) {
+                        descriptor.getKeyTags(index)?.let { keyTags ->
+                            if (!(keyTags contentEquals tags)) throw CborDecodingException("CBOR tags $tags do not match declared tags $keyTags")


General observation (for all throw expressions): It is would be much easier to debug errors if they also included locations. Here we don't have a json path analog, but appending descriptor.toString() will already make the job easier significantly.

done where we have access to the descriptor (which we do not always have)

sandwwraith · 2023-12-12T17:52:18Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

            readByte()
        }
+
+        return (if (collectedTags.isEmpty()) null else collectedTags.toULongArray()).also { collected ->


IMO also creates unnecessary nesting here.

it needs to be transformed into an ULongArray for comparison and for returning it. the alternative of adding a temp variable just for two accesses seems a bit cumbersome to me

sandwwraith · 2023-12-12T17:54:25Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

+
+        return (if (collectedTags.isEmpty()) null else collectedTags.toULongArray()).also { collected ->
+            tags?.let {
+                if (!it.contentEquals(collected)) throw CborDecodingException(


Note that contentEquals has both nullable receiver and parameter. It results in behavior like this: intArrayOf(1,2,3).contentEquals(null) -> false. I'm not sure it is what you intended to have here.

We only want to compare if tags are actually set. Otherwise, we don't care.

I'll also add that comment to the code.

sandwwraith · 2023-12-12T17:54:53Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Encoding.kt

@@ -633,9 +952,55 @@ private fun Iterable<ByteArray>.flatten(): ByteArray {

 @OptIn(ExperimentalSerializationApi::class)
 private fun SerialDescriptor.isByteString(index: Int): Boolean {
-    return getElementAnnotations(index).find { it is ByteString } != null
+    return kotlin.runCatching { getElementAnnotations(index).find { it is ByteString } != null }.getOrDefault(false)


Why runCatching is needed here?

it's only a theoretical possibility, but an IndexOutOfBoundsException could be thrown. I'll remove it.

well, such a call fails for the WASM target, even though we use runCatching. I have no explanation

Can you elaborate on the problem? I do not see any related WASM failures

it was an implementation bug on my end, that has ben resolved in the meantime. I removed the runCatching in a1a7dad

develar · 2023-12-17T08:54:22Z

Globally preferring major type 2 for byte array encoding

It would be awesome, as currently it's somewhat surprising that the default encoded size of byte arrays is nearly twice as large.

JesusMcCloud · 2024-01-12T10:59:11Z

WASM still fails, because even runCatching does not catch a runtimeException for an expected IndexOutOfBoundsException when checking for tags. Help would be much appreachiated.

JesusMcCloud · 2024-01-12T14:33:57Z

WASM still fails, because even runCatching does not catch a runtimeException for an expected IndexOutOfBoundsException when checking for tags. Help would be much appreachiated.

Here are the details:

  Current test: testOpenPolymorphism
  Process output:
   wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:1
  
  
  RuntimeError: array element access out of bounds
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlin.Array.get (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[2207]:0x4247e)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.descriptors.SerialDescriptorImpl.getElementAnnotations (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[3974]:0x595a3)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.getKeyTags (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5369]:0x6b6e2)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.Preamble.encode (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5180]:0x68fa4)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.Token.encode (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5173]:0x68d56)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.internal.CborWriter.encode (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5294]:0x6a0c3)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.Cbor.encodeToByteArray (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5042]:0x6743b)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlinx.serialization.cbor.CborPolymorphismTest.testOpenPolymorphism (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[6111]:0x79d2a)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlin.wasm.internal.testContainer$kotlinx.serialization.cbor test fun$CborPolymorphismTest test fun$testOpenPolymorphism test fun.invoke (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[6715]:0x8440e)
      at <org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>.kotlin.test.TeamcityAdapter$test$lambda.invoke (wasm://wasm/<org.jetbrains.kotlinx:kotlinx-serialization-cbor_test>-00616f4e:wasm-function[5605]:0x6f4a9)

EDIT: seems it's already been reported and there's really not much we can do: https://youtrack.jetbrains.com/issue/KT-59081/WASM-Cant-catch-index-out-of-bounds-and-divide-by-0-exceptions

Co-authored-by: Leonid Startsev <sandwwraith@users.noreply.github.com>

JesusMcCloud · 2024-07-12T06:22:52Z

In my browser, this PR is now behaves uber-glitchy when all comments are shown, maybe it is just too large. The page jumps around when scrolling and is utterly unusable for me. I really tried to address all comments and i sincerely hope I did, because every single one was very valuable and improved the overall quality. If something slipped through, I'm really sorry!
If this page still renders fine for you, hopefully everything shows up as outdated. The gliches are also the reason I stopped accepting suggestion through here and simply applied them and committed them regularly.

whyoleg

Look like I have nothing more to add here except for a few minor final touches :)
Tremendous amount of work! Thank you for your contribution and fixing all the comments!

P.S. I've mostly reviewed public API all the time as I don’t know all the details of CBOR implementation

whyoleg · 2024-07-12T06:33:31Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborConfiguration.kt

+ * to annotate every `ByteArray` in a class hierarchy.
+ *
+ */
+public class CborConfiguration(


missing ExperimentalSerializationApi as Cbor is all experimental and probably it is better to make constructor internal. toString would be nice, like in JsonConfiguration, but optional

of course, you are right. will do!

done in 167877b

whyoleg · 2024-07-12T08:41:13Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

        return reader.decodeSerializableValue(deserializer)
    }
 }

 @OptIn(ExperimentalSerializationApi::class)
-private class CborImpl(encodeDefaults: Boolean, ignoreUnknownKeys: Boolean, serializersModule: SerializersModule) :
-    Cbor(encodeDefaults, ignoreUnknownKeys, serializersModule)
+private class CborImpl(


minor: I believe it can accept just configuration and redirect to Cbor constructor (like in Json)
Overall, IMO, it would be nice to have even just small implementation details be aligned between formats, so that those who will want to develop custom formats will see the pattern and the whole ecosystem of formats will be aligned on public API and internals somehow. But it is a minor and can be done in a follow-up PR if needed.

I was working with what I got and did not really consider aligning across formats, but you are right, of course!

done in 2294cc7

whyoleg · 2024-07-12T08:45:47Z

formats/cbor/commonTest/src/kotlinx/serialization/cbor/CborArrayTest.kt

+        val reference = ClassAs1Array(alg = -7)
+
+        val cbor = Cbor(from = Cbor.CoseCompliant) {
+            useDefiniteLengthEncoding = true


this flag is not needed here anymore as it is set in CoseCompilant. Same in several more places in this file

fixed in 507ccd1

whyoleg · 2024-07-12T08:46:27Z

formats/cbor/commonTest/src/kotlinx/serialization/cbor/CborDefLenTest.kt

+import kotlin.test.*
+
+
+class CborDefLenTest {


minor: would be nice to have full name here instead of shortcuts :)

fixed in 15bbc3f

whyoleg · 2024-07-12T08:48:06Z

formats/cbor/commonTest/src/kotlinx/serialization/cbor/CborReaderTest.kt

@@ -13,9 +15,9 @@ class CborReaderTest {

    private val ignoreUnknownKeys = Cbor { ignoreUnknownKeys = true }

-    private fun withDecoder(input: String, block: CborDecoder.() -> Unit) {
+    private fun withDecoder(input: String, block: CborParser.() -> Unit) {


minor, optional: class is now called CborParser so both test name and function can be renamed

Split up into tow distinct Test classes in 46227cf

sandwwraith · 2024-07-17T18:07:41Z

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Decoder.kt

+                        keyTags.joinToString(
+                            prefix = "[",
+                            postfix = "]"
+                        ) { it.toString() }


Since you copied it.toString()from CborParser.processTags here, CborReader now also requires @SuppressAnimalSniffer annotation. But I would prefer if you just add another method to CborParser — e.g. verifyTagsAndThrow(expected: ULongArray, actual: ULongArray) to reduce the amount of copy-pasted code

I hope I understood correctly. See 7ec8e02

whyoleg

LGTM :)

sandwwraith

Yeah, Github is not very good at tracking large conversations.
Looks good to me after fixing minor comments!

formats/cbor/commonMain/src/kotlinx/serialization/cbor/Cbor.kt

docs/formats.md

formats/cbor/commonMain/src/kotlinx/serialization/cbor/CborConfiguration.kt

formats/cbor/commonMain/src/kotlinx/serialization/cbor/internal/Decoder.kt

docs/formats.md

Co-authored-by: Leonid Startsev <sandwwraith@users.noreply.github.com>

JesusMcCloud · 2024-07-19T20:55:57Z

I have to say, I now understand why kotlinx.serialization works as well and reliably as it does. you really do take QA seriously!

sandwwraith · 2024-07-22T18:08:49Z

I want to thank you again for this huge amount of work and your time. Well done!

JesusMcCloud changed the base branch from master to dev August 18, 2023 11:18

This was referenced Aug 18, 2023

CBOR: Use Serial Labels #2371

Closed

CBOR: Basic Support for Tags #2359

Closed

Support definite-length Arrays and Maps in CBOR #1955

Open

CBOR tagged data can't be deserialized #1560

Open

sandwwraith reviewed Sep 22, 2023

View reviewed changes

JesusMcCloud requested a review from sandwwraith October 6, 2023 18:21

sandwwraith requested changes Dec 12, 2023

View reviewed changes

sandwwraith mentioned this pull request Dec 15, 2023

Support encoding/decoding ByteArrays in parameterized types as ByteStrings. #2383

Open

JesusMcCloud requested a review from sandwwraith January 12, 2024 10:59

sandwwraith self-assigned this Apr 17, 2024

JesusMcCloud and others added 15 commits June 7, 2024 19:30

CBOR basic value tagging

b1182ec

CBOR: handle tagged keys

47d6233

CBOR: make verifying and writing tags optional

87bcda3

CBOR: fix decoding value tags, add tests

d3eac6b

CBOR: refactor Tagged → ValueTags

2788ec2

CBOR: Fixes (docs, whitespace, names, superflous params)

c620fdb

CBOR: serialize cbor to tree

0d1b31b

CBOR: tree fixes

4f82ec8

CBOR: frankensteined data tree

1356666

CBOR: fix direct encoding of primitives

76afa35

CBOR: write definite lengths

ce9eefd

CBOR: experiment with pruning null properties

8b328f4

CBOR: Catch errors on finding annotations

ad97cbb

CBOR: Implement labels to use as map keys

ef1f77d

WIP: Ignore failing tests

83824b6

JesusMcCloud and others added 6 commits July 12, 2024 07:16

CBOR: move convenience method used for testing to test sources

b0d8885

CBOR: clarify and test object tags behaviour

86fc01b

CBOR: Fix docs

707b9b7

CBOR: Fix Docs

069fb77

Co-authored-by: Leonid Startsev <sandwwraith@users.noreply.github.com>

CBOR: fix builder switch names and polisch docs

69ae08f

CBOR: Update API dump and run knit

29b6605

JesusMcCloud requested review from sandwwraith and whyoleg July 12, 2024 06:22

whyoleg reviewed Jul 12, 2024

View reviewed changes

JesusMcCloud added 5 commits July 15, 2024 05:51

CBOR: refactor CborImpl C'tor

2294cc7

CBOR: remove superfluouos config switch for CborArrayTest

507ccd1

CBOR: mv CborDefLenTest -> CborDefiniteLengthTest

15bbc3f

CBOR: Refactor Tests

46227cf

CBOR: polish CborConfiguration

167877b

JesusMcCloud requested a review from whyoleg July 15, 2024 04:19

sandwwraith reviewed Jul 17, 2024

View reviewed changes

CBOR: cleanup CborDecoder and CborParser

7ec8e02

JesusMcCloud requested a review from sandwwraith July 17, 2024 18:49

whyoleg reviewed Jul 18, 2024

View reviewed changes

sandwwraith requested changes Jul 19, 2024

View reviewed changes

JesusMcCloud force-pushed the feature/coseUpstream branch from d3d49e8 to 3ef9fe1 Compare July 19, 2024 20:43

CBOR: Fix documentation

a272a2d

Co-authored-by: Leonid Startsev <sandwwraith@users.noreply.github.com>

JesusMcCloud force-pushed the feature/coseUpstream branch from 3ef9fe1 to a272a2d Compare July 19, 2024 20:46

CBOR: cleanup AnimalSniffer

f12a5bf

JesusMcCloud requested a review from sandwwraith July 19, 2024 20:54

sandwwraith approved these changes Jul 22, 2024

View reviewed changes

sandwwraith merged commit 2017084 into Kotlin:dev Jul 22, 2024
3 checks passed

CBOR Feature Drop: COSE #2412

CBOR Feature Drop: COSE #2412

Conversation

JesusMcCloud commented Aug 18, 2023

sandwwraith commented Aug 21, 2023

sandwwraith left a comment

Choose a reason for hiding this comment

sandwwraith left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

develar commented Dec 17, 2023

JesusMcCloud commented Jan 12, 2024

JesusMcCloud commented Jan 12, 2024 • edited Loading

JesusMcCloud commented Jul 12, 2024

whyoleg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyoleg left a comment

Choose a reason for hiding this comment

sandwwraith left a comment

Choose a reason for hiding this comment

JesusMcCloud commented Jul 19, 2024

sandwwraith commented Jul 22, 2024

sandwwraith left a comment •

edited

Loading

JesusMcCloud commented Jan 12, 2024 •

edited

Loading