Collecting items for conformity / plug tests

[chrysn]

I don't think there are plans yet to have high-level tests similar to those we had for OSCORE as the plug test spec. What I envision there is a set of corner cases that implementers could try to interoperate with, some of which deliberately hard.

This would be distinct from the traces / vectors as I understand them: Those are highly detailed step-by-step data sets that ensure that all data items go to the right places and can be debugged if they don't. For this here, I think more of a few high-level items that can be tested on a live systems too (ie. don't require predefined ephemeral values), and their failure indicates some particular set of things going wrong. Also, unlike the traces / vectors, not all of this necessarily reflects a MUST, some only SHOULDs.

Concretely (and usually because I either just fixed that or know that my implementation will get it wrong), this would include:

• CRED_x encoded noncanonically ("SHOULD use an available authentication credential (transported in EDHOC or otherwise provisioned) without re-encoding")
• ID_CRED_x encoded noncanonically (not sure if that is even allowed as it's a header map)
• Presence of unknown (eg. not-yet-specified) cipher suites.

Do you think it is useful to gather such cases? If so, what are others you'd like to see covered?

[gselander]

Yes, please list such cases here as they become known. It would be good with additional tests beyond current interop testing. Is OSCORE style tests preferable (defined initial state, actions and expected outcome)? What cases do people find important to cover?

[chrysn]

My takeaway from using the OSCORE tests was that they work best if at least one side can just work as a server (which is generally stateless, and can accept clients performing any test; for EDHOC that may require having multiple servers run or name-based-virtual-hosting). Thus initial state would be empty by test case (unless they build upon each other, in which case it is "whatever test X turned out to produce"). This property allowed the last versions of the OSCORE tests to be run unilaterally, only coming back to the server operator if trouble occurred.

I generally prefer tests in which nothing needs to be configured that is not also configurable at run time (as such configurations might conflict with general API design; for example, I'd hope to use truly ephemeral keys and not hard-coded ones), but that may not be possible completely for EDHOC. For example, implementations would need to be configured with externally generated private keys to work with a peer that is configured to the plug test specs, even though it'd typically generate its private key pair and keep it isolated. (But the alternative of having an out-of-band live plug test configuration, eg. by the client GETting the server's public keys and PUTting its own in some CWT format would IMO be more error prone than the EDHOC tests themselves).