How to organise inter-op testing

[mirjak]

Some groups do frequent inter-op tests, either by explicit inter-op events or during the hackathon. Is there a way to better support these activities, e.g. regarding how to describe what feature to test and also how to run the tests?

[mirjak]

And of course how to find this information more easily, especially for implementators who are otherwise not active in the wg.

[mirjak]

At minimum I think it would be nice to mark a certain draft version as interop testing version in the datatracker with a link to some wiki page or external source with a description of what part of the protocol is stable to test. (disclaimer to not steal the glory: his is not my idea; this came up at some point doing some QUIC discussion but I can't remember who proposed it.)

[obonaventure]

There are different ways to address this interoperability problem. Manual tests such as those carried out during hackathons or similar events have been used by several working groups. Some working groups also use implementers surveys and some working groups of the routing area maintain this information on their trac server. Within the security area, some RFCs contain some test vectors for cryptographic solutions. These test vectors are a first step to validate such mechanisms.

However, RFCs rarely contain detailed information on how to test a given protocol. Test suites have been developed for different protocols, but usually outside the IETF. Without attempting to "certify" implementations by defining a test suite, the IETF could consider to recommend some test vectors that represent the minimum features that a conformant implementation needs to support. These test suites are often written by developers or by protocol test experts, but unfortunately, these efforts are rarely documented within the IETF. Some examples include packetdrill to test TCP implementations, using formal methods to test QUIC implementations, ...