Interop testsuite between JS/TS, Python and Go clients

[ivelin]

There are now 3 different peerjs client implementations that I am aware of. The JS/TS one hosted here. The python port, and now a new Go port by @muka.

Is there an ongoing conversation about an end to end testsuite that covers peer to peer interactions?

I am thinking about a cross language version that allows new language ports of peerjs to test for interoperability.

[kidandcat]

We already tried time ago, but testing WebRTC is hard, because you must set a real scenario with multiple devices in real network conditions, and also there are many external conditions that affect the usability (browser, networking, devices,...).

It's easy to setup offline testing, but it will not give much feedback about the real usage...

Anyway if you find an interesting approach, we will consider it.

[ivelin]

There are several reasons where I can see a peerjs interop testsuite helpful:

1. Ensure that there is API and protocol level compatibility between peer implementations. The messages sent on one end arrive and are interpreted as expected. There are no losses in translation.

2. Signaling handshake protocol compatibility.

3. p2p messages compatibility with various payloads.

4. End to end p2p connectivity works in various network and firewall topologies.

1, 2, and 3 do not require complicated networking setup, because they would assume peers see the signaling server's IP and that their ICE implementations are good enough to establish an initial RTCPeerConnection.

For 4 - I believe a number of different network topologies and firewall settings can be simulated within a github action with docker container settings and iptables/nftables firewall settings. Each peer app can run in its own container with unique network settings. Same for the signaling server.

See a diagram attached for a potential setup:

Link to diagram source if anyone wants to suggest edits:

[muka]

Thanks @ivelin @kidandcat to boost the discussion, I do agree having a shared platform for testing would help assessing interoperability and ensure users expectations are met.

In Go for example, porting the binarypack-js is pretty challenging involving a good amount of runtime types reflection work. While just supporting plain bytes array would work out of the box (delegating to the user any encoding).

So a tradeoff on different language implementation could drive to a more balanced API or other adaptations, as an example: based on the peer declared capabilities at connection time, select an appropriate exchange format type or abort the connection.

Thank you!