Multidim Interop tests are broken in master

[achingbrain]

This PR was merged with green CI but now these tests fail in master.

It looks like master runs more test variations there than there were on the PR:

Master:

https://github.com/libp2p/js-libp2p/actions/runs/5251996577

#1788:

https://github.com/libp2p/js-libp2p/actions/runs/5201404450

CC @MarcoPolo @maschad

[maschad]

It looks like master runs more test variations there than there were on the PR:

I believe that's because the workflow file is going to run the tests against the latest JS versions in test plans but @MarcoPolo can confirm this. Regardless, I don't running those variations is a drawback as that provides better coverage.

It seems like #1804 introduced some inconsistency in the master CI runs (given that the workflow on that PR check passed) and that previous runs tested against this webrtc and webrtc-direct matrix were passing prior to this PR being merged on master, and subsequent runs post-merge of #1804 are failing.

[achingbrain]

Hmm, interesting. The multidim tests passed for the #1804 PR though so that might be a red herring?

[maschad]

Hmm, interesting. The multidim tests passed for the #1804 PR though so that might be a red herring?

Yah that's what I was referring to by stating (given that the workflow on that PR check passed) but I don't think it's a red herring because all subsequent master CI Multidim interop runs have failed post-merge of #1804 so that seems to be consistent. The error also seems to be the same in that the streams are aborting early (there is also another error related to monorepo refactor but seems that will be fixed by #1828)

But these tests pass locally when I run them, perhaps it's a timeout issue ?

[MarcoPolo]

The extra tests that happened in master are because I merged this PR on test-plans libp2p/test-plans#189 that introduced js-libp2p v0.45 to the test-plans repo so that every implementation (including js-libp2p) tests against this released version of js-libp2p. #1788 Was merged before libp2p/test-plans#189 was, so it didn't know about that released version yet.

The timelime:

• test: Interop test for webrtc and webrtc-direct #1788 was merged. Adds webrtc and webrtc-direct interop tests.
• Reintroduce js-libp2p v0.45.0 and fix build from cache test-plans#189 was merged. Adds js-libp2p v0.45 to test-plans so every libp2p implementation tests against it.
• js-libp2p master runs use the latest test-plans commit when running tests so they are testing against js-libp2p v0.45. These are the "more test variations".

[MarcoPolo]

I wonder if the test passed in #1804 because dependency resolution worked out then, but failed after because of a different dependency resolution (just a guess). I'm going to try running the multidim CI for 1804 and see if it still passes.

[achingbrain]

If I'm reading this correctly that means the tests that are run are controlled by whatever is in master of https://github.com/libp2p/test-plans?

Is there a way to depend on a specific version of the multidim tests instead? Otherwise commits in an unrelated repo can break CI in this repo which makes the build non-deterministic and leads to confusion like above.

Previously this was handled by having releases of https://github.com/libp2p/interop with semver and all that.

[achingbrain]

Ah, wait - we can control that here.

[p-shahi]

@achingbrain
It looks like all of the interop tests are now failing as a result of failing to import modules post the monorepo consolidation: https://github.com/libp2p/js-libp2p/actions/runs/5315540166/jobs/9624024987

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@libp2p/crypto' imported from /app/packages/libp2p/dist/src/libp2p.js\n" +

Ah there's a PR in flight that will likely address this #1831

[achingbrain]

#1842 fixes that particular error but the tests still fail so there's more work to do.

One error I'm seeing in the logs right now is:

unexpected exception in ping test: Error: The operation was aborted

This seems like a timeout to me. @maschad has had success running these tests locally so this could be the test suite overwhelming the weaker CI machines.

The other is:

unexpected exception in ping test: Error: stream ended before 1 bytes became available

This happens when the remote closes the stream before it sends any data (e.g. before multistream select has a chance to run). It's symptomatic of the issue described here and is probably related to the above.

[MarcoPolo]

WebRTC tests seem flaky to me. I don't think this is a resource issue in CI. Running these tests locally (in compose and npm test) also sometimes fail for me.

Let's remove the webrtc tests for now, and add them back in when they're not flaky.