test: introduce `libp2p-swarm-test` #2888

thomaseizinger · 2022-09-12T07:55:25Z

Description

This patch-set introduces libp2p-swarm-test. It provides utilities for quick and safe bootstrapping of tests for NetworkBehaviours. The main design features are:

Everything has timeouts
APIs don't get in your way
Minimal boilerplate

Closes #2884.

Notes

I did not port libp2p-kad because the testing logic was too complicated. I also did not add tests for protocols that don't already have some. That is out of scope for this PR.

Open Questions

~~Should this testing crate also pull together utilities like ArbitraryPeerId?~~
- No. That should go into quickcheck-ext.
Can we make the Swarm::new_ephemeral API any better? At the moment it only allows to configure the underlying transport but the authentication and muxer are hardcoded. I don't want to introduce any more parameters because it will get too verbose and there wouldn't be a benefit in having the function in the first place. Should we perhaps introduce a dedicated SwarmBuilder for our tests that start with some reasonable test defaults, like a MemoryTransport?
Trying to port the new harness to libp2p-ping, I noticed that we'd lose the MuxerChoice as part of the prop test. If we go forward with the SwarmBuilder extension idea, we could make a function there that cycles through various muxers and authentication algorithms to make sure things work across various stacks. I am not sure it is worth it though to be honest.

Open tasks

~~Port to more crates~~

Change checklist

I have performed a self-review of my own code
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have added tests that prove my fix is effective or that my feature works~~
~~- [ ] A changelog entry has been made in the appropriate crates~~

Returns an event or panics in case nothing happens for X seconds.

melekes

I'm in great support of this ❤️

This forces us to get rid of the muxer choice which I am not yet sure about.

mxinden

Direction looks good to me. Thanks for pushing this forward.

swarm-test/src/lib.rs

swarm/src/lib.rs

swarm-test/src/lib.rs

mxinden · 2022-09-16T15:45:46Z

Should this testing crate also pull together utilities like ArbitraryPeerId?

I am guessing that defining Arbitrary on PeerId directly under the cfg[test] doesn't expose outside of the crate. Is that correct?

mxinden · 2022-09-16T15:46:44Z

Can we make the Swarm::new_ephemeral API any better? At the moment it only allows to configure the underlying transport but the authentication and muxer are hardcoded. I don't want to introduce any more parameters because it will get too verbose and there wouldn't be a benefit in having the function in the first place. Should we perhaps introduce a dedicated SwarmBuilder for our tests that start with some reasonable test defaults, like a MemoryTransport?

I think all our non Transport tests should use the MemoryTransport, thus I am in favor of this default.

mxinden · 2022-09-16T15:48:06Z

Trying to port the new harness to libp2p-ping, I noticed that we'd lose the MuxerChoice as part of the prop test. If we go forward with the SwarmBuilder extension idea, we could make a function there that cycles through various muxers and authentication algorithms to make sure things work across various stacks. I am not sure it is worth it though to be honest.

I still have to put more thoughts into this, though as an intuition I suggest that a test only tests one thing. E.g. the libp2p-ping tests test libp2p-ping and not some permutation of different transports and muxers. I think that should happen in a separate dedicated test.

thomaseizinger · 2022-09-19T05:49:30Z

Should this testing crate also pull together utilities like ArbitraryPeerId?

I am guessing that defining Arbitrary on PeerId directly under the cfg[test] doesn't expose outside of the crate. Is that correct?

That is correct.

Can we make the Swarm::new_ephemeral API any better? At the moment it only allows to configure the underlying transport but the authentication and muxer are hardcoded. I don't want to introduce any more parameters because it will get too verbose and there wouldn't be a benefit in having the function in the first place. Should we perhaps introduce a dedicated SwarmBuilder for our tests that start with some reasonable test defaults, like a MemoryTransport?

I think all our non Transport tests should use the MemoryTransport, thus I am in favor of this default.

Cool, that makes this simpler :)

Trying to port the new harness to libp2p-ping, I noticed that we'd lose the MuxerChoice as part of the prop test. If we go forward with the SwarmBuilder extension idea, we could make a function there that cycles through various muxers and authentication algorithms to make sure things work across various stacks. I am not sure it is worth it though to be honest.

I still have to put more thoughts into this, though as an intuition I suggest that a test only tests one thing. E.g. the libp2p-ping tests test libp2p-ping and not some permutation of different transports and muxers. I think that should happen in a separate dedicated test.

Yeah I am tending towards the same. We should have a good muxer test harness instead!

NB implementations should not make any assumptions about the underlying transport implementations. Defaulting this to `MemoryTransport` allows us to simplify the API a great deal.

We remove the dynamic muxer choice because it is not the responsibility of the ping tests to check that the muxers behave identically.

Not sure where that came from!

mxinden · 2022-09-21T10:42:08Z

Should this testing crate also pull together utilities like ArbitraryPeerId?

I am guessing that defining Arbitrary on PeerId directly under the cfg[test] doesn't expose outside of the crate. Is that correct?

That is correct.

That is unfortunate. In that case, I am in favor of providing an ArbitraryPeerId from this new crate.

elenaf9

Nice!

swarm-test/src/lib.rs

swarm/src/lib.rs

Co-authored-by: Elena Frank <elena.frank@protonmail.com>

…-test

The timeout is implied by the testing lib and this is more consistent with `next_behaviour_event`.

thomaseizinger · 2023-03-02T03:48:48Z

swarm-test/src/lib.rs

+///
+/// Especially (2) is crucial for our usage of this function.
+/// If a [`Swarm`] is not polled, nothing within it makes progress.
+/// This can "starve" the other swarm which for example may wait for another message to be sent on a connection.


I spent a good two hours of debugging timeouts in the test and unwoken tasks until it dawned on me why things were not working. @elenaf9 I believe this might also be the reason for your timeouts in #3345.

This function may seem like a beast but after trying various solutions locally, I am convinced it makes sense to add this to the test-harness and use this instead of future::join for testing Swarms.

mxinden

Just looking at the diff, this is already a great step forward.

A couple of comments. Nothing major. Good to merge from my end.

swarm-test/src/lib.rs

mxinden · 2023-03-07T08:11:19Z

swarm-test/src/lib.rs

+    while res1.len() < NUM_EVENTS_SWARM_1 || res2.len() < NUM_EVENTS_SWARM_2 {
+        match futures::future::select(swarm1.next_swarm_event(), swarm2.next_swarm_event()).await {
+            Either::Left((o1, _)) => {
+                if let Ok(o1) = o1.try_into_output() {
+                    res1.push(o1);
+                }
+            }
+            Either::Right((o2, _)) => {
+                if let Ok(o2) = o2.try_into_output() {
+                    res2.push(o2);
+                }
+            }
+        }
+    }


Long term it would be nice to detect missed wake-ups in a NetworkBehaviour implementation with libp2p-swarm-test. If I am not mistaken, that is not possible with the above while select! loop as long as the other swarm triggers a wake-up.

Lets defer to a future pull request.

This is kind of a conflicting requirement. I added this because it is very easy to poll a Swarm in a wrong way in a test, thus it is often the test that is buggy, not the NetworkBehaviour implementation.

swarm-test/src/lib.rs

mxinden · 2023-03-07T08:15:48Z

swarm-test/src/lib.rs

+        let peer_id = PeerId::from(identity.public());
+
+        let transport = MemoryTransport::default()
+            .or_transport(libp2p_tcp::async_io::Transport::default())


Why do we need the TCP transport? Is this needed for libp2p-autonat?

Yes, and dcutr needs it too.

mxinden · 2023-03-07T08:23:01Z

protocols/request-response/tests/ping.rs

+    let event = loop {
+        futures::select!(
+            event = swarm1.select_next_some() => {
+                if let SwarmEvent::Behaviour(request_response::Event::Message {
+                    peer,
+                    message: request_response::Message::Request { request, channel, .. }
+                }) = event {
+                    assert_eq!(&request, &ping);
+                    assert_eq!(&peer, &peer2_id);
+
+                    drop(channel);
+                    continue;
+                }
+            },
+            event = swarm2.select_next_some() => {
+                if let SwarmEvent::Behaviour(ev) = event {
+                    break ev;
+                }
+            },
+        )
+    };


Can this not be done via the new shiny drive function?

Probably, I've introduced that one only at the very end!

I am not sure actually, this one is a bit different because the two peers operate autonomously. drive is useful if you want to "step" through a test in a more linear fashion because it gives you access to the Swarms inbetween. Here, we can just create to tasks that operate independently.

Co-authored-by: Max Inden <mail@max-inden.de>

Circular dependencies are problematic in several ways: - They result in cognitive overhead for developers, in trying to figure out what depends on what. - They present `cargo` with limits in what order the crates can be compiled in. - They invalidate build caches unnecessarily thus forcing `cargo` to rebuild certain crates. - They cause problems with tooling such as `release-please`. To actually break the circular dependencies, this patch inlines the uses of `development_transport` in the examples and tests for all sub-crates. This is only meant to be a short-term fix until libp2p#3111 and libp2p#2888 are fixed. To ensure we don't accidentally reintroduce this dependency, we add a basic CI that queries `cargo metadata` using `jq`. Resolves libp2p#3053. Fixes libp2p#3223. Related: libp2p#2918 (comment) Related: googleapis/release-please#1662

This patch-set introduces `libp2p-swarm-test`. It provides utilities for quick and safe bootstrapping of tests for `NetworkBehaviour`s. The main design features are: - Everything has timeouts - APIs don't get in your way - Minimal boilerplate Closes libp2p#2884. Pull-Request: libp2p#2888.

thomaseizinger added 3 commits September 12, 2022 13:32

Initialise libp2p-swarm-test with bits of rendezvous' test-harness

c814911

Introduce SwarmExt::next_within

7c7cd69

Returns an event or panics in case nothing happens for X seconds.

Rewrite libp2p-ping tests using new test framework

f00d954

thomaseizinger requested review from mxinden and elenaf9 September 12, 2022 07:55

melekes reviewed Sep 14, 2022

View reviewed changes

thomaseizinger added 5 commits September 16, 2022 11:57

Merge branch 'master' into swarm-test

9ac8c8a

Only use yamux

ae33efc

Introduce SwarmExt::new_ephemeral

909de35

Add missing features

246d10c

Re-write first test of libp2p-ping with SwarmExt::new_ephemeral

0a679cd

This forces us to get rid of the muxer choice which I am not yet sure about.

mxinden reviewed Sep 16, 2022

View reviewed changes

swarm-test/src/lib.rs Outdated Show resolved Hide resolved

swarm-test/src/lib.rs Outdated Show resolved Hide resolved

swarm/src/lib.rs Show resolved Hide resolved

swarm-test/src/lib.rs Outdated Show resolved Hide resolved

thomaseizinger added 9 commits September 19, 2022 15:52

Rename connect function

3ec9ab3

Fix typo

ba91fc8

Default transport in new_ephemeral to MemoryTransport

099eec3

NB implementations should not make any assumptions about the underlying transport implementations. Defaulting this to `MemoryTransport` allows us to simplify the API a great deal.

Update docs

aeb7224

Make ping tests pass by fully converting to test harness

96ebbf2

We remove the dynamic muxer choice because it is not the responsibility of the ping tests to check that the muxers behave identically.

Remove unused imports

82ede8f

Apply clippy suggestion

2aa7f5e

Remove extern crate statement

3c0ac6a

Not sure where that came from!

Remove duplication in test

973a66f

elenaf9 approved these changes Sep 21, 2022

View reviewed changes

swarm-test/src/lib.rs Outdated Show resolved Hide resolved

swarm-test/src/lib.rs Outdated Show resolved Hide resolved

swarm/src/lib.rs Show resolved Hide resolved

thomaseizinger and others added 3 commits September 22, 2022 17:23

Update swarm-test/src/lib.rs

ac30260

Co-authored-by: Elena Frank <elena.frank@protonmail.com>

Merge branch 'master' into swarm-test

2bc39ff

Merge branch 'swarm-test' of github.com:libp2p/rust-libp2p into swarm…

61ed59c

…-test

thomaseizinger added 15 commits February 28, 2023 14:08

Don't depend on async-std executor of swarm

a74a058

Make gossipsub tests compile

f4122d8

Fix formatting

4f7eb04

Fix dcutr tests

81ca805

Fix gossipsub doc tests

70d8d14

Fix ping tests

ee295af

Make autonat test more resilient

7029868

Fix req-res test

e4203e6

Fix test

3183125

Rename next_or_timeout to next_swarm_event

e8600f0

The timeout is implied by the testing lib and this is more consistent with `next_behaviour_event`.

Get rid of rendezvous harness

e942586

Use defaults of derive macro

0e83a95

Merge branch 'master' into swarm-test

09e34c5

Introduce swarm_test::drive to avoid starvation in rendezvous protocol

8aad527

Further improve drive

e47dbf7

thomaseizinger commented Mar 2, 2023

View reviewed changes

thomaseizinger added 4 commits March 2, 2023 14:50

Fix clippy and docs

2f0f200

Fix clippy

cd93320

Use drive in ping test

9f8015d

Lazily compute vec length

20ccbcb

mxinden approved these changes Mar 7, 2023

View reviewed changes

thomaseizinger and others added 3 commits March 8, 2023 19:53

Update swarm-test/src/lib.rs

3ba7eb0

Co-authored-by: Max Inden <mail@max-inden.de>

Add license header

96fd331

Fix docs

9f5f280

thomaseizinger added the send-it label Mar 8, 2023

Merge branch 'master' into swarm-test

56ea5e1

mergify bot merged commit 7069d78 into master Mar 8, 2023

mergify bot deleted the swarm-test branch March 8, 2023 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: introduce `libp2p-swarm-test` #2888

test: introduce `libp2p-swarm-test` #2888

thomaseizinger commented Sep 12, 2022 •

edited

melekes left a comment

mxinden left a comment

mxinden commented Sep 16, 2022

mxinden commented Sep 16, 2022

mxinden commented Sep 16, 2022

thomaseizinger commented Sep 19, 2022

mxinden commented Sep 21, 2022

elenaf9 left a comment

thomaseizinger Mar 2, 2023

mxinden left a comment

mxinden Mar 7, 2023

thomaseizinger Mar 8, 2023

mxinden Mar 7, 2023

thomaseizinger Mar 8, 2023

mxinden Mar 7, 2023

thomaseizinger Mar 8, 2023

thomaseizinger Mar 8, 2023

test: introduce libp2p-swarm-test #2888

test: introduce libp2p-swarm-test #2888

Conversation

thomaseizinger commented Sep 12, 2022 • edited

Description

Notes

Open Questions

Open tasks

Change checklist

melekes left a comment

Choose a reason for hiding this comment

mxinden left a comment

Choose a reason for hiding this comment

mxinden commented Sep 16, 2022

mxinden commented Sep 16, 2022

mxinden commented Sep 16, 2022

thomaseizinger commented Sep 19, 2022

mxinden commented Sep 21, 2022

elenaf9 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxinden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

test: introduce `libp2p-swarm-test` #2888

test: introduce `libp2p-swarm-test` #2888

thomaseizinger commented Sep 12, 2022 •

edited