Add `mixnode` and `mixnet-client` crate #302

youngjoon-lee · 2023-08-14T09:53:05Z

Implemented the mixnode and mixnet-client crate, according to https://www.notion.so/Mixnet-Architecture-613f53cf11a245098c50af6b191d31d2.

NOTE:

This PR is going to be merged to the mixnet branch, not the master.
I recommend you to look at the test files first.

CHANGES:

mixnode: A core crate for implementing the mixnode binary
- forwards Sphinx packets to next mixnodes.
mixnet-client: A client library for mixnet users (such as nomos-node)
- splits a message into multiple Sphinx packets
- sends Sphinx packets to the mixnodes.
- receives Sphinx packets from mixnodess and re-assemble the message.

TODO:

Implement a nomos-mixnode binary using the mixnode crate.
Integrate the mixnet_client crate with Nomos network backends.
Refactoring
Many TODOs written in source code.
Cover traffic (We could postpone this a bit, as it's complex).

* Add pruning methods * Call pruning on node step method * Fix tally retain closure

* Update sim engine on timeout qc * Remove rebuild method

* implement research overlay * Add branch overlay * Branch overlay config in simulation * Remove research related extensions * Use branch_depth and chunks --------- Co-authored-by: Gusto <bacvinka@gmail.com>

zeegomo · 2023-08-14T10:13:19Z

mixnet/client/src/receiver.rs

+        message_reconstructor: Arc<Mutex<MessageReconstructor>>,
+    ) -> Result<(), Box<dyn Error>> {
+        let mut buf = Vec::new();
+        socket.read_to_end(&mut buf).await?;


Just to confirm, I guess this means a new connections is opened for each new message? This would indeed make sense given route is dynamic

Yes, for now. Later, I think we need to introduce better connection managements at some point, depending on how often the mixnet topology is updated. For example, Nym mixnodes stay connected with mixnodes in the next layer because they updated the topology once a day or an hour, iirc.

zeegomo · 2023-08-14T10:16:34Z

mixnet/client/src/receiver.rs

+        socket.read_to_end(&mut buf).await?;
+
+        let payload = Payload::from_bytes(&buf)?.recover_plaintext()?;
+        let fragment = Fragment::try_from_bytes(&payload)?;


If I understand your comment correctly, this function just receives a single fragment (chunk) and throws it to the message reconstructor. The message reconstructor keeps all fragments previously received, in memory. If the message can be finally reconstructed with the newly received fragment, the message reconstrcutor returns the message fully reconstructed.

mixnet/client/src/sender.rs

mixnet/mixnode/src/lib.rs

zeegomo · 2023-08-14T14:22:07Z

mixnet/topology/src/lib.rs

+        let mut route: Vec<route::Node> = Vec::new();
+
+        for layer_id in 0..num_hops {
+            let layer = self.layers.get(layer_id).unwrap();


it's not clear from the api that layer_ids are incremental

Thank you. I just realized that we don't actually need the LayerId type: ea23c6e

I also simplified the API by removing the num_hops: b8d3bad. We can revive it later after the first iteration is done.

zeegomo · 2023-08-14T14:22:31Z

mixnet/topology/src/lib.rs

+}
+
+impl Layer {
+    pub fn random_node(&self) -> Option<&Mixnode> {


we should probably ask the caller to provide the source of randomness

Good point: 2fd055b.
I had to downgrade the rand and rand_xoshiro in the integration test, because nym-sphinx depends on the rand@v0.7.3. If we use the rand@v0.8, the following error occurs:

--> mixnet/client/src/sender.rs:65:35 | 65 | .split_into_fragments(rng, plaintext_size_per_packet) | -------------------- ^^^ the trait `rand_core::RngCore` is not implemented for `R`

* Handle finished simulation * Flush remaining records to the sink * Write runtime metadata last

* Add rs lib, implement encoding * Implement decoding

youngjoon-lee · 2023-08-17T09:12:14Z

tests/src/tests/mixnet.rs

+    let mut msg = [0u8; 100 * 1024];
+    rand::thread_rng().fill_bytes(&mut msg);
+
+    let res = client1.send(msg.to_vec(), destination, &mut OsRng, topology.layers.len());


@zeegomo @danielSanchezQ
I just changed the mixnet design a bit in the PR #307, so that the message sender doesn't need to specify the mixnet destination explicitly. Please review that PR together with this PR. I put detailed descriptions there.

* optimize simulation app serialization

danielSanchezQ

We can go ahead with this. 🚀

youngjoon-lee · 2023-08-21T10:14:48Z

@zeegomo I'm merging this PR since I've applied your suggestions and there have been many architectural changes (especially in PR #307). I think it's better to merge this now to the mixnet branch, so that we can improve it gradually together from there after you come back. I guess there would be many things to be refined/improved when we integrate the mixnet with network-backend.

* Add `mixnode` and `mixnet-client` crate (#302) * Add `mixnode` binary (#317) * Integrate mixnet with libp2p network backend (#318) * Fix #312: proper delays (#321) * proper delays * add missing duration param * tiny fix: compilation error caused by `rand` 0.8 -> 0.7 * use `get_available_port()` for mixnet integration tests (#333) * add missing comments * Overwatch mixnet node (#339) * Add mixnet service and overwatch app * remove #[tokio::main] --------- Co-authored-by: Youngjoon Lee <taxihighway@gmail.com> * fix tests for the overwatch mixnode (#342) * fix panic when corner case happen in RandomDelayIter (#335) * Use `log` service for `mixnode` bin (#341) * Use `wire` for MixnetMessage in libp2p (#347) * Prevent tmixnet tests from running forever (#363) * Use random delay when sending msgs to mixnet (#362) * fix a minor compilation error caused by the latest master * Fix run output fd (#343) * add a connection pool * Exp backoff (#332) * move mixnet listening into separate task * add exponential retry for insufficient peers in libp2p * fix logging * Fix MutexGuard across await (#373) * Fix MutexGuard across await Holding a MutexGuard across an await point is not a good idea. Removing that solves the issues we had with the mixnet test * Make mixnode handle bodies coming from the same source concurrently (#372) --------- Co-authored-by: Youngjoon Lee <taxihighway@gmail.com> * Move wait at network startup (#338) We now wait after the call to 'subscribe' to give the network the time to register peers in the mesh before starting to publish messages * Remove unused functions from mixnet connpool (#374) * Mixnet benchmark (#375) * merge fixes * add `connection_pool_size` field to `config.yaml` * Simplify mixnet topology (#393) * Simplify bytes and duration range ser/de (#394) * optimize bytes serde and duration serde --------- Co-authored-by: Al Liu <scygliu1@gmail.com> Co-authored-by: Daniel Sanchez <sanchez.quiros.daniel@gmail.com> Co-authored-by: Giacomo Pasini <Zeegomo@users.noreply.github.com>

bacv and others added 6 commits August 10, 2023 12:56

Parallel node init (#300)

da60d8f

Simulation prune tally (#299)

ef5935d

* Add pruning methods * Call pruning on node step method * Fix tally retain closure

Sim update overlay (#298)

44367a1

* Update sim engine on timeout qc * Remove rebuild method

Add missing build and test flags (#301)

07ae5a4

Implement research overlay (#291)

2612c77

* implement research overlay * Add branch overlay * Branch overlay config in simulation * Remove research related extensions * Use branch_depth and chunks --------- Co-authored-by: Gusto <bacvinka@gmail.com>

Add mixnet crate

2d9e909

youngjoon-lee added this to the Network Privacy Measurements milestone Aug 14, 2023

github-actions bot assigned youngjoon-lee Aug 14, 2023

youngjoon-lee changed the title ~~Add mixnet crate~~ Add mixnode and mixnet-client crate Aug 14, 2023

youngjoon-lee added the mixnet label Aug 14, 2023

zeegomo requested changes Aug 14, 2023

View reviewed changes

youngjoon-lee added 4 commits August 16, 2023 12:48

undo refmt Cargo.toml

df93f0e

use map

20be7cf

refactor: use a common function

9b13f44

do not hardcode the source of rand

2fd055b

youngjoon-lee force-pushed the mixnet-node branch from bf10cc3 to 9a1d94f Compare August 16, 2023 05:22

remove LayerId

ea23c6e

youngjoon-lee force-pushed the mixnet-node branch from 9a1d94f to ea23c6e Compare August 16, 2023 05:24

youngjoon-lee requested review from zeegomo and danielSanchezQ August 16, 2023 06:27

bacv and others added 2 commits August 16, 2023 11:08

Simulation graceful shutdown fixes (#305)

fe3d390

* Handle finished simulation * Flush remaining records to the sink * Write runtime metadata last

rename packages/structs

9b53c10

youngjoon-lee force-pushed the mixnet-node branch from b7563e4 to 9b53c10 Compare August 17, 2023 05:47

RS encoding/decoding base methods (#303)

61b587a

* Add rs lib, implement encoding * Implement decoding

youngjoon-lee force-pushed the mixnet-node branch from 80e49f2 to 9b53c10 Compare August 17, 2023 09:03

youngjoon-lee mentioned this pull request Aug 17, 2023

Improved Mixnet design: Random mixnet destination #307

Merged

youngjoon-lee commented Aug 17, 2023

View reviewed changes

youngjoon-lee and others added 3 commits August 17, 2023 18:40

simplify api

b8d3bad

Optimize serialization/deserialization for [u8; 32] (#306)

87f49c0

* optimize simulation app serialization

Using CSV format by default for simulations (#304)

5391382

danielSanchezQ approved these changes Aug 21, 2023

View reviewed changes

Improved Mixnet design: Random mixnet destination (#307)

79d20ef

typo

84dfb0f

youngjoon-lee force-pushed the mixnet-node branch 2 times, most recently from 9be1b8f to 84dfb0f Compare August 21, 2023 12:42

youngjoon-lee added 3 commits August 21, 2023 22:13

Pick ports more randomly for integration tests (#315)

ebbdf14

Merge remote-tracking branch 'origin/master' into mixnet-node

913c5c0

fix the version issue of rand

77dc749

youngjoon-lee force-pushed the mixnet-node branch from c338f15 to 77dc749 Compare August 21, 2023 13:18

youngjoon-lee merged commit aec6258 into mixnet Aug 21, 2023
2 checks passed

youngjoon-lee deleted the mixnet-node branch August 21, 2023 13:34

youngjoon-lee added a commit that referenced this pull request Aug 21, 2023

Add mixnode and mixnet-client crate (#302)

d57efc3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `mixnode` and `mixnet-client` crate #302

Add `mixnode` and `mixnet-client` crate #302

youngjoon-lee commented Aug 14, 2023 •

edited

zeegomo Aug 14, 2023

youngjoon-lee Aug 16, 2023 •

edited

zeegomo Aug 14, 2023

youngjoon-lee Aug 16, 2023 •

edited

zeegomo Aug 14, 2023

youngjoon-lee Aug 16, 2023 •

edited

youngjoon-lee Aug 17, 2023

zeegomo Aug 14, 2023

youngjoon-lee Aug 16, 2023

youngjoon-lee Aug 17, 2023 •

edited

danielSanchezQ left a comment

youngjoon-lee commented Aug 21, 2023 •

edited

Add mixnode and mixnet-client crate #302

Add mixnode and mixnet-client crate #302

Conversation

youngjoon-lee commented Aug 14, 2023 • edited

zeegomo Aug 14, 2023

Choose a reason for hiding this comment

youngjoon-lee Aug 16, 2023 • edited

Choose a reason for hiding this comment

zeegomo Aug 14, 2023

Choose a reason for hiding this comment

youngjoon-lee Aug 16, 2023 • edited

Choose a reason for hiding this comment

zeegomo Aug 14, 2023

Choose a reason for hiding this comment

youngjoon-lee Aug 16, 2023 • edited

Choose a reason for hiding this comment

youngjoon-lee Aug 17, 2023

Choose a reason for hiding this comment

zeegomo Aug 14, 2023

Choose a reason for hiding this comment

youngjoon-lee Aug 16, 2023

Choose a reason for hiding this comment

youngjoon-lee Aug 17, 2023 • edited

Choose a reason for hiding this comment

danielSanchezQ left a comment

Choose a reason for hiding this comment

youngjoon-lee commented Aug 21, 2023 • edited

Add `mixnode` and `mixnet-client` crate #302

Add `mixnode` and `mixnet-client` crate #302

youngjoon-lee commented Aug 14, 2023 •

edited

youngjoon-lee Aug 16, 2023 •

edited

youngjoon-lee Aug 16, 2023 •

edited

youngjoon-lee Aug 16, 2023 •

edited

youngjoon-lee Aug 17, 2023 •

edited

youngjoon-lee commented Aug 21, 2023 •

edited