Basic Waku simulation

[oskarth]

Problem

We don't know if Waku mode will actually significantly decrease bandwidth vacp2p/rfc#49

We have a theoretical model that claims it well, but we need more confirmation. Core integration is one way (and the ultimate goal), but it'd be good to know earlier if we are actually solving the problem.

Acceptance criteria

This is a minimal simulation for Waku. To start with, it should

• launch N waku nodes
• that send M messages over some basic (possibly hardcoded configuration)
• prints statistics that can be correlated to bandwidth
• do this with both bloom filter and topic interest (i.e. similar to the model)

I.e. you can run a script and it shows you some basic BW usage for some time period. Essentially testing the model and ensure it reflects reality.

As well as:

• more precise follow-up issues created for specific conditions (mailserver, discovery, geth, bad peers, message confirmations, etc).

Solution

Start with shh client, then figure out how get them talk. Static nodes / bootstrap nodes etc. See nim-eth for additional changes (also related: https://github.com/vacp2p/pm/issues/4)

Notes

See https://github.com/status-im/nim-beacon-chain for how they do it as inspiration. Start with basics though, something that can be done in a few days.

https://github.com/status-im/nimbus/issues/426

Further work

After basics are done:

• Extend with mailserver
• Extend with better monitoring/logging (Prometheus/Grafana)
• Extend with multiple clients (geth)
• Extend with more adverse conditions (network drops, spikes, etc)

[oskarth]

Basic very early WIP here in PR. Having some issues with nim-eth connectivity (though to be fair I only played with it very briefly so far). @kdeme any ideas? Based it on shh client status-im/nim-eth#145

We want this to be separate unix procs, so we can have a more realistic simulation with various components, etc. So can't take current in-proc tests as is, hence basing it on basic client

Expected result: See messages (handler log) appearing in watch process.

Actual result: Connected to Whisper nodes, messages appear to be sent and filters installed, but no messages coming through.

Note: Noticed some WRN 2019-12-11 13:46:56+08:00 Message PoW too low topics="whisper" tid=8053 minPow=0.2 pow=0.0003289473684210526, unclear if related or mainnet artifact.

[kdeme]

Yes, definitely separate procs would be the way to go. Both for being able to gather metrics/bandwidth usage/memory usage/ etc. per node and to keep it understandable and scalable.

That warning is likely to be a mainnet artifact, as same clients (with same PoW requirement I assume) are used to post/subscribe. Also, I see that the postMessage call is not providing a maximum PoW calculation time, so it should normally reach the PoW target.

[oskarth]

Trying to run with https://github.com/status-im/status-go/tree/develop/_assets/systemd/bootnode and using that enode with local config I get:

INF 2019-12-12 12:50:50+08:00 Failed to bond with bootstrap nodes, retrying topics="discovery" tid=6707
TRC 2019-12-12 12:50:50+08:00 Bonding to peer topics="discovery" tid=6707 n=Node[1.171.46.175:30303]
TRC 2019-12-12 12:50:50+08:00 >>> ping topics="discovery" tid=6707 n=Node[1.171.46.175:30303]
TRC 2019-12-12 12:50:51+08:00 Bonding failed, didn't receive pong from topics="discovery" tid=6707 n=Node[1.171.46.175:30303]
INF 2019-12-12 12:50:51+08:00 Failed to bond with bootstrap nodes, retrying topics="discovery" tid=6707

The bootstrap service is running (lsof -i :30303) and nim-eth is trying to connect to that node, similar to Dockerbootnode placeholder

[kdeme]

You could now use the wakunode at https://github.com/status-im/nimbus/pull/437 .

It is far from finished but should be usable to start a bootnode, and several other nodes and post messages through the RPC interface (via web3, plain curl, and hopefully soon nim-web3).

I will try to add the actual Waku Mode settings soon.

[oskarth]

You can run nodes e.g. like this:
./build/wakunode --log-level:DEBUG --bootnode-only --nodekey:5dc5381cae54ba3174dc0d46040fe11614d0cc94d41185922585198b4fcef9d3
./build/wakunode --log-level:DEBUG --bootnodes:enode://e5fd642a0f630bbb1e4cd7df629d7b8b019457a9a74f983c0484a045cebb176def86a54185b50bbba6bbf97779173695e92835d63109c23471e6da382f922fdb@0.0.0.0:30303 --rpc --tcp-port:30304 --udp-port:30304
./build/wakunode --log-level:DEBUG --bootnodes:enode://e5fd642a0f630bbb1e4cd7df629d7b8b019457a9a74f983c0484a045cebb176def86a54185b50bbba6bbf97779173695e92835d63109c23471e6da382f922fdb@0.0.0.0:30303 --rpc --tcp-port:30305 --udp-port:30305 --rpc-binds:localhost:8546
And then through the RPC interfaces subscribe filters and/or post messages, etc.
You can test this just with curl or use web3 + node.js for now.
some test code I still had laying around: https://gist.github.com/kdeme/17006627154cbb796df845a0b7a14fb2
Gist
whisper web3 tests
whisper web3 tests. GitHub Gist: instantly share code, notes, and snippets.
The curl way, quite less user friendly :):
curl --header "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"shh_version","params":[],"id":1}' localhost:8545
Anyway, the idea would be that I update nim-web3 to include these JSON RPC calls. And then simulations could be written in nim I think.
https://github.com/status-im/nim-web3/
GitHub
status-im/nim-web3
Contribute to status-im/nim-web3 development by creating an account on GitHub.
Some scripting to startup the node setup (could be also via some docker images), and then some nim app to send/subscribe messages for several nodes. And then ideally some metrics enabled for monitoring also.
But other suggestions are welcome

[oskarth]

Brief update: Using a local network with bootstrap node etc it is running over lo, and with RPC tools like nethog (process oriented) become less useful. Worthwhile to:

• TODO: Try running through Docker for cleaner stats
• TODO: Get envelope counter up and running to get basic multiplier stats

See https://github.com/status-im/nimbus/pull/437/files#diff-a3fe75c397a0d5a4da33c4d6a48f75b7 for current runner code

FYI @adambabik too

[oskarth]

Brief update:

1. Messed around a bit with Docker, but need to setup a better base image with all Waku deps etc (had some issues with Nimbus base image), then we could a similar style to https://github.com/status-im/status-protocol-go-bandwidth-test for out of band verification. Also see go build -mod vendor' returned a non-zero code:  status-im/status-protocol-go-bandwidth-test#1 for current issue with go setup (didn't try to debug/fix)

2. We have metrics now, thanks to @kdeme!
   See https://github.com/status-im/nimbus/pull/437 README for latest instructions.

3. Next step is probably to add some background noise to show diff with waku-server and waku-client, as well as to script multiple nodes, etc.

4. I'll probably not have time to look at this more before holidays. Didn't get as far as I would've liked with this but oh well. If anyone wants to pick it up please go ahead.

Addendum: Check beacon chain Dockerfile, possibly more up to date

[kdeme]

Updated https://github.com/status-im/nimbus/pull/437, needs further work on network set up + traffic generation and some other items but a hardcoded (star) network can be launched and traffic can be generated by running quicksim, see readme.

[kdeme]

So I think the basic version fulfils the requirements:
https://github.com/status-im/nimbus/tree/master/waku#testing-waku-protocol

Network can be started in Star, FullMesh or via discovery (v4). Configurable amount of peers can be started. Additionally a light node with using bloom filter and a light node using set topic list is started. Metrics are collected of all nodes and some are printed also on the console.

Follow-up issue(s) would contain still:

• Extend with mailserver -> mail client not finished in nim-eth Waku, mail server not started yet
• Extend with multiple clients (geth) + option to user Docker
• Extend with more adverse conditions (network drops, spikes, etc)
• Add/measure message confirmations? -> no confirmations yet in nim-eth Waku

The Waku version tested does not use latest spec yet (alist).

[oskarth]

Awesome work @kdeme, do you want to post a mini write up in Vac forum or Discuss with screenshots of the configurations and roughly what it means? Think a lot of people would be interested

[oskarth]

I'll probably include simulation numbers in upcoming write-up, so if you prefer I can do above (or we can do both)

[kdeme]

Created an issue with some follow-up items: https://github.com/status-im/nimbus/issues/458
Feel free to comment on what would be still important to add

[oskarth]

Thanks! This issue can probably be closed