feat: implement p2p based networking using `iroh-net` #404

dignifiedquire · 2024-05-15T19:52:43Z

This the start to run the networking connections over iroh, allowing direct connections between players.

General Architecture

The online matchmaking server does not proxy connections anymore, it only introduces players to each other. After that iroh connections are used (which are either direct or relayed, in all cases e2e encrypted)
The Lan and Online discovery services, only function as such, discovering other players, after this has happened, all work is passed to Socket which is the p2p version of transferring game state between players

Testing it out

There is a branch of jumpy using this, which can be used to try it out
https://github.com/dignifiedquire/jumpy/tree/feat-iroh-networking

Matchmaker Servers

Europe: dkv5qztdu75wgtkyukkmhemz25adco7jrplzockgqgzvzl3d3z4q
US: tkj2gohdqx2fjqm24s6l6m7ehecntlcz72kzliwjy5ezs6yfdo6q

Work to be done

browser, will likely have to be in some form of disabling this when running in wasm (for now)
cleanup
better error handling

for bones-matchmaker server & client

zicklag

I just reviewed the code, and I'm very happy with it. 👍

I haven't tested it yet, I'll try to test it on LAN if I get the chance, but all the changes look straight-forward. If you don't count the Cargo.lock file, you deleted more than twice as much code as you added!

My only minor thought is that before merging we should try to figure out which tokio features we need and remove the full feature.

Huge thanks for this! The stuff this can help enable in the bigger picture is really exciting!

MaxCWhitehead · 2024-05-16T02:28:44Z

Yeah this is great work - excited to play with it more. Once we get this in bones can deploy another matchmaker and switch jumpy over and see how things go.

MaxCWhitehead · 2024-05-16T04:03:57Z

I tried testing this with a docker imagine running on my same machine as clients, and found there is high latency and spam of this:

2024-05-16T03:58:55.046961Z  INFO poll_send{me=juiy6aqwnxrhan5j}:get_send_addrs{node=g3vsalpcnyeyqk6m}: iroh_net::magicsock::node_map::node_state: new connection type typ=mixed

I am curious how publishing the IP of server works when trying to connect to node ID from inside the LAN (or in my case on the same host) - it seemed to connect, but did not seem to have a reliable / direct UDP connection. (It seems possible there is something we have to change to handle this better - just wanted to note as while I would normally specify the localhost or LAN address to do this, I am no longer sure how to do so with node id.

anywho haven't debugged this much but going to throw this up in a more legit configuration on a VPS in same region as another non-iroh matchmaker to compare with.

EDIT:
I think I missed the fact we are on p2p now and not just relaying through matchmaker with iroh connections - testing with a remote server with two local clients I still get same logs / not great ping (seems that using relay https://use1-1.relay.iroh.network which I am 85ms away from).

But the logs show my two clients are sending to each other (our server is not MITM anymore). So issue is not related to location of the matchmaking server, but is impacted by relay as not able to negotiate direct connections on same host. Will have to test with two clients on separate hosts in same lan next.

MaxCWhitehead · 2024-05-16T06:39:53Z

In case it's helpful for anyone got a tiny server running matchmaker (as of commit e075fc6a, in Texas, USA) can use for testing.

node id:
37zobpkk3vyheqonfph55vh5p5w5wn2ipx6qixcffkfox4gdzpsq

dignifiedquire · 2024-05-16T09:20:47Z

But the logs show my two clients are sending to each other (our server is not MITM anymore). So issue is not related to location of the matchmaking server, but is impacted by relay as not able to negotiate direct connections on same host. Will have to test with two clients on separate hosts in same lan next.

Yes, that is what I was suspecting the issue to be as well, especially the log line including mixed connection type, means the system is not able to determine if the direct connections are fully reliable.

Running on my machine locally (two games, same host) things worked out well, it might be we are having some trouble detecting appropriate IP addresses due to docker isolations.

dignifiedquire · 2024-05-16T09:21:39Z

My only minor thought is that before merging we should try to figure out which tokio features we need and remove the full feature.

Yes, fully agreed, I use this to just say give me everything during development and then remove the pieces that I don't end up needing in the end.

Arqu · 2024-05-16T10:29:40Z

I'm just going to sling a couple wild guesses, maybe they lead somewhere.

Is the docker running the network in host or bridge mode for the image. Bridge might interfere here as you basically get a docker "subnet" which then gives bogus IP addresses internally. Technically you need to have a local quasi stun so that it gives the right IP addr of the machine not the container.
Port matching might be at play, dunno if it's possible to just expose all ports in bridge mode

Ideally the next thing that gets tested is to try running in HOST mode which binds to your local network interface/stack. That should hopefully resolve both and give us a better understanding of the issue.

dignifiedquire · 2024-05-16T18:23:51Z

framework_crates/bones_framework/src/networking/socket.rs

+    }
+}
+
+impl NetworkSocket for Socket {


Socket is the only implementor of this trait now, could be removed as well, but that's your call

MaxCWhitehead · 2024-05-16T20:27:11Z

I'll try adjusting the container's network mode on local / remote deployments and see if that helps. I'll also test matchmaker running locally without a container. (Problem doesn't have to block merge, just sorting it out)

MaxCWhitehead · 2024-05-16T23:23:48Z

Updating the remote mm server container to run with host networking (docker run -p 8943:8943/udp --network host matchmaker-iroh) did not resolve the issue with two clients on same host. It is up at 37zobpkk3vyheqonfph55vh5p5w5wn2ipx6qixcffkfox4gdzpsq in case anyone wants to test the configuration - but fairly certain not a docker issue.

I am running another mm server on VPS in same region natively (cargo run) instead of in container at 7v77r7qofyrz4ms4y6offrfji377l5qqatxjqa34z57tgcv7fpkq, I get the same issue. So likely an issue with my local network configuration.

Will do some more tests and look into it further as I have the time. (Next test is two clients separate hosts under same LAN, then can figure out if should be looking host configuration or router / NAT).

MaxCWhitehead · 2024-05-16T23:28:39Z

@dignifiedquire re:

browser, will likely have to be in some form of disabling this when running in wasm (for now)

Not sure if you are tracking this but we do not support multiplayer in wasm, basically just a demo atm. So disabling all networking in wasm is sufficient resolution, just need to make sure wasm builds.

MaxCWhitehead · 2024-05-16T02:26:44Z

other_crates/bones_matchmaker/src/lib.rs


-    // Generate certificate
-    let (cert, key) = certs::generate_self_signed_cert()?;
+    let secret_key = iroh_net::key::SecretKey::generate();


Thinking about how we deploy our matchmaker - if I understood correctly, we want a persistent secret key for server so the node ID (public key) is the same and if we set a default matchmaking server by node_id in Jumpy / on game side, so if restart / move server it will still be connectable.

Maybe read from env var or generate if not found so we can configure a deployment with persistent ID?

If I understood correctly, looks like we are using the global discovery method mentioned here, the alternative would be ship a default ticket? I think discovery sounds like a good bet so don't have to reissue ticket if move server away from closest DERP. I was curious about the mentioning that DNS record must be republished as DHT nodes may go away - is this something we need to handle, or is there a service / loop somewhere here refreshing this occasionally while server is up? Not sure how much of a concern this is.

I read the other article again on DNS - it sounds like we may be using a centralized DNS server / the DHT stuff is not a concern here. Correct me if I misunderstood though. Thanks!

Yeah, as far as I understand so far, it updates a centralized DNS record so that other nodes can resolve our node ID, so I think if we persist our key, then we can keep the same node ID and people will be able to find us, even on a different server/ip.

We should probably read the secret key from an environment variable, and I like the idea of generating if one isn't supplied, we should just log a warning to make it clear that it will be different every time the server is started.

in the iroh main library we store the keys on disk in the same format as openssh does, this soul easy to add here, but could also use an env variable

dignifiedquire added 11 commits May 14, 2024 22:46

feat: use iroh-net instead of quinn for networking

21eec13

for bones-matchmaker server & client

draft for using iroh-net for networking in bones_framework

12cdc3d

cleanup

68674e8

make network init work

3cedef2

improve runtime handling

a86276c

fixup tokio runtime

e274f11

fmt

13a5147

feat: p2p play matchmaking for the example

4636e06

p2p networking working

61955ba

increase size limit for matchmaking message

70ace6e

remove dead code

e075fc6

zicklag reviewed May 16, 2024

View reviewed changes

dignifiedquire added 5 commits May 16, 2024 11:37

simplify network loops

b196ca6

remove dead crate: quinn_runtime_bevy

8ec37c9

improved error handling in online search

607df47

improve error handling in socket.rs

75eadc7

improve error handling in lan.rs

68ac4db

dignifiedquire added 2 commits May 16, 2024 20:13

remove unused tokio features

063581f

cleanup: remove boxednonblockingsocket

912df1b

dignifiedquire commented May 16, 2024

View reviewed changes

dignifiedquire changed the title ~~[WIP] Implement p2p based networking using iroh-net~~ Implement p2p based networking using iroh-net May 16, 2024

dignifiedquire marked this pull request as ready for review May 16, 2024 20:30

fixup: docs and clippy

5f8b0f3

dignifiedquire changed the title ~~Implement p2p based networking using iroh-net~~ feat: implement p2p based networking using iroh-net May 17, 2024

MaxCWhitehead approved these changes May 17, 2024

View reviewed changes

MaxCWhitehead enabled auto-merge May 17, 2024 23:23

MaxCWhitehead added this pull request to the merge queue May 17, 2024

Merged via the queue into fishfolk:main with commit 3cb6a88 May 17, 2024
10 of 11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement p2p based networking using `iroh-net` #404

feat: implement p2p based networking using `iroh-net` #404

dignifiedquire commented May 15, 2024 •

edited

Loading

zicklag left a comment

MaxCWhitehead commented May 16, 2024

MaxCWhitehead commented May 16, 2024 •

edited

Loading

MaxCWhitehead commented May 16, 2024 •

edited

Loading

dignifiedquire commented May 16, 2024

dignifiedquire commented May 16, 2024

Arqu commented May 16, 2024

dignifiedquire May 16, 2024

MaxCWhitehead commented May 16, 2024 •

edited

Loading

MaxCWhitehead commented May 16, 2024 •

edited

Loading

MaxCWhitehead commented May 16, 2024

MaxCWhitehead May 16, 2024

MaxCWhitehead May 16, 2024

zicklag May 18, 2024 •

edited

Loading

dignifiedquire May 18, 2024

feat: implement p2p based networking using iroh-net #404

feat: implement p2p based networking using iroh-net #404

Conversation

dignifiedquire commented May 15, 2024 • edited Loading

General Architecture

Testing it out

Matchmaker Servers

Work to be done

zicklag left a comment

Choose a reason for hiding this comment

MaxCWhitehead commented May 16, 2024

MaxCWhitehead commented May 16, 2024 • edited Loading

MaxCWhitehead commented May 16, 2024 • edited Loading

dignifiedquire commented May 16, 2024

dignifiedquire commented May 16, 2024

Arqu commented May 16, 2024

dignifiedquire May 16, 2024

Choose a reason for hiding this comment

MaxCWhitehead commented May 16, 2024 • edited Loading

MaxCWhitehead commented May 16, 2024 • edited Loading

MaxCWhitehead commented May 16, 2024

MaxCWhitehead May 16, 2024

Choose a reason for hiding this comment

MaxCWhitehead May 16, 2024

Choose a reason for hiding this comment

zicklag May 18, 2024 • edited Loading

Choose a reason for hiding this comment

dignifiedquire May 18, 2024

Choose a reason for hiding this comment

feat: implement p2p based networking using `iroh-net` #404

feat: implement p2p based networking using `iroh-net` #404

dignifiedquire commented May 15, 2024 •

edited

Loading

MaxCWhitehead commented May 16, 2024 •

edited

Loading

MaxCWhitehead commented May 16, 2024 •

edited

Loading

MaxCWhitehead commented May 16, 2024 •

edited

Loading

MaxCWhitehead commented May 16, 2024 •

edited

Loading

zicklag May 18, 2024 •

edited

Loading