Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: node discovery via DNS #2045

Merged
merged 80 commits into from
Apr 15, 2024
Merged

feat: node discovery via DNS #2045

merged 80 commits into from
Apr 15, 2024

Conversation

Frando
Copy link
Member

@Frando Frando commented Feb 27, 2024

Description

This enables global node discovery over DNS, i.e. dialing nodes by just their node id.

Current setup is as follows:

  • When dialing a node only by its NodeId, the new DnsDiscovery service is invoked. It will lookup a TXT record at (by default) _iroh_node.b32encodednodeid.testdns.iroh.link over regular DNS or DNS-over-http. Right now the Cloudflare DNS servers are configured. At testdns.iroh.link we run a custom DNS server
  • Nodes publish their Derp address to this DNS server through Pkarr signed packets. This is an intermediate step, we decided that the publishing by default should not happen by the nodes directly but mediated through the Derp servers. Work for the latter happens in feat: Publish self-announces as pkarr signed packets through the derper #2052

This PR thus allows for the following:

# terminal/computer 1
$ iroh console --start
Iroh is running
Node ID: qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua
$ blob add ./myfile
...
Blob: o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba
...

# terminal/computer 2
iroh console --start
blob get o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba --node qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua

Notes & open questions

  • Misses node configuration in the CLI for the node origin domain (right now hardcoded to testdns.iroh.link). How do we want to expose this - CLI flag? Or in the config file? I'd say the latter.

  • Offload publishing to the Derpers - see feat: Publish self-announces as pkarr signed packets through the derper #2052

  • Right now the records published via pkarr have a TTL of 30s - the iroh-dns-server will use that TTL as-is when serving the records over DNS. both can/should change?

  • We can also very easily allow to lookup nodes not only by NodeId, but by any domain name. In the iroh-dns crate I included an example resolve that does just that. By setting a CNAME record you can even use any domain and simply point to the record hosted at the testdns.iroh.link server.
    So if, on your custom domain, you added a record like this

_iroh_node.frando.n0.computer CNAME _iroh_node.qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua.iroh.link.

You can use this with the example to resolve to the node id and derp addresses:

cargo run --example resolve -- domain frando.n0.computer

Change checklist

  • Self-review.
  • Documentation updates if relevant.
  • Tests if relevant.

Closes #1248

@b5
Copy link
Member

b5 commented Mar 6, 2024

Does this close #1248?

@Frando Frando changed the base branch from main to feat/combined-discovery March 7, 2024 16:16
@Frando Frando changed the title (WIP) feat: node discovery via DNS feat: node discovery via DNS Mar 7, 2024
@Frando
Copy link
Member Author

Frando commented Mar 7, 2024

Does this close #1248?

Yes

@Frando
Copy link
Member Author

Frando commented Apr 8, 2024

@Frando I think this should also have an integration test, running the actual iroh-dns server

~~Added an online test that uses dns.iroh.link for now. ~~
Edit: Removed the online test. The rate limits we set in the n0 pkarr relay are hit in CI. We should think about this some more actually: If we enable pkarr publishing by default, which this PR does, then our DNS server / pkarr relay will be hit a lot from CI (not only ours, but any CI that creates an Iroh node with the defaults). At least with the current, very low, limits. Of course we can increase the limits. But CI is very close to bad actor, when it comes to the "amount of publishes from a single IP address".

The PR also has an integration test with a very simple test server, see iroh-net/src/discovery.rs.

I will add a test that spawns an instance of the actual iroh-dns server code once we have a PR to move the server from https://github.com/n0-computer/iroh-dns-server/ into the iroh repo.

Frando added 15 commits April 8, 2024 22:53
## Description

Imports https://github.com/n0-computer/iroh-dns-server into this repo.

See n0-computer/iroh-dns-server#5 for previous review/discussion.

Now includes an integration smoke test in `iroh-dns-server/src/lib.rs`.

## Notes & open questions

I *think* I addressed most review points that came up in the initial
review. Prominently still open is:

* The `redb` store is used from async context but only exposes a sync
interface (redb default). I think this is fine for medium load. However
for better performance we should reuse transactions, which likely means
we need an actor on a separate thread, as we do in iroh-bytes and
iroh-sync.

## Change checklist

- [ ] Self-review.
- [ ] Documentation updates if relevant.
- [ ] Tests if relevant.
@Frando Frando enabled auto-merge April 15, 2024 10:48
@Frando Frando added this pull request to the merge queue Apr 15, 2024
Merged via the queue into main with commit 72384ce Apr 15, 2024
20 checks passed
@Arqu Arqu deleted the feat/dns branch April 19, 2024 12:50
ppodolsky pushed a commit to izihawa/iroh that referenced this pull request Apr 20, 2024
## Description

This enables global node discovery over DNS, i.e. dialing nodes by just
their node id.

Current setup is as follows:

* When dialing a node only by its NodeId, the new `DnsDiscovery` service
is invoked. It will lookup a TXT record at (by default)
`_iroh_node.b32encodednodeid.testdns.iroh.link` over regular DNS or
DNS-over-http. Right now the Cloudflare DNS servers are configured. At
`testdns.iroh.link` we run a custom [DNS
server](https://github.com/n0-computer/iroh-dns-server/tree/main)
* Nodes publish their Derp address to this DNS server through Pkarr
signed packets. This is an intermediate step, we decided that the
publishing by default should not happen by the nodes directly but
mediated through the Derp servers. Work for the latter happens in n0-computer#2052

This PR thus allows for the following:
```sh
# terminal/computer 1
$ iroh console --start
Iroh is running
Node ID: qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua
$ blob add ./myfile
...
Blob: o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba
...

# terminal/computer 2
iroh console --start
blob get o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba --node qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua
```


<!-- A summary of what this pull request achieves and a rough list of
changes. -->

## Notes & open questions

* Misses node configuration in the CLI for the node origin domain (right
now hardcoded to `testdns.iroh.link`). How do we want to expose this -
CLI flag? Or in the config file? I'd say the latter.

* Offload publishing to the Derpers - see n0-computer#2052 

* Right now the records published via pkarr have a TTL of 30s - the
iroh-dns-server will use that TTL as-is when serving the records over
DNS. both can/should change?

* We can also *very* easily allow to lookup nodes not only by NodeId,
but by any domain name. In the `iroh-dns` crate I included an example
`resolve` that does just that. By setting a `CNAME` record you can even
use any domain and simply point to the record hosted at the
`testdns.iroh.link` server.
So if, on your custom domain, you added a record like this
```
_iroh_node.frando.n0.computer CNAME _iroh_node.qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua.iroh.link.
```
You can use this with the example to resolve to the node id and derp
addresses:
```
cargo run --example resolve -- domain frando.n0.computer
```

<!-- Any notes, remarks or open questions you have to make about the PR.
-->

## Change checklist

- [x] Self-review.
- [x] Documentation updates if relevant.
- [x] Tests if relevant.

Closes n0-computer#1248

---------

Co-authored-by: Kasey <kasey@n0.computer>
Co-authored-by: Asmir Avdicevic <asmir.avdicevic64@gmail.com>
Co-authored-by: Ruediger Klaehn <rklaehn@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

magic_endpoint: allow dialing with just PeerId
7 participants