One lighthouse supports multiple overly networks

[samuelxhu]

This would be a feature request instead of a bug report.

In order to setup an overlay network, one needs at least one public IP address for lighthouse, but in practice, one would like one lighthouse (with a single IP address ) to host (support) multiple overlay networks.

As far as I know, Tinc VPN has this feature, which enables one single VM with a public IPv4 addess to support multiple instances of networks for multi tenants.

thanks,

samuel

[forfuncsake]

You can achieve this now by running multiple instances of nebula on the lighthouse machine, but using a different config/cert/listen port for each instance (and therefore "network").

[nbrownus]

Assuming the address spaces issued under each CA root does not overlap, you can trust each root issuing for devices at the lighthouse and re-use the single process. Just make sure to trust the root that signed the lighthouse certificate on each device.

[samuelxhu]

Thanks a lot. In this case , I would first try to run multiple instances of nebula on the lighthouse machine, for the purpose of serving multiple overlay networks for different users with a single machine. Concerning config/cert/listen port for each instance, could you please elaborate a bit more by giving an example or tutorial? best regards, Samuel

…

On Mon, Sep 28, 2020 at 3:44 AM Nathan Brown ***@***.***> wrote: Assuming the address spaces issued under each CA root *does not* overlap, you can trust each root issuing for devices at the lighthouse and re-use the single process. Just make sure to trust the root that signed the lighthouse certificate on each device. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#306 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYMCAOMCRV3AB2A2T4J7OTSH7TBLANCNFSM4R3BMX2Q> .

[victorhooi]

I asked something similar in this issue as well:

#251

so am very curious about the answer here.

Would it be possible to add a feature to allow a single Nebula client to connect to multiple instances? (Similar to what ZeroTier allows):

For right now - would be awesome if there was something in the docs about how to launch multiple instances, with a valid example setup for certs/config/ports etc?

[victorhooi]

Does anybody have a working example of multiple Lighthouse instances running on the same host, connecting to different overlay networks?

[caguiclajmg]

Does anybody have a working example of multiple Lighthouse instances running on the same host, connecting to different overlay networks?

What problems are you hitting? There should be no issue provided you run the nebula instances on distinct ports.

[mrbluecoat]

Assuming the address spaces issued under each CA root does not overlap, you can trust each root issuing for devices at the lighthouse and re-use the single process. Just make sure to trust the root that signed the lighthouse certificate on each device.

@nbrownus can you elaborate on this? If I have a single public IP (1.2.3.4) and I use a single port (4242) are you saying you can still achieve multi-tenant support using the overlay network address space? Something like this, perhaps:

mkdir /path/to/nebula_ca/abc
nebula-cert ca -name "abc-network" -ips "10.10.10.0/24" -out-crt "/path/to/nebula_ca/abc/ca.crt" -out-key "/path/to/nebula_ca/abc/ca.key"
nebula-cert sign -name "abc-lighthouse" -ip "10.10.10.1/24" -out-crt "/path/to/nebula_ca/abc/lighthouse.crt" -out-key "/path/to/nebula_ca/abc/lighthouse.key" -ca-crt "/path/to/nebula_ca/abc/ca.crt" -ca-key "/path/to/nebula_ca/abc/ca.key"
nebula-cert sign -name "abc-client1" -ip "10.10.10.2/24" -out-crt "/path/to/nebula_ca/abc/client1.crt" -out-key "/path/to/nebula_ca/abc/client1.key" -ca-crt "/path/to/nebula_ca/abc/ca.crt" -ca-key "/path/to/nebula_ca/abc/ca.key"

mkdir /path/to/nebula_ca/xzy
nebula-cert ca -name "xzy-network" -ips "10.20.20.0/24" -out-crt "/path/to/nebula_ca/xzy/ca.crt" -out-key "/path/to/nebula_ca/xzy/ca.key"
nebula-cert sign -name "xzy-lighthouse" -ip "10.20.20.1/24" -out-crt "/path/to/nebula_ca/xzy/lighthouse.crt" -out-key "/path/to/nebula_ca/xzy/lighthouse.key" -ca-crt "/path/to/nebula_ca/xzy/ca.crt" -ca-key "/path/to/nebula_ca/xzy/ca.key"
nebula-cert sign -name "xzy-client1" -ip "10.20.20.2/24" -out-crt "/path/to/nebula_ca/xzy/client1.crt" -out-key "/path/to/nebula_ca/xzy/client1.key" -ca-crt "/path/to/nebula_ca/xzy/ca.crt" -ca-key "/path/to/nebula_ca/xzy/ca.key"

So when ABC client1 connects to 1.2.3.4:4242 Nebula knows how to route it because it's using overlay network 10.10.10.* instead of 10.20.20.* ? And there's no security risk because the overlay networks aren't bridged, right?

[mrbluecoat]

I'm guessing the IP space approach above could also be combined with the port option as well, right? So 10.10.10.* on port 49152 could service a different network than 10.10.10.* on port 49153

[johnmaguire]

Related to #251.

[ZelnickB]

Assuming the address spaces issued under each CA root does not overlap, you can trust each root issuing for devices at the lighthouse and re-use the single process. Just make sure to trust the root that signed the lighthouse certificate on each device.

@nbrownus Sorry to resurrect an old issue, but what would happen if the address spaces did overlap?

For example, consider the following setup:

• FooNet, for my personal devices
• BarNet, for a company

FooNet and BarNet both have their own CAs and both use the 100.64.0.0/10 network. Each device on FooNet and BarNet has a pki.ca file containing both the certificate authority for the correct network (FooNet or BarNet, respectively) and a third certificate authority BazNet (used to sign the lighthouse certificate), also using 100.64.0.0/10. On the Internet, the lighthouse is publicly accessible 192.0.0.203:4242, and it has the IP address 100.64.0.1 in BazNet (which is unused except for the lighthouse nodes). The lighthouse is configured to trust the BazNet, FooNet, and BarNet certificates.

Now let's say that I have a personal device connected to FooNet with mesh IP 100.64.2.17 and a work device connected to BarNet with mesh IP 100.64.2.17. This shouldn't cause an issue since there is no device connected to both FooNet and BarNet (except for the lighthouse, which has tun.disabled set to true and therefore will not connect to either device). If all nodes on both FooNet and BarNet are configured to use the same lighthouse (at 192.0.0.203:4242 on the Internet), will an issue arise?

[johnmaguire]

@ZelnickB The Lighthouse tracks hosts by their Nebula IP. If multiple hosts connect to a Lighthouse with the same Nebula IP, there will be confusion across the network over who owns said IP.

For example, when the device at 100.64.2.123 requests the list of IP addresses for 100.64.2.17, how would the Lighthouse know which IPs to provide it? Those belonging to the device in BarNet, or those belonging to the device in FooNet? It doesn't... instead, the two devices will be "fighting" over that IP address, and you'll end up with a flaky, race-y split network.

This is not a supported configuration. Your best bet is to run separate Lighthouse processes for separate networks.

[ZelnickB]

Got it.

I haven't looked at the protocol specification, so I'm not sure if this would work, but one possible way to implement this feature request (as described by @samuelxhu) would be to store two key-value maps, one for FooNet and one for BarNet. Depending on the issuing CA for the certificate of the device that was contacting the lighthouse, the lighthouse could decide which map to use for reading/writing IP addresses.

For example, two networks with a lighthouse on the same IP at the same port might have transactions that look like this:

Clients: Alice (FooNet - 100.64.2.2/10), Bob (FooNet - 100.64.2.3/10), Charlie (BarNet - 100.64.2.2/10), and David (BarNet - 100.64.2.3/10)
Lighthouse: Frank
Networks: FooNet and BarNet

1. Alice → Frank: handshake — {authenticated with Alice's certificate}
2. Frank → Alice: handshake — (ack); (Frank stores 100.64.2.2 → Alice's WAN IP address in the FooNet key-value map, since Alice's certificate is from the FooNet CA)
3. Charlie → Frank: handshake — {authenticated with Charlie's certificate}
4. Frank → Charlie: handshake — (ack); (Frank stores 100.64.2.2 → Charlie's WAN IP address in the BarNet key-value map, since Charlie's certificate is from the BarNet CA)
5. David → Frank: handshake — {authenticated with David's certificate}
6. Frank → David: handshake — (ack); (Frank stores 100.64.2.3 → David's WAN IP address in the BarNet key-value map, since David's certificate is from the BarNet CA)
7. Bob → Frank: handshake — {authenticated with Bob's certificate}
8. Frank → Bob: handshake — (ack); (Frank stores 100.64.2.3 → Bob's WAN IP address in the FooNet key-value map, since Bob's certificate is from the FooNet CA)
9. David → Frank: IP resolution — {authenticated with David's certificate, 100.64.2.2}
10. (Frank finds the right IP to send to David)
    1. Frank checks the issuing CA of David's certificate and sees that it is the BarNet CA
    2. Frank looks up 100.64.2.2 in the BarNet hash table and finds Charlie's IP address but not Alice's IP address (since that is in the FooNet hash table)
11. Frank responds to David's resolution request
12. Bob → Frank: IP resolution — {authenticated with Bob's certificate, 100.64.2.2}
13. (Frank finds the right IP to send to Bob)
    1. Frank checks the issuing CA of Bob's certificate and sees that it is the FooNet CA
    2. Frank looks up 100.64.2.2 in the FooNet hash table and finds Alice's IP address but not Charlie's IP address (since that is in the BarNet hash table)
14. Frank responds to Bob's resolution request

[johnmaguire]

@ZelnickB Indeed, this is a possible approach. However, it is supported for an organization to run with host configs that trust multiple CAs in a bundle. In other words, a CA is not a network, a network can have many CAs (and in fact, must have two CAs during a CA rotation event) - so using a CA as the "boundaries" of a network may create other problems. (To say nothing of the complexity of configuring the network mapping, and ensuring that all Lighthouses are running the same config.)

This isn't meant to rule the option out, just pointing out that a solution isn't trivial here.

[maggie44]

Cross posting here to some relevant content on this issue for others to find: #823 (comment)