DN42 Project Decomposition
This was created with MindMup for Google Drive and full map can be found at: MindMap for DN42 Project
Distributed Network 42, a dynamic interconnected network
The process of connecting to other members of the network, to share routes and enable connectivity to the rest of the network
There are a number of VPN/tunneling protocols to choose from, with varying advantages and adoption on DN42.
Testing should consider adoption on the DN42 network, performance, security, and compatibility with hardware and software.
Wireguard is a modern, fast, and secure VPN protocol which is well-suited for road-warrior and site-to-site scenarios but operates at Layer 3, making it difficult to use for internetworking (despite this, it is more commonly used in DN42 than other protocols)
OpenVPN is a mature VPN protocol which is typically used for road-warrior and site-to-site scenarios but can be customized to fit most use cases. It is also quite slow compared to the alternatives, and does not run well on commodity routers.
Tinc is a mesh peer-to-peer VPN protocol which is useful for internetworking scenarios, and has seen some adoption in DN42
GRE, or Generic Routing Encapsulation, is well-suited to peering since it can carry any Layer 3 protocol and does not have overhead for encryption (by default)
SIT is a commonly used encapsulation protocol for transporting IP packets with minimal MTU overhead
IPsec, a protocol for encrypting packets, supports several transport modes that can be used to establish encrypted connections allowing IP transport. Like OpenVPN, it has quite a bit of encryption overhead, but this can be mitigated using devices with IPsec offloading or hardware-accelerated encryption.
There are a number of BGP daemons with varying features to choose from.
Testing should consider how easy it is to configure, which features it supports, the level of maturity and maintenance, and compatibility with operating systems.
Bird or Bird2 are stable BGP implementations which support most necessary functionality, though they can be clunky to administer
Quagga is a powerful and well-maintained BGP daemon which supports advanced features like BGP multipathing, and is fairly easy to administer
OpenBGPD is a mature BGP daemon created for BSD-based operating systems by the OpenBSD team
All BGP daemons must support RPKI. which is used to verify route signatures to prevent BGP hijacking
If using a routing OS or routing hardware, then the existing implementation is likely the only choice
There are several operating systems and firmware options to choose from for running the router.
Testing should consider performance, support for various BGP/VPN software, hardware/virtualization support, and how easy it is to configure and automate. A bonus is that it is compatible with the Looking Glass software.
Any flavor of Linux, using open source BGP daemons and VPN software, is easily capable of doing the routing needed
FreeBSD, much like Linux, has many options for BGP daemons and VPN software, and is often used as it is considered a more stable and hardened operating system
VyOS is a powerful router firmware based on Debian which is configured via a CLI and supports most of the needed functionality
EdgeOS is a fork of VyOS created for the Ubiquiti EdgeRouter line, which also features a web GUI, however it typically lags behind VyOS in updates
Untangle is a powerful router OS configured with a web GUI, often used for its advanced firewalling/filtering solutions
pfSense is a popular router firmware configured using a web GUI which implements most of the needed features and supports plugins, though there has been some controversy about their recent development practices
There are a number of hardware router options to evaluate, including virtualizing the router.
Testing should consider performance (including acceleration features), OS/firmware support, and management/orchestration.
Protectli devices are designed to run open-source router firmware, and are likely the most flexible hardware option
The EdgeRouter runs a fork of VyOS with good support for most of the required technologies, though the firmware may be slightly out of date
Any generic PC with the necessary networking hardware can run various router OSs supporting BGP and VPNs, or run Linux/FreeBSD with the necessary packages installed
One of the most flexible options for running many router OSs is to virtualize them in vCenter and use Software-Defined Networking to connect them
In order to connect to the DN42 network, you must find peers to connect to that have routes to the rest of the network. The more the better.
The DN42 PingFinder is a good way to find the lowest-latency peers
People often come to the IRC channel looking for peers
Some people advertise that they are on DN42 on their blogs or social media and can be contacted to request to peer
In order to participate in the DN42 network, you must register for a number of internetwork resources like address space
Objects in the DN42 registry are usually associated with a person object containing contact details and a unique identifier
Objects in the DN42 registry can also be associated with organization objects, which have member Person objects
DN42 supports both IPv4 addresses (in the range 172.20.0.0/14) and IPv6 ULAs (fd00::/8), which must be selected from the available pool of addresses
A public key can optionally be included in the registration, which is used to verify changes to the registry and routes that are signed using ROA
The primary identifier for a registered DN42 network is the Autonomous System Number, which always starts with the AS424242
Domains ending in .dn42 can optionally be registered for and pointed to shared or self-hosted authoritative nameservers
Services that can be hosted on the configured network, which will become available to other DN42 members
Validates route signatures that arrive in BGP messages, used to prevent BGP hijacking
Responds to requests by the router to verify a route's signature, given its internal datastore of valid keys
Collects signing keys from the registry to determine whether a route must be checked and is valid, or should not be checked
A BGP looking glass displays what routes are known by the router in a web portal, and can look up additional information like ASNs
The webserver looks up the collected routes and displays them to a visitor
A daemon reaches out to the router and collects information about which routes have been added
An authoritative DNS server responds to queries for a specific domain's records
A data storage backend, either BIND files or a database as required by the particular DNS server, stores the records
An authoritative DNS server like PowerDNS or BIND, which responds to queries for a specific .dn42 domain(s)
Adding tooling to simplify what will likely become common tasks
Adding peers, including setting up VPN tunnels, is a repetitive and relatively easy-to-automate process
Setting up services with the right IPs and reachability via DNS can be automated
Using tools like DNSControl, adding and removing DNS records can be easily automated
Documentation that can help for maintaining and recreating the network
A high-level overview of how the BGP protocol works, including a technical breakdown, RPKI overview, and generic troubleshooting steps
Instructions for how to set up BGP on the router
Instructions for both setting up an RPKI validator and connecting the router to it
Step-by-step instructions for registering to join DN42 and finding peers
Low-level explanation of how BGP works over VPN tunnels
Instructions for how to set up the looking glass
Additional resources which must be requests
To peer with others, the VPN server must be exposed to the outside world via a public IP address, which should be static
To make it easier to connect, the VPN server should have a DNS record
Used for both testing and deployment of the network
A network which end-user devices can join to access the DN42 network
A NAT gateway used to allow access to the rest of the IPv4 DN42 network
VPN concentrator which allows access to the internal, NATted network