-
Notifications
You must be signed in to change notification settings - Fork 0
Capstone Implementation Weekly Reflection
This week, I made excellent progress, starting with formally joining the DN42 network. I completed the registration process, including reserving an ASN and IPv4 and IPv6 address space. I also began the process of setting up infrastructure, specifically provisioning the Supermicro server with Proxmox and deploying RTR servers onto it, which will enable the router to validate route origins to help prevent BGP route hijacking. There is still more work to be done on this front, including deploying the other services like DNS. I also identified several good starting peers with whom I can connect once the router is set up, but at this point I am blocked by not having access to the Protectli which I will install VyOS on, which will hopefully be resolved this coming week. I also do not know whether I will receive access to a public IP, which is essential to start sprint 2, when I will set up my first peer.
One of the major learnings this week was simply understanding how ROA actually works. Route Origin Authorization involves the use of a cache server which downloads route signatures (typically, this file contains signatures proving the ownership of individual netblocks by their associated ASNs, but DN42 has opted not to use that feature), then uses an RPKI to Router (RTR) server which consumes the database to allow the router to check incoming route advertisements and ensure they come from the correct ASN.
Minus a few hiccups getting Proxmox installed, primarily to do with the IPMI but also with bugs in Proxmox's ZFS install, I did not encounter any issues completing this week's tasks. As a result of not having access to the Protectli, I did shuffle around some objectives. Until the router is available, I will continue working on the supporting services, including DNS and an API to scrape the registry and build the RPKI database. Once it is available, these may be pushed back to accomodate for setting it up in preparation for peering. Regardless, sprint 1 is currently on track.
Despite being blocked on a couple of external dependencies, I made some strides in regards to planning and setting up the necessary infrastructure for next Sprint's actual joining to the network. During the beginning of the week, I began working on IP address assignments and planning firewall rules and zones for the Production Network. This document gradually evolved as I weighed the pros and cons of segmenting the network further or decreasing the complexity of the rules.
I also set up a "staging" VyOS virtual machine, which allowed me to run the necessary commands to configure the router without actually having access to the Protectli. Once in my posession, I will be able to copy and paste the commands in to set up most of the necessary services and interface configurations. It also allowed me to experiment further with different services and firewall setups, so that the configuration can be more or less complete and optimized by the time I have access to the router. I have yet to figure out what is hopefully a simple issue with SLAAC for distributing IPv6 addresses, but otherwise the configuration is functional so far.
Using the staging router, I've also prepared the supporting services and networks for deployment, as well as setting up the Proxmox networks to bridge with the router. The staging router is able to reach both RTR servers running on Proxmox, and I can even VPN in using an address from my DN42 allocation, although it of course is not routed to the rest of the DN42 network, so this isn't useful yet. I'm still on track for this sprint, although somewhat concerned that there doesn't appear to be any progress getting a public IP yet.
This week unfortunately had the slowest progress of the entire sprint, but it was not without its successes. I continued to be blocked by several dependencies, though at this point the Protectli is available and Leahy Center is an option for a public IP if needed. I finished some of the last required firewall configurations, at least in the staging environment, meaning it will take much less time to set up the Protectli when I get my hands on it. Part of this involved fixing the SLAAC issue, which turned out to be a simple firewalling issue preventing router advertisements from escaping.
I've also set up a Linux Mint management box, giving me a desktop environment accessible through the Proxmox SPICE console to remotely administer devices using SSH or a web browser. Because it is a management box, however, it does not currently have access to DN42, nor do I plan to add that functionality, since the network will be accessed by Wireguard VPN.
Unfortunately, due to the hold-ups on the router and public IP, I will have to get these set up at the beginning of Sprint 2 instead. This should not prevent me from connecting to the first peer soon, but it may result in some other supporting services like DNS being pushed back further.
Once again, I made little progress this week. While the public IP should be authorized soon, I've made the decision to transition to a virtual router since the Protectli fell through due to it being a barebones model, at least for the time being. I've also made the decision to work on the DNS servers ahead of time, before connecting to a peer. They will be practically useless until I do peer, but I can hopefully get a partially working server within a few days.
I'll also be updating the networking topology of the Proxmox hypervisor prior to connecting, since I had originally set up vSwitches for an external physical router. I'll need to know more about the physical topology and/or VLANs used in order to properly hook up the router to the outside world.
This week, there was some good news, in that I'll soon be receiving the public IP I need. In Week 3, I'll receive some guidelines and have it hooked up, hopefully meaning I can quickly peer to give the router a touch of the DN42 network by the end of the sprint. There has been essentially no other progress, unfortunately, as this was a major blocking milestone. Depending on the topology of the public link, I may need to mess with some of the host networking, but it should otherwise be fairly trivial to interface with the internet and from there, establish a peering.
Despite the presumed timeline from the previous week, and some minimal progress, the public IP still is not ready. I have made no additional progress this week, nor for the sprint as a whole.
This week, I was finally able to connect to the public IP provided by IS. I've begun the process of peering, but ran into a few issues which seem to be related to the VPN not connecting properly. Besides this, I'm now able to properly manage the other servers thanks to the internet being available again to them. I also took some time to confirm that the IP is isolated from the Champlain network, although this will make it difficult to VPN into the network from my laptop without chaining several VPN servers.
Next week, I'll have to sort out these issues first, but hopefully should be able to successfully peer and being exploring DN42.
This week, I sorted out the above issues. First, while I was struggling to connect to Kioubit, my first peer, updating to VyOS 1.4 made it possible to connect over IPv6 link-local addresses, so that I could properly exchange IPv6 routes. I tracked down the other issue to being related to Multiprotocol BGP, which allows a single-stack BGP session to exchange dual-stack routes. If two neighbors are set up for the same AS, some other BGP routers may reject one, which Kioubit appears to do. The fix was to instruct VyOS to only connect with BGP over link-local IPv6, and to exchange both IPv4 and IPv6 routes over this session. The IPv4 address is still used as a next-hop for actual traffic. Besides this, I was able to circumvent the inability to connect to the access VPN from my laptop by instead connecting from the management box, so that I can browse the network from there. Initial results are very promising, including speedtest results greater than 150mbps. Documentation for VyOS updated, and I included peer-specific and technology considerations in Peering.
Next week, I'll try to automate this process, and possible will get a jump start on another peer. I'll also try to set up DNS resolvers and authoritative DNS.
With DN42 peering set up, I got DNS turned up. Now, DNS works both as a resolver for any other .dn42 domains and reverse resolution, and as an authoritative server for champlain.dn42. This is thanks to BIND, which could handle both, including restricting recursive resolution to my local DN42 subnets. I also submitted a pull request to the DN42 registry to add champlain.dn42 so it could be resolved by other network members, though there is nothing for them to do right now except use the DNS servers.
I opted not to proceed with automation (at least not yet), since not every peering is the same and the time spent customizing any automation to fit the variety of scenarios may not be worth it. I will reevaluate after peering with more people, to see how similar or different, and how tedious it is.
Following last week, my pull request was accepted and now the domain champlain.dn42 is recognized by DNS servers across DN42, though the only records for now are just ns-1.champlain.dn42 and ns-2.champlain.dn42. However, I'm working on changing that. I started work on a new service, lg.champlain.dn42, a BGP looking glass service running Hyperglass. This application would use SSH to connect to the router and pull information about routes from it, as well as provide ping and traceroute services useful for troubleshooting network connectivity or simply collecting statistics about peers. Unfortunately, although the application is being served correctly, I've been unable to get the frontend working properly, it fails to even send requests for information from the router. I hope to solve this soon, but if not I may have to use an alternative application, though these may also have issues since like Hyperglass, they are also lacking in recent commits.
I have not yet started on other peering tasks including automation steps, though from my research I can still use Ansible, I just need to ensure my code is written to be idempotent since Ansible will not fully handle this.
While I started with the Hyperglass BGP looking glass last week, this didn't end up panning out (due to the issues mentioned last week), and I had to pivot to a different LG. The new one, Looking-Glass, supports the same commands, even though the UI is a little less desirable. This tool works fine, it also relies on connecting to the router using SSH, and while it claims to support VyOS, this is really only true for versions prior to 1.2, since it really support Vyatta. Vyatta used Quagga for BGP, and VyOS 1.2 and later use FRR, which is a Quagga fork with different syntax. Luckily, it also supports FRR, so simply changing this setting works perfectly. Luckily, Looking-Glass also supports setting a source IP for ping and traceroute commands, since running these commands normally on the router actually fails since it has to infer the IP, and chooses the interface IPv4, which is a dummy address used only for the single hop to the peer.
I'll be working on peering again this week, and writing some automation if it fits. The advantage of adding more peers, besides improving the bandwidth capacity, is to add redundancy in the event one goes down, which is where BGP really shines.