Btw, out of the box that is not as easy as you would think. The problem:
- OpenNSL, the Open Network Switch Library that is required by the Trident II switch chip only supports CentOS 6.5.
- Facebook Wedge 16X (pdf) comes with CentOS 6.5 pre-installed, reason see above.
- FBOSS, the Facebook Open Switching System Software for controlling the switch comes with build instructions for Ubuntu 14.04. But it requires OpenNSL kernel modules which are only available for CentOS 6.5.
Nowadays if you have to run Ubuntu on CentOS you probably plan to use Docker. But the 2.6 kernel of CentOS 6.5 is not officially supported by Docker. Fortunately we found that Docker version "1.7.1, build 786b29d/1.7.1" works surprisingly reliable on our Wedge.
Yet that is still not the end of the hack. The OpenNSL library wants to run a real-time scheduling task. That task needs to run exclusively for longer
than allowed by default for a cgroup-managed task. We had to do some additional hacking around cpu.rt_runtime_us as described in Docker Issue 13983.
We added some backdoor to our FBOSS container and some special startup script to be able to orchestrate that multi-step startup procedure.
This repository contains all the scripts and steps required to get that hack going. We finally not only got FBOSS running under CentOS on our Wedge but we also ended up with a reliable Docker environment which allowed us to do things that where beyond our imagination.
Picture, slide 33: Facebook Keynote - Open Networking Summit 2016 (slides)
We will contribute the complete ToRC setup including all source code to Open Source as a reference architecture for a Top-of-Rack-Controller based autonomous rack .. stay tuned.
As a starting point we have some documentation in the ./docs folder that shows you how to docker-enable your Facebook Wedge Switch, how to build fboss, how to containerize fboss, and how to run it.