wgkex

wgkex (WireGuard key exchange) is the node management service of Freifunk München. Mesh nodes register their WireGuard public key with a central broker, which picks a gateway for them and distributes the key via MQTT to the workers running on the gateways. The workers add the peer to WireGuard and the VXLAN FDB, so that nodes can run BATMAN over VXLAN over WireGuard as encrypted, high-performance mesh links.

On the node side, wgkex is used by the Gluon package ffmuc-mesh-vpn-wireguard-vxlan.

• Architecture
• Quick start
  • Docker
  • Bazel
  • Python
• Configuration
• API
• Operations
• MQTT topics
• Development
• Contributing
• License

flowchart TD
    node["Gluon node<br/>(ffmuc-mesh-vpn-wireguard-vxlan)"]
    broker["wgkex broker<br/>(HTTP API)"]
    mqtt[("MQTT broker<br/>(e.g. Mosquitto)")]
    subgraph gw["Gateway (one per worker)"]
        worker["wgkex worker"]
        wg["WireGuard<br/>wg-{domain}"]
        vx["VXLAN FDB<br/>vx-{domain}"]
        route["Routes<br/>(netlink)"]
    end

    node -- "POST /api/v2/wg/key/exchange" --> broker
    broker -- "endpoint of chosen gateway" --> node
    broker -- "wireguard/{domain}/{worker|all}<br/>wireguard-worker/{worker}/{domain}/remove<br/>wireguard-worker/{worker}/rebalance" --> mqtt
    mqtt --> worker
    worker -- "wireguard-worker/{worker}/status<br/>wireguard-worker/{worker}/{domain}/data<br/>wireguard-metrics/{domain}/{worker}/connected_peers" --> mqtt
    mqtt --> broker
    worker -- pyroute2 --> wg & vx & route
    node -. "WireGuard + VXLAN tunnel" .-> wg

• Broker (wgkex/broker): a Flask app served by waitress. It validates key exchange requests, chooses a gateway weighted by load (/api/v2), and publishes the key via MQTT. It also serves Prometheus metrics and runs the optional rebalancing loop.
• Worker (wgkex/worker): runs on each gateway. It subscribes to the keys for its domains and adds them as WireGuard peers, routes and VXLAN FDB entries via netlink (pyroute2). It reports its status, endpoint data and connected peers back to the broker, removes stale peers, and supports drain mode for planned maintenance.
• WireGuard and VXLAN interface names are derived from the domain: the matching domain_prefixes entry is replaced by wg- / vx- (e.g. ffmuc_welt → wg-welt, vx-welt).

The broker publishes /api/v1 keys to all workers and /api/v2 keys only to the chosen worker; see MQTT topics and Docs/operations.md for details.

All components read their configuration from /etc/wgkex.yaml, or from the file in the environment variable WGKEX_CONFIG_FILE. Start from wgkex.yaml.example.

Images are published to ghcr.io/freifunkmuc/wgkex on every push to main (tags main and sha-<commit>) and for release tags vX.Y.Z. The image command selects the component (broker, the default, or worker):

docker run -d --name wgkex-broker -p 5000:5000 \
  -v "$PWD/wgkex.yaml:/etc/wgkex.yaml:ro" \
  ghcr.io/freifunkmuc/wgkex:main broker

docker-compose.yml starts Mosquitto, a broker and a worker. Without a mounted /etc/wgkex.yaml, the entrypoint generates a minimal config from the environment variables in env.example (WGKEX_DOMAINS, WGKEX_DOMAIN_PREFIXES, WGKEX_DEBUG, WGKEX_BROKER_LISTEN_HOST/_PORT (default 0.0.0.0:5000), MQTT_*). WGKEX_IMAGE_TAG selects the image tag (default main; the latest tag still points to v0.2.1). docker-compose.override.yaml.example shows how to mount configs or disable services.

bazel build //wgkex/broker:app //wgkex/worker:app
export WGKEX_CONFIG_FILE=$PWD/wgkex.yaml
./bazel-bin/wgkex/broker/app
# the worker needs root (or CAP_NET_ADMIN) to manage the interfaces
sudo -E ./bazel-bin/wgkex/worker/app

Bazelisk picks the Bazel version from .bazelversion. See Docs/development.md for a local setup with dummy interfaces.

Bazel uses Python 3.13 (see MODULE.bazel). To run from a checkout without Bazel:

python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements_lock.txt
export WGKEX_CONFIG_FILE=$PWD/wgkex.yaml
python3 -m wgkex.broker.app
sudo -E .venv/bin/python3 -m wgkex.worker.app

wgkex.yaml.example is the source of truth: it documents every option with its default and whether it applies to the broker, the worker or both. Overview:

The broker exposes:

Docs/openapi.yaml (OpenAPI 3) is the authoritative specification of requests, responses and status codes, and the reference for other client and broker implementations. View it with e.g. the Swagger Editor; //wgkex/broker:openapi_test checks the broker against it.

curl -s -X POST http://127.0.0.1:5000/api/v2/wg/key/exchange \
  -d '{"domain": "ffmuc_welt", "public_key": "o52Ge+Rpj4CUSitVag9mS7pSXUesNM0ESnvj/wwehkg=", "location": "muc"}'

{
  "Endpoint": {
    "Address": "gw04.ext.ffmuc.net",
    "Port": "40011",
    "AllowedIPs": ["fe80::27c:16ff:fec0:6c74"],
    "PublicKey": "TszFS3oFRdhsJP3K0VOlklGMGYZy+oFCtlaghXJqW2g="
  }
}

location is optional; workers configured with the same location are preferred.

Detailed guides are in Docs/operations.md and Docs/metrics.md:

• Load balancing and rebalancing: weighted, randomized gateway selection with pending assignments, location preference, optional active rebalancing, and multiple broker instances.
• Drain mode and planned reboots: SIGUSR1/SIGUSR2 or a drain file, plus a step-by-step reboot runbook.
• Stale peer cleanup and worker internals.
• Rate limiting and key block-/allowlists.
• Running behind a reverse proxy: broker.trusted_proxies and broker.log_client_ip.
• Prometheus metrics and Grafana dashboards: metric reference and the dashboards in contrib/grafana for Prometheus and InfluxDB/Telegraf.

All topics are defined in wgkex/common/mqtt.py. {worker} is the worker's hostname.

The remove, rebalance and assignment topics, the targeted wireguard/{domain}/{worker} keys and status 2 were added without breaking older components, so brokers and workers can be upgraded one at a time:

• Older workers subscribe to wireguard/{domain}/+ and add every key; newer workers only act on all and their own hostname. Older brokers publish every key to all. The bandwidth savings of targeted publishing only apply once both sides are updated.
• Older workers ignore remove and rebalance messages; older brokers ignore assignment messages.
• Older brokers treat any status >= 1, including draining (2), as online.

bazel build //...
# all tests, as run in CI
bazel test //... --test_output=errors --action_env=WGKEX_CONFIG_FILE=$PWD/wgkex.yaml.example

Without Bazel, run the test modules one at a time (they share module-level state, so a single unittest discover run does not work):

export WGKEX_CONFIG_FILE=$PWD/wgkex.yaml.example
for t in $(git ls-files 'wgkex/*_test.py'); do python3 -m unittest "$(echo "${t%.py}" | tr / .)" || break; done

CI (.github/workflows) runs the Bazel tests with coverage, ruff check on ./wgkex (ruff is pinned to 0.16.1, so new default rules don't break CI) and black --check (the psf/black action, v26.5.1):

pip install ruff==0.16.1 black==26.5.1
ruff check ./wgkex
black --check .

Python dependencies are declared in requirements.txt and locked in requirements_lock.txt via Bazel's compile_pip_requirements. After changing requirements.txt, regenerate the lock file:

bazel run //:requirements.update

See Docs/development.md for running broker and worker locally with dummy interfaces and example clients.

Issues and pull requests are welcome. Please keep CI green (tests, ruff, black), add tests for new behaviour, and update wgkex.yaml.example and Docs/openapi.yaml when changing configuration or the API. New MQTT topics and payload fields must stay compatible with older brokers and workers.

Questions? Reach us on the Freifunk München Mattermost. Background on the architecture: FFMUC goes wild – infrastructure recap 2020 (rC3).

wgkex is licensed under the GNU General Public License v3.0.