Proposal: platform topology

[Zaba505]

Note

Wave 1 of Proposal: rebuilding infra from scratch, as amended by revision 3. It depends only on the charter. Proposal: definitions, values and secrets and Proposal: drills build on the environments it names.

Summary

This proposal reaffirms the three tiers ADR-0001 chose:

• the edge, Cloudflare, where every end-user request enters;
• the public-cloud anchor, on Google Cloud, which is always on and is where a rebuild starts;
• the home lab, the operator's own hardware.

Four paths connect them, and each carries one kind of traffic:

Path	From, to	Carries
application	end user → edge → tenant, in either environment	end users reaching tenants
operations	cloud anchor ↔ home lab, through the operations tunnel	the operator and the rebuild managing the home lab, and nothing else
boot	home-lab host → cloud anchor, over HTTPS	a host's boot artifacts
provider APIs	wherever the rebuild runs → Google Cloud's and Cloudflare's APIs	the rebuild's changes

Tenants run in either environment. A tenant is placed whole, in one environment, by a rule that matches what it declares against what each environment offers. Placement never changes how a tenant is reached, and a placed tenant doesn't move.

Background

• ADR-0001 chose the shape (ADR-0001): an edge with mutual TLS and DDoS protection, in front of a public-cloud anchor, with a secure tunnel to the private home lab. It also decided that:

  • end users reach tenants only through the edge, and the tunnel is for operations only;
  • both environments host tenants, and they may offer different things, so placement depends on what a tenant needs;
  • moving a tenant between environments isn't supported;
  • the hop from the edge to an origin is mutually authenticated in both environments;
  • the tunnel is a checkpoint of its own.

  It left one question open: how tenants in different environments reach each other.

• ADR-0002 decided placement (ADR-0002). A tenant's environment is resolved from the needs it already declares, matched against each environment's offerings. The cloud anchor is the default when both qualify. The operator may override, with the reason recorded on the onboarding issue. A tenant is placed whole, and placement isn't part of what a tenant is promised.

• Some of what those ADRs leaned on is gone.

  • The engagement process is out of scope, so there's no onboarding issue to record an override on, and no "new offering needed" hold.
  • ADR-0001's strongest reason was reuse: its definitions "already exist and are proven in cloud/". The reset deleted them, and nothing is deployed.

• The home lab today (Hardware): a Ubiquiti Dream Machine Pro router, a 48-port PoE switch, HP ProLiant DL360 Gen 9 servers, and a Ryzen 5 PC with a GPU. The network has three VLANs: IPMI, network boot, and the main home-lab network.

• ADR-0004 placed two things on these paths (ADR-0004): hosts boot over HTTPS from the cloud anchor, at a DNS name that doesn't change, and the tunnel's home-lab end lives on always-on network equipment.

Proposed design

The environments

• The edge: Cloudflare. DNS for the platform's domain, TLS for end users, DDoS protection, and the client certificate it presents to every origin.
• The public-cloud anchor: Google Cloud. It's up even when the home lab isn't, so a rebuild starts here. It holds the state backend (phase 0), the boot endpoint home-lab hosts boot from, and the cloud end of the operations tunnel. It can host tenants.
• The home lab. The operator's hardware behind the router. It can host tenants, and its hosts are rebuilt by network boot from the cloud anchor.

Both vendors pass the charter's test:

Vendor	Configuration	Data	Credentials
Cloudflare	set through its OpenTofu provider, from this repository	holds no data of record; its DNS records are definitions	API tokens the operator creates and revokes; origin keys generated by the rebuild, never by Cloudflare
Google Cloud	set through its OpenTofu provider, from this repository	anything the platform keeps there can be exported through its APIs	the operator's own login; service accounts and keys the rebuild creates and deletes

The paths

flowchart LR
    user([End user])
    rebuild([The rebuild])
    subgraph edge[Edge: Cloudflare]
        cf[DNS, TLS, DDoS,<br/>client certificate]
    end
    subgraph cloud[Cloud anchor: Google Cloud]
        ctenant[Tenants]
        boot[Boot endpoint]
        cend[Tunnel end]
    end
    subgraph home[Home lab]
        htenant[Tenants]
        hosts[Hosts]
        hend[Tunnel end, on the router]
    end

    user -- application --> cf
    cf -- application --> ctenant
    cf -- application --> htenant
    cend -- operations --- hend
    hosts -- boot --> boot
    rebuild -- provider APIs --> cf
    rebuild -- provider APIs --> cloud

Loading

1. Application. Every end-user request enters through the edge, wherever its tenant runs. Every origin, in either environment, refuses a connection that doesn't present the edge's client certificate. So reaching an origin directly fails at TLS, not at a firewall rule that could be loosened. Proposal: tenant networking decides how the edge reaches a home-lab origin.
2. Operations. The operations tunnel joins the cloud anchor and the home lab. It carries the operator's and the rebuild's management traffic, and never a tenant's. Its home-lab end is on always-on network equipment, never on a host the rebuild provisions, so a rebuild can reach the LAN before any host is up. Proposal: operations tunnel builds it.
3. Boot. A home-lab host fetches its boot artifacts from the cloud anchor over HTTPS, at a DNS name that stays the same across rebuilds. The artifacts hold no secrets, and TLS protects their integrity. Proposal: cloud anchor and edge serves them, and Proposal: home-lab hosts boots from them.
4. Provider APIs. The rebuild changes Google Cloud and Cloudflare through their APIs, over the internet, with the credentials Proposal: definitions, values and secrets supplies.

Tenants in different environments have no path to each other. Two tenants that call each other are placed in the same environment. Proposal: tenant networking decides how tenants reach each other. If it needs a path between environments, it proposes one as a revision of this topology.

Where a tenant is placed

• Both environments host tenants, and they needn't offer the same things. An offering is built in an environment when a tenant placed there needs it, not in advance.
• A tenant is placed whole, in one environment. A split tenant would need a path between environments for its own traffic, and there's none.
• Placement is a rule. It matches what the tenant declares, through the declaration Proposal: tenant compute defines, against what each environment offers:
  • if one environment offers everything the tenant needs, the tenant goes there;
  • if both do, it goes to the default environment;
  • if neither does, it isn't hosted until one does.
• The operator may override the rule, but only between environments that both qualify. The override is recorded in the tenant's definitions, with its reason.
• A placed tenant doesn't move. Moving one means tearing it down and hosting it again in the other environment.
• Placement isn't promised to a tenant. Like a DNS name, it's the platform's business.

What this fixes in the rebuild

The cloud anchor and the edge come up first, in phases 0 and 1a. The tunnel comes next, in 1b, and then home-lab hosts, in 1c. Tenants come only after both environments are up. This is ADR-0004's order, and the topology is why: the home lab boots from the cloud anchor, and is managed through the tunnel.

Files

This proposal lands no code. Its stories write its docs pages.

Design decisions

• Three tiers, still. The edge keeps end users off the home lab's internet connection, and gives both environments one front door. The cloud anchor gives a rebuild somewhere always on to start from. The home lab is hardware the operator already owns.
• One path, one kind of traffic. If tenant traffic used the operations tunnel, a tenant's uptime would depend on the operator's maintenance link, and the tunnel's firewall would have to admit end users.
• The boot path is public. Serving boot artifacts over the internet from the cloud anchor means the home lab needs no always-on boot server that a rebuild doesn't build.
• The override lives in the tenant's definitions. The engagement process is out of scope, and definitions are where a rebuild can read the override back.
• No migration. As ADR-0001 found, supporting it would oblige every stateful offering in both environments to support it too, forever.

Drill

The topology has no phase of its own. It's drilled through the phases that build each path, and each of their proposals' drills must show these checks passing against a scratch target:

Check	Shown by
a request to a cloud origin without the edge's client certificate is refused	cloud anchor and edge
a request to a home-lab origin without the edge's client certificate is refused	tenant networking
the tunnel refuses traffic to any tenant's ports	operations tunnel
a host boots from the boot name, with nothing on the LAN but DHCP	home-lab hosts
a tenant whose needs no environment meets isn't placed	tenant compute

Documentation

• Fundamentals → Topology (docs/content/fundamentals/topology.md, weight 10): the environments, the paths and what each carries, and how a tenant is placed. It's the first page of Fundamentals, and links to The rebuild and its phases.
• Overview: a short Environments paragraph on the home page, naming the three tiers and linking to Topology, as the charter's docs sketch says.

What it replaces

• ADR-0001, reaffirmed, with its open items moved:
  • its home-lab "peer surface alongside cloud/" goes: each part's proposal says where its files live;
  • vendoring the edge's trust anchor (its residual risk 2) goes to Proposal: cloud anchor and edge;
  • the hand-configured home-lab tunnel end (its residual risk 3) goes to Proposal: operations tunnel;
  • tenants reaching each other across environments goes to Proposal: tenant networking.
• ADR-0002, reaffirmed, with these changes:
  • the override is recorded in the tenant's definitions, not on an onboarding issue;
  • a tenant that no environment can host waits for an offering, without the engagement process's hold;
  • the shape of the declaration goes to Proposal: tenant compute;
  • the default environment is an open question below.

Open questions

1. Are Google Cloud and Cloudflare still the right vendors? ADR-0001's strongest reason was reusing definitions that already ran, and nothing runs now. Every module is rewritten either way. Keep both, or compare them with alternatives before wave 2 builds on them?
2. Which environment is the default? ADR-0002 chose the cloud anchor because only its offerings existed. Now both are built from scratch. The home lab costs nothing more to run a tenant on. The cloud anchor keeps running through a power or internet outage at home. Which one does a tenant go to when both qualify?

Out of scope

• How each path is built: cloud anchor and edge, operations tunnel, home-lab hosts and tenant networking.
• How administrative interfaces reject everyone but the operator: Proposal: operator access.
• The hardware itself: Proposal: home-lab hosts, which updates the Hardware page.

References

• Proposal: rebuilding infra from scratch, the charter
• ADR-0001, ADR-0002 and ADR-0004
• Hardware