Shared Proxy

Shared Proxy

One Nginx can serve several ɳSelf projects on a host. The project that owns the Nginx sets NGINX_MODE=shared. Each other project is nested in a sub-directory of the owner and sets NGINX_FRONTED_BY=<owner directory name>. This page covers what nself build does for each side and what nself start and nself stop do at run time (see Runtime).

Without NGINX_MODE=shared on the owner nothing changes: a fronted project still writes plain, unprefixed site files into the owner's nginx/sites, exactly as before.

---

Layout

stack/                    owner: NGINX_MODE=shared
  nginx/sites/            site confs of every project
  nginx/shared/           registrations (written by nself build)
  ssl/certificates/       certificate pairs of every project
  backend/                fronted project: NGINX_FRONTED_BY=stack

The fronted project must live directly under a directory named after NGINX_FRONTED_BY. Any other layout is refused, as before.

A fronted project named backend inside the owner directory makes nself build run in the owner directory treat the owner as a monorepo and build backend instead ("Detected monorepo layout"). Build the owner with nself build --no-monorepo, or give the fronted project another directory name.

What the owner's build does

• Creates nginx/shared/ and writes _owner.json (service names, server names, upstream hosts that stay as written).
• Writes its own routes with container names as upstreams: <PROJECT_NAME>_hasura:8080, not hasura:8080. A bare service name is ambiguous once one Nginx reaches two projects, because every project has a hasura. Container names are unique per host.
• Leaves plugin and hand-managed upstreams as written.
• Never deletes a file that a registration lists.
• Warns about each registration whose project directory is gone, and prints how to remove it.

What a fronted project's build does

When the owner has nginx/shared/ with a _owner.json (the owner's build writes it; a nginx/shared/ without it means the owner dropped shared mode, and the project's files are then removed, see Leaving):

• Prefixes the http-level names a plugin snippet declares (upstream X, limit_req_zone zone=X, keys_zone=X, map/geo variables) with <PROJECT_NAME>_, so two projects running one plugin never declare a name twice in the owner's nginx.
• Registers every server name its confs serve, including docs.<base>. A conf left by a build that failed before its registration was written still counts as a claim.
• Writes container-name upstreams for its own services (core, optional, custom, installed plugins). Frontend routes point at host.docker.internal, which is not a service, so they stay as written. A plugin upstream is rewritten when its host is a service of this project (plugin containers are <PROJECT_NAME>_<name>).
• Names its site files <PROJECT_NAME>.<file> (bproj.hasura.conf) and stamps them # Generated by nself build (project bproj).
• Copies each certificate pair its confs reference into the owner's ssl/certificates/<dir> (directory mode 0700, key 0600, plus a .nself-project marker; only a directory carrying this project's marker is ever removed by it).
• Writes nginx/shared/<PROJECT_NAME>.json listing every file it put under the owner.
• Removes only files its own previous registration listed and the new one does not.

Certificates

Situation	Result
Directory does not exist in the owner's ssl/	Pair is copied
Directory is listed in this project's registration	Replaced on rotation (temp directory, then rename)
Own directory already holds a certificate that expires later	Kept (an ACME renewal installed it)
Identical pair held by someone else	Reused, not listed
Different pair not owned by this project	E503
Directory managed by ACME (symlink to a generation directory, recorded in ssl/.acme/lineages.json)	Left alone, never listed, never removed
Any other symlink on the path, or no write permission	E503, nothing written
This project has no local pair yet (first build, or ssl/ cleaned) and the directory exists in the owner's ssl/	Own directory: kept and stays listed. Anyone else's: E503 (it cannot be proven identical)

Runtime

A fronted project publishes no port 80 or 443 and runs no Nginx of its own. The owner's Nginx serves its domains by joining the fronted project's own <project>_network, so each fronted project stays unreachable from every other fronted project (the owner's Nginx is the only foreign container on a fronted network).

Command	What happens
nself start in a fronted project	After the stack is healthy: connect the owner's Nginx to <project>_network (no change when already connected), run nginx -t, then nginx -s reload.
nself start in the owner	Does the same for every registration whose network exists. A registered project that is stopped gives one warning per project; the owner still starts.
nself stop, restart, start --fresh, reset, uninstall in a fronted project	Before compose takes the project down, the owner's Nginx is disconnected from <project>_network. Compose cannot remove a network while a foreign container is attached, so without this step nself stop would fail.
nself uninstall in a fronted project	Also removes the project's files and registration from the owner and reloads the owner's Nginx. If that cannot be done (for example a registration cannot be read, nginx/shared/quarantine or a certificate directory is a symlink, or the owner's Nginx cannot be looked up), uninstall returns E506 and does not report the files as removed.

A fronted nself start returns E506 when the registrations under the owner cannot be read (one invalid registration file of any project is enough), because its domains would not be served. The owner's own start only warns. Before a compose down, the same condition is a warning that names the cause; the down continues.

Changes to one owner's shared proxy (attach, detach, leave, quarantine, restore, and a nself build that writes into the owner) take turns: each holds an exclusive lock on nginx/shared/.lock in the owner's directory (mode 0600) for the whole call and waits up to 60 seconds. A wait that runs out is an E506. A build holds the lock from the moment it reads the registrations until it returns; nself build --check and plan mode write nothing and do not wait. An attach drops a project whose registration was removed or changed while it waited. On Windows the lock only covers one nself process.

If the owner's Nginx is not running, the fronted project starts and stops anyway with one warning ("start the owner project first"). Nothing is attached in that case; run nself start in the fronted project again once the owner is up, or nself start in the owner, which attaches every running registered project.

Restarting the owner's Nginx

A restarted or recreated Nginx container loses its extra networks. Run nself start in the owner afterwards: it reattaches every registered, running project.

A failed nginx -t (E506)

The test always runs before the reload, so a broken conf never replaces the running configuration. When nginx -t fails after an attach:

1. The files of the project the error output names are moved to nginx/shared/quarantine/<project>/ in the owner's directory (directories 0700, keys 0600; symlinks are refused). That project may be a different one than the project that was starting.
2. The test runs again. If the output now names another registered project, that project is quarantined too and the test repeats (once per project). If it still fails, every connection made by this call is undone, the owner keeps its previous configuration and nself start returns E506 with the Nginx error.
3. Whenever a start fails, the files it had just restored from quarantine go back into quarantine, so nginx/sites never keeps a conf that fails nginx -t.
4. The next successful nself start of the quarantined project restores its files.

E506 is also returned when the connect or the reload itself fails on a network that exists. A missing network or a stopped owner is a warning, not an error.

Troubleshooting

Symptom	Check
Domain of a fronted project answers 502 or not at all	docker inspect <owner>_nginx --format '{{json .NetworkSettings.Networks}}' should list <project>_network. Run nself start in the fronted project.
nself stop in a fronted project warns that the owner's Nginx is not running	Harmless: nothing was attached.
E506 naming a conf file	Fix that conf in the project that owns it (nself build), then nself start there.
Registration lists a conf you did not expect	A site file is moved or removed for a project only when it is named <project>.* and starts with # Generated by nself build (project <project>). A hand-edited registration cannot move the owner's own confs.
Files under nginx/shared/quarantine/	Left by a failed test; the owning project restores them on its next successful start.

Refusals

Code	When
E503	A project name that equals one of its own conf stems (PROJECT_NAME=hasura); a server name or certificate directory is already served by the owner or another registered project; a registration does not validate; the owner's ssl/ cannot be written safely
E504	An upstream host that stays as written (a single label, such as sidecar) is a service name or upstream host of the owner or another registered project, or the container name (<project>_<service>) of one of that project's services
E505	NGINX_MODE is not shared; NGINX_MODE=shared together with NGINX_FRONTED_BY; NGINX_MODE=shared with a proxy provider other than nginx

The checks run before any file is written, so a refused build changes nothing. E505 for an unknown mode or for shared with NGINX_FRONTED_BY is a warning in v1.4 and an error from v1.5. NGINX_MODE=shared with the traefik provider is an error in both.

Host names with a dot, IP addresses and localhost resolve the same for every project and are never refused by E504. Upper case and a trailing dot are folded first (HASURA. is hasura). The owner's own build only warns about an ambiguous host, because it leaves plugin upstreams as written.

Leaving

Unsetting NGINX_FRONTED_BY in the fronted project, or the owner unsetting NGINX_MODE, makes the next build of the fronted project remove every file its registration lists, then the registration. Files in an ACME-managed directory are not touched.

Registration file

nginx/shared/<project>.json, generated, schema schemas/shared-proxy.v1.schema.json:

{
  "_generated": "GENERATED BY nself build - DO NOT HAND EDIT",
  "files": ["nginx/sites/bproj.hasura.conf", "ssl/certificates/b-test/fullchain.pem"],
  "network": "bproj_network",
  "project": "bproj",
  "project_dir": "/srv/stack/backend",
  "schema_version": "1",
  "server_names": ["api.b.test"],
  "service_names": ["auth", "hasura", "postgres"],
  "verbatim_upstream_hosts": []
}

The file name must equal <project>.json, a project is registered once, and project_dir must be under the owner. Keys and lists are sorted.