v2.14.0
plug --dockerrun puts a container in the cluster, which prefixing docker never could
plug docker run my-image looks like it works. plug says the tunnel is ready,
the container starts, and nothing it does reaches the cluster. plug carries the
traffic of the process it launches, and that process is the docker CLI, which
posts a request to a socket and exits: the container is made by the docker
daemon, which is nobody's child. The failure is silent, which is what makes it
expensive. People look at their image and at their cluster, never at what
prefixing docker with plug actually means.
plug -p prod -c --dockerrun docker run --rm my-image
holds the datapath in a sidecar container and runs yours in its network
namespace, unmodified. Nothing is rebuilt, nothing is added to the image. It
works the same on macOS, Windows and Linux for a reason worth stating: a
container is a Linux environment everywhere, so the Linux datapath is what runs
in all three cases, VM or no VM. The host needs no privilege at all, since the
capabilities are granted by docker inside the sidecar.
-s and -c mean here exactly what they mean elsewhere, and are handed to the
sidecar rather than swallowed: a container in a cluster is a member of it like
any process, so it either answers to a name or says it is a pure client. -s
costs nothing extra, because the two containers share one network and a port
yours listens on is already on the sidecar's loopback.
One branch of it had never executed anywhere, and now something says what is and
is not covered. A standalone cluster checks no personal key, so the profile key
is empty on every cluster this repository can stand up, e2e included, and the
mount that carries it into the sidecar was written, compiled and shipped without
running once. Its argv is asserted now: mounted read-only, named to the core by
path, and absent entirely when the profile has none. Whether a container then
authenticates with that key against a gateway is the gateway's integration to
test, and saying so beats a green tick that would not have covered it.
It has an e2e cell now, in the block the three families share: an unmodified
curlimages/curl reaching httpbin by name, from inside a container, against a
cluster provisioned by compose, by swarm and by k8s in turn. That is the family
question asked of a container rather than of a process. It runs on the Linux legs
and reports a skip on the others, which have no Linux docker daemon, and it costs
no wall clock: those legs finish two to five minutes ahead of the macOS ones,
which is where the image pull it adds fits.
The sidecar runs the client of the same FLAVOUR as the launcher that started it,
without being told to. The flavour rides in the version string, so a hosted
launcher composes the hosted image tag, which is what its own release publishes.
That holds by composition rather than by design, so there is a test on it: a
standalone client serving a gateway-managed cluster would fail at authentication,
one container away from anything that mentions flavours.
Two things it deliberately does not do. It does not parse docker's command line:
its two flags go in right after run, where docker takes options in any order,
and the rest of the line is passed through untouched. Working out where the
user's options end and the image begins would mean maintaining a table of which
of docker's sixty-odd flags take a value, and getting it wrong means editing a
command plug does not understand. When the line does conflict, docker refuses it
and plug translates the refusal into the flag the user actually typed. And it
covers docker run only, in the foreground: docker compose up, docker create and podman build containers differently, and a detached container would
outlive the network it was given, so both are refused by name rather than
half-supported.
Connecting through plug on macOS was paying 92ms to ask who was connecting
Since 2.13.1 a single cluster is no longer open to every account on the machine:
before a flow is carried, plug asks which process opened the socket and whether
its user is one that registered a client. Asking cost more than anyone had
measured. The answer came from lsof, which walks every open file descriptor of
every process on the machine to report on one port: 92ms, on the first packet of
every new connection, which capped plug at about a dozen connections a second and
made a connection pool feel like a stall.
The same question is answered by netstat -anv, which reads the table the kernel
already keeps, in 3ms. Measured on the real function, not on the two commands in
isolation: 92.2ms to 3.26ms. Windows never had this problem, because it asks the
kernel directly through GetExtendedTcpTable, and macOS is now the same shape.
The reason to prefer it is not only the cost. lsof lists open DESCRIPTORS,
netstat lists live CONNECTIONS, and the difference has teeth: a process that has
not closed the descriptor of a finished connection still appears in lsof, in
state CLOSED, holding a port the kernel has already released and may already have
given to someone else. Three such ghosts were sitting on the machine this was
written on. Attributing a flow through one of them answers with whoever held the
port BEFORE the process actually connecting, which in the multicluster router is
the wrong cluster. A port with no live connection has no row in the kernel's
table, so that answer can no longer be produced.
One thing found while testing it, which nobody had asked for: the parser rejected
a process whose name is empty. Rejecting means "no owner", and no owner is what
the ownership check lets through. A process able to blank its own name would have
outranked the check. It is attributed like any other now, and there is a test that
fails if that guard is put back.
A cluster belongs to one account, and now it actually does
2.13.1 said a single cluster was no longer open to every account on the machine.
Looking again at how that was enforced, it was not.
The check ran per flow: it asked which account opened the socket and compared it
against everyone with a live client marker. But a client writes its marker BEFORE
anything authenticates it. A second account on the same machine had only to run
plug -p <the other account's cluster>, which put its own uid in the very set it
was about to be compared against, and its traffic then went through a tunnel
opened with somebody else's key. It never needed a key, only the host and port.
And with two or more clusters up, no account check ran at all: that path walks the
process ancestry to the registered launcher and hands over its cluster without
asking whose launcher it is.
The question is asked one step earlier now, when a client registers, which is what
makes it a member in the first place. Another account holding the cluster is
refused there, by name, instead of having its connections reset by something that
cannot explain itself. Both paths are covered because both are downstream of that
marker: no marker, no membership, and no ancestor to walk to.
Windows is covered too, and it had never been. Every process there reports the
same uid, so every client recorded the same owner, no account could be told from
another, and the per-flow check fell through on that platform for the same reason.
A client now records the SID of its own token, which is a question a process is
never in doubt about, and the rule reads the same sentence on both systems. Root
and LocalSystem are exempt: they already own the machine, and the service's own
work must not be refused by the rule the service enforces.
READ THIS BEFORE WORRYING ABOUT YOUR TEAM: the scope is ONE COMPUTER. Everything
the rule consults is a local directory, and nothing is asked of the agent or of
the network. Ten developers on ten machines share a cluster exactly as before,
each with their own key and their own tunnel, and none of them can see that the
others exist. What is refused is two accounts of the SAME computer holding one
cluster at the same time, for as long as the first session lives. Different
clusters, or the same cluster one after the other, are unaffected.
Two allocations per packet, and a marker that could outlive its process
Every packet coming back from a cluster was allocating twice: a buffer, and the
one-element list wrapping it for the device. 165ns and 1560 bytes each time,
against 12.6ns and nothing when both are reused. The time is the small half; the
bytes were feeding the garbage collector at the rate of the tunnel.
Reusing them is only allowed if the device consumes what it is handed before it
returns, and wireguard-go does not promise that in its interface, so the three
implementations were read rather than assumed: macOS writes the buffer to the
device file and returns, Windows copies it into a WinTun ring packet, and Linux,
the one that coalesces packets and writes a header in front of them, keeps only
indices in a slice it resets on every call. None holds the memory. A test now
sends a long packet, then a short one, then a long one, and fails if any arrives
with the previous one's tail attached or if the device is handed memory that
changes underneath it.
The second half closes a window that yesterday's ownership change opened. A
client marker records a pid, and the rule reads it to decide whether another
account holds a cluster. A pid alone only says "something is alive there": a
client that crashes without unregistering leaves its marker, and the kernel
reissues pid numbers, so the first unrelated process to land on that number
resurrects a membership nobody holds. That was a small leak while the marker
merely granted access. It is a lockout now that it refuses, and it would name an
account that had already left. The marker carries the client's start time as
well, so the question is no longer "is that pid alive" but "is it still the same
process". A marker written by an older client carries no stamp and is still
trusted, because turning a version skew into a lockout would be the worse bug.
One correction that came out of the Windows leg of the build rather than from
reading: whether an account can hold a cluster is now decided on the account
RECORDED in the marker, not only on the one asking. Guarding just the caller left
the answer depending on who happened to be enquiring, and a marker written where
identity means nothing could have been reported as holding a cluster against
somebody.
Getting started asks which system you are on, and stops answering three at once
The page had grown a tail. After "run your process" it dissolved into how
plug update rewrites a deployment's tag, which moving tags are left alone, and
what happens when a pinned deployment is already current: real material, and none
of it something you need before your first session. It has moved to Profiles &
versions, under a heading that says so.
What is left is numbered, and each number is one thing you might want to do:
deploy the agent, install the CLI, run a process as a service, run one as a pure
client, run a container, let plug pick the local port. The instructions that
differ by system no longer sit three abreast in the prose. A selector next to the
title picks one, guessed from your browser, and the install line and the examples
follow it.
The Docker page introduced with --dockerrun is gone with it. It promised a
chapter on working with Docker and delivered one flag, which is a section of
Getting started, not a place of its own; the full detail lives in the CLI
reference, where the other flags are.
The help showed a command plug refuses, and claimed an order it never required
Three things wrong in plug --help, two of them mine and new.
The --dockerrun example read plug -p prod --dockerrun docker run my-image,
which plug turns down: a member of a cluster either names itself with -s or
declares itself a client with -c, and a container is a member like any other.
The help was teaching a command that stops with an error. The container form was
also missing from the Usage block, where both process forms are listed.
And -s carried "place after the other options", which was not true. Every
permutation parses: the options may be written in any order, and only the command
has to come last. A sentence like that is worse than saying nothing, because a
reader obeys it and never finds out it was unnecessary.
--profile=prod did not work either, and the failure pointed elsewhere: the
equals form fell through to the command, so plug complained that nothing said
whether the process was a service or a client. plug advertises long options in
its help; taking only the spaced form was a promise half kept. Both forms work
now, and -- ends plug's options for a command whose own name starts with a
dash.
What did NOT change, after an attempt: an unknown flag is still not refused. It
stops the parse and travels to the core untouched, which is how a launcher
carries a flag it has never heard of to a core that has. A launcher is installed
once and runs cached cores for months, so refusing what it does not recognise
would freeze the flag surface at whatever it knew on install day. The test that
states that contract is what stopped the change, and the price of keeping it is
that a mistyped option is still treated as a command name.
All three are now held by tests rather than by attention. One walks every example
in the help through serveRequired, the same function the launcher runs, so what
is refused at runtime is refused at build time. Another permutes the options and
checks they parse identically, and a third checks that everything after the
command belongs to it: npm run dev -p x must reach npm with its -p, not hand
plug a profile.
The run now says how much of the code its tests ran, and what that leaves out
Coverage was measured nowhere. It is measured now, in the runs that were already
happening: the same go test, one -coverprofile heavier, and a job that merges
them into the run summary. Nothing new executes.
Merged ACROSS the three operating systems, which is the part that matters here. A
file behind a build tag is not compiled on a runner that does not need it, so it
is not "uncovered" on that runner, it is absent from both sides of the fraction:
graft_darwin.go, pidroute_windows.go and nsshim_linux.go would each vanish from a
single-OS number rather than lower it. The merge is a plain concatenation, which
is valid because Go sums the counts of identical blocks, checked both ways before
it was relied on: the same profile twice gives the same total, and an empty
profile plus a real one gives the real one.
The number is 41%, and the report says in the same breath what that is not.
Whole files sit at 0% while being exercised hard: a daemon, a selftest, a swarm
backend only run against a real datapath or a real cluster, and the e2e matrix is
what proves them, on three systems and three orchestrators, where no counter
looks. So nothing here fails a build. A floor would push someone to write unit
tests for exactly those files, duplicating worse what already covers them.
There is a line-by-line HTML report per module too, as an artifact. Per module
rather than per repository because go tool cover resolves sources through the
module it runs from, and a profile mixing the two fails on whichever one is not
current. That was found by trying it, not by reasoning about it.
Giving up on a name now says what kept failing, wherever it gave up
plug -s waits for the cluster to actually carry the name before it hands you a
session, retrying inside a budget. When that budget runs out it has two ways of
saying so, and only one of them named a cause. The exit after a failed attempt
wrapped that attempt's error; the exit at the top of the next iteration returned
"gave up after 90s of retries" and threw away the error it was already holding,
leaving a message that says it insisted without saying at what.
Which of the two fires is a matter of milliseconds, so the quality of the answer
depended on the weather. It surfaced when coverage instrumentation slowed a
Windows runner just enough to shift it. Both exits now carry the cause, and the
one case with genuinely nothing to name, no attempt completing at all, says that
rather than implying a reason it does not have.
The same slowdown found a race in a test, not in the code: the watcher that
adopts new nameservers sets them and logs them one line apart, and the test
waited for the first and read the second. It waits for both now.
plug doctor could name the wrong core version, and say it as a fact
Beside a running daemon the doctor prints "core v2.13.2", read out of the cached
core's path. It scanned that path from the LEFT for the versions directory,
and the store is ~/.plug/versions: a developer whose home path contained that
word anywhere earlier got a different segment back, printed as their version,
next to a pid that was perfectly real. Wrong, and of the shape nobody
double-checks. It reads from the right now, where the store's own directory is.
Found by asking which uncovered code would be worth a unit test, rather than
which would raise the number. Two more came with it, both contracts rather than
percentages: the exit status of --dockerrun mirrors the container's, so a
script reading $? is told the truth about work it did not watch; and the
resolver a container is pointed at is the one the datapath actually installs,
which if it drifted would break name resolution inside the container while
everything on the host kept working.