You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Design feedback before a PR: fixing minion presence under NAT (#66295)
#70582
CkMinions.connected_ids() decides who is connected by matching each minion's cached grainipv4/ipv6 against the addresses connected to the publish port. A minion whose connection address differs from its grains — NAT, a floating IP, a master managing minions in another network — never matches and is reported absent while test.ping succeeds against it.
This reaches manage.present / not_present / alived, the minions.connected wheel, ping_all_connected_minions, and max_minions enforcement at authentication.
This is why we see issues like #66295 ([BUG] Minion behind NAT incorrectly shows not connected).
I have a working branch and would like feedback on the design before opening a PR.
The address table and the grain cache are two independent sources that only agree when no address translation sits between master and minion. There is no amount of care in the comparison that recovers the identity, because the identity was never on the wire — the master sees a TCP connection and nothing else.
Proposed design
1. Record what the publish server already knows, and union it.
Where the publish server knows which minion is behind each connection it records that, and connected_ids() returns the union of that record and the existing address matching. The two cover each other: the record names minions address matching cannot, address matching sees connections the record cannot name. For an id the record names, its address wins, so show_ip reports the real one.
Unioning rather than replacing is what makes it safe to roll out: the answer can only ever be a superset of what Salt reported before, so max_minions can fail closed but never open, and a partly-upgraded fleet is never reported absent.
This alone fixes the bug on the tcp transport, where the publish connection is authenticated and the master has always known each subscriber.
2. Give the zeromq publish socket an identity.
zeromq's publish socket carries none, which is why the bug bites hardest there. A subscriber announces itself by SUBSCRIBE-ing to a topic nothing is ever published to; the subscription exists purely to carry identity, and libzmq hands the master the frame.
The topic is assigned by the master, not derived by both ends. The master keeps one long-lived secret (pki_dir, or cluster_pki_dir when clustered; 0600; created once with O_EXCL) and assigns each minion HMAC(secret, minion_id). The minion derives nothing — it receives the topic inside the RSA-encrypted part of its auth reply and subscribes to it once.
The design decision I would most like a second opinion on
I first derived the topic from the per-minion session key, which both ends already hold. That seemed obviously right and was wrong, in a way worth recording:
A topic derived from a rotating secret inherits that secret's lifecycle. The session key rotates on its own schedule and whenever any other process on the minion's host authenticates — a salt-call run is enough. So the topic moved underneath connected minions, which forced a re-announcement path, an ordering constraint between subscribing to the new topic and cancelling the old one, and a window in which a perfectly healthy minion could not be named. Three rounds of review kept finding bugs in that machinery; they were all the same bug wearing different clothes.
Assigning a topic from a secret that never rotates removes the entire class: there is nothing to keep in sync, so there is no re-announcement, no ordering, and no window. It also deleted more code than it added.
A plain hash of the id would not work: ids are usually hostnames, so anything able to reach the publish port could claim presence for any minion — and, since cancelling a subscription only requires naming the topic, withdraw it for one that is really connected. (The bare sha1(id) that zmq_filtering makes minions subscribe to for delivery is refused as identity for exactly this reason, so filtering has no bearing on presence.)
What changes for existing deployments
max_minions is enforced against a count that can only grow. A fleet quietly over its limit will begin rejecting authentications. Syndics count too. With con_cache the change reaches enforcement at that cache's interval.
On tcp this is not opt-in and there is no way to opt out — that transport already knows the identities, so the record is simply accurate now. This is the consequence I am least comfortable with and would particularly like an opinion on.
On zeromq it is opt-in via a new zmq_presence_subscriptions master option, and a minion too old to be assigned a topic simply does not announce.
salt/presence/* events become edge-triggered where the publish server owns presence, rather than firing every loop_interval.
Security posture
This is a reporting aid, not an access-control input. An attacker already on the network path can read a subscription off the wire and replay or cancel it; ZeroMQ without CURVE concedes on-path attackers regardless. What the assigned topic buys is that an attacker who cannot see a minion's traffic cannot derive its topic either — which the previous sha1(id)-shaped options did not give.
What this deliberately does not fix
Address matching's false positives: a minion that has gone away but whose stale cached grain address still matches some other connection is still reported present. That is unchanged, and out of scope for #66295, which is about minions wrongly reported absent.
State of the branch
Four commits on v3008.3, each standing alone and passing tests independently:
Three macOS-only test fixes (no feature code — happy to submit separately)
The presence record and the connected_ids() union — fixes tcp on its own
The zeromq assigned-topic half
Docs and changelog
Docs, changelog and tests are in place. Verified end to end in Docker with a NAT topology and a control: with the option off test.ping succeeds while manage.present is empty (the bug); with it on the minion is named and show_ip reports the gateway address.
Questions
Is the union the right call, or would you rather the record be authoritative when present? Union is strictly safer for max_minions but means the cached-grain scan still runs on every call, so it is not a performance win.
tcp getting this with no opt-out — acceptable, or should it be gated too?
Branch targeting.CONTRIBUTING.rst says bug fixes go to the oldest supported branch and features to master. This is a bug fix implemented as an opt-in feature plus a new config option and a new field in the auth reply. Which branch do you want it against? It is currently based on v3008.3.
The new field in the auth reply — any objection to adding presence alongside aes/session, or is there a preferred way to hand a minion per-minion configuration at authentication?
Is a Discussion the right venue for this, or would you rather it were an issue?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
CkMinions.connected_ids()decides who is connected by matching each minion's cached grainipv4/ipv6against the addresses connected to the publish port. A minion whose connection address differs from its grains — NAT, a floating IP, a master managing minions in another network — never matches and is reported absent whiletest.pingsucceeds against it.This reaches
manage.present/not_present/alived, theminions.connectedwheel,ping_all_connected_minions, andmax_minionsenforcement at authentication.This is why we see issues like #66295 ([BUG] Minion behind NAT incorrectly shows not connected).
I have a working branch and would like feedback on the design before opening a PR.
Branch: https://github.com/cstrahan/salt/tree/cstrahan/zmq-xpub-presence-split
Why the existing mechanism can't just be patched
The address table and the grain cache are two independent sources that only agree when no address translation sits between master and minion. There is no amount of care in the comparison that recovers the identity, because the identity was never on the wire — the master sees a TCP connection and nothing else.
Proposed design
1. Record what the publish server already knows, and union it.
Where the publish server knows which minion is behind each connection it records that, and
connected_ids()returns the union of that record and the existing address matching. The two cover each other: the record names minions address matching cannot, address matching sees connections the record cannot name. For an id the record names, its address wins, soshow_ipreports the real one.Unioning rather than replacing is what makes it safe to roll out: the answer can only ever be a superset of what Salt reported before, so
max_minionscan fail closed but never open, and a partly-upgraded fleet is never reported absent.This alone fixes the bug on the tcp transport, where the publish connection is authenticated and the master has always known each subscriber.
2. Give the zeromq publish socket an identity.
zeromq's publish socket carries none, which is why the bug bites hardest there. A subscriber announces itself by SUBSCRIBE-ing to a topic nothing is ever published to; the subscription exists purely to carry identity, and libzmq hands the master the frame.
The topic is assigned by the master, not derived by both ends. The master keeps one long-lived secret (
pki_dir, orcluster_pki_dirwhen clustered; 0600; created once withO_EXCL) and assigns each minionHMAC(secret, minion_id). The minion derives nothing — it receives the topic inside the RSA-encrypted part of its auth reply and subscribes to it once.The design decision I would most like a second opinion on
I first derived the topic from the per-minion session key, which both ends already hold. That seemed obviously right and was wrong, in a way worth recording:
A topic derived from a rotating secret inherits that secret's lifecycle. The session key rotates on its own schedule and whenever any other process on the minion's host authenticates — a
salt-callrun is enough. So the topic moved underneath connected minions, which forced a re-announcement path, an ordering constraint between subscribing to the new topic and cancelling the old one, and a window in which a perfectly healthy minion could not be named. Three rounds of review kept finding bugs in that machinery; they were all the same bug wearing different clothes.Assigning a topic from a secret that never rotates removes the entire class: there is nothing to keep in sync, so there is no re-announcement, no ordering, and no window. It also deleted more code than it added.
A plain hash of the id would not work: ids are usually hostnames, so anything able to reach the publish port could claim presence for any minion — and, since cancelling a subscription only requires naming the topic, withdraw it for one that is really connected. (The bare
sha1(id)thatzmq_filteringmakes minions subscribe to for delivery is refused as identity for exactly this reason, so filtering has no bearing on presence.)What changes for existing deployments
max_minionsis enforced against a count that can only grow. A fleet quietly over its limit will begin rejecting authentications. Syndics count too. Withcon_cachethe change reaches enforcement at that cache's interval.zmq_presence_subscriptionsmaster option, and a minion too old to be assigned a topic simply does not announce.salt/presence/*events become edge-triggered where the publish server owns presence, rather than firing everyloop_interval.Security posture
This is a reporting aid, not an access-control input. An attacker already on the network path can read a subscription off the wire and replay or cancel it; ZeroMQ without CURVE concedes on-path attackers regardless. What the assigned topic buys is that an attacker who cannot see a minion's traffic cannot derive its topic either — which the previous
sha1(id)-shaped options did not give.What this deliberately does not fix
Address matching's false positives: a minion that has gone away but whose stale cached grain address still matches some other connection is still reported present. That is unchanged, and out of scope for #66295, which is about minions wrongly reported absent.
State of the branch
Four commits on
v3008.3, each standing alone and passing tests independently:connected_ids()union — fixes tcp on its ownDocs, changelog and tests are in place. Verified end to end in Docker with a NAT topology and a control: with the option off
test.pingsucceeds whilemanage.presentis empty (the bug); with it on the minion is named andshow_ipreports the gateway address.Questions
max_minionsbut means the cached-grain scan still runs on every call, so it is not a performance win.CONTRIBUTING.rstsays bug fixes go to the oldest supported branch and features to master. This is a bug fix implemented as an opt-in feature plus a new config option and a new field in the auth reply. Which branch do you want it against? It is currently based onv3008.3.presencealongsideaes/session, or is there a preferred way to hand a minion per-minion configuration at authentication?All reactions