queue/rabbitmq: a Publisher can't set a message's user-id, so no consumer can trust a scenario a humus service publishes

[Zaba505]

What happens

A queue/rabbitmq Publisher can't set a message's user-id. So every message a humus service publishes arrives with an empty Message.UserID, and a consumer whose TrustScenario trusts by UserID, as the package doc and the Failure injection page recommend, strips every failure scenario a humus service publishes or forwards.

PublisherSettings.ForwardScenario still puts the scenario in the message's baggage header. No consumer can trust it, though, so forwarding a scenario over RabbitMQ between humus services never works.

The docs' own example shows it. It trusts "the end user's own services, which connect as orders-api", with m.UserID == "orders-api". If orders-api is a humus service, none of its messages carry a user-id, so that func trusts none of them.

Why

• rabbitmq.Publishing has no field for AMQP's user-id property (publisher.go#L42-L91), and its conversion to amqp091's Publishing sets no UserId (publisher.go#L96-L118).
• The Service's strip asks TrustScenario with the delivery's Message (faults.go#L34-L44), whose UserID is the delivery's user-id (message.go#L68-L69).
• RabbitMQ validates user-id only when a publisher sets it. A message without one is accepted with none.

How it was found

While prototyping the RabbitMQ processor walkthrough, against RabbitMQ 4.3.6 in a container. A humus App ran rabbitmq.NewService with a TrustScenario of m.UserID == <the test's login>, under humus.WithFaults and jsonscenario. Its Processor republished each delivery with a Publisher whose ForwardScenario was set.

• A delivery published with amqp091 directly, with UserId set to the trusted login, reached Process with its scenario.
• The humus Publisher's copy of it carried the scenario in its baggage header, but arrived with an empty UserID, so the Service stripped it.

Neither the unit tests nor the container tests publish with a humus Publisher to a Service that trusts by UserID, so they don't reach this.

The docs

• The package doc (doc.go#L732-L744) and the Failure injection page's RabbitMQ: UserID recommend trusting by UserID, with the end user's own services as the example. That holds only for publishers that aren't humus services.
• The RabbitMQ page's Failure injection section says the same in passing.

Expected

A humus Publisher can publish a message whose user-id RabbitMQ validates against the Publisher's login. A consumer can then trust what a humus service publishes and forwards. Some ways to get there:

• a UserID field on Publishing, passed through as amqp091's UserId. A publish whose user-id isn't the connection's user is refused by RabbitMQ, which the Publisher reports as a failure that isn't retried;
• or a Publisher that sets every message's user-id to the user it logged in as. That needs no new field, but a Publisher that logs in with a client certificate, through amqp091's ExternalAuth, doesn't know its user's name.

Either way:

• a container test shows a humus Publisher's message, with its scenario forwarded, trusted by a Service's TrustScenario on UserID;
• the package doc, the RabbitMQ page and the Failure injection page say how a humus service's messages get their user-id.

The fix adds to queue/rabbitmq's API. It changes nothing for a message published without a user-id, TrustScenario's default, or the Publisher's retries for other failures.

References

• Proposal: RabbitMQ, and Proposal: failure injection
• story(rabbitmq): fail or delay a Publish that a failure scenario names, and strip scenarios from untrusted deliveries #897, failure injection for RabbitMQ (story(rabbitmq): fail or delay a Publish that a failure scenario names, and strip scenarios from untrusted deliveries #938)
• Proposal: RabbitMQ processor walkthrough, whose case where the provider is down and then replayed waits for the fix

[Zaba505]

Triage

Verdict: this is a reproducible bug in queue/rabbitmq. It isn't a misuse of humus, and it isn't a feature request, though any fix adds to the API. A consumer that trusts by UserID, as the docs recommend, strips every scenario a humus Publisher forwards, whether a job or a Processor publishes it. #960 is filed to fix it.

Is this a misuse of humus?

No. The end user writes TrustScenario and decides which Publishers forward. What goes on the wire is the package's: Publish builds amqp091's Publishing from a rabbitmq.Publishing, and nothing the end user passes reaches the user-id property (publisher.go#L41-L118).

The end user did what the docs ask. The package doc says "Who a message came from is best told by its UserID", and its example trusts "the end user's own services, which connect as the user orders-api" (doc.go#L732-L744). The Failure injection page says the same (failure-injection.md#L363-L377). A humus service is one of the end user's own services, and that example trusts none of its messages.

The strongest argument for misuse is that the end user can trust something else. strip keeps a scenario whenever TrustScenario returns true (faults.go#L34-L44). So a TrustScenario on Exchange, which RabbitMQ's write permissions guard as Kafka's ACLs guard a topic, would keep a forwarded scenario. I reasoned that from the code and RabbitMQ's access control, and didn't test it. It doesn't hold, though. The docs name UserID as the way to trust a RabbitMQ publisher, and never offer Exchange. The only other way to send a user-id is to publish with amqp091 directly, without the Publisher's telemetry, retries, breaker and injection point.

Is it a feature request?

No. It contradicts what humus promises:

• the package doc's Failure injection: "Who a message came from is best told by its UserID", with the end user's own services as the example (doc.go#L732-L744), and "Set ForwardScenario for a destination whose consumers take part in failure injection with the same codec, such as another service of the end user's, so that a scenario can reach the points they publish to in turn" (doc.go#L714-L717);
• the Failure injection page's "RabbitMQ: UserID" and "Sending a scenario on: ForwardScenario" (failure-injection.md#L363-L420), and the RabbitMQ page's Failure injection (rabbitmq.md#L1201-L1216);
• Proposal: failure injection #869's design, which trusts a RabbitMQ delivery by "Message.UserID" under "Stripping at servers", and has every client strip "unless ForwardScenario says that dependency takes part".

Together they promise that a scenario can go from one of the end user's services to the next over RabbitMQ, trusted by UserID. Between two humus services it can't.

The gap is between two proposals. #765 gave Message a UserID but Publishing none, and #869 later built trust on UserID. #938's judgment calls noticed it: "Publishing has no UserID, and the container test sets user-id through amqp091 directly."

Any fix changes the API. A Publishing's fields and what Publish sends are both documented, and neither gives a message a user-id. The docs that describe trust by UserID change with the fix.

Is it reproducible?

Yes. I wrote three container tests that extend faults_testcontainers_test.go. I ran them, and that file's existing test as a baseline, against rabbitmq:4.3.6 at 4ddf294. The versions were amqp091-go v1.15.0, testcontainers-go v0.44.0, Go 1.27.0 and Podman 5.8.9. Each test ran 4 times under -race.

Test	Expected	Runs failed
A job's Publisher forwards its scenario to a Service that trusts the Publisher's login	the Service keeps the scenario	4 of 4
A Processor forwards a trusted delivery's scenario to the next Service	both Services keep the scenario	4 of 4
RabbitMQ validates a user-id, and gives a message published without one none	passes	0 of 4
Baseline: TestService_strips_the_scenario_of_an_untrusted_delivery_and_keeps_one_trusted_by_its_UserID	passes	0 of 4

A job's Publisher: the Service strips the scenario it forwards. One App runs a job, a Publisher with ForwardScenario set, and a Service whose TrustScenario is m.UserID == <login>, the login the Publisher connects with. The job's BAGGAGE carries a scenario that spares the Publisher's point.

1. The job's context carries the scenario: the job publishes, scenario in its context: true.
2. The Publisher evaluates its point and forwards the scenario: the send amq.default span has the events [humus.fit.evaluation] and no humus.fit.strip.
3. The message arrives with the scenario in its header, but with no user-id: user-id "", baggage header "tenant=acme,acme.fault={%22rules%22:[…]}".
4. TrustScenario returns false, and the Service strips the scenario: scenario in its context: false, and the process span has [humus.fit.strip].

A Processor that publishes on: the first Service trusts, the second strips. This is the report's case. One App runs two Services that trust the same login, and a Publisher with ForwardScenario set.

1. A delivery published with amqp091, with user-id set to the login, reaches the first Process with its scenario: user-id "humus-…", scenario in its context: true, and its process span has no events.
2. Process publishes the delivery on with the Publisher: send amq.default, with [humus.fit.evaluation] and no strip.
3. The copy reaches the second Service with the scenario in its header, but user-id "", so it is stripped: scenario in its context: false, [humus.fit.strip].

RabbitMQ's half. A message published with amqp091 and no user-id arrives with none. One whose user-id names another user closes the channel with 406. The broker logs operation basic.publish caused a channel exception precondition_failed: user_id property set to 'not-humus-…' but authenticated user was 'humus-…', and the message never reaches the queue.

One claim of the report is too strong. "Forwarding a scenario over RabbitMQ between humus services never works" holds for a consumer that trusts by UserID, as the docs recommend. A consumer that trusts by Exchange or Queue keeps the scenario. I reasoned that from strip, and didn't test it.

Why

• rabbitmq.Publishing has no user-id field, and amqpPublishing copies each field it has into amqp091's Publishing, leaving UserId empty (publisher.go#L41-L118). Reproduced: every message the Publisher published arrived with user-id "".
• amqp091 sets the user-id flag only when UserId isn't empty (write.go#L127-L128), so the property isn't sent at all. I got that from amqp091's code.
• RabbitMQ validates a user-id only when one is sent, and adds none of its own. Reproduced: a message published without one arrived with none, and one naming another user was refused with 406 PRECONDITION_FAILED.
• The Service copies the delivery's UserId into Message.UserID (message.go#L68-L69, #L100), and strip asks TrustScenario about that Message (faults.go#L34-L44). With UserID empty, a TrustScenario on UserID strips. Reproduced.

So the user-id can only come from the publishing side. Nothing the consuming Service sees tells it who published a message that has none. I reasoned that from the points above.

No test reached this. The unit tests' stand-in implements no part of AMQP. #938's container test publishes with amqp091 and sets the user-id itself (faults_testcontainers_test.go#L28-L46). No test publishes with a humus Publisher to a Service that trusts by UserID.

#960 is filed to fix it:

• a humus Publisher's message can carry a user-id that RabbitMQ validates against the Publisher's login;
• a Service that trusts that login by UserID keeps the scenario the Publisher forwards, from a job and from a Processor;
• container tests of both, from this triage;
• the package doc, the RabbitMQ page and the Failure injection page say how a humus service's messages get their user-id.

[Zaba505]

Fixed in #962, merged as 41e53e2.

rabbitmq.Publishing now has a UserID field, which Publish sends as the message's user-id. RabbitMQ validates it against the user the Publisher's Connection logs in as, so a consumer whose TrustScenario trusts that login by Message.UserID keeps the scenario a Publisher with ForwardScenario set forwards, from a job or from a Processor that publishes a delivery on. The end user sets it on each Publishing to the login they source for Connection.SASL, or to the user a client certificate maps to under ExternalAuth.

• A UserID that isn't the Publisher's login is refused with 406 PRECONDITION_FAILED, a channel error: Publish returns it at once, with no retry, and the breaker counts it a success.
• An empty UserID sends no user-id, as before. The defaults of TrustScenario (trusts no publisher) and ForwardScenario (strips) are unchanged.
• The triage's three reproductions are in faults_testcontainers_test.go, with each humus Publish now setting UserID, and pass against RabbitMQ 4.3.6.
• The package doc, the comments on Publishing, Settings.TrustScenario and PublisherSettings.ForwardScenario, ExampleSettings_trustScenario, the Failure injection page and the RabbitMQ page now say how a humus service's messages get their user-id.