fix: avoid nested RabbitMQ pub channel acquire on tenant register - #4433
Merged
Conversation
pubMessage held a pooled publish channel and then called RegisterTenant, which acquired a second channel. Under top-of-hour cron bursts this exhausted the pub pool and timed out, dropping cron triggers. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Benchmark resultsCompared against |
Contributor
|
|
abelanger5
approved these changes
Jul 15, 2026
Comment on lines
+476
to
+478
| // Reuse the already-held publish channel. Calling RegisterTenant here would | ||
| // Acquire a second pub channel while this one is still checked out and can | ||
| // exhaust the pool under concurrent publish load (e.g. top-of-hour crons). |
Contributor
There was a problem hiding this comment.
nit: remove unnecessary comment
Contributor
|
|
5 tasks
mrshu
added a commit
to mrshu/sre-world
that referenced
this pull request
Jul 25, 2026
Add the graded scenario for the fault that hatchet-dev/hatchet#4433 fixed. The publish path keeps one channel from the pool. It then takes a second channel from the same pool to register the tenant exchange. Under load the pool becomes empty and the service loses messages. The fault ships as a Tier-2 layer image. The layer compiles the publish service from the source before the fix, and it sets the pool size to 2 so the fault appears quickly. The healthy base image keeps the merged fix. A layer fault carries no chart overlay, so both the mechanism and the pool size live in the image. The repair is an operator action: change the image of svc-pub back to the healthy base. Two gates grade the result. The outcome gate sees the lost messages. The image_state gate proves the rollback happened. The second gate is necessary, because a larger channel pool hides the lost messages while the faulted code stays in the service. The scenario is publication_pending. The images are not published, so the generator writes no task and the scenario is not in tasks/. The bands are provisional until a calibration runs on a cluster. - Add spec.yaml, ground-truth.yaml, instruction.md, solve.sh and the fault layer with its patch - pub.pub-pool is the decoy in the closed component inventory. It is the place where the symptom appears, and not the cause.
mrshu
added a commit
to mrshu/sre-world
that referenced
this pull request
Jul 25, 2026
Add a standalone reproduction of the fault from hatchet-dev/hatchet#4433. One command starts it. The reproduction runs the true Hatchet message queue against a true RabbitMQ broker, at the pinned commit 245c06e. It changes only two controls. It sets the pub channel pool to 2, which the Hatchet option WithMaxPubChannels supports, and it applies a patch that puts back the code from before the fix. An external load generator built on the shared arrival engine applies the load. Measured result at a pool of 2, with the same load for both runs. The armed build loses 291 of 324 messages and the log shows 291 "cannot acquire channel" errors. The fixed build loses none. verify_dual_gate.py proves that the two gates of the scenario separate the three cases, and it does so without a cluster. It runs three episodes and grades each one with the true oracle against the committed answer key. It states the expected result for each episode and exits with an error if the gates stop separating them. faulted image, small pool outcome FAIL, image_state FAIL FAIL faulted image, larger pool outcome PASS, image_state FAIL FAIL healthy image after rollback outcome PASS, image_state PASS PASS The second case is the reason the image_state gate exists. A larger pool hides the lost messages, so every gate for the symptom passes. Only the gate for the cause finds that the faulted code is still in the service. The scripts provision the broker user through the image entrypoint and then verify the credential. An earlier version added the user after the broker started and did not check the result. On a slow start that command failed in silence, every publish returned 403, and both runs then showed a total loss that looked the same as the incident.
mrshu
added a commit
to mrshu/sre-world
that referenced
this pull request
Jul 25, 2026
Add the graded scenario for the fault that hatchet-dev/hatchet#4433 fixed. The publish path keeps one channel from the pool. It then takes a second channel from the same pool to register the tenant exchange. Under load the pool becomes empty and the service loses messages. The fault ships as a Tier-2 layer image. The layer compiles the publish service from the source before the fix, and it sets the pool size to 2 so the fault appears quickly. The healthy base image keeps the merged fix. A layer fault carries no chart overlay, so both the mechanism and the pool size live in the image. The repair is an operator action: change the image of svc-pub back to the healthy base. Two gates grade the result. The outcome gate sees the lost messages. The image_state gate proves the rollback happened. The second gate is necessary, because a larger channel pool hides the lost messages while the faulted code stays in the service. The scenario is publication_pending. The images are not published, so the generator writes no task and the scenario is not in tasks/. The bands are provisional until a calibration runs on a cluster. - Add spec.yaml, ground-truth.yaml, instruction.md, solve.sh and the fault layer with its patch - pub.pub-pool is the decoy in the closed component inventory. It is the place where the symptom appears, and not the cause.
mrshu
added a commit
to mrshu/sre-world
that referenced
this pull request
Jul 25, 2026
Add a standalone reproduction of the fault from hatchet-dev/hatchet#4433. One command starts it. The reproduction runs the true Hatchet message queue against a true RabbitMQ broker, at the pinned commit 245c06e. It changes only two controls. It sets the pub channel pool to 2, which the Hatchet option WithMaxPubChannels supports, and it applies a patch that puts back the code from before the fix. An external load generator built on the shared arrival engine applies the load. Measured result at a pool of 2, with the same load for both runs. The armed build loses 291 of 324 messages and the log shows 291 "cannot acquire channel" errors. The fixed build loses none. verify_dual_gate.py proves that the two gates of the scenario separate the three cases, and it does so without a cluster. It runs three episodes and grades each one with the true oracle against the committed answer key. It states the expected result for each episode and exits with an error if the gates stop separating them. faulted image, small pool outcome FAIL, image_state FAIL FAIL faulted image, larger pool outcome PASS, image_state FAIL FAIL healthy image after rollback outcome PASS, image_state PASS PASS The second case is the reason the image_state gate exists. A larger pool hides the lost messages, so every gate for the symptom passes. Only the gate for the cause finds that the faulted code is still in the service. The scripts provision the broker user through the image entrypoint and then verify the credential. An earlier version added the user after the broker started and did not check the result. On a slow start that command failed in silence, every publish returned 403, and both runs then showed a total loss that looked the same as the incident.
mrshu
added a commit
to abundant-ai/sre-world
that referenced
this pull request
Jul 28, 2026
Add the graded scenario for the fault that hatchet-dev/hatchet#4433 fixed. The publish path keeps one channel from the pool. It then takes a second channel from the same pool to register the tenant exchange. Under load the pool becomes empty and the service loses messages. The fault ships as a Tier-2 layer image. The layer compiles the publish service from the source before the fix, and it sets the pool size to 2 so the fault appears quickly. The healthy base image keeps the merged fix. A layer fault carries no chart overlay, so both the mechanism and the pool size live in the image. The repair is an operator action: change the image of svc-pub back to the healthy base. Two gates grade the result. The outcome gate sees the lost messages. The image_state gate proves the rollback happened. The second gate is necessary, because a larger channel pool hides the lost messages while the faulted code stays in the service. The scenario is publication_pending. The images are not published, so the generator writes no task and the scenario is not in tasks/. The bands are provisional until a calibration runs on a cluster. - Add spec.yaml, ground-truth.yaml, instruction.md, solve.sh and the fault layer with its patch - pub.pub-pool is the decoy in the closed component inventory. It is the place where the symptom appears, and not the cause.
mrshu
added a commit
to abundant-ai/sre-world
that referenced
this pull request
Jul 28, 2026
Add a standalone reproduction of the fault from hatchet-dev/hatchet#4433. One command starts it. The reproduction runs the true Hatchet message queue against a true RabbitMQ broker, at the pinned commit 245c06e. It changes only two controls. It sets the pub channel pool to 2, which the Hatchet option WithMaxPubChannels supports, and it applies a patch that puts back the code from before the fix. An external load generator built on the shared arrival engine applies the load. Measured result at a pool of 2, with the same load for both runs. The armed build loses 291 of 324 messages and the log shows 291 "cannot acquire channel" errors. The fixed build loses none. verify_dual_gate.py proves that the two gates of the scenario separate the three cases, and it does so without a cluster. It runs three episodes and grades each one with the true oracle against the committed answer key. It states the expected result for each episode and exits with an error if the gates stop separating them. faulted image, small pool outcome FAIL, image_state FAIL FAIL faulted image, larger pool outcome PASS, image_state FAIL FAIL healthy image after rollback outcome PASS, image_state PASS PASS The second case is the reason the image_state gate exists. A larger pool hides the lost messages, so every gate for the symptom passes. Only the gate for the cause finds that the faulted code is still in the service. The scripts provision the broker user through the image entrypoint and then verify the credential. An earlier version added the user after the broker started and did not check the result. On a slow start that command failed in silence, every publish returned 403, and both runs then showed a total loss that looked the same as the incident.
mrshu
added a commit
to abundant-ai/sre-world
that referenced
this pull request
Jul 28, 2026
Add the graded scenario for the fault that hatchet-dev/hatchet#4433 fixed. The publish path keeps one channel from the pool. It then takes a second channel from the same pool to register the tenant exchange. Under load the pool becomes empty and the service loses messages. The fault ships as a Tier-2 layer image. The layer compiles the publish service from the source before the fix, and it sets the pool size to 2 so the fault appears quickly. The healthy base image keeps the merged fix. A layer fault carries no chart overlay, so both the mechanism and the pool size live in the image. The repair is an operator action: change the image of svc-pub back to the healthy base. Two gates grade the result. The outcome gate sees the lost messages. The image_state gate proves the rollback happened. The second gate is necessary, because a larger channel pool hides the lost messages while the faulted code stays in the service. The scenario is publication_pending. The images are not published, so the generator writes no task and the scenario is not in tasks/. The bands are provisional until a calibration runs on a cluster. - Add spec.yaml, ground-truth.yaml, instruction.md, solve.sh and the fault layer with its patch - pub.pub-pool is the decoy in the closed component inventory. It is the place where the symptom appears, and not the cause.
mrshu
added a commit
to abundant-ai/sre-world
that referenced
this pull request
Jul 28, 2026
Add a standalone reproduction of the fault from hatchet-dev/hatchet#4433. One command starts it. The reproduction runs the true Hatchet message queue against a true RabbitMQ broker, at the pinned commit 245c06e. It changes only two controls. It sets the pub channel pool to 2, which the Hatchet option WithMaxPubChannels supports, and it applies a patch that puts back the code from before the fix. An external load generator built on the shared arrival engine applies the load. Measured result at a pool of 2, with the same load for both runs. The armed build loses 291 of 324 messages and the log shows 291 "cannot acquire channel" errors. The fixed build loses none. verify_dual_gate.py proves that the two gates of the scenario separate the three cases, and it does so without a cluster. It runs three episodes and grades each one with the true oracle against the committed answer key. It states the expected result for each episode and exits with an error if the gates stop separating them. faulted image, small pool outcome FAIL, image_state FAIL FAIL faulted image, larger pool outcome PASS, image_state FAIL FAIL healthy image after rollback outcome PASS, image_state PASS PASS The second case is the reason the image_state gate exists. A larger pool hides the lost messages, so every gate for the symptom passes. Only the gate for the cause finds that the faulted code is still in the service. The scripts provision the broker user through the image entrypoint and then verify the credential. An earlier version added the user after the broker started and did not check the result. On a slow start that command failed in silence, every publish returned 403, and both runs then showed a total loss that looked the same as the incident.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Avoid nested RabbitMQ pub-channel acquires when registering a tenant exchange during publish. Under concurrent load (e.g. top-of-hour crons), holding one pooled channel and then acquiring a second exhausted the pool and timed out, dropping cron triggers.
Type of change
What's Changed
pubMessagedeclareTenantExchangefor shared use byRegisterTenantandpubMessageChecklist
Changes have been:
Testing
go build ./internal/msgqueue/rabbitmq/[RegisterTenant] cannot acquire channel/could not run cron workflow🤖 AI Disclosure
I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
Details: Investigating production cron misses via o11y, identifying nested channel acquire, implementing fix and PR