docs(self-hosted): provide a detailed example on how to separate ingest taskbroker #15601
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull request overview
This PR adds detailed documentation on how to separate ingest taskbrokers in self-hosted Sentry installations, addressing a common recommendation for users experiencing delays in notifications or alerts.
Key Changes:
- Added hyperlinks to referenced source code files for easier navigation
- Added a new "Separate Ingest Workers" section with concrete implementation examples
- Provided complete docker-compose and configuration examples for setting up dedicated ingest task processing
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
develop-docs/self-hosted/tasks.mdx
Outdated
| environment: | ||
| TASKBROKER_KAFKA_CLUSTER: "kafka:9092" | ||
| TASKBROKER_KAFKA_DEADLETTER_CLUSTER: "kafka:9092" | ||
| TASKBROKER_DB_PATH: "/opt/sqlite/taskbroker-activations.sqlite" |
There was a problem hiding this comment.
The TASKBROKER_DB_PATH uses the same filename "taskbroker-activations.sqlite" as might be used by the default taskbroker. Based on the earlier example in the file (lines 76 and 87), each taskbroker replica should have a unique SQLite database file to prevent contention and locks. Consider using "taskbroker-activations-ingest.sqlite" instead.
| TASKBROKER_DB_PATH: "/opt/sqlite/taskbroker-activations.sqlite" | |
| TASKBROKER_DB_PATH: "/opt/sqlite/taskbroker-activations-ingest.sqlite" |
| taskbroker-ingest: | ||
| restart: "unless-stopped" | ||
| image: "$TASKBROKER_IMAGE" | ||
| environment: | ||
| TASKBROKER_KAFKA_CLUSTER: "kafka:9092" | ||
| TASKBROKER_KAFKA_DEADLETTER_CLUSTER: "kafka:9092" | ||
| TASKBROKER_DB_PATH: "/opt/sqlite/taskbroker-activations.sqlite" | ||
| volumes: | ||
| - sentry-taskbroker-ingest:/opt/sqlite | ||
| depends_on: | ||
| - kafka |
There was a problem hiding this comment.
The TASKBROKER_KAFKA_TOPIC environment variable is missing from the taskbroker-ingest configuration. According to step 2 in the instructions above (line 133), you need to use this environment variable to define which topic a taskbroker consumes from. Without this, the ingest taskbroker will consume from the default topic instead of "taskworker-ingest".
| ### Separate Ingest Workers | ||
|
|
||
| Having separate ingest `taskbroker` and `taskworker` is useful for high-throughput | ||
| installation, therefore you can receive timely alerts and not having to wait for | ||
| ingest-related tasks to finish. As an implementation of the above steps, | ||
| you need to add a few new containers on your `docker-compose.override.yml` file: |
There was a problem hiding this comment.
The documentation is missing step 1 from the "Isolate Workload Separation" section: provisioning the additional Kafka topic. Users need to know how to create the "taskworker-ingest" topic in Kafka before deploying the taskbroker that will consume from it. Consider adding a note about this prerequisite step.
develop-docs/self-hosted/tasks.mdx
Outdated
| ### Separate Ingest Workers | ||
|
|
||
| Having separate ingest `taskbroker` and `taskworker` is useful for high-throughput | ||
| installation, therefore you can receive timely alerts and not having to wait for |
There was a problem hiding this comment.
The phrase "therefore you can receive timely alerts and not having to wait" has a grammatical issue with mixed verb forms. It should be "without having to wait" or "and not have to wait" to maintain parallel structure.
| installation, therefore you can receive timely alerts and not having to wait for | |
| installation, therefore you can receive timely alerts and not have to wait for |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Bundle ReportChanges will increase total bundle size by 1.74kB (0.01%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: sentry-docs-server-cjsAssets Changed:
view changes for bundle: sentry-docs-client-array-pushAssets Changed:
|
DESCRIBE YOUR PR
Whenever self-hosted users has delay on their notifications or alert, I would recommend them to do this first. Previously there's no clear documentation on how to implement this.
IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
LEGAL BOILERPLATE
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
EXTRA RESOURCES