Skip to content

Add evals devservices mode#115526

Merged
JoshFerge merged 1 commit into
masterfrom
jferg-codex/evals-devservices-mode
May 13, 2026
Merged

Add evals devservices mode#115526
JoshFerge merged 1 commit into
masterfrom
jferg-codex/evals-devservices-mode

Conversation

@JoshFerge
Copy link
Copy Markdown
Member

Summary

  • add an evals devservices mode for local seeded eval ingestion
  • keep the mode focused on error-event evals: Relay, Snuba/Postgres, objectstore, taskbroker, ingest-events, post-process-forwarder-errors, and dedicated taskworkers for relay, ingest.errors, ingest.errors.postprocess, and issues
  • avoid using the broader ingest mode or a catch-all taskworker for this local eval path

Why

Seeded Seer evals need enough of local Sentry ingestion to accept error events, save them, post-process them, and handle issue-side work such as auto source-code config. The existing ingest mode is broader than this path needs, while the generic taskworker can let slower issue-side work starve ingest.errors saves. This mode keeps that topology in Sentry devservices rather than in Seer helper scripts.

Verification

  • git diff --check origin/master..HEAD
  • .venv/bin/prek run -q check-yaml --files devservices/config.yml
  • devservices config loader check for evals mode and supervisor dependency types
  • pre-push hooks

@JoshFerge JoshFerge requested review from a team as code owners May 13, 2026 22:35
Comment thread devservices/config.yml
taskworker:
command: sentry run taskworker
taskworker-evals-ingest-errors:
command: sentry run taskworker --namespace ingest.errors --concurrency 2 --processing-pool-name eval-ingest-errors
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did we expect this to have concurrency 2 compared to the others?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, trying to make seeding local scenarios fast as possible. may add more depending on if machines can take it as well. 🙏🏼

@JoshFerge JoshFerge enabled auto-merge (squash) May 13, 2026 22:39
@JoshFerge JoshFerge merged commit cdf4cc0 into master May 13, 2026
56 checks passed
@JoshFerge JoshFerge deleted the jferg-codex/evals-devservices-mode branch May 13, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants