Dagster Orchestration

Dagster Orchestration

Slackquery exposes Dagster definitions from:

slackquery.definitions:definitions

Asset graph

flowchart LR
    P["search/document_projection"] --> E["search/message_embeddings"]
    E --> C["search/artifact_candidate"]
    C --> K["candidate_integrity\nblocking asset check"]
    K --> U["search/published_artifact"]

Loading

Assets and checks

search/document_projection

Projects the canonical DuckDB archive into Slackquery state. Materialization metadata reports projection, active, tombstoned, and source-watermark summaries. The canonical database is attached read-only.

search/message_embeddings

Runs the resumable worker against active projected rows. Metadata reports claimed, succeeded, retryable-failed, and terminal-failed work. A healthy no-change run may claim no items because successful vectors are durable.

search/artifact_candidate

Builds a new immutable artifact for the selected projection watermark and vector generation. Build lineage is stored in durable state so downstream steps choose the candidate produced by the same orchestration run without relying on a published run identifier in documentation.

candidate_integrity

This blocking asset check opens the candidate through a fresh read-only connection and verifies:

• schema and required metadata;
• active document/vector coverage;
• unique and matching document keys;
• vector shape, finite values, and normalization;
• manifest checksum.

Publication cannot proceed when the check fails.

search/published_artifact

Selects the validated candidate for the current run and publishes it by atomically replacing current.duckdb. Materialization metadata records the selected artifact without mutating it.

Job and schedule

Definition	Value
Asset job	slackquery_reconcile
Schedule	slackquery_hourly_reconciliation
Cron	0 * * * *

The schedule performs periodic reconciliation rather than depending on an upstream event. Projection and embedding are idempotent and content-addressed, so a no-change run can safely validate and publish without re-embedding unchanged content.

Enable and set the schedule timezone according to deployment policy. A schedule shown as RUNNING is enabled; that status alone does not prove successful ticks, fresh assets, a passing candidate, or a healthy MCP service.

Operational inspection

For a complete health check:

1. Verify the Dagster instance, daemons, and code location.
2. Inspect recent schedule ticks for failure, skips, and launched runs.
3. Inspect the latest slackquery_reconcile status and failed step logs.
4. Compare asset materialization metadata and vector coverage.
5. Confirm candidate_integrity passed.
6. Verify MCP /readyz after publication and run a search smoke test.

Failure patterns

Projection failure

Check the canonical file mount, schema compatibility, read permissions, and database readability. Do not grant canonical write access as a workaround.

Embedding failure

Check selected backend, network access, model revision, dimensions, endpoint health, and per-item failure classes. Existing successful vectors remain durable.

Candidate build failure

Common causes are incomplete vector coverage, unavailable FTS extension, disk space, or permissions. A failed candidate does not affect the published artifact.

Integrity-check failure

Do not publish around the check. Investigate metadata counts, key mismatch, vector validation, and manifest checksum while the known-good artifact continues serving.

Publication failure

Check artifact-directory ownership, selector permissions, checksum, and disk state. Publication should accept candidates only from the configured artifact directory.

Concurrency rule

Do not run a second manual pipeline writer while scheduled reconciliation is active. DuckDB state is operated as a single-writer store. Read-only MCP traffic remains isolated because it opens the immutable artifact.

See Deployment and Operations and Troubleshooting.