refactor(dagster): clarify bootstrap vs runtime config#37
Merged
Conversation
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
- Isolate API container env (no Dagster/GitHub/LLM secrets) - Move healthcheck to production compose (not just override) - Remove shared volumes from API service - Add rate limiting (60 req/min/IP) on all routes via slowapi decorators - Extract limiter to rate_limit.py to avoid circular imports Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
- CLAUDE.md: add API run command, env vars (API_HOST, API_PORT, API_RATE_LIMIT) - architecture.md: add REST API section, update Docker services count, add serving layer to data flow - Bump docs submodule with rest-api.mdx page Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Verify API response shapes match ost-mcp TypeScript types exactly. Catches breaking changes in field names, types, or structure before they reach the MCP server. Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com>
Use snake_case column names (created_at) matching the Prisma model, add gen_random_uuid() for id, remove non-existent updatedAt column.
…line Trending repos now also upsert into github.raw_github_project alongside raw_trending_project, so they flow through the standard pipeline (fetcher → dbt → classification → sync) and land in public.Project.
Adds stg_public__project_bookmark staging and a LEFT JOIN-based exclusion in match_user_recommendation so users never receive a recommendation for a project they have already bookmarked. A singular test enforces the invariant in CI.
… SDK Replace the OpenAI-client-over-OpenRouter wrapper with the official mistralai Python SDK. Simpler stack, direct provider, typed messages for mypy strict. Default model is now `mistral-small-latest` (tracks current Small release). Env var renamed from OPENROUTER_API_KEY to MISTRAL_API_KEY and updated across code, compose, docs, and tests.
New endpoint embeds free-text queries with the same SentenceTransformer used by the pipeline (MiniLM-L6-v2, 384d) and ranks projects by pgvector cosine similarity. Optional hard filters (language, domain, category, techstack) narrow the candidate set before ranking. Model is eagerly loaded in the FastAPI lifespan to avoid cold-request latency. Unit tests cover query validation, ranking shape, and filter propagation to SQL.
New RecommendationEvent Prisma model (mirrors the one in ost-backend) and dbt staging expose a feedback signal: projects shown ≥N times in the lookback window without being clicked or starred are excluded from the mart. Keeps the reco surface fresh instead of repeating projects the user has already dismissed implicitly. Lookback and threshold are configurable via dbt vars (ignored_lookback_days=30, ignored_min_shown=3). A singular test enforces the invariant in CI.
… add user FK Addresses 3 schema gaps flagged in review: 1. `eventType` is now a Postgres enum (RecommendationEventType) — database rejects typos and unknown event types at INSERT time instead of relying on application-level validation alone. 2. `source` promoted from `context.source` to a dedicated enum column (RecommendationSource: PERSONALIZED | TRENDING | SIMILAR | SEMANTIC_SEARCH) + indexed `(source, occurredAt)` for analytics. `rank` promoted to an Int column too. `context` stays jsonb for unstructured metadata (A/B variant, session_id, etc.). 3. Added FK on `userId` with ON DELETE CASCADE — prevents orphan events and ensures RGPD-compliant deletion when users leave. dbt staging, mart exclusion, and singular test updated to match uppercase enum values. 46 dbt tests pass.
Hardens the LLM classification harness for production scale: 1. Parallelization via ThreadPoolExecutor (5 workers default) — ~7× speedup measured on 23 projects (4.9s vs ~35s sequential). 2. Cost tracking: ClassificationResult now carries token usage + model; asset aggregates into Output metadata (prompt_tokens, completion_tokens, estimated_cost_usd, model_version) so a bad prompt change is visible at the next run. 3. DLQ (`match.project_classification_failure`): persistent failures stop consuming LLM budget on each run. Exponential backoff (2h, 4h, 8h, ..., capped at 7d), max 5 attempts. RateLimitError is a distinct exception so 429s don't look like ordinary errors in the logs. Also: modelVersion column added to match.project_classification — every classification now carries the model that produced it, so future model migrations are auditable.
When a worker hits Mistral's 429, the previous behavior was to DLQ the project immediately. A 1-minute 429 window hitting 5 concurrent workers would poison the DLQ with 5+ projects whose only fault was being in the wrong second. _classify_one now sleeps _RATE_LIMIT_COOLDOWN_SECONDS + jitter on the first RateLimitError and retries once. Only the second failure (or any non-429 error) is routed to the DLQ. Successful retries are logged as warnings via the `rate_limit_hits` counter so operators can see 429 pressure at a glance.
…sage for embeddings
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
settings.pyfor import-time Dagster setupassets,resources,jobs,schedules, andsensorsassembly into helper functions indefinitions.pyGO_TRENDING_PATHwith the other required runtime environment variablesValidation
python -m py_compile src/linker/definitions.py