From 2aa06e1270afe8cacd5d52d9dde7e82ce2da8172 Mon Sep 17 00:00:00 2001 From: spideystreet Date: Sat, 7 Mar 2026 18:00:23 +0100 Subject: [PATCH 1/4] docs(ai): complete rewrite of AI Engine documentation - Rewrite overview, pipeline, structure, installation, troubleshooting - Add new pages: database schema, dbt layer, Go services - Update docs.json navigation with Architecture section - All content sourced from current codebase state Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com> --- ai/database.mdx | 134 ++++++++++++++++++++ ai/dbt.mdx | 180 +++++++++++++++++++++++++++ ai/go-services.mdx | 153 +++++++++++++++++++++++ ai/installation.mdx | 241 +++++++++++++++++++++--------------- ai/overview.mdx | 183 +++++++++++++++------------ ai/pipeline.mdx | 253 ++++++++++++++++++++++++------------- ai/structure.mdx | 192 ++++++++++++++++++++++++---- ai/troubleshooting.mdx | 275 ++++++++++++++++++++++++++++++++++++++++- docs.json | 10 +- 9 files changed, 1326 insertions(+), 295 deletions(-) create mode 100644 ai/database.mdx create mode 100644 ai/dbt.mdx create mode 100644 ai/go-services.mdx diff --git a/ai/database.mdx b/ai/database.mdx new file mode 100644 index 0000000..917348a --- /dev/null +++ b/ai/database.mdx @@ -0,0 +1,134 @@ +--- +title: "Database Schema" +description: "PostgreSQL schemas, pgvector setup, and table reference for OST Linker" +--- + +## Overview + +OST Linker uses a single PostgreSQL database with the **pgvector** extension, organized into 4 schemas. The schema is defined in Prisma (`prisma/schema.prisma`) and serves as the single source of truth shared with the backend. + +```mermaid +graph LR + subgraph PostgreSQL + PUB[public] + GH[github] + ML[ml] + MA[match] + end + + PUB -->|"Users, Projects, Categories"| GH + GH -->|"Raw + Staging data"| ML + ML -->|"Embeddings"| MA + MA -->|"Recommendations"| PUB +``` + +## Schemas + +### `public` -- User-Facing Data + +The primary schema used by the backend API. Contains all user-facing models. + +| Table | Description | +|-------|-------------| +| `user` | User accounts with social links, GitHub/GitLab identities | +| `Project` | Published projects with metadata (title, description, URLs, trending flag) | +| `Category` | Project categories (e.g., Framework, Library, CLI Tool) | +| `Domain` | Project domains (e.g., Web Development, DevOps, Data Science) | +| `tech_stack` | Technologies and languages (typed as `TECH` or `LANGUAGE`) | +| `user_tech_stack` | Junction: user <-> tech stack preferences | +| `user_categories` | Junction: user <-> category preferences | +| `user_domain` | Junction: user <-> domain preferences | +| `project_tech_stack` | Junction: project <-> tech stacks | +| `project_category` | Junction: project <-> categories | +| `project_domain` | Junction: project <-> domains | +| `project_bookmark` | User bookmarks on projects | +| `match_global_recommendation` | Top-N global project recommendations (dbt-materialized) | +| `match_user_recommendation` | Per-user personalized recommendations (dbt-materialized) | + +### `github` -- Raw and Staged Ingestion Data + +All data scraped from GitHub lives here, from raw JSON through staged and enriched layers. + +| Table | Managed By | Description | +|-------|-----------|-------------| +| `raw_github_project` | Go scraper | Raw JSON from GitHub Search API | +| `raw_github_readme` | Go fetcher | README content per project | +| `raw_github_languages` | Go fetcher | Language byte counts per project | +| `raw_github_topics` | Go fetcher | Topic arrays per project | +| `int_github_detection` | Python (FastText) | Language detection results and filtering | +| `stg_github__project` | dbt | Cleaned and typed project data | +| `stg_github__readme` | dbt | Staged README content | +| `stg_github__languages` | dbt | Staged language breakdowns | +| `stg_github__topics` | dbt | Staged topics | +| `stg_github__detection` | dbt | Staged detection metadata | +| `int_project_enriched` | dbt | Joined enriched project data | +| `fct_github_project` | dbt | Final fact table with stars, forks, pushed_at | + +### `ml` -- Machine Learning Artifacts + +Stores embeddings and intermediate ML data. + +| Table | Managed By | Description | +|-------|-----------|-------------| +| `embd_github_project` | Python | 384-dim project embedding vectors | +| `embd_user` | Python | 384-dim user embedding vectors | +| `stg_public__project` | dbt | Project data staged for ML processing | +| `stg_public__user` | dbt | User data staged for ML processing | +| `int_user_enriched` | dbt | User context strings for embedding | +| `int_project_contextualized` | dbt | Project context strings for embedding | +| `int_project_embedding_candidate` | dbt | Projects ready for embedding | +| `fct_public_user` | dbt | Final user fact table | + +### `match` -- Classification Results + +| Table | Managed By | Description | +|-------|-----------|-------------| +| `project_classification` | Python (LLM) | Category and domain assignments with confidence scores | + +## pgvector + +The database uses the [pgvector](https://github.com/pgvector/pgvector) extension for vector similarity search. + +| Property | Value | +|----------|-------| +| Extension | `vector` (enabled via Prisma) | +| Vector dimension | 384 (MiniLM-L6-v2 output) | +| Distance function | Cosine distance (`<=>` operator) | +| Similarity formula | `1 - (vector_a <=> vector_b)` | +| Similarity threshold | 0.25 (configurable via dbt var) | + + +Cosine similarity is computed directly in SQL within dbt models. The `match_user_recommendation` model uses `1 - (uv.vector <=> pv.vector)` to score user-project pairs. + + +The vector columns are defined in Prisma as `Unsupported("vector")` since Prisma does not natively support the pgvector type. The actual column type in PostgreSQL is `vector(384)`. + +## Schema Routing in dbt + +dbt models are routed to specific PostgreSQL schemas using the `generate_schema_name` macro. The target schema is set per model in `dbt_project.yml` via the `+schema` property: + +```yaml +# Example from dbt_project.yml +models: + ost_linker: + staging: + stg_github__project: + +schema: github # Writes to github.stg_github__project + stg_public__user: + +schema: ml # Writes to ml.stg_public__user + marts: + match_global_recommendation: + +schema: public # Writes to public.match_global_recommendation +``` + +The custom `generate_schema_name` macro overrides dbt's default behavior to use the configured schema name directly, without prepending the target schema. + +## Prisma as Schema Manager + +Prisma manages the database schema for both the AI engine and the backend: + +- **Schema definition:** `prisma/schema.prisma` declares all models across all 4 schemas +- **Extensions:** pgvector and uuid-ossp are enabled via `extensions = [uuidOssp, vector]` +- **Multi-schema support:** Uses `@@schema("public")`, `@@schema("github")`, `@@schema("ml")`, `@@schema("match")` annotations +- **Seed data:** `prisma/seed/seed.ts` populates Categories, Domains, and TechStacks +- **Cross-repo sync:** The Prisma schema is automatically synced to the backend repo via the `sync-prisma-backend.yml` GitHub Actions workflow diff --git a/ai/dbt.mdx b/ai/dbt.mdx new file mode 100644 index 0000000..315d12d --- /dev/null +++ b/ai/dbt.mdx @@ -0,0 +1,180 @@ +--- +title: "dbt Layer" +description: "Data transformation models, macros, and recommendation scoring in dbt" +--- + +## Overview + +The dbt project (`dbt/`) handles all SQL-based data transformations, from cleaning raw ingested data to computing recommendation scores. Models are organized into 3 layers: staging, intermediate, and marts. + +All models are materialized as tables and routed to specific PostgreSQL schemas via the `generate_schema_name` macro. + +## Model Organization + +```mermaid +graph TD + subgraph Staging + SG1[stg_github__project] + SG2[stg_github__readme] + SG3[stg_github__languages] + SG4[stg_github__topics] + SG5[stg_github__detection] + SP1[stg_public__project] + SP2[stg_public__user] + end + + subgraph Intermediate + IE[int_project_enriched] + IC[int_project_contextualized] + IEC[int_project_embedding_candidate] + IU[int_user_enriched] + end + + subgraph Marts + FG[fct_github_project] + FU[fct_public_user] + MG[match_global_recommendation] + MU[match_user_recommendation] + end + + SG1 & SG2 & SG3 & SG4 & SG5 --> IE + IE --> FG + SP1 --> IC --> IEC + SP2 --> IU --> FU + FG --> MG + FG & IEC --> MU +``` + +## Models Reference + +### Staging Models + +Staging models clean and type-cast raw data. They follow the `stg___` naming convention. + +| Model | Schema | Dagster Group | Description | +|-------|--------|---------------|-------------| +| `stg_github__project` | `github` | ingestion | Flattens raw JSON from `raw_github_project` into typed columns (name, description, stars, URL, etc.) | +| `stg_github__readme` | `github` | ingestion | Stages README content from `raw_github_readme` | +| `stg_github__languages` | `github` | ingestion | Stages language breakdowns from `raw_github_languages` | +| `stg_github__topics` | `github` | ingestion | Stages topic arrays from `raw_github_topics` | +| `stg_github__detection` | `github` | ingestion | Stages FastText detection results from `int_github_detection` | +| `stg_public__project` | `ml` | project_ml | Stages public project data for ML processing | +| `stg_public__user` | `ml` | user_ml | Stages user data for ML processing | + +### Intermediate Models + +Intermediate models join, enrich, and prepare data for consumption. + +| Model | Schema | Dagster Group | Description | +|-------|--------|---------------|-------------| +| `int_project_enriched` | `github` | ingestion | Joins project with readme, languages, topics, and detection data | +| `int_project_contextualized` | `ml` | project_ml | Builds rich context strings for embedding using `build_project_context` macro | +| `int_project_embedding_candidate` | `ml` | project_ml | Filters projects with valid context strings, ready for embedding | +| `int_user_enriched` | `ml` | user_ml | Builds user context strings for embedding using `build_user_context` macro | + +### Mart Models + +Mart models are the final consumption layer, used directly by the backend and ML assets. + +| Model | Schema | Dagster Group | Description | +|-------|--------|---------------|-------------| +| `fct_github_project` | `github` | ingestion | Fact table with stars, forks, pushed_at, and all enriched metadata | +| `fct_public_user` | `ml` | user_ml | User fact table with aggregated preferences | +| `match_global_recommendation` | `public` | project_ml | Top-N global recommendations (trending/published, ordered by recency and stars) | +| `match_user_recommendation` | `public` | user_ml | Per-user personalized recommendations with hybrid scoring | + +## Macros + +### Data Cleaning + +| Macro | Description | +|-------|-------------| +| `clean_text(column)` | Strips HTML, normalizes whitespace, removes special characters | +| `deduplicate(relation, partition_by, order_by)` | Window-function dedup (keeps first row per partition) | +| `jsonb_to_list(column)` | Converts a JSONB array to a comma-separated string | + +### Context Building + +| Macro | Description | +|-------|-------------| +| `build_project_context(...)` | Concatenates project title, description, topics, languages, and README into a single string for embedding | +| `build_user_context(...)` | Concatenates user bio, job title, tech stacks, categories, and domains into a single string for embedding | + +### Scoring Helpers + +| Macro | Description | +|-------|-------------| +| `safe_divide(numerator, denominator)` | Returns NULL instead of dividing by zero | +| `clamp(expression)` | Clamps a numeric expression to the [0, 1] range | + +### Schema Management + +| Macro | Description | +|-------|-------------| +| `generate_schema_name(custom_schema_name, node)` | Overrides dbt default to use the `+schema` value directly without prefix | + +## Recommendation Scoring + +### Global Recommendations + +The `match_global_recommendation` model selects the top-N projects (default: 20) that are trending or published, ordered by last sync time and star count. + +### User Recommendations (Hybrid Scoring) + +The `match_user_recommendation` model computes personalized scores by blending 4 signals: + +| Signal | Weight | Description | +|--------|--------|-------------| +| **Similarity** | 40% | Cosine similarity between user and project embeddings (`1 - (user_vec <=> proj_vec)`) | +| **Preference** | 35% | Weighted overlap between user and project attributes (tech stacks, categories, domains) | +| **Freshness** | 15% | Linear decay based on last push date (configurable decay window: 90 days) | +| **Popularity** | 10% | Log-normalized star count, scaled to [0, 1] | + +The final formula: + +``` +final_score = 0.40 * similarity + 0.35 * preference + 0.15 * freshness + 0.10 * popularity +``` + +#### Preference Score Breakdown + +The preference signal itself is a weighted combination of 3 overlap dimensions: + +| Dimension | Sub-Weight | Calculation | +|-----------|-----------|-------------| +| Tech stacks | 30% | `shared_tech_stacks / user_total_tech_stacks` | +| Categories | 45% | `shared_categories / user_total_categories` | +| Domains | 25% | `shared_domains / user_total_domains` | + + +If a user has no items in a dimension (e.g., no tech stacks selected), that dimension is excluded and its weight is redistributed proportionally among active dimensions. This prevents penalizing users with incomplete profiles. + + +#### Filtering and Thresholds + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `similarity_threshold` | 0.25 | Minimum cosine similarity to be considered | +| `reco_top_n` | 30 | Maximum recommendations per user | +| `freshness_decay_days` | 90 | Days until freshness score reaches 0 | +| `global_reco_top_n` | 20 | Number of global recommendations | + +All scoring parameters are defined as dbt vars in `dbt_project.yml` and can be overridden at runtime. + +## Data Contracts + +Mart models use dbt data contracts with `contract: {enforced: true}`. Each column specifies a `data_type` and optional `constraints`, ensuring schema stability for downstream consumers (backend API, ML assets). + +## Profiles + +| Profile | Host | Port | Usage | +|---------|------|------|-------| +| `local` (default) | `localhost` | 5433 | Local development (DB exposed via Docker) | +| `docker` | `db` | 5432 | Inside Docker Compose network | + +Switch profiles by setting the `DBT_TARGET` environment variable: + +```bash +export DBT_TARGET=docker +dbt build +``` diff --git a/ai/go-services.mdx b/ai/go-services.mdx new file mode 100644 index 0000000..f3ba9eb --- /dev/null +++ b/ai/go-services.mdx @@ -0,0 +1,153 @@ +--- +title: "Go Services" +description: "GitHub scraper and fetcher binaries used for data ingestion" +--- + +OST Linker uses two independent Go binaries for high-performance data ingestion from the GitHub API. Both are invoked by Dagster assets as subprocesses with a 600-second timeout. + +## Scraper + +**Location:** `src/services/go/scraper/` + +The scraper searches the GitHub Search API for open-source projects matching specific criteria and upserts the raw JSON data into `github.raw_github_project`. + +### How It Works + +1. Reads search queries from the `GITHUB_SCRAPING_QUERIES` environment variable (JSON array) +2. Launches one goroutine per query for parallel scraping +3. Each query paginates through results (100 per page, up to 1,000 per query) +4. Results are batch-upserted into PostgreSQL using `pgx.Batch` +5. Outputs a JSON summary to stdout (parsed by the Dagster asset) + +### Search Queries + +Queries are built dynamically by `PipelineConfig` in Python and cover 3 star ranges: + +| Range | Stars | +|-------|-------| +| Range 1 | 300 -- 1,000 | +| Range 2 | 1,000 -- 3,000 | +| Range 3 | 3,000 -- 5,000 | + +Each query includes these filters: +- `good-first-issues:>1` -- at least one good first issue +- `help-wanted-issues:>0` -- at least one help wanted issue +- `topics:>2` -- at least 2 topics defined +- `fork:false` -- exclude forks +- `pushed:>=<7 days ago>` -- active in the last week +- `is:public archived:false` -- public and not archived +- Excludes terms: `awesome`, `roadmap`, `cheatsheet`, `interview` + +### Rate Limiting + +The scraper uses a shared `searchRateLimiter` across all goroutines that tracks GitHub's `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers. + +| Parameter | Value | +|-----------|-------| +| Initial budget | 30 requests (GitHub Search API limit per minute for authenticated users) | +| Rate limit handling | Sleeps until reset time + 1 second | +| 403 response handling | Reads `Retry-After` header, falls back to 60 seconds | +| Retry policy | 3 attempts with exponential backoff (2s, 4s) | +| HTTP timeout | 30 seconds per request | +| Global timeout | 8 minutes for all queries | + +### Environment Variables + +| Variable | Required | Description | +|----------|----------|-------------| +| `GITHUB_SCRAPING_QUERIES` | Yes (or `GITHUB_SCRAPING_QUERY`) | JSON array of search queries | +| `GITHUB_ACCESS_TOKEN` | Recommended | GitHub API token (unauthenticated requests have lower limits) | +| `DATABASE_URL` | Yes | PostgreSQL connection string | +| `GITHUB_API_URL` | No | Override API endpoint (default: `https://api.github.com/search/repositories`) | + +### Output + +The scraper writes a JSON summary to stdout: + +```json +{ + "queries": [ + {"query": "stars:300..1000 ...", "collected_count": 450, "upserted_count": 448, "failed_upserts": 2} + ], + "total_collected": 1200, + "total_upserted": 1195, + "total_failed": 5, + "status": "partial", + "duration_seconds": 45.2 +} +``` + +## Fetcher + +**Location:** `src/services/go/fetcher/` + +The fetcher enriches scraped projects by fetching additional data from the GitHub REST API. It operates in 3 modes, each targeting a different API endpoint and database table. + +### Modes + +| Mode | API Endpoint | Output Table | Description | +|------|-------------|-------------|-------------| +| `readme` | `GET /repos/{owner}/{repo}/readme` | `github.raw_github_readme` | Fetches raw README content | +| `languages` | `GET /repos/{owner}/{repo}/languages` | `github.raw_github_languages` | Fetches language byte counts | +| `topics` | `GET /repos/{owner}/{repo}/topics` | `github.raw_github_topics` | Fetches repository topics | + +### CLI Flags + +```bash +ost-fetcher --mode readme --concurrency 20 --limit 100 +``` + +| Flag | Default | Description | +|------|---------|-------------| +| `--mode` | (required) | One of: `readme`, `languages`, `topics` | +| `--concurrency` | 10 | Number of concurrent worker goroutines | +| `--limit` | 0 (no limit) | Maximum number of projects to process | + +### Incremental Fetching + +The fetcher only processes projects that do not already have data in the target table. It queries `github.int_github_detection` (the output of FastText language filtering) and LEFT JOINs against the target table to find new projects. + +### Rate Limiting and Retries + +| Parameter | Value | +|-----------|-------| +| Initial budget | 5,000 requests (GitHub REST API limit per hour for authenticated users) | +| Rate limit tracking | Reads `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers | +| Retry policy | Up to 3 attempts with exponential backoff (1s, 2s, 3s) | +| 403 handling | Reads `Retry-After` header, falls back to 60-second sleep | +| 404/422 handling | No retry, returns empty result | +| HTTP timeout | 30 seconds per request | +| Global timeout | 30 minutes | +| Response body limit | 10 MB per response | + +### README Truncation + +README content is truncated to 50,000 bytes (preserving valid UTF-8 boundaries) before insertion into the database. + + +The `truncateUTF8` function ensures multi-byte characters are never split, preventing invalid UTF-8 sequences in the database. + + +### Batch Upserts + +Results are collected via channels and flushed in batches of 100 using `pgx.Batch` for efficient database writes. Each batch uses `ON CONFLICT ... DO UPDATE` to handle re-runs gracefully. + +## Dagster Integration + +Both Go binaries are invoked by Dagster Python assets using `subprocess.run()`: + +```python +result = subprocess.run( + [binary_path], # or [binary_path, "--mode", "readme", "--concurrency", "20"] + capture_output=True, + text=True, + env=env, # DATABASE_URL, GITHUB_ACCESS_TOKEN, queries + timeout=600, # 10 minutes +) +``` + +The environment is constructed by `build_scraper_env()` and `build_fetcher_env()` helpers in `cfg_resource.py`, which read values from the `PipelineConfig` resource. + + +If the Go binary is not found at the configured path, the asset will raise a `RuntimeError` with a clear message. Make sure `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` point to compiled binaries. + diff --git a/ai/installation.mdx b/ai/installation.mdx index 136f0cc..bf2921c 100644 --- a/ai/installation.mdx +++ b/ai/installation.mdx @@ -1,127 +1,174 @@ -# OST AI Engine – Installation & Deployment Guide - -This guide explains how to install, configure, and deploy the OST AI Engine platform, with a focus on centralized configuration, multi-language management, Dockerization, and Dagster orchestration. - -## Requirements -- OS: Linux, macOS, Windows -- Python: 3.13+ -- Go: ≥ 1.20 -- Node.js (for Prisma) -- Docker - -## Centralized Configuration Management - -Project configuration is centralized in the `config/config.yaml` file, which is automatically generated by the Python module `config.py` using environment variables and business logic (e.g., calculation of `seven_days_ago`). - -- **Why this choice?** YAML is accessible by all languages (Python, Go, Node.js), ensuring consistent parameters (tokens, queries, N projects, etc.) throughout the pipeline. -- **Workflow:** - 1. Environment variables are defined in `.env`. - 2. On startup or build, `config/config.py` reads these variables and generates/updates `config/config.yaml`. - 3. All components (Dagster, Go scrapers, Prisma) read config from YAML. - -**Example config.example.yaml:** -```yaml -DATABASE_URL: postgresql://postgres:postgres@ost-db:5432/ost_dev -GITHUB_ACCESS_TOKEN: ... -GITLAB_ACCESS_TOKEN: ... -GITHUB_SCRAPING_QUERY: stars:>100 stars:<500 created:>=2025-10-11 is:public archived:false -GITHUB_TOP_N: 30 +--- +title: "Installation & Setup" +description: "Prerequisites, environment configuration, and development setup for OST Linker" +--- + +## Prerequisites + +| Tool | Version | Purpose | +|------|---------|---------| +| Docker | Latest | Container runtime for all services | +| Go | 1.24+ | Compile scraper and fetcher binaries | +| Python | 3.11 | Dagster, dbt, ML models | +| Node.js | 18+ | Prisma CLI for schema management and seeding | +| uv | Latest | Python dependency management (replaces pip/poetry) | + +## Quick Start with Docker + +The fastest way to get OST Linker running: + +```bash +# 1. Clone and configure +git clone https://github.com/opensource-together/ost-linker.git +cd ost-linker +cp .env.example .env +# Edit .env with your values (see Environment Variables below) + +# 2. Launch all services +docker compose up --build -d ``` +This starts the Dagster webserver (port 3000), daemon, and a PostgreSQL database with pgvector (port 5433). + + +The Dagster UI is available at [http://localhost:3000](http://localhost:3000) once the containers are running. + + ## Environment Variables -Copy `.env.example` to `.env` and fill in the values: -```ini -OST_CONFIG_PATH=config/config.yaml +All configuration is driven by environment variables. Copy `.env.example` to `.env` and fill in the required values. + +### Required + +| Variable | Description | +|----------|-------------| +| `DATABASE_URL` | PostgreSQL connection string (e.g., `postgresql://user:pass@localhost:5433/dbname`) | +| `GITHUB_ACCESS_TOKEN` | GitHub fine-grained personal access token for API access | +| `OPENROUTER_API_KEY` | API key for OpenRouter (LLM classification) | +| `GO_SCRAPER_PATH` | Absolute path to the compiled Go scraper binary | +| `GO_FETCHER_PATH` | Absolute path to the compiled Go fetcher binary | + +### Optional -DATABASE_URL=postgresql://ai-engine:ai-engine@localhost:7777/ai-engine -POSTGRES_DB=ai-engine -POSTGRES_USER=ai-engine -POSTGRES_PASSWORD=ai-engine +| Variable | Default | Description | +|----------|---------|-------------| +| `FASTTEXT_MODEL_PATH` | `models/lid.176.ftz` | Path to FastText language detection model | +| `DBT_TARGET` | `local` | dbt profile target (`local` for port 5433, `docker` for port 5432) | +| `DBT_PROJECT_DIR` | `/dbt` | dbt project directory (set to `/app/dbt` in Docker) | +| `DAGSTER_HOME` | `./dagster_home` | Dagster metadata and run storage directory | -GITHUB_ACCESS_TOKEN=your_github_access_token_here -GITLAB_ACCESS_TOKEN=your_gitlab_access_token_here + +Never commit `.env` or hardcode secrets in code. The `.env` file is gitignored by default. + + +## Local Development Setup + +For development outside of Docker, follow these steps in order. + +### 1. Install Python Dependencies + +```bash +uv sync ``` -**Propagation:** -- Python loads variables via `dotenv`. -- `config.py` injects them into YAML. -- Go and Dagster read config from YAML (never directly from `.env`). - -| Variable | Description | Example | -|-----------------------|------------------------------|---------| -| DATABASE_URL | PostgreSQL connection URL | postgresql://db_user:db_password@localhost:port/db_name | -| POSTGRES_DB | Database name | db_name | -| POSTGRES_USER | Database user | db_user | -| POSTGRES_PASSWORD | Database password | db_password | -| GITHUB_ACCESS_TOKEN | GitHub API token | your_github_access_token_here | -| GITLAB_ACCESS_TOKEN | GitLab API token | your_gitlab_access_token_here | - -## Install Dependencies +### 2. Compile Go Binaries + +```bash +cd src/services/go/scraper && go build -o github-scraper main.go +cd ../fetcher && go build -o ost-fetcher main.go +``` + +Or use the convenience script: + ```bash -poetry install -go mod tidy ./src/infrastructure/services/go/github -go mod tidy ./src/infrastructure/services/go/gitlab -cd prisma -npm install +scripts/go_binary_gen.sh ``` -## Database Setup -Start PostgreSQL with Docker Compose: +Then set `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` in your `.env` to the absolute paths of the compiled binaries. + +### 3. Download FastText Model + +The language detection model (`lid.176.ftz`) is not included in the repo due to its size. Download it: + ```bash -docker compose up -d +mkdir -p models +wget -O models/lid.176.ftz https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz ``` -## Prisma Migrations -Apply database migrations: +### 4. Start the Database + +If not using Docker Compose for the full stack, start just the database: + ```bash -cd prisma -npx prisma migrate deploy +docker compose up db -d ``` -## Dockerization & Multi-language Build +This starts PostgreSQL with pgvector on port 5433. -The Dockerfile builds the entire stack: -- Installs Python dependencies (Poetry), Go, Node.js -- Compiles Go scrapers -- Generates Prisma client -- Copies and generates centralized YAML config -- Sets `DAGSTER_HOME` for Dagster -- Entrypoint: launches Dagster daemon +### 5. Initialize the Database -**Dockerfile excerpt:** -```dockerfile -FROM python:3.13-slim AS base +Apply the Prisma schema and seed reference data: -WORKDIR /app +```bash +npx prisma db push +npx ts-node prisma/seed/seed.ts +``` -COPY pyproject.toml poetry.lock ./ -RUN pip install poetry && poetry install --no-root --only main + +`prisma db push` applies the schema without creating migration files. For production, use `prisma migrate deploy`. + -COPY src/ src/ -COPY prisma/ prisma/ -COPY .env .env -COPY config/ config/ +### 6. Install dbt Dependencies -RUN poetry run python config/config.py -RUN poetry run prisma generate -ENV GOARCH=arm64 -RUN cd src/infrastructure/services/go/github && go build -o /app/github-scraper main.go -ENV DAGSTER_HOME=/app/src/dagster +```bash +cd dbt && dbt deps +``` -EXPOSE 3000 -CMD ["poetry", "run", "dagster-daemon", "run"] +### 7. Run Dagster Locally + +```bash +dagster dev -h 0.0.0.0 -p 3000 ``` -## Dagster Orchestration & Cron +The Dagster UI will be available at `http://localhost:3000`. -- The main job (`github_scraper_job`) is scheduled via a cron table (e.g., every 6 hours: `0 */6 * * *`). -- Dagster config (`dagster.yaml`) allows customization of storage, logs, etc. -- Dagster assets are modular: each pipeline step is an asset, including Go scrapers. +## Docker Build Details + +The Dockerfile uses a 3-stage build: + +| Stage | Base Image | Purpose | +|-------|-----------|---------| +| Go Builder | `golang:1.24-alpine` | Compiles both Go binaries to `/app/bin/` | +| Python Builder | `python:3.11-slim` | Exports dependencies via `uv export` to `requirements.txt` | +| Runtime | `python:3.11-slim` | Installs deps, copies Go binaries to `/usr/local/bin/`, runs Dagster | + +### Compose Services + +| Service | Port | Description | +|---------|------|-------------| +| `webserver` | 3000 | Dagster webserver (UI + GraphQL API) | +| `daemon` | -- | Dagster daemon (schedules, sensors, run monitoring) | +| `db` (dev only) | 5433 | PostgreSQL with pgvector (`ankane/pgvector:v0.4.1`) | + + +The `db` service is defined in `docker-compose.override.yml` and only runs in local development. In production, use an external managed PostgreSQL instance. + + +## Verifying the Setup + +After completing the setup, verify everything is working: -**Dagster local launch** : ```bash -export DAGSTER_HOME="$PWD/src/dagster" -poetry run dagster dev -m src.dagster.definitions --host 127.0.0.1 --port 3000 +# Check Python deps and Dagster module +dagster definitions validate + +# Run dbt models +cd dbt && dbt build + +# Run tests +pytest + +# Check Go binaries +src/services/go/scraper/github-scraper --help +src/services/go/fetcher/ost-fetcher --help ``` -Accès UI Dagster : [http://localhost:3000](http://localhost:3000) \ No newline at end of file diff --git a/ai/overview.mdx b/ai/overview.mdx index bb5da8b..01b1e4a 100644 --- a/ai/overview.mdx +++ b/ai/overview.mdx @@ -1,90 +1,115 @@ +--- +title: "Architecture Overview" +description: "High-level architecture of the OST Linker AI recommendation engine" +--- +## What is OST Linker? -Learn about the OST AI-Engine architecture and features - -OST Chevalier - -## Introduction - -The AI-Engine is a modular, extensible data & machine learning platform designed to orchestrate, analyze, and enrich open-source project data. -Its mission is to automate the collection, transformation, and ranking of data from sources like GitHub and GitLab, powering advanced analytics and dashboards for the [OpenSource Together](https://opensource-together.com) ecosystem. +OST Linker is the AI-powered recommendation engine behind [OpenSourceTogether](https://opensource-together.com/). It continuously scrapes GitHub for open-source projects, classifies them using LLMs, computes semantic embeddings, and surfaces personalized recommendations to users via cosine similarity with pgvector. - Core Principle: The AI Engine centralizes data intelligence for open-source communities, enabling automated workflows, unified data models, and scalable enrichment pipelines. +OST Linker runs as a fully automated pipeline orchestrated by **Dagster**. Once deployed, it requires no manual intervention -- projects are discovered, enriched, classified, and recommended on schedule. -## Key Features - - - - Collects and updates project data from GitHub & GitLab sources automatically - - - Transforms raw data into a unified, queryable format for analytics and dashboards - - - Ranks projects by popularity, activity, and custom metrics - - - Seamless integration with a PostgreSQL database using Prisma ORM - - - Robust pipeline orchestration, scheduling, and asset checks with Dagster - - - Step-by-step guide to install dependencies, configure environment, and deploy the AI Engine - - - -## Technology Stack - - - - - Dagster: Data pipeline orchestration, scheduling, and asset management - - - - Prisma: Type-safe ORM for PostgreSQL - - PostgreSQL: Scalable relational database - - - - Python 3.13+: Core engine and pipeline logic - - Go 1.20+: Ingestion services and high-performance modules - - - - Docker: Containerized deployment - - GitHub Actions: CI/CD workflows - - - -## Architecture Principles - -The AI Engine follows a feature-based architecture with clear separation of concerns: - - - Extensibility: Easily add new data sources, assets, and custom logic via modular connectors and pipelines. - - -### Key Architectural Decisions - -1. Modular Connectors: Each data source (GitHub, GitLab, etc.) is handled by an autonomous connector module -2. Unified Data Model: All raw data is mapped to a consistent schema for analytics and dashboards -3. Automated Pipelines: Dagster orchestrates scraping, transformation, and enrichment workflows -4. Type Safety & Reliability: Prisma ensures robust database operations and schema validation -5. Scalability: Designed to handle thousands of projects and contributors +## Data Flow + +The pipeline follows a linear progression through 5 stages: + +```mermaid +graph LR + A[Ingestion] --> B[Classification] + B --> C[Sync] + C --> D[Project ML] + D --> E[User ML] + + style A fill:#2d6a4f,color:#fff + style B fill:#40916c,color:#fff + style C fill:#52b788,color:#fff + style D fill:#74c69d,color:#000 + style E fill:#95d5b2,color:#000 +``` + +### Detailed Pipeline + +```mermaid +graph TD + subgraph Ingestion + GH[GitHub Search API] -->|Go scraper| RAW[raw_github_project] + RAW -->|dbt staging| STG[stg_github__project] + STG -->|FastText| DET[int_github_detection] + DET -->|Go fetcher| README[raw_github_readme] + DET -->|Go fetcher| LANG[raw_github_languages] + DET -->|Go fetcher| TOPICS[raw_github_topics] + README & LANG & TOPICS -->|dbt| FCT[fct_github_project] + end + + subgraph Classification + FCT --> LLM[LLM Classifier] + LLM -->|Category + Domain| CLASS[project_classification] + end + + subgraph Sync + CLASS --> SYNC[public.Project] + end + + subgraph Project ML + SYNC -->|dbt| CTX[int_project_contextualized] + CTX -->|dbt| CAND[int_project_embedding_candidate] + CAND -->|SentenceTransformer| PEMB[embd_github_project] + PEMB -->|dbt| GREC[match_global_recommendation] + end + + subgraph User ML + USR[public.User] -->|dbt| UENR[int_user_enriched] + UENR -->|SentenceTransformer| UEMB[embd_user] + UEMB -->|dbt| UREC[match_user_recommendation] + end +``` + +## Pipeline Stages + +| Stage | Purpose | Key Technology | +|-------|---------|----------------| +| **Ingestion** | Scrape GitHub, detect languages, fetch READMEs/topics/languages | Go binaries, FastText, dbt | +| **Classification** | Assign Category and Domain to each project via LLM | Mistral Small 3.2 via OpenRouter | +| **Sync** | Write enriched project data to the public-facing `Project` table | Python, PostgreSQL | +| **Project ML** | Build project context strings, compute 384-dim embeddings, generate global recommendations | dbt, SentenceTransformer (MiniLM-L6-v2) | +| **User ML** | Build user context strings, compute user embeddings, generate personalized recommendations | dbt, SentenceTransformer, pgvector cosine similarity | + +## Tech Stack + +| Layer | Technology | +|-------|-----------| +| **Orchestration** | Dagster (assets, jobs, schedules, sensors) | +| **Data transformation** | dbt (staging, intermediate, marts) | +| **Ingestion services** | Go 1.24 (scraper + fetcher binaries) | +| **ML / Classification** | Python 3.11, SentenceTransformer, FastText, OpenRouter (Mistral) | +| **Database** | PostgreSQL + pgvector (384-dim vectors, cosine similarity) | +| **Schema management** | Prisma (4 schemas: `public`, `github`, `ml`, `match`) | +| **Containerization** | Docker (3-stage build: Go builder, Python builder, runtime) | +| **CI/CD** | GitHub Actions (quality checks, Docker publish, submodule sync) | + +## Key Design Decisions + +1. **Go for ingestion** -- High-performance concurrent HTTP requests with goroutines and rate limiting, compiled to static binaries invoked by Dagster via `subprocess.run()` +2. **dbt for transformations** -- SQL-first data modeling with contracts, tests, and documentation baked in +3. **Hybrid recommendation scoring** -- Combines semantic similarity (40%), user preference overlap (35%), project freshness (15%), and popularity (10%) +4. **pgvector for similarity search** -- Native PostgreSQL extension for cosine similarity on 384-dimensional embeddings, no external vector DB required +5. **Prisma as schema source of truth** -- Single schema definition shared between the backend (Node.js) and the AI engine ## Getting Started -Ready to build with the AI Engine? Follow our Quick Start Guide to set up your development environment. - - - Get up and running in minutes - - - Deep dive into the AI Engine architecture - + + Understand every asset, job, and schedule in detail + + + Set up the development environment from scratch + + + Explore the 4 PostgreSQL schemas and pgvector setup + + + Navigate the codebase directory layout + diff --git a/ai/pipeline.mdx b/ai/pipeline.mdx index 494b7eb..a32a183 100644 --- a/ai/pipeline.mdx +++ b/ai/pipeline.mdx @@ -1,89 +1,164 @@ -The OST AI Engine pipeline orchestrates the scraping, ranking, normalization, and insertion of trending open-source projects into the database. Dagster manages the workflow, ensuring each step is executed reliably and data quality is maintained. - -## Job Scheduling & Definition - -Below are visual diagrams showing how the GitHub scraping job is scheduled and defined in Dagster: - -
-
- GitHub Scraper Job Schedule -
-

- Job Schedule: The pipeline is scheduled to run automatically every 6 hours, keeping the database up to date with the latest trending projects and respecting the rate-limiting of the GitHub API. -

-
- -
- GitHub Scraper Job Definition -
-

- Job Definition: The job includes four main assets: scraping, ranking, mapping, and database insertion, with data quality checks at each stage. -

- - -
-
- GitHub Scraper Assets -
-

- Assets Overview: Each asset represents a distinct step in the pipeline, from data collection to database insertion, ensuring modularity and maintainability. -

-
- - -## Asset Checks - -Dagster asset checks validate the quality and consistency of data at each pipeline step. Here are the main checks used in the OST AI Engine pipeline: - -| Check Name | Purpose | -|-------------------------------------- |--------------------------------------------------------------------------| -| github_top_projects_description_check | Ensures all projects have a non-empty description | -| github_mapping_type_check | Checks that the mapping output is a non-empty list | -| github_mapping_required_fields_check | Validates required fields and types (title, repoUrl, provider, published, trending) | -| github_mapping_duplicate_url_check | Detects duplicate repoUrl or githubUrl values | -| github_to_db_insert_count_check | Confirms the number of DB inserts matches the number of mapped projects | -| github_to_db_error_check | Checks for insertion errors in the logs | -| github_to_db_consistency_check | Verifies that inserted projects exist in the database | -| github_to_db_uniqueness_check | Ensures repoUrl values are unique in the database | -| github_to_db_mapping_match_check | Checks that DB fields match the mapping (e.g., title/repoUrl) | - -Below are visual examples of asset checks in the Dagster UI: - -
- GitHub Mapping Checks Overview -
-

- Checks Overview: The Dagster UI displays the status of each asset check, making it easy to monitor data quality in real time. -

- -
- GitHub Mapping Checks Details -
-

- Checks Details: Detailed results show which projects passed or failed each check, with error messages for troubleshooting and data validation. All errors and warnings are surfaced in the Dagster UI for rapid debugging. -

\ No newline at end of file +--- +title: "Pipeline Deep-Dive" +description: "Complete reference for Dagster assets, jobs, schedules, and resources" +--- + +## Asset Groups + +OST Linker organizes its Dagster assets into 5 groups that execute in sequence. Each group contains both Python assets and dbt models. + +### Ingestion + +The ingestion group collects raw data from GitHub and transforms it through staging and enrichment layers. + +| Asset | Type | Description | +|-------|------|-------------| +| `raw_github__extract_projects` | Python (Go subprocess) | Runs the Go scraper to search GitHub and upsert raw project data | +| `stg_github__project` | dbt | Cleans and flattens raw JSON into typed columns | +| `core_github__detect_languages` | Python | Filters non-English repos using FastText language detection | +| `core_github__fetch_readme` | Python (Go subprocess) | Fetches README content for detected projects | +| `core_github__fetch_repo_languages` | Python (Go subprocess) | Fetches repository language breakdowns | +| `core_github__fetch_repo_topics` | Python (Go subprocess) | Fetches repository topics | +| `stg_github__readme`, `stg_github__languages`, `stg_github__topics`, `stg_github__detection` | dbt | Staging models for fetched data | +| `int_project_enriched` | dbt | Joins all staging sources into a single enriched project view | +| `fct_github_project` | dbt | Final fact table with stars, forks, pushed_at, and all metadata | + +### Classification + +| Asset | Type | Description | +|-------|------|-------------| +| `core_match__classify_projects` | Python | Sends project context to Mistral Small 3.2 (via OpenRouter) to assign a Category and Domain | + +The LLM receives a truncated context (max 8,000 chars) containing the project title, description, topics, and README. It returns a JSON object with `category` and `domain` fields matched against the valid labels from the database. + +### Sync + +| Asset | Type | Description | +|-------|------|-------------| +| `core_public__sync_projects` | Python | Syncs enriched and classified project data into the `public.Project` table | + +### Project ML + +| Asset | Type | Description | +|-------|------|-------------| +| `stg_public__project` | dbt | Stages public project data for ML processing | +| `int_project_contextualized` | dbt | Builds rich context strings from project metadata using the `build_project_context` macro | +| `int_project_embedding_candidate` | dbt | Selects projects ready for embedding (non-null context) | +| `core_ml__embed_projects` | Python | Computes 384-dim embeddings with SentenceTransformer and upserts to `ml.embd_github_project` | +| `match_global_recommendation` | dbt | Top-N global recommendations ranked by recency and stars | + +### User ML + +| Asset | Type | Description | +|-------|------|-------------| +| `stg_public__user` | dbt | Stages user data for ML processing | +| `int_user_enriched` | dbt | Builds user context strings using the `build_user_context` macro | +| `fct_public_user` | dbt | Final user fact table | +| `core_ml__embed_users` | Python | Computes 384-dim user embeddings and upserts to `ml.embd_user` | +| `match_user_recommendation` | dbt | Personalized recommendations using hybrid scoring | + +## Data Flow + +The full asset dependency graph follows this order: + +```mermaid +graph TD + A[raw_github_project] --> B[stg_github__project] + B --> C[core_github__detect_languages] + C --> D1[core_github__fetch_readme] + C --> D2[core_github__fetch_repo_languages] + C --> D3[core_github__fetch_repo_topics] + D1 --> E1[stg_github__readme] + D2 --> E2[stg_github__languages] + D3 --> E3[stg_github__topics] + B --> F[int_project_enriched] + E1 & E2 & E3 --> F + F --> G[fct_github_project] + G --> H[core_match__classify_projects] + H --> I[core_public__sync_projects] + I --> J[stg_public__project] + J --> K[int_project_contextualized] + K --> L[int_project_embedding_candidate] + L --> M[core_ml__embed_projects] + M --> N[match_global_recommendation] +``` + +## Jobs + +| Job | Groups | Description | Retry Policy | +|-----|--------|-------------|-------------| +| `project_enrichment_job` | ingestion, classification, sync, project_ml | Full project pipeline from scraping to recommendations | 2 retries, exponential backoff, full jitter | +| `user_recommendation_job` | user_ml | User embedding and recommendation refresh | Default | +| `run_all_job` | All groups | Manual-only job for initial setup or recovery | Default | +| `cleanup_dagster_history_job` | N/A | Cleans up old Dagster run history | Default | + + +The `project_enrichment_job` is tagged with `dagster/max_concurrent_runs: 1` to prevent overlapping runs. + + +## Schedules + +| Schedule | Job | Cron | Timezone | Status | +|----------|-----|------|----------|--------| +| `project_enrichment_schedule` | `project_enrichment_job` | `0 3 * * *` (daily at 3 AM) | Europe/Paris | Running | +| `user_recommendation_schedule` | `user_recommendation_job` | `*/10 * * * *` (every 10 min) | Europe/Paris | Running | +| `cleanup_dagster_history_schedule` | `cleanup_dagster_history_job` | `0 23 */2 * *` (every 2 days at 11 PM) | Europe/Paris | Running | + +## Resources + +All resources are configured in `src/linker/definitions.py` and injected into assets via `required_resource_keys`. + +### PipelineConfig + +Central configuration resource that reads environment variables at runtime. + +| Field | Source | Description | +|-------|--------|-------------| +| `db_url` | `DATABASE_URL` | PostgreSQL connection string | +| `github_token` | `GITHUB_ACCESS_TOKEN` | GitHub API token for scraping | +| `go_scraper_path` | `GO_SCRAPER_PATH` | Path to compiled Go scraper binary | +| `go_fetcher_path` | `GO_FETCHER_PATH` | Path to compiled Go fetcher binary | +| `github_api_url` | Hardcoded | `https://api.github.com/search/repositories` | + +The config resource also builds GitHub search queries dynamically. Queries target 3 star ranges (300-1000, 1000-3000, 3000-5000) with filters for good first issues, recent activity, and excluded terms (awesome, roadmap, cheatsheet, interview). + +### LLMClassifierResource + +| Property | Value | +|----------|-------| +| Provider | OpenRouter (`https://openrouter.ai/api/v1`) | +| Model | `mistralai/mistral-small-3.2-24b-instruct` | +| Temperature | 0.0 | +| Response format | JSON object | +| Timeout | 45s hard timeout (thread-based) | +| Context truncation | 8,000 characters | + +### SentenceTransformerResource + +| Property | Value | +|----------|-------| +| Model | `sentence-transformers/all-MiniLM-L6-v2` | +| Embedding dimension | 384 | +| Device | CPU (configurable) | +| Normalization | Enabled (for cosine similarity) | + +Supports both single text encoding (`encode`) and batch encoding (`encode_batch`). + +### FastTextModelResource + +| Property | Value | +|----------|-------| +| Model file | `models/lid.176.ftz` | +| Purpose | Language detection on project text (name, description, README) | +| Loading | Lazy-loaded singleton, reused across all runs | + +### PandasPostgresIOManager + +Custom IO manager that transfers DataFrames between assets via PostgreSQL. Uses a schema/table allowlist to prevent SQL injection. Supports truncate-then-append writes and full table reads. + +### Other Resources + +| Resource | Type | Purpose | +|----------|------|---------| +| `dbt` | `DbtCliResource` | Runs dbt CLI commands from Dagster | +| `fs_io_manager` | `FilesystemIOManager` | Default file-based IO for non-DB assets | diff --git a/ai/structure.mdx b/ai/structure.mdx index aad3b38..b9a6ff3 100644 --- a/ai/structure.mdx +++ b/ai/structure.mdx @@ -1,25 +1,169 @@ -# Project Structure -```python -config/ - config.py # Generates YAML from env - config.yaml # Centralized config file -src/ - dagster/ - assets.py # Dagster assets definition - dagster.yaml # Dagster config instance - definitions.py # Dagster jobs, schedules, sensors - config/ # Centralized config files - config.py # Source YAML generated on project root config/config.yaml - github_mapping.py - infrastructure/ - services/ - go/ - github/ - go.mod, main.go, main_test.go - gitlab/ - go.mod, main.go, main_test.go +--- +title: "Project Structure" +description: "Directory layout and key files of the OST Linker codebase" +--- + +## Top-Level Layout + +``` +ost-linker/ + src/ + linker/ # Main Dagster module (Python) + services/ # External services (Go + Python) + dbt/ # dbt project (SQL transformations) + prisma/ # Database schema and seeds + scripts/ # Utility shell scripts + models/ # ML model files (FastText) + dagster_home/ # Dagster metadata (local dev) + docs/ # Documentation (Mintlify, git submodule) + .github/workflows/ # CI/CD workflows + pyproject.toml # Python project config (uv, pytest, ruff, mypy, dagster) + Dockerfile # 3-stage Docker build + docker-compose.yml # Production compose (webserver + daemon) + docker-compose.override.yml # Dev overrides (db, volumes, env) +``` + +## `src/linker/` -- Dagster Module + +This is the core Python module registered with Dagster. + +``` +src/linker/ + definitions.py # Dagster Definitions (wires everything together) + assets/ + scraper/ + raw_github__extract_projects.py # Go scraper invocation + core_github__detect_languages.py # FastText language filtering + core_github__fetch_readme.py # Go fetcher (readme mode) + core_github__fetch_repo_languages.py # Go fetcher (languages mode) + core_github__fetch_repo_topics.py # Go fetcher (topics mode) + classification/ + core_match__classify_projects.py # LLM classification via OpenRouter + sync/ + core_public__sync_projects.py # Sync to public.Project + embedding/ + core_ml__embed_projects.py # Project embedding computation + core_ml__embed_users.py # User embedding computation + resources/ + cfg_resource.py # PipelineConfig + query builder + env helpers + llm_classifier_resource.py # OpenRouter LLM client + sentence_transformer_resource.py # MiniLM-L6-v2 embeddings + fasttext_resource.py # FastText language detection model + io_manager.py # PandasPostgresIOManager (DataFrame <-> DB) + jobs/ + project_enrichment_job.py # Classification + sync + project_ml + user_recommendation_job.py # User ML pipeline + run_all_job.py # Full pipeline (manual) + cleanup_dagster_job.py # Dagster history cleanup + schedules/ + project_enrichment_schedule.py # Daily at 3 AM + user_recommendation_schedule.py # Every 10 minutes + cleanup_dagster_schedule.py # Every 2 days at 11 PM + sensors/ + __init__.py # (Reserved for future sensors) + utils/ + language_detection.py # Non-Latin detection, FastText label parsing, blacklists + serialization.py # JSON serialization helpers (datetime, UUID, LLM cleanup) + __init__.py +``` + +## `src/services/go/` -- Go Binaries + +Two independent Go modules, each compiled to a standalone binary. + +``` +src/services/go/ + scraper/ + main.go # Entry point: multi-query parallel scraping + common.go # HTTP client, rate limiter, GitHub API types + main_test.go # Scraper tests + common_test.go # Rate limiter and parser tests + go.mod / go.sum + fetcher/ + main.go # Entry point: --mode readme|languages|topics + common.go # GitHubFetcher struct, rate limiter, retry logic + fetch_readme.go # README fetching with concurrent workers + fetch_languages.go # Language breakdown fetching + fetch_topics.go # Topics fetching + common_test.go # Fetcher tests + go.mod / go.sum +``` + +## `src/services/python/` + +``` +src/services/python/ + db.py # get_db_cursor() context manager for direct DB access +``` + +## `dbt/` -- Data Transformation Layer + +``` +dbt/ + dbt_project.yml # Project config, vars (weights, thresholds), model schemas + profiles.yml # Connection profiles (local port 5433, docker port 5432) + models/ + staging/ + stg_github__project.sql / .yml + stg_github__readme.sql / .yml + stg_github__languages.sql / .yml + stg_github__topics.sql / .yml + stg_github__detection.sql / .yml + stg_public__project.sql / .yml + stg_public__user.sql / .yml + intermediate/ + int_project_enriched.sql / .yml + int_project_contextualized.sql / .yml + int_project_embedding_candidate.sql / .yml + int_user_enriched.sql / .yml + marts/ + fct_github_project.sql / .yml + fct_public_user.sql / .yml + match_global_recommendation.sql / .yml + match_user_recommendation.sql / .yml + macros/ + clean_text.sql / .yml + build_project_context.sql / .yml + build_user_context.sql / .yml + safe_divide.sql / .yml + clamp.sql / .yml + deduplicate.sql / .yml + jsonb_to_list.sql / .yml + generate_schema_name.sql / .yml + tests/ # Singular dbt tests + sources/ # Source definitions (YAML) +``` + +## `prisma/` -- Schema Management + +``` prisma/ - schema.prisma, migrations/, seed/ -docs/ - ai/*.mdx -``` \ No newline at end of file + schema.prisma # Single source of truth for all 4 DB schemas + seed/ + seed.ts # Seeds categories, domains, tech stacks + data/ # JSON seed data files + migrations/ # Prisma migration history +``` + +## Key Configuration Files + +| File | Purpose | +|------|---------| +| `pyproject.toml` | Python deps (uv), pytest config, ruff/mypy settings, Dagster module registration | +| `dagster.yaml` | Dagster instance config (storage, logs) | +| `workspace.yaml` | Dagster workspace definition (points to `src.linker.definitions`) | +| `Dockerfile` | 3-stage build: Go builder, Python builder, runtime | +| `docker-compose.yml` | Production services: `webserver` + `daemon` | +| `docker-compose.override.yml` | Dev overrides: `db` service, volume mounts, `.env` loading | +| `.env.example` | Template for required environment variables | +| `dbt/dbt_project.yml` | dbt config, model materialization, schema routing, scoring variables | +| `dbt/profiles.yml` | Database connection profiles (local vs docker) | + +## Utility Scripts + +| Script | Purpose | +|--------|---------| +| `scripts/go_binary_gen.sh` | Compile both Go binaries for local development | +| `scripts/clean_dagster.sh` | Clear Dagster run history and storage | +| `scripts/sync_prisma.sh` | Sync Prisma schema to backend repo | +| `scripts/clean_docker_images.sh` | Remove dangling Docker images | diff --git a/ai/troubleshooting.mdx b/ai/troubleshooting.mdx index efb8904..ae570a9 100644 --- a/ai/troubleshooting.mdx +++ b/ai/troubleshooting.mdx @@ -1,6 +1,271 @@ +--- +title: "Troubleshooting" +description: "Common issues, cleanup procedures, and debugging commands for OST Linker" +--- + ## Common Issues -- Missing environment variables -- Restarting dagster to clear cache -- Port conflicts -- Prisma errors (migrations) -- Database connection problems + +### Missing Environment Variables + +**Symptom:** Dagster fails to load definitions or assets crash at startup. + +**Solution:** Ensure all required variables are set in `.env`: + +```bash +# Check which variables are missing +grep -v '^#' .env.example | while IFS='=' read -r key _; do + [ -z "${!key}" ] && echo "MISSING: $key" +done +``` + +Required variables: `DATABASE_URL`, `GITHUB_ACCESS_TOKEN`, `OPENROUTER_API_KEY`, `GO_SCRAPER_PATH`, `GO_FETCHER_PATH`. + +### dbt Manifest Not Found + +**Symptom:** `FileNotFoundError: Could not find manifest.json` when Dagster starts. + +**Solution:** The dbt manifest must be generated before Dagster can discover dbt assets. Run: + +```bash +cd dbt && dbt parse +``` + + +In development, `dbt_project.prepare_if_dev()` is called automatically in `definitions.py` to generate the manifest. If this fails, run `dbt parse` manually. + + +### Go Binary Not Found + +**Symptom:** `RuntimeError: Go scraper binary not found at ` or similar for the fetcher. + +**Solution:** Compile the Go binaries and update your `.env`: + +```bash +cd src/services/go/scraper && go build -o github-scraper main.go +cd ../fetcher && go build -o ost-fetcher main.go +``` + +Then set the absolute paths in `.env`: + +``` +GO_SCRAPER_PATH=/absolute/path/to/src/services/go/scraper/github-scraper +GO_FETCHER_PATH=/absolute/path/to/src/services/go/fetcher/ost-fetcher +``` + +### Database Connection Errors + +**Symptom:** `connection refused` or `FATAL: password authentication failed`. + +**Checklist:** +1. Is PostgreSQL running? `docker compose ps` +2. Is the port correct? Local dev uses **5433** (mapped from container's 5432) +3. Does the `DATABASE_URL` match your Docker Compose config? +4. Is pgvector installed? Check with `SELECT * FROM pg_extension WHERE extname = 'vector';` + +### Port Conflicts + +**Symptom:** `Address already in use` when starting Dagster or PostgreSQL. + +**Solution:** Check what is using the port and stop it: + +```bash +# Check port 3000 (Dagster) +lsof -i :3000 + +# Check port 5433 (PostgreSQL) +lsof -i :5433 +``` + +### FastText Model Missing + +**Symptom:** `FileNotFoundError: FastText model not found at: models/lid.176.ftz` + +**Solution:** Download the model file: + +```bash +mkdir -p models +wget -O models/lid.176.ftz https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz +``` + +Or set `FASTTEXT_MODEL_PATH` to the correct location if the file is stored elsewhere. + +### Dagster Cache / Stale State + +**Symptom:** Assets show outdated status, schedules do not trigger, or the UI behaves unexpectedly. + +**Solution:** Clear Dagster's local storage: + +```bash +scripts/clean_dagster.sh +``` + +Or manually: + +```bash +rm -rf dagster_home/ +mkdir dagster_home +``` + + +Clearing Dagster storage deletes all run history, asset materializations, and schedule state. Use this as a last resort. + + +### OpenRouter API Errors + +**Symptom:** `RuntimeError: OpenRouter API error for ` or `TimeoutError: OpenRouter API hard timeout after 45s`. + +**Checklist:** +1. Is `OPENROUTER_API_KEY` set and valid? +2. Is the OpenRouter service reachable? `curl https://openrouter.ai/api/v1/models` +3. Check your OpenRouter account for rate limits or credit balance +4. The LLM classifier has a 45-second hard timeout per project -- transient failures are expected and logged + +### Prisma Schema Drift + +**Symptom:** dbt models fail because tables or columns do not match expectations. + +**Solution:** Re-apply the Prisma schema: + +```bash +npx prisma db push +``` + +For a full reset (destructive): + +```bash +npx prisma db push --force-reset +npx ts-node prisma/seed/seed.ts +``` + +## Cleanup Procedures + +### dbt Artifacts + +Remove compiled SQL and cached packages: + +```bash +cd dbt && dbt clean +``` + +### Dagster History + +```bash +scripts/clean_dagster.sh +``` + +### Docker Resources + +Remove dangling images and unused volumes: + +```bash +scripts/clean_docker_images.sh + +# Or manually +docker compose down -v +docker system prune -f +``` + +### Full Reset + +To start completely fresh: + +```bash +# Stop everything +docker compose down -v + +# Clean artifacts +cd dbt && dbt clean && cd .. +rm -rf dagster_home/ + +# Rebuild +docker compose up --build -d + +# Re-initialize database +npx prisma db push +npx ts-node prisma/seed/seed.ts +``` + +## Useful Debugging Commands + +### Dagster + +```bash +# Validate definitions without starting the server +dagster definitions validate + +# Check asset materializations +dagster asset list + +# Run a specific job manually +dagster job execute -j project_enrichment_job +``` + +### dbt + +```bash +# Run a single model +dbt run --select stg_github__project + +# Test a single model +dbt test --select fct_github_project + +# Show compiled SQL for a model +dbt compile --select match_user_recommendation + +# Preview model output (first 5 rows) +dbt show --select int_project_contextualized --limit 5 +``` + +### Go Services + +```bash +# Test scraper +cd src/services/go/scraper && go test ./... + +# Test fetcher +cd src/services/go/fetcher && go test ./... + +# Run scraper manually (requires env vars) +./github-scraper + +# Run fetcher manually +./ost-fetcher --mode readme --concurrency 5 --limit 10 +``` + +### Python + +```bash +# Run all tests +pytest + +# Run only unit tests +pytest -m unit + +# Run with verbose output +pytest -v --tb=short + +# Lint and format +ruff check src/ +ruff format src/ + +# Type check +mypy src/ +``` + +### Database + +```bash +# Connect to the database +psql "postgresql://user:pass@localhost:5433/dbname" + +# Check table counts +SELECT schemaname, tablename, n_tup_ins FROM pg_stat_user_tables ORDER BY schemaname; + +# Check pgvector embeddings +SELECT count(*) FROM ml.embd_github_project; +SELECT count(*) FROM ml.embd_user; + +# Check recommendations +SELECT count(*) FROM public.match_global_recommendation; +SELECT count(*) FROM public.match_user_recommendation; +``` diff --git a/docs.json b/docs.json index 3658759..6f97c07 100644 --- a/docs.json +++ b/docs.json @@ -73,9 +73,17 @@ "pages": ["ai/overview"] }, { - "group": "Developer Guide", + "group": "Architecture", "pages": [ "ai/pipeline", + "ai/database", + "ai/dbt", + "ai/go-services" + ] + }, + { + "group": "Developer Guide", + "pages": [ "ai/structure", "ai/installation", "ai/troubleshooting" From 260827edc43ae554a4bfd5af7969be490616baa4 Mon Sep 17 00:00:00 2001 From: spideystreet Date: Sat, 7 Mar 2026 21:53:24 +0100 Subject: [PATCH 2/4] docs(ai): simplify and condense AI Engine documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove verbose diagrams, reduce redundancy, and improve readability. Total reduction: 1167 → 524 lines (-55%). Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com> --- ai/database.mdx | 148 ++++++------------------ ai/dbt.mdx | 208 ++++++++-------------------------- ai/go-services.mdx | 152 +++++-------------------- ai/installation.mdx | 134 +++++----------------- ai/overview.mdx | 111 ++++-------------- ai/pipeline.mdx | 199 ++++++++------------------------ ai/structure.mdx | 167 +++++++-------------------- ai/troubleshooting.mdx | 251 +++-------------------------------------- 8 files changed, 267 insertions(+), 1103 deletions(-) diff --git a/ai/database.mdx b/ai/database.mdx index 917348a..14bfd2b 100644 --- a/ai/database.mdx +++ b/ai/database.mdx @@ -1,134 +1,60 @@ --- title: "Database Schema" -description: "PostgreSQL schemas, pgvector setup, and table reference for OST Linker" +description: "PostgreSQL schemas and pgvector setup" --- ## Overview -OST Linker uses a single PostgreSQL database with the **pgvector** extension, organized into 4 schemas. The schema is defined in Prisma (`prisma/schema.prisma`) and serves as the single source of truth shared with the backend. +Single PostgreSQL database with **pgvector**, organized into 4 schemas. Schema is defined in Prisma (`prisma/schema.prisma`) and shared with the backend. -```mermaid -graph LR - subgraph PostgreSQL - PUB[public] - GH[github] - ML[ml] - MA[match] - end - - PUB -->|"Users, Projects, Categories"| GH - GH -->|"Raw + Staging data"| ML - ML -->|"Embeddings"| MA - MA -->|"Recommendations"| PUB -``` - -## Schemas - -### `public` -- User-Facing Data - -The primary schema used by the backend API. Contains all user-facing models. +## `public` -- User-Facing Data | Table | Description | |-------|-------------| -| `user` | User accounts with social links, GitHub/GitLab identities | -| `Project` | Published projects with metadata (title, description, URLs, trending flag) | -| `Category` | Project categories (e.g., Framework, Library, CLI Tool) | -| `Domain` | Project domains (e.g., Web Development, DevOps, Data Science) | -| `tech_stack` | Technologies and languages (typed as `TECH` or `LANGUAGE`) | -| `user_tech_stack` | Junction: user <-> tech stack preferences | -| `user_categories` | Junction: user <-> category preferences | -| `user_domain` | Junction: user <-> domain preferences | -| `project_tech_stack` | Junction: project <-> tech stacks | -| `project_category` | Junction: project <-> categories | -| `project_domain` | Junction: project <-> domains | -| `project_bookmark` | User bookmarks on projects | -| `match_global_recommendation` | Top-N global project recommendations (dbt-materialized) | +| `user` | User accounts with social links and GitHub/GitLab identities | +| `Project` | Published projects with metadata | +| `Category` | Project categories (Framework, Library, CLI Tool, etc.) | +| `Domain` | Project domains (Web Dev, DevOps, Data Science, etc.) | +| `tech_stack` | Technologies and languages | +| `user_tech_stack`, `user_categories`, `user_domain` | User preference junctions | +| `project_tech_stack`, `project_category`, `project_domain` | Project attribute junctions | +| `project_bookmark` | User bookmarks | +| `match_global_recommendation` | Top-N global recommendations (dbt-materialized) | | `match_user_recommendation` | Per-user personalized recommendations (dbt-materialized) | -### `github` -- Raw and Staged Ingestion Data - -All data scraped from GitHub lives here, from raw JSON through staged and enriched layers. +## `github` -- Ingestion Data -| Table | Managed By | Description | -|-------|-----------|-------------| -| `raw_github_project` | Go scraper | Raw JSON from GitHub Search API | -| `raw_github_readme` | Go fetcher | README content per project | -| `raw_github_languages` | Go fetcher | Language byte counts per project | -| `raw_github_topics` | Go fetcher | Topic arrays per project | -| `int_github_detection` | Python (FastText) | Language detection results and filtering | -| `stg_github__project` | dbt | Cleaned and typed project data | -| `stg_github__readme` | dbt | Staged README content | -| `stg_github__languages` | dbt | Staged language breakdowns | -| `stg_github__topics` | dbt | Staged topics | -| `stg_github__detection` | dbt | Staged detection metadata | -| `int_project_enriched` | dbt | Joined enriched project data | -| `fct_github_project` | dbt | Final fact table with stars, forks, pushed_at | - -### `ml` -- Machine Learning Artifacts +| Table | Description | +|-------|-------------| +| `raw_github_project` | Raw JSON from GitHub Search API (Go scraper) | +| `raw_github_readme`, `raw_github_languages`, `raw_github_topics` | Fetched enrichment data (Go fetcher) | +| `int_github_detection` | FastText language detection results (Python) | +| `stg_github__*` | Staged data (dbt) | +| `int_project_enriched` | Joined enriched project data (dbt) | +| `fct_github_project` | Final fact table (dbt) | -Stores embeddings and intermediate ML data. +## `ml` -- Machine Learning -| Table | Managed By | Description | -|-------|-----------|-------------| -| `embd_github_project` | Python | 384-dim project embedding vectors | -| `embd_user` | Python | 384-dim user embedding vectors | -| `stg_public__project` | dbt | Project data staged for ML processing | -| `stg_public__user` | dbt | User data staged for ML processing | -| `int_user_enriched` | dbt | User context strings for embedding | -| `int_project_contextualized` | dbt | Project context strings for embedding | -| `int_project_embedding_candidate` | dbt | Projects ready for embedding | -| `fct_public_user` | dbt | Final user fact table | +| Table | Description | +|-------|-------------| +| `embd_github_project` | 384-dim project embedding vectors (Python) | +| `embd_user` | 384-dim user embedding vectors (Python) | +| `stg_public__project`, `stg_public__user` | Staged data for ML (dbt) | +| `int_user_enriched`, `int_project_contextualized`, `int_project_embedding_candidate` | ML prep models (dbt) | +| `fct_public_user` | User fact table (dbt) | -### `match` -- Classification Results +## `match` -- Classification -| Table | Managed By | Description | -|-------|-----------|-------------| -| `project_classification` | Python (LLM) | Category and domain assignments with confidence scores | +| Table | Description | +|-------|-------------| +| `project_classification` | LLM-assigned category and domain with confidence scores | ## pgvector -The database uses the [pgvector](https://github.com/pgvector/pgvector) extension for vector similarity search. - -| Property | Value | -|----------|-------| -| Extension | `vector` (enabled via Prisma) | -| Vector dimension | 384 (MiniLM-L6-v2 output) | -| Distance function | Cosine distance (`<=>` operator) | -| Similarity formula | `1 - (vector_a <=> vector_b)` | -| Similarity threshold | 0.25 (configurable via dbt var) | - - -Cosine similarity is computed directly in SQL within dbt models. The `match_user_recommendation` model uses `1 - (uv.vector <=> pv.vector)` to score user-project pairs. - - -The vector columns are defined in Prisma as `Unsupported("vector")` since Prisma does not natively support the pgvector type. The actual column type in PostgreSQL is `vector(384)`. - -## Schema Routing in dbt - -dbt models are routed to specific PostgreSQL schemas using the `generate_schema_name` macro. The target schema is set per model in `dbt_project.yml` via the `+schema` property: - -```yaml -# Example from dbt_project.yml -models: - ost_linker: - staging: - stg_github__project: - +schema: github # Writes to github.stg_github__project - stg_public__user: - +schema: ml # Writes to ml.stg_public__user - marts: - match_global_recommendation: - +schema: public # Writes to public.match_global_recommendation -``` - -The custom `generate_schema_name` macro overrides dbt's default behavior to use the configured schema name directly, without prepending the target schema. +The `vector` extension enables cosine similarity search on 384-dimensional embeddings (MiniLM-L6-v2). Similarity is computed as `1 - cosine_distance` directly in dbt SQL models. Minimum similarity threshold is 0.25 (configurable via dbt var). -## Prisma as Schema Manager +Vectors are defined in Prisma as `Unsupported("vector")` since Prisma does not natively support the pgvector type. -Prisma manages the database schema for both the AI engine and the backend: +## Schema Routing -- **Schema definition:** `prisma/schema.prisma` declares all models across all 4 schemas -- **Extensions:** pgvector and uuid-ossp are enabled via `extensions = [uuidOssp, vector]` -- **Multi-schema support:** Uses `@@schema("public")`, `@@schema("github")`, `@@schema("ml")`, `@@schema("match")` annotations -- **Seed data:** `prisma/seed/seed.ts` populates Categories, Domains, and TechStacks -- **Cross-repo sync:** The Prisma schema is automatically synced to the backend repo via the `sync-prisma-backend.yml` GitHub Actions workflow +dbt models are routed to PostgreSQL schemas via the `generate_schema_name` macro and `+schema` in `dbt_project.yml`. The macro uses the configured schema name directly without prefix. diff --git a/ai/dbt.mdx b/ai/dbt.mdx index 315d12d..9a74843 100644 --- a/ai/dbt.mdx +++ b/ai/dbt.mdx @@ -1,180 +1,68 @@ --- title: "dbt Layer" -description: "Data transformation models, macros, and recommendation scoring in dbt" +description: "Data transformation models, macros, and recommendation scoring" --- -## Overview - -The dbt project (`dbt/`) handles all SQL-based data transformations, from cleaning raw ingested data to computing recommendation scores. Models are organized into 3 layers: staging, intermediate, and marts. - -All models are materialized as tables and routed to specific PostgreSQL schemas via the `generate_schema_name` macro. - -## Model Organization - -```mermaid -graph TD - subgraph Staging - SG1[stg_github__project] - SG2[stg_github__readme] - SG3[stg_github__languages] - SG4[stg_github__topics] - SG5[stg_github__detection] - SP1[stg_public__project] - SP2[stg_public__user] - end - - subgraph Intermediate - IE[int_project_enriched] - IC[int_project_contextualized] - IEC[int_project_embedding_candidate] - IU[int_user_enriched] - end - - subgraph Marts - FG[fct_github_project] - FU[fct_public_user] - MG[match_global_recommendation] - MU[match_user_recommendation] - end - - SG1 & SG2 & SG3 & SG4 & SG5 --> IE - IE --> FG - SP1 --> IC --> IEC - SP2 --> IU --> FU - FG --> MG - FG & IEC --> MU -``` - -## Models Reference - -### Staging Models - -Staging models clean and type-cast raw data. They follow the `stg___` naming convention. - -| Model | Schema | Dagster Group | Description | -|-------|--------|---------------|-------------| -| `stg_github__project` | `github` | ingestion | Flattens raw JSON from `raw_github_project` into typed columns (name, description, stars, URL, etc.) | -| `stg_github__readme` | `github` | ingestion | Stages README content from `raw_github_readme` | -| `stg_github__languages` | `github` | ingestion | Stages language breakdowns from `raw_github_languages` | -| `stg_github__topics` | `github` | ingestion | Stages topic arrays from `raw_github_topics` | -| `stg_github__detection` | `github` | ingestion | Stages FastText detection results from `int_github_detection` | -| `stg_public__project` | `ml` | project_ml | Stages public project data for ML processing | -| `stg_public__user` | `ml` | user_ml | Stages user data for ML processing | - -### Intermediate Models - -Intermediate models join, enrich, and prepare data for consumption. - -| Model | Schema | Dagster Group | Description | -|-------|--------|---------------|-------------| -| `int_project_enriched` | `github` | ingestion | Joins project with readme, languages, topics, and detection data | -| `int_project_contextualized` | `ml` | project_ml | Builds rich context strings for embedding using `build_project_context` macro | -| `int_project_embedding_candidate` | `ml` | project_ml | Filters projects with valid context strings, ready for embedding | -| `int_user_enriched` | `ml` | user_ml | Builds user context strings for embedding using `build_user_context` macro | - -### Mart Models - -Mart models are the final consumption layer, used directly by the backend and ML assets. - -| Model | Schema | Dagster Group | Description | -|-------|--------|---------------|-------------| -| `fct_github_project` | `github` | ingestion | Fact table with stars, forks, pushed_at, and all enriched metadata | -| `fct_public_user` | `ml` | user_ml | User fact table with aggregated preferences | -| `match_global_recommendation` | `public` | project_ml | Top-N global recommendations (trending/published, ordered by recency and stars) | -| `match_user_recommendation` | `public` | user_ml | Per-user personalized recommendations with hybrid scoring | +## Models + +All models are materialized as tables and routed to PostgreSQL schemas via `generate_schema_name`. + +| Model | Layer | Schema | Description | +|-------|-------|--------|-------------| +| `stg_github__project` | staging | `github` | Flattens raw JSON into typed columns | +| `stg_github__readme` | staging | `github` | Stages README content | +| `stg_github__languages` | staging | `github` | Stages language breakdowns | +| `stg_github__topics` | staging | `github` | Stages topic arrays | +| `stg_github__detection` | staging | `github` | Stages FastText detection results | +| `stg_public__project` | staging | `ml` | Stages projects for ML | +| `stg_public__user` | staging | `ml` | Stages users for ML | +| `int_project_enriched` | intermediate | `github` | Joins project + readme + languages + topics + detection | +| `int_project_contextualized` | intermediate | `ml` | Builds context strings for embedding | +| `int_project_embedding_candidate` | intermediate | `ml` | Filters projects ready for embedding | +| `int_user_enriched` | intermediate | `ml` | Builds user context strings | +| `fct_github_project` | mart | `github` | Final project fact table | +| `fct_public_user` | mart | `ml` | Final user fact table | +| `match_global_recommendation` | mart | `public` | Top-N global recommendations | +| `match_user_recommendation` | mart | `public` | Per-user personalized recommendations | ## Macros -### Data Cleaning - -| Macro | Description | -|-------|-------------| -| `clean_text(column)` | Strips HTML, normalizes whitespace, removes special characters | -| `deduplicate(relation, partition_by, order_by)` | Window-function dedup (keeps first row per partition) | -| `jsonb_to_list(column)` | Converts a JSONB array to a comma-separated string | - -### Context Building - -| Macro | Description | -|-------|-------------| -| `build_project_context(...)` | Concatenates project title, description, topics, languages, and README into a single string for embedding | -| `build_user_context(...)` | Concatenates user bio, job title, tech stacks, categories, and domains into a single string for embedding | - -### Scoring Helpers - -| Macro | Description | -|-------|-------------| -| `safe_divide(numerator, denominator)` | Returns NULL instead of dividing by zero | -| `clamp(expression)` | Clamps a numeric expression to the [0, 1] range | - -### Schema Management - -| Macro | Description | -|-------|-------------| -| `generate_schema_name(custom_schema_name, node)` | Overrides dbt default to use the `+schema` value directly without prefix | +| Macro | Purpose | +|-------|---------| +| `clean_text(column)` | Strip HTML, normalize whitespace | +| `deduplicate(relation, partition_by, order_by)` | Window-function dedup | +| `jsonb_to_list(column)` | JSONB array to comma-separated string | +| `build_project_context(...)` | Concatenate project fields into embedding input | +| `build_user_context(...)` | Concatenate user fields into embedding input | +| `safe_divide(num, denom)` | Returns NULL on division by zero | +| `clamp(expr)` | Clamps value to 0-1 range | +| `generate_schema_name(...)` | Uses `+schema` directly, no prefix | ## Recommendation Scoring -### Global Recommendations - -The `match_global_recommendation` model selects the top-N projects (default: 20) that are trending or published, ordered by last sync time and star count. - -### User Recommendations (Hybrid Scoring) - -The `match_user_recommendation` model computes personalized scores by blending 4 signals: - -| Signal | Weight | Description | -|--------|--------|-------------| -| **Similarity** | 40% | Cosine similarity between user and project embeddings (`1 - (user_vec <=> proj_vec)`) | -| **Preference** | 35% | Weighted overlap between user and project attributes (tech stacks, categories, domains) | -| **Freshness** | 15% | Linear decay based on last push date (configurable decay window: 90 days) | -| **Popularity** | 10% | Log-normalized star count, scaled to [0, 1] | +**Global:** Top-N projects (default 20) that are trending or published, ordered by recency and stars. -The final formula: +**User (hybrid scoring):** ``` -final_score = 0.40 * similarity + 0.35 * preference + 0.15 * freshness + 0.10 * popularity +final_score = 0.40 * similarity + + 0.35 * preference + + 0.15 * freshness + + 0.10 * popularity ``` -#### Preference Score Breakdown +- **Similarity** -- cosine similarity between user and project embeddings (pgvector) +- **Preference** -- weighted overlap: tech stacks (30%), categories (45%), domains (25%) +- **Freshness** -- linear decay over 90 days from last push +- **Popularity** -- log-normalized star count, clamped to 0-1 -The preference signal itself is a weighted combination of 3 overlap dimensions: - -| Dimension | Sub-Weight | Calculation | -|-----------|-----------|-------------| -| Tech stacks | 30% | `shared_tech_stacks / user_total_tech_stacks` | -| Categories | 45% | `shared_categories / user_total_categories` | -| Domains | 25% | `shared_domains / user_total_domains` | - - -If a user has no items in a dimension (e.g., no tech stacks selected), that dimension is excluded and its weight is redistributed proportionally among active dimensions. This prevents penalizing users with incomplete profiles. - - -#### Filtering and Thresholds - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `similarity_threshold` | 0.25 | Minimum cosine similarity to be considered | -| `reco_top_n` | 30 | Maximum recommendations per user | -| `freshness_decay_days` | 90 | Days until freshness score reaches 0 | -| `global_reco_top_n` | 20 | Number of global recommendations | - -All scoring parameters are defined as dbt vars in `dbt_project.yml` and can be overridden at runtime. - -## Data Contracts - -Mart models use dbt data contracts with `contract: {enforced: true}`. Each column specifies a `data_type` and optional `constraints`, ensuring schema stability for downstream consumers (backend API, ML assets). +Minimum similarity threshold: 0.25. Max recommendations per user: 30. All params are dbt vars in `dbt_project.yml`. ## Profiles -| Profile | Host | Port | Usage | -|---------|------|------|-------| -| `local` (default) | `localhost` | 5433 | Local development (DB exposed via Docker) | -| `docker` | `db` | 5432 | Inside Docker Compose network | +| Profile | Host | Port | +|---------|------|------| +| `local` (default) | `localhost` | 5433 | +| `docker` | `db` | 5432 | -Switch profiles by setting the `DBT_TARGET` environment variable: - -```bash -export DBT_TARGET=docker -dbt build -``` +Switch with `export DBT_TARGET=docker`. diff --git a/ai/go-services.mdx b/ai/go-services.mdx index f3ba9eb..76c0a8d 100644 --- a/ai/go-services.mdx +++ b/ai/go-services.mdx @@ -1,153 +1,55 @@ --- title: "Go Services" -description: "GitHub scraper and fetcher binaries used for data ingestion" +description: "GitHub scraper and fetcher binaries for data ingestion" --- -OST Linker uses two independent Go binaries for high-performance data ingestion from the GitHub API. Both are invoked by Dagster assets as subprocesses with a 600-second timeout. +## Overview + +Two independent Go binaries handle GitHub data ingestion. Both are called by Dagster assets via `subprocess.run()` with a 600-second timeout. ## Scraper **Location:** `src/services/go/scraper/` -The scraper searches the GitHub Search API for open-source projects matching specific criteria and upserts the raw JSON data into `github.raw_github_project`. - -### How It Works - -1. Reads search queries from the `GITHUB_SCRAPING_QUERIES` environment variable (JSON array) -2. Launches one goroutine per query for parallel scraping -3. Each query paginates through results (100 per page, up to 1,000 per query) -4. Results are batch-upserted into PostgreSQL using `pgx.Batch` -5. Outputs a JSON summary to stdout (parsed by the Dagster asset) - -### Search Queries - -Queries are built dynamically by `PipelineConfig` in Python and cover 3 star ranges: - -| Range | Stars | -|-------|-------| -| Range 1 | 300 -- 1,000 | -| Range 2 | 1,000 -- 3,000 | -| Range 3 | 3,000 -- 5,000 | - -Each query includes these filters: -- `good-first-issues:>1` -- at least one good first issue -- `help-wanted-issues:>0` -- at least one help wanted issue -- `topics:>2` -- at least 2 topics defined -- `fork:false` -- exclude forks -- `pushed:>=<7 days ago>` -- active in the last week -- `is:public archived:false` -- public and not archived -- Excludes terms: `awesome`, `roadmap`, `cheatsheet`, `interview` - -### Rate Limiting - -The scraper uses a shared `searchRateLimiter` across all goroutines that tracks GitHub's `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers. - -| Parameter | Value | -|-----------|-------| -| Initial budget | 30 requests (GitHub Search API limit per minute for authenticated users) | -| Rate limit handling | Sleeps until reset time + 1 second | -| 403 response handling | Reads `Retry-After` header, falls back to 60 seconds | -| Retry policy | 3 attempts with exponential backoff (2s, 4s) | -| HTTP timeout | 30 seconds per request | -| Global timeout | 8 minutes for all queries | - -### Environment Variables - -| Variable | Required | Description | -|----------|----------|-------------| -| `GITHUB_SCRAPING_QUERIES` | Yes (or `GITHUB_SCRAPING_QUERY`) | JSON array of search queries | -| `GITHUB_ACCESS_TOKEN` | Recommended | GitHub API token (unauthenticated requests have lower limits) | -| `DATABASE_URL` | Yes | PostgreSQL connection string | -| `GITHUB_API_URL` | No | Override API endpoint (default: `https://api.github.com/search/repositories`) | - -### Output - -The scraper writes a JSON summary to stdout: - -```json -{ - "queries": [ - {"query": "stars:300..1000 ...", "collected_count": 450, "upserted_count": 448, "failed_upserts": 2} - ], - "total_collected": 1200, - "total_upserted": 1195, - "total_failed": 5, - "status": "partial", - "duration_seconds": 45.2 -} -``` +Searches the GitHub Search API for open-source projects and upserts results into `github.raw_github_project`. + +- Reads queries from `GITHUB_SCRAPING_QUERIES` env var (JSON array) +- One goroutine per query, paginating up to 1,000 results each +- Batch upserts via `pgx.Batch` +- Outputs JSON summary to stdout (parsed by Dagster) + +**Search filters:** stars 300-5000, good first issues, recent pushes, public, not archived, not forks. Excludes awesome lists, roadmaps, cheatsheets, interview repos. + +**Rate limiting:** 30 req/min budget, tracks `X-RateLimit-*` headers, 3 retries with exponential backoff. ## Fetcher **Location:** `src/services/go/fetcher/` -The fetcher enriches scraped projects by fetching additional data from the GitHub REST API. It operates in 3 modes, each targeting a different API endpoint and database table. +Enriches scraped projects by fetching additional data. Operates in 3 modes: -### Modes - -| Mode | API Endpoint | Output Table | Description | -|------|-------------|-------------|-------------| -| `readme` | `GET /repos/{owner}/{repo}/readme` | `github.raw_github_readme` | Fetches raw README content | -| `languages` | `GET /repos/{owner}/{repo}/languages` | `github.raw_github_languages` | Fetches language byte counts | -| `topics` | `GET /repos/{owner}/{repo}/topics` | `github.raw_github_topics` | Fetches repository topics | +| Mode | Endpoint | Output Table | +|------|----------|-------------| +| `readme` | `GET /repos/owner/repo/readme` | `github.raw_github_readme` | +| `languages` | `GET /repos/owner/repo/languages` | `github.raw_github_languages` | +| `topics` | `GET /repos/owner/repo/topics` | `github.raw_github_topics` | ### CLI Flags -```bash -ost-fetcher --mode readme --concurrency 20 --limit 100 -``` - | Flag | Default | Description | |------|---------|-------------| -| `--mode` | (required) | One of: `readme`, `languages`, `topics` | -| `--concurrency` | 10 | Number of concurrent worker goroutines | -| `--limit` | 0 (no limit) | Maximum number of projects to process | - -### Incremental Fetching - -The fetcher only processes projects that do not already have data in the target table. It queries `github.int_github_detection` (the output of FastText language filtering) and LEFT JOINs against the target table to find new projects. +| `--mode` | (required) | `readme`, `languages`, or `topics` | +| `--concurrency` | 10 | Number of concurrent workers | +| `--limit` | 0 (unlimited) | Max projects to process | -### Rate Limiting and Retries +**Incremental:** only fetches projects missing data in the target table. -| Parameter | Value | -|-----------|-------| -| Initial budget | 5,000 requests (GitHub REST API limit per hour for authenticated users) | -| Rate limit tracking | Reads `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers | -| Retry policy | Up to 3 attempts with exponential backoff (1s, 2s, 3s) | -| 403 handling | Reads `Retry-After` header, falls back to 60-second sleep | -| 404/422 handling | No retry, returns empty result | -| HTTP timeout | 30 seconds per request | -| Global timeout | 30 minutes | -| Response body limit | 10 MB per response | - -### README Truncation - -README content is truncated to 50,000 bytes (preserving valid UTF-8 boundaries) before insertion into the database. - - -The `truncateUTF8` function ensures multi-byte characters are never split, preventing invalid UTF-8 sequences in the database. - - -### Batch Upserts - -Results are collected via channels and flushed in batches of 100 using `pgx.Batch` for efficient database writes. Each batch uses `ON CONFLICT ... DO UPDATE` to handle re-runs gracefully. +**Rate limiting:** 5,000 req/hour budget, 3 retries, 30-min global timeout. READMEs truncated to 50KB. ## Dagster Integration -Both Go binaries are invoked by Dagster Python assets using `subprocess.run()`: - -```python -result = subprocess.run( - [binary_path], # or [binary_path, "--mode", "readme", "--concurrency", "20"] - capture_output=True, - text=True, - env=env, # DATABASE_URL, GITHUB_ACCESS_TOKEN, queries - timeout=600, # 10 minutes -) -``` - -The environment is constructed by `build_scraper_env()` and `build_fetcher_env()` helpers in `cfg_resource.py`, which read values from the `PipelineConfig` resource. +Both binaries are invoked via `subprocess.run()`. Environment is built by `build_scraper_env()` / `build_fetcher_env()` helpers in `cfg_resource.py`. -If the Go binary is not found at the configured path, the asset will raise a `RuntimeError` with a clear message. Make sure `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` point to compiled binaries. +If the Go binary is not found at the configured path, the asset raises a `RuntimeError`. Make sure `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` point to compiled binaries. diff --git a/ai/installation.mdx b/ai/installation.mdx index bf2921c..f3c40c0 100644 --- a/ai/installation.mdx +++ b/ai/installation.mdx @@ -1,174 +1,92 @@ --- title: "Installation & Setup" -description: "Prerequisites, environment configuration, and development setup for OST Linker" +description: "Get OST Linker running locally" --- -## Prerequisites - -| Tool | Version | Purpose | -|------|---------|---------| -| Docker | Latest | Container runtime for all services | -| Go | 1.24+ | Compile scraper and fetcher binaries | -| Python | 3.11 | Dagster, dbt, ML models | -| Node.js | 18+ | Prisma CLI for schema management and seeding | -| uv | Latest | Python dependency management (replaces pip/poetry) | - -## Quick Start with Docker - -The fastest way to get OST Linker running: +## Quick Start (Docker) ```bash -# 1. Clone and configure git clone https://github.com/opensource-together/ost-linker.git cd ost-linker cp .env.example .env -# Edit .env with your values (see Environment Variables below) - -# 2. Launch all services +# Edit .env with your values (see below) docker compose up --build -d ``` -This starts the Dagster webserver (port 3000), daemon, and a PostgreSQL database with pgvector (port 5433). - - -The Dagster UI is available at [http://localhost:3000](http://localhost:3000) once the containers are running. - +Dagster UI will be at `http://localhost:3000`. ## Environment Variables -All configuration is driven by environment variables. Copy `.env.example` to `.env` and fill in the required values. - ### Required | Variable | Description | |----------|-------------| -| `DATABASE_URL` | PostgreSQL connection string (e.g., `postgresql://user:pass@localhost:5433/dbname`) | -| `GITHUB_ACCESS_TOKEN` | GitHub fine-grained personal access token for API access | -| `OPENROUTER_API_KEY` | API key for OpenRouter (LLM classification) | -| `GO_SCRAPER_PATH` | Absolute path to the compiled Go scraper binary | -| `GO_FETCHER_PATH` | Absolute path to the compiled Go fetcher binary | +| `DATABASE_URL` | PostgreSQL connection string | +| `GITHUB_ACCESS_TOKEN` | GitHub fine-grained personal access token | +| `OPENROUTER_API_KEY` | API key for LLM classification | +| `GO_SCRAPER_PATH` | Path to compiled Go scraper binary | +| `GO_FETCHER_PATH` | Path to compiled Go fetcher binary | ### Optional -| Variable | Default | Description | -|----------|---------|-------------| -| `FASTTEXT_MODEL_PATH` | `models/lid.176.ftz` | Path to FastText language detection model | -| `DBT_TARGET` | `local` | dbt profile target (`local` for port 5433, `docker` for port 5432) | -| `DBT_PROJECT_DIR` | `/dbt` | dbt project directory (set to `/app/dbt` in Docker) | -| `DAGSTER_HOME` | `./dagster_home` | Dagster metadata and run storage directory | +| Variable | Default | +|----------|---------| +| `FASTTEXT_MODEL_PATH` | `models/lid.176.ftz` | +| `DBT_TARGET` | `local` (port 5433) | +| `DBT_PROJECT_DIR` | `./dbt` | +| `DAGSTER_HOME` | `./dagster_home` | -Never commit `.env` or hardcode secrets in code. The `.env` file is gitignored by default. +Never commit `.env` or hardcode secrets. The file is gitignored. ## Local Development Setup -For development outside of Docker, follow these steps in order. - -### 1. Install Python Dependencies +### 1. Python dependencies ```bash uv sync ``` -### 2. Compile Go Binaries - -```bash -cd src/services/go/scraper && go build -o github-scraper main.go -cd ../fetcher && go build -o ost-fetcher main.go -``` - -Or use the convenience script: +### 2. Compile Go binaries ```bash scripts/go_binary_gen.sh ``` -Then set `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` in your `.env` to the absolute paths of the compiled binaries. - -### 3. Download FastText Model +Then set `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` in `.env` to the absolute paths. -The language detection model (`lid.176.ftz`) is not included in the repo due to its size. Download it: +### 3. Download FastText model ```bash mkdir -p models wget -O models/lid.176.ftz https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz ``` -### 4. Start the Database - -If not using Docker Compose for the full stack, start just the database: +### 4. Start the database ```bash docker compose up db -d ``` -This starts PostgreSQL with pgvector on port 5433. - -### 5. Initialize the Database - -Apply the Prisma schema and seed reference data: +### 5. Initialize the database ```bash npx prisma db push npx ts-node prisma/seed/seed.ts ``` - -`prisma db push` applies the schema without creating migration files. For production, use `prisma migrate deploy`. - - -### 6. Install dbt Dependencies - -```bash -cd dbt && dbt deps -``` - -### 7. Run Dagster Locally +### 6. Install dbt deps and run Dagster ```bash +cd dbt && dbt deps && cd .. dagster dev -h 0.0.0.0 -p 3000 ``` -The Dagster UI will be available at `http://localhost:3000`. - -## Docker Build Details - -The Dockerfile uses a 3-stage build: - -| Stage | Base Image | Purpose | -|-------|-----------|---------| -| Go Builder | `golang:1.24-alpine` | Compiles both Go binaries to `/app/bin/` | -| Python Builder | `python:3.11-slim` | Exports dependencies via `uv export` to `requirements.txt` | -| Runtime | `python:3.11-slim` | Installs deps, copies Go binaries to `/usr/local/bin/`, runs Dagster | - -### Compose Services - -| Service | Port | Description | -|---------|------|-------------| -| `webserver` | 3000 | Dagster webserver (UI + GraphQL API) | -| `daemon` | -- | Dagster daemon (schedules, sensors, run monitoring) | -| `db` (dev only) | 5433 | PostgreSQL with pgvector (`ankane/pgvector:v0.4.1`) | - - -The `db` service is defined in `docker-compose.override.yml` and only runs in local development. In production, use an external managed PostgreSQL instance. - - -## Verifying the Setup - -After completing the setup, verify everything is working: +## Verify Setup ```bash -# Check Python deps and Dagster module dagster definitions validate - -# Run dbt models -cd dbt && dbt build - -# Run tests +cd dbt && dbt build && cd .. pytest - -# Check Go binaries -src/services/go/scraper/github-scraper --help -src/services/go/fetcher/ost-fetcher --help ``` diff --git a/ai/overview.mdx b/ai/overview.mdx index 01b1e4a..d713fca 100644 --- a/ai/overview.mdx +++ b/ai/overview.mdx @@ -5,111 +5,50 @@ description: "High-level architecture of the OST Linker AI recommendation engine ## What is OST Linker? -OST Linker is the AI-powered recommendation engine behind [OpenSourceTogether](https://opensource-together.com/). It continuously scrapes GitHub for open-source projects, classifies them using LLMs, computes semantic embeddings, and surfaces personalized recommendations to users via cosine similarity with pgvector. +OST Linker is the AI-powered recommendation engine for [OpenSourceTogether](https://opensource-together.com/). It scrapes GitHub for open-source projects, classifies them via LLM, computes embeddings, and surfaces personalized recommendations using pgvector cosine similarity. - -OST Linker runs as a fully automated pipeline orchestrated by **Dagster**. Once deployed, it requires no manual intervention -- projects are discovered, enriched, classified, and recommended on schedule. - +The pipeline is fully automated and orchestrated by **Dagster**. -## Data Flow - -The pipeline follows a linear progression through 5 stages: - -```mermaid -graph LR - A[Ingestion] --> B[Classification] - B --> C[Sync] - C --> D[Project ML] - D --> E[User ML] - - style A fill:#2d6a4f,color:#fff - style B fill:#40916c,color:#fff - style C fill:#52b788,color:#fff - style D fill:#74c69d,color:#000 - style E fill:#95d5b2,color:#000 -``` - -### Detailed Pipeline - -```mermaid -graph TD - subgraph Ingestion - GH[GitHub Search API] -->|Go scraper| RAW[raw_github_project] - RAW -->|dbt staging| STG[stg_github__project] - STG -->|FastText| DET[int_github_detection] - DET -->|Go fetcher| README[raw_github_readme] - DET -->|Go fetcher| LANG[raw_github_languages] - DET -->|Go fetcher| TOPICS[raw_github_topics] - README & LANG & TOPICS -->|dbt| FCT[fct_github_project] - end - - subgraph Classification - FCT --> LLM[LLM Classifier] - LLM -->|Category + Domain| CLASS[project_classification] - end - - subgraph Sync - CLASS --> SYNC[public.Project] - end - - subgraph Project ML - SYNC -->|dbt| CTX[int_project_contextualized] - CTX -->|dbt| CAND[int_project_embedding_candidate] - CAND -->|SentenceTransformer| PEMB[embd_github_project] - PEMB -->|dbt| GREC[match_global_recommendation] - end +## Tech Stack - subgraph User ML - USR[public.User] -->|dbt| UENR[int_user_enriched] - UENR -->|SentenceTransformer| UEMB[embd_user] - UEMB -->|dbt| UREC[match_user_recommendation] - end -``` +- **Orchestration:** Dagster (assets, jobs, schedules, sensors) +- **Data transformation:** dbt (staging, intermediate, marts) +- **Ingestion:** Go 1.24 (scraper + fetcher binaries) +- **ML / Classification:** Python 3.11, SentenceTransformer (MiniLM-L6-v2), FastText, OpenRouter (Mistral) +- **Database:** PostgreSQL + pgvector (384-dim vectors, cosine similarity) +- **Schema management:** Prisma (4 schemas: `public`, `github`, `ml`, `match`) +- **Containerization:** Docker (3-stage build) +- **CI/CD:** GitHub Actions ## Pipeline Stages -| Stage | Purpose | Key Technology | -|-------|---------|----------------| -| **Ingestion** | Scrape GitHub, detect languages, fetch READMEs/topics/languages | Go binaries, FastText, dbt | -| **Classification** | Assign Category and Domain to each project via LLM | Mistral Small 3.2 via OpenRouter | -| **Sync** | Write enriched project data to the public-facing `Project` table | Python, PostgreSQL | -| **Project ML** | Build project context strings, compute 384-dim embeddings, generate global recommendations | dbt, SentenceTransformer (MiniLM-L6-v2) | -| **User ML** | Build user context strings, compute user embeddings, generate personalized recommendations | dbt, SentenceTransformer, pgvector cosine similarity | - -## Tech Stack - -| Layer | Technology | -|-------|-----------| -| **Orchestration** | Dagster (assets, jobs, schedules, sensors) | -| **Data transformation** | dbt (staging, intermediate, marts) | -| **Ingestion services** | Go 1.24 (scraper + fetcher binaries) | -| **ML / Classification** | Python 3.11, SentenceTransformer, FastText, OpenRouter (Mistral) | -| **Database** | PostgreSQL + pgvector (384-dim vectors, cosine similarity) | -| **Schema management** | Prisma (4 schemas: `public`, `github`, `ml`, `match`) | -| **Containerization** | Docker (3-stage build: Go builder, Python builder, runtime) | -| **CI/CD** | GitHub Actions (quality checks, Docker publish, submodule sync) | +1. **Ingestion** -- Go binaries scrape GitHub, FastText filters languages, dbt stages the data +2. **Classification** -- LLM (Mistral Small 3.2) assigns Category and Domain to each project +3. **Sync** -- Enriched data is written to the public-facing `Project` table +4. **Project ML** -- dbt builds context strings, SentenceTransformer computes embeddings, global recommendations are generated +5. **User ML** -- Same flow for users, producing personalized recommendations via hybrid scoring ## Key Design Decisions -1. **Go for ingestion** -- High-performance concurrent HTTP requests with goroutines and rate limiting, compiled to static binaries invoked by Dagster via `subprocess.run()` -2. **dbt for transformations** -- SQL-first data modeling with contracts, tests, and documentation baked in -3. **Hybrid recommendation scoring** -- Combines semantic similarity (40%), user preference overlap (35%), project freshness (15%), and popularity (10%) -4. **pgvector for similarity search** -- Native PostgreSQL extension for cosine similarity on 384-dimensional embeddings, no external vector DB required -5. **Prisma as schema source of truth** -- Single schema definition shared between the backend (Node.js) and the AI engine +- **Go for ingestion** -- concurrent HTTP with goroutines, compiled to static binaries called by Dagster +- **dbt for transformations** -- SQL-first modeling with contracts and tests +- **Hybrid scoring** -- similarity (40%) + preference overlap (35%) + freshness (15%) + popularity (10%) +- **pgvector** -- native PostgreSQL extension, no external vector DB needed +- **Prisma** -- single schema definition shared with the backend ## Getting Started - Understand every asset, job, and schedule in detail + Assets, jobs, schedules, and resources - Set up the development environment from scratch + Set up the development environment - Explore the 4 PostgreSQL schemas and pgvector setup + PostgreSQL schemas and pgvector setup - Navigate the codebase directory layout + Codebase directory layout diff --git a/ai/pipeline.mdx b/ai/pipeline.mdx index a32a183..e658a33 100644 --- a/ai/pipeline.mdx +++ b/ai/pipeline.mdx @@ -1,164 +1,55 @@ --- title: "Pipeline Deep-Dive" -description: "Complete reference for Dagster assets, jobs, schedules, and resources" +description: "Dagster assets, jobs, schedules, and resources reference" --- -## Asset Groups - -OST Linker organizes its Dagster assets into 5 groups that execute in sequence. Each group contains both Python assets and dbt models. - -### Ingestion - -The ingestion group collects raw data from GitHub and transforms it through staging and enrichment layers. - -| Asset | Type | Description | -|-------|------|-------------| -| `raw_github__extract_projects` | Python (Go subprocess) | Runs the Go scraper to search GitHub and upsert raw project data | -| `stg_github__project` | dbt | Cleans and flattens raw JSON into typed columns | -| `core_github__detect_languages` | Python | Filters non-English repos using FastText language detection | -| `core_github__fetch_readme` | Python (Go subprocess) | Fetches README content for detected projects | -| `core_github__fetch_repo_languages` | Python (Go subprocess) | Fetches repository language breakdowns | -| `core_github__fetch_repo_topics` | Python (Go subprocess) | Fetches repository topics | -| `stg_github__readme`, `stg_github__languages`, `stg_github__topics`, `stg_github__detection` | dbt | Staging models for fetched data | -| `int_project_enriched` | dbt | Joins all staging sources into a single enriched project view | -| `fct_github_project` | dbt | Final fact table with stars, forks, pushed_at, and all metadata | - -### Classification - -| Asset | Type | Description | -|-------|------|-------------| -| `core_match__classify_projects` | Python | Sends project context to Mistral Small 3.2 (via OpenRouter) to assign a Category and Domain | - -The LLM receives a truncated context (max 8,000 chars) containing the project title, description, topics, and README. It returns a JSON object with `category` and `domain` fields matched against the valid labels from the database. - -### Sync - -| Asset | Type | Description | -|-------|------|-------------| -| `core_public__sync_projects` | Python | Syncs enriched and classified project data into the `public.Project` table | - -### Project ML - -| Asset | Type | Description | -|-------|------|-------------| -| `stg_public__project` | dbt | Stages public project data for ML processing | -| `int_project_contextualized` | dbt | Builds rich context strings from project metadata using the `build_project_context` macro | -| `int_project_embedding_candidate` | dbt | Selects projects ready for embedding (non-null context) | -| `core_ml__embed_projects` | Python | Computes 384-dim embeddings with SentenceTransformer and upserts to `ml.embd_github_project` | -| `match_global_recommendation` | dbt | Top-N global recommendations ranked by recency and stars | - -### User ML - -| Asset | Type | Description | -|-------|------|-------------| -| `stg_public__user` | dbt | Stages user data for ML processing | -| `int_user_enriched` | dbt | Builds user context strings using the `build_user_context` macro | -| `fct_public_user` | dbt | Final user fact table | -| `core_ml__embed_users` | Python | Computes 384-dim user embeddings and upserts to `ml.embd_user` | -| `match_user_recommendation` | dbt | Personalized recommendations using hybrid scoring | - -## Data Flow - -The full asset dependency graph follows this order: - -```mermaid -graph TD - A[raw_github_project] --> B[stg_github__project] - B --> C[core_github__detect_languages] - C --> D1[core_github__fetch_readme] - C --> D2[core_github__fetch_repo_languages] - C --> D3[core_github__fetch_repo_topics] - D1 --> E1[stg_github__readme] - D2 --> E2[stg_github__languages] - D3 --> E3[stg_github__topics] - B --> F[int_project_enriched] - E1 & E2 & E3 --> F - F --> G[fct_github_project] - G --> H[core_match__classify_projects] - H --> I[core_public__sync_projects] - I --> J[stg_public__project] - J --> K[int_project_contextualized] - K --> L[int_project_embedding_candidate] - L --> M[core_ml__embed_projects] - M --> N[match_global_recommendation] -``` - -## Jobs - -| Job | Groups | Description | Retry Policy | -|-----|--------|-------------|-------------| -| `project_enrichment_job` | ingestion, classification, sync, project_ml | Full project pipeline from scraping to recommendations | 2 retries, exponential backoff, full jitter | -| `user_recommendation_job` | user_ml | User embedding and recommendation refresh | Default | -| `run_all_job` | All groups | Manual-only job for initial setup or recovery | Default | -| `cleanup_dagster_history_job` | N/A | Cleans up old Dagster run history | Default | +## Assets + +| Asset | Type | Group | +|-------|------|-------| +| `raw_github__extract_projects` | Python (Go subprocess) | ingestion | +| `stg_github__project` | dbt | ingestion | +| `core_github__detect_languages` | Python (FastText) | ingestion | +| `core_github__fetch_readme` | Python (Go subprocess) | ingestion | +| `core_github__fetch_repo_languages` | Python (Go subprocess) | ingestion | +| `core_github__fetch_repo_topics` | Python (Go subprocess) | ingestion | +| `stg_github__readme`, `stg_github__languages`, `stg_github__topics`, `stg_github__detection` | dbt | ingestion | +| `int_project_enriched` | dbt | ingestion | +| `fct_github_project` | dbt | ingestion | +| `core_match__classify_projects` | Python (LLM) | classification | +| `core_public__sync_projects` | Python | sync | +| `stg_public__project` | dbt | project_ml | +| `int_project_contextualized` | dbt | project_ml | +| `int_project_embedding_candidate` | dbt | project_ml | +| `core_ml__embed_projects` | Python | project_ml | +| `match_global_recommendation` | dbt | project_ml | +| `stg_public__user` | dbt | user_ml | +| `int_user_enriched` | dbt | user_ml | +| `fct_public_user` | dbt | user_ml | +| `core_ml__embed_users` | Python | user_ml | +| `match_user_recommendation` | dbt | user_ml | + +## Jobs and Schedules + +| Job | Schedule | Groups | +|-----|----------|--------| +| `project_enrichment_job` | Daily at 3 AM (Europe/Paris) | ingestion, classification, sync, project_ml | +| `user_recommendation_job` | Every 10 min | user_ml | +| `run_all_job` | Manual only | All groups | +| `cleanup_dagster_history_job` | Every 2 days at 11 PM | Housekeeping | -The `project_enrichment_job` is tagged with `dagster/max_concurrent_runs: 1` to prevent overlapping runs. +`project_enrichment_job` is limited to 1 concurrent run with 2 retries and exponential backoff. -## Schedules - -| Schedule | Job | Cron | Timezone | Status | -|----------|-----|------|----------|--------| -| `project_enrichment_schedule` | `project_enrichment_job` | `0 3 * * *` (daily at 3 AM) | Europe/Paris | Running | -| `user_recommendation_schedule` | `user_recommendation_job` | `*/10 * * * *` (every 10 min) | Europe/Paris | Running | -| `cleanup_dagster_history_schedule` | `cleanup_dagster_history_job` | `0 23 */2 * *` (every 2 days at 11 PM) | Europe/Paris | Running | - ## Resources -All resources are configured in `src/linker/definitions.py` and injected into assets via `required_resource_keys`. - -### PipelineConfig - -Central configuration resource that reads environment variables at runtime. - -| Field | Source | Description | -|-------|--------|-------------| -| `db_url` | `DATABASE_URL` | PostgreSQL connection string | -| `github_token` | `GITHUB_ACCESS_TOKEN` | GitHub API token for scraping | -| `go_scraper_path` | `GO_SCRAPER_PATH` | Path to compiled Go scraper binary | -| `go_fetcher_path` | `GO_FETCHER_PATH` | Path to compiled Go fetcher binary | -| `github_api_url` | Hardcoded | `https://api.github.com/search/repositories` | - -The config resource also builds GitHub search queries dynamically. Queries target 3 star ranges (300-1000, 1000-3000, 3000-5000) with filters for good first issues, recent activity, and excluded terms (awesome, roadmap, cheatsheet, interview). - -### LLMClassifierResource - -| Property | Value | -|----------|-------| -| Provider | OpenRouter (`https://openrouter.ai/api/v1`) | -| Model | `mistralai/mistral-small-3.2-24b-instruct` | -| Temperature | 0.0 | -| Response format | JSON object | -| Timeout | 45s hard timeout (thread-based) | -| Context truncation | 8,000 characters | - -### SentenceTransformerResource - -| Property | Value | -|----------|-------| -| Model | `sentence-transformers/all-MiniLM-L6-v2` | -| Embedding dimension | 384 | -| Device | CPU (configurable) | -| Normalization | Enabled (for cosine similarity) | - -Supports both single text encoding (`encode`) and batch encoding (`encode_batch`). - -### FastTextModelResource - -| Property | Value | -|----------|-------| -| Model file | `models/lid.176.ftz` | -| Purpose | Language detection on project text (name, description, README) | -| Loading | Lazy-loaded singleton, reused across all runs | - -### PandasPostgresIOManager - -Custom IO manager that transfers DataFrames between assets via PostgreSQL. Uses a schema/table allowlist to prevent SQL injection. Supports truncate-then-append writes and full table reads. - -### Other Resources - -| Resource | Type | Purpose | -|----------|------|---------| -| `dbt` | `DbtCliResource` | Runs dbt CLI commands from Dagster | -| `fs_io_manager` | `FilesystemIOManager` | Default file-based IO for non-DB assets | +| Resource | Purpose | +|----------|---------| +| `PipelineConfig` | Reads env vars (`DATABASE_URL`, `GITHUB_ACCESS_TOKEN`, Go binary paths), builds search queries | +| `LLMClassifierResource` | OpenRouter API, Mistral Small 3.2, temp 0.0, 45s timeout, 8K char context | +| `SentenceTransformerResource` | `all-MiniLM-L6-v2`, 384-dim embeddings, CPU, normalized | +| `FastTextModelResource` | `lid.176.ftz` for language detection, lazy-loaded singleton | +| `PandasPostgresIOManager` | DataFrame transfers between assets via PostgreSQL | +| `DbtCliResource` | Runs dbt CLI commands from Dagster | +| `FilesystemIOManager` | Default file-based IO for non-DB assets | diff --git a/ai/structure.mdx b/ai/structure.mdx index b9a6ff3..5913035 100644 --- a/ai/structure.mdx +++ b/ai/structure.mdx @@ -1,6 +1,6 @@ --- title: "Project Structure" -description: "Directory layout and key files of the OST Linker codebase" +description: "Directory layout of the OST Linker codebase" --- ## Top-Level Layout @@ -8,162 +8,77 @@ description: "Directory layout and key files of the OST Linker codebase" ``` ost-linker/ src/ - linker/ # Main Dagster module (Python) - services/ # External services (Go + Python) - dbt/ # dbt project (SQL transformations) - prisma/ # Database schema and seeds - scripts/ # Utility shell scripts - models/ # ML model files (FastText) - dagster_home/ # Dagster metadata (local dev) - docs/ # Documentation (Mintlify, git submodule) - .github/workflows/ # CI/CD workflows - pyproject.toml # Python project config (uv, pytest, ruff, mypy, dagster) - Dockerfile # 3-stage Docker build - docker-compose.yml # Production compose (webserver + daemon) - docker-compose.override.yml # Dev overrides (db, volumes, env) + linker/ # Dagster module (Python) + services/ # Go binaries + Python DB helpers + dbt/ # dbt project (SQL transformations) + prisma/ # Database schema and seeds + scripts/ # Utility shell scripts + models/ # ML model files (FastText) + dagster_home/ # Dagster metadata (local dev) + docs/ # Documentation (Mintlify) + .github/workflows/ # CI/CD workflows + pyproject.toml # Python config (uv, pytest, ruff, mypy, dagster) + Dockerfile # 3-stage Docker build + docker-compose.yml # Production compose + docker-compose.override.yml # Dev overrides (db, volumes) ``` ## `src/linker/` -- Dagster Module -This is the core Python module registered with Dagster. - ``` src/linker/ - definitions.py # Dagster Definitions (wires everything together) - assets/ - scraper/ - raw_github__extract_projects.py # Go scraper invocation - core_github__detect_languages.py # FastText language filtering - core_github__fetch_readme.py # Go fetcher (readme mode) - core_github__fetch_repo_languages.py # Go fetcher (languages mode) - core_github__fetch_repo_topics.py # Go fetcher (topics mode) - classification/ - core_match__classify_projects.py # LLM classification via OpenRouter - sync/ - core_public__sync_projects.py # Sync to public.Project - embedding/ - core_ml__embed_projects.py # Project embedding computation - core_ml__embed_users.py # User embedding computation - resources/ - cfg_resource.py # PipelineConfig + query builder + env helpers - llm_classifier_resource.py # OpenRouter LLM client - sentence_transformer_resource.py # MiniLM-L6-v2 embeddings - fasttext_resource.py # FastText language detection model - io_manager.py # PandasPostgresIOManager (DataFrame <-> DB) - jobs/ - project_enrichment_job.py # Classification + sync + project_ml - user_recommendation_job.py # User ML pipeline - run_all_job.py # Full pipeline (manual) - cleanup_dagster_job.py # Dagster history cleanup - schedules/ - project_enrichment_schedule.py # Daily at 3 AM - user_recommendation_schedule.py # Every 10 minutes - cleanup_dagster_schedule.py # Every 2 days at 11 PM - sensors/ - __init__.py # (Reserved for future sensors) - utils/ - language_detection.py # Non-Latin detection, FastText label parsing, blacklists - serialization.py # JSON serialization helpers (datetime, UUID, LLM cleanup) - __init__.py + definitions.py # Dagster Definitions entry point + assets/ # All Dagster assets (scraper, classification, sync, embedding) + resources/ # Config, LLM, SentenceTransformer, FastText, IO manager + jobs/ # Job definitions + schedules/ # Schedule definitions + sensors/ # Reserved for future sensors + utils/ # Language detection, serialization helpers ``` ## `src/services/go/` -- Go Binaries -Two independent Go modules, each compiled to a standalone binary. - ``` src/services/go/ - scraper/ - main.go # Entry point: multi-query parallel scraping - common.go # HTTP client, rate limiter, GitHub API types - main_test.go # Scraper tests - common_test.go # Rate limiter and parser tests - go.mod / go.sum - fetcher/ - main.go # Entry point: --mode readme|languages|topics - common.go # GitHubFetcher struct, rate limiter, retry logic - fetch_readme.go # README fetching with concurrent workers - fetch_languages.go # Language breakdown fetching - fetch_topics.go # Topics fetching - common_test.go # Fetcher tests - go.mod / go.sum -``` - -## `src/services/python/` - -``` -src/services/python/ - db.py # get_db_cursor() context manager for direct DB access + scraper/ # GitHub Search API scraper + fetcher/ # README, languages, topics fetcher (3 modes) ``` -## `dbt/` -- Data Transformation Layer +## `dbt/` -- Data Transformation ``` dbt/ - dbt_project.yml # Project config, vars (weights, thresholds), model schemas - profiles.yml # Connection profiles (local port 5433, docker port 5432) - models/ - staging/ - stg_github__project.sql / .yml - stg_github__readme.sql / .yml - stg_github__languages.sql / .yml - stg_github__topics.sql / .yml - stg_github__detection.sql / .yml - stg_public__project.sql / .yml - stg_public__user.sql / .yml - intermediate/ - int_project_enriched.sql / .yml - int_project_contextualized.sql / .yml - int_project_embedding_candidate.sql / .yml - int_user_enriched.sql / .yml - marts/ - fct_github_project.sql / .yml - fct_public_user.sql / .yml - match_global_recommendation.sql / .yml - match_user_recommendation.sql / .yml - macros/ - clean_text.sql / .yml - build_project_context.sql / .yml - build_user_context.sql / .yml - safe_divide.sql / .yml - clamp.sql / .yml - deduplicate.sql / .yml - jsonb_to_list.sql / .yml - generate_schema_name.sql / .yml - tests/ # Singular dbt tests - sources/ # Source definitions (YAML) + dbt_project.yml # Config, vars (weights, thresholds), schema routing + profiles.yml # Connection profiles (local / docker) + models/ # staging/ intermediate/ marts/ + macros/ # clean_text, build_project_context, safe_divide, etc. + tests/ # Singular dbt tests + sources/ # Source definitions ``` ## `prisma/` -- Schema Management ``` prisma/ - schema.prisma # Single source of truth for all 4 DB schemas - seed/ - seed.ts # Seeds categories, domains, tech stacks - data/ # JSON seed data files - migrations/ # Prisma migration history + schema.prisma # Single source of truth for all 4 DB schemas + seed/ # Seeds categories, domains, tech stacks ``` -## Key Configuration Files +## Key Config Files | File | Purpose | |------|---------| -| `pyproject.toml` | Python deps (uv), pytest config, ruff/mypy settings, Dagster module registration | -| `dagster.yaml` | Dagster instance config (storage, logs) | -| `workspace.yaml` | Dagster workspace definition (points to `src.linker.definitions`) | -| `Dockerfile` | 3-stage build: Go builder, Python builder, runtime | -| `docker-compose.yml` | Production services: `webserver` + `daemon` | -| `docker-compose.override.yml` | Dev overrides: `db` service, volume mounts, `.env` loading | -| `.env.example` | Template for required environment variables | -| `dbt/dbt_project.yml` | dbt config, model materialization, schema routing, scoring variables | -| `dbt/profiles.yml` | Database connection profiles (local vs docker) | +| `pyproject.toml` | Python deps, pytest, ruff/mypy, Dagster module | +| `dagster.yaml` | Dagster instance config | +| `workspace.yaml` | Points to `src.linker.definitions` | +| `Dockerfile` | 3-stage build (Go, Python, runtime) | +| `.env.example` | Required environment variables template | ## Utility Scripts | Script | Purpose | |--------|---------| -| `scripts/go_binary_gen.sh` | Compile both Go binaries for local development | -| `scripts/clean_dagster.sh` | Clear Dagster run history and storage | -| `scripts/sync_prisma.sh` | Sync Prisma schema to backend repo | +| `scripts/go_binary_gen.sh` | Compile Go binaries | +| `scripts/clean_dagster.sh` | Clear Dagster storage | +| `scripts/sync_prisma.sh` | Sync Prisma schema to backend | | `scripts/clean_docker_images.sh` | Remove dangling Docker images | diff --git a/ai/troubleshooting.mdx b/ai/troubleshooting.mdx index ae570a9..6543e80 100644 --- a/ai/troubleshooting.mdx +++ b/ai/troubleshooting.mdx @@ -1,271 +1,56 @@ --- title: "Troubleshooting" -description: "Common issues, cleanup procedures, and debugging commands for OST Linker" +description: "Common issues and solutions for OST Linker" --- ## Common Issues ### Missing Environment Variables -**Symptom:** Dagster fails to load definitions or assets crash at startup. - -**Solution:** Ensure all required variables are set in `.env`: - -```bash -# Check which variables are missing -grep -v '^#' .env.example | while IFS='=' read -r key _; do - [ -z "${!key}" ] && echo "MISSING: $key" -done -``` - -Required variables: `DATABASE_URL`, `GITHUB_ACCESS_TOKEN`, `OPENROUTER_API_KEY`, `GO_SCRAPER_PATH`, `GO_FETCHER_PATH`. +Dagster fails to load definitions. Ensure all required vars are set in `.env`: `DATABASE_URL`, `GITHUB_ACCESS_TOKEN`, `OPENROUTER_API_KEY`, `GO_SCRAPER_PATH`, `GO_FETCHER_PATH`. ### dbt Manifest Not Found -**Symptom:** `FileNotFoundError: Could not find manifest.json` when Dagster starts. - -**Solution:** The dbt manifest must be generated before Dagster can discover dbt assets. Run: - -```bash -cd dbt && dbt parse -``` - - -In development, `dbt_project.prepare_if_dev()` is called automatically in `definitions.py` to generate the manifest. If this fails, run `dbt parse` manually. - +`FileNotFoundError: Could not find manifest.json` at Dagster startup. Fix: run `cd dbt && dbt parse`. ### Go Binary Not Found -**Symptom:** `RuntimeError: Go scraper binary not found at ` or similar for the fetcher. - -**Solution:** Compile the Go binaries and update your `.env`: - -```bash -cd src/services/go/scraper && go build -o github-scraper main.go -cd ../fetcher && go build -o ost-fetcher main.go -``` - -Then set the absolute paths in `.env`: - -``` -GO_SCRAPER_PATH=/absolute/path/to/src/services/go/scraper/github-scraper -GO_FETCHER_PATH=/absolute/path/to/src/services/go/fetcher/ost-fetcher -``` +`RuntimeError: Go scraper binary not found`. Compile with `scripts/go_binary_gen.sh` and set absolute paths in `.env`. ### Database Connection Errors -**Symptom:** `connection refused` or `FATAL: password authentication failed`. - -**Checklist:** -1. Is PostgreSQL running? `docker compose ps` -2. Is the port correct? Local dev uses **5433** (mapped from container's 5432) -3. Does the `DATABASE_URL` match your Docker Compose config? -4. Is pgvector installed? Check with `SELECT * FROM pg_extension WHERE extname = 'vector';` +`connection refused` or `password authentication failed`. Check: is PostgreSQL running (`docker compose ps`)? Is the port 5433? Does `DATABASE_URL` match your config? ### Port Conflicts -**Symptom:** `Address already in use` when starting Dagster or PostgreSQL. - -**Solution:** Check what is using the port and stop it: - -```bash -# Check port 3000 (Dagster) -lsof -i :3000 - -# Check port 5433 (PostgreSQL) -lsof -i :5433 -``` +`Address already in use`. Check with `lsof -i :3000` or `lsof -i :5433` and stop the conflicting process. ### FastText Model Missing -**Symptom:** `FileNotFoundError: FastText model not found at: models/lid.176.ftz` - -**Solution:** Download the model file: +`FileNotFoundError: FastText model not found`. Download it: ```bash mkdir -p models wget -O models/lid.176.ftz https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz ``` -Or set `FASTTEXT_MODEL_PATH` to the correct location if the file is stored elsewhere. - -### Dagster Cache / Stale State +### Dagster Stale State -**Symptom:** Assets show outdated status, schedules do not trigger, or the UI behaves unexpectedly. - -**Solution:** Clear Dagster's local storage: - -```bash -scripts/clean_dagster.sh -``` - -Or manually: - -```bash -rm -rf dagster_home/ -mkdir dagster_home -``` - - -Clearing Dagster storage deletes all run history, asset materializations, and schedule state. Use this as a last resort. - +Assets show outdated status or schedules do not trigger. Fix: `scripts/clean_dagster.sh` (deletes all run history). ### OpenRouter API Errors -**Symptom:** `RuntimeError: OpenRouter API error for ` or `TimeoutError: OpenRouter API hard timeout after 45s`. - -**Checklist:** -1. Is `OPENROUTER_API_KEY` set and valid? -2. Is the OpenRouter service reachable? `curl https://openrouter.ai/api/v1/models` -3. Check your OpenRouter account for rate limits or credit balance -4. The LLM classifier has a 45-second hard timeout per project -- transient failures are expected and logged +Check that `OPENROUTER_API_KEY` is valid, the service is reachable (`curl https://openrouter.ai/api/v1/models`), and you have credits. The 45s per-project timeout means transient failures are expected. ### Prisma Schema Drift -**Symptom:** dbt models fail because tables or columns do not match expectations. - -**Solution:** Re-apply the Prisma schema: - -```bash -npx prisma db push -``` - -For a full reset (destructive): - -```bash -npx prisma db push --force-reset -npx ts-node prisma/seed/seed.ts -``` - -## Cleanup Procedures - -### dbt Artifacts - -Remove compiled SQL and cached packages: - -```bash -cd dbt && dbt clean -``` - -### Dagster History +dbt models fail due to missing tables/columns. Fix: `npx prisma db push`. For full reset: `npx prisma db push --force-reset && npx ts-node prisma/seed/seed.ts`. -```bash -scripts/clean_dagster.sh -``` - -### Docker Resources - -Remove dangling images and unused volumes: - -```bash -scripts/clean_docker_images.sh - -# Or manually -docker compose down -v -docker system prune -f -``` - -### Full Reset - -To start completely fresh: - -```bash -# Stop everything -docker compose down -v - -# Clean artifacts -cd dbt && dbt clean && cd .. -rm -rf dagster_home/ - -# Rebuild -docker compose up --build -d - -# Re-initialize database -npx prisma db push -npx ts-node prisma/seed/seed.ts -``` - -## Useful Debugging Commands - -### Dagster - -```bash -# Validate definitions without starting the server -dagster definitions validate - -# Check asset materializations -dagster asset list - -# Run a specific job manually -dagster job execute -j project_enrichment_job -``` - -### dbt - -```bash -# Run a single model -dbt run --select stg_github__project +## Cleanup Commands -# Test a single model -dbt test --select fct_github_project - -# Show compiled SQL for a model -dbt compile --select match_user_recommendation - -# Preview model output (first 5 rows) -dbt show --select int_project_contextualized --limit 5 -``` - -### Go Services - -```bash -# Test scraper -cd src/services/go/scraper && go test ./... - -# Test fetcher -cd src/services/go/fetcher && go test ./... - -# Run scraper manually (requires env vars) -./github-scraper - -# Run fetcher manually -./ost-fetcher --mode readme --concurrency 5 --limit 10 -``` - -### Python - -```bash -# Run all tests -pytest - -# Run only unit tests -pytest -m unit - -# Run with verbose output -pytest -v --tb=short - -# Lint and format -ruff check src/ -ruff format src/ - -# Type check -mypy src/ -``` - -### Database - -```bash -# Connect to the database -psql "postgresql://user:pass@localhost:5433/dbname" - -# Check table counts -SELECT schemaname, tablename, n_tup_ins FROM pg_stat_user_tables ORDER BY schemaname; - -# Check pgvector embeddings -SELECT count(*) FROM ml.embd_github_project; -SELECT count(*) FROM ml.embd_user; - -# Check recommendations -SELECT count(*) FROM public.match_global_recommendation; -SELECT count(*) FROM public.match_user_recommendation; -``` +| Task | Command | +|------|---------| +| Clean dbt artifacts | `cd dbt && dbt clean` | +| Clear Dagster history | `scripts/clean_dagster.sh` | +| Remove Docker resources | `scripts/clean_docker_images.sh` | +| Full reset | `docker compose down -v && cd dbt && dbt clean && cd .. && rm -rf dagster_home/ && docker compose up --build -d` | From cf8f7012f91dff4e73b2178add5fe501bd1be195 Mon Sep 17 00:00:00 2001 From: spideystreet Date: Sat, 7 Mar 2026 22:21:06 +0100 Subject: [PATCH 3/4] docs: rename AI Engine tab to Linker Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com> --- docs.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs.json b/docs.json index 6f97c07..30a6218 100644 --- a/docs.json +++ b/docs.json @@ -66,7 +66,7 @@ }, { - "tab": "AI Engine", + "tab": "Linker", "groups": [ { "group": "Overview", From f4f3bf3780d60f54cfbe580e1b2d4e82144426f6 Mon Sep 17 00:00:00 2001 From: spideystreet Date: Tue, 10 Mar 2026 17:37:12 +0100 Subject: [PATCH 4/4] docs(ai): add REST API & MCP documentation page - New rest-api.mdx covering API endpoints, rate limiting, Docker config, and MCP server - Update overview.mdx with API/MCP in tech stack and card grid - Update structure.mdx with src/services/api/ directory layout - Add rest-api page to Mintlify navigation Co-Authored-By: spidecode-bot <263227865+spicode-bot@users.noreply.github.com> --- ai/overview.mdx | 5 ++ ai/rest-api.mdx | 120 +++++++++++++++++++++++++++++++++++++++++++++++ ai/structure.mdx | 17 +++++++ docs.json | 3 +- 4 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 ai/rest-api.mdx diff --git a/ai/overview.mdx b/ai/overview.mdx index d713fca..37b8f58 100644 --- a/ai/overview.mdx +++ b/ai/overview.mdx @@ -15,6 +15,8 @@ The pipeline is fully automated and orchestrated by **Dagster**. - **Data transformation:** dbt (staging, intermediate, marts) - **Ingestion:** Go 1.24 (scraper + fetcher binaries) - **ML / Classification:** Python 3.11, SentenceTransformer (MiniLM-L6-v2), FastText, OpenRouter (Mistral) +- **REST API:** FastAPI (read-only, consumed by MCP server) +- **MCP Server:** TypeScript ([ost-mcp](https://github.com/opensource-together/ost-mcp)), exposes project discovery tools to AI assistants - **Database:** PostgreSQL + pgvector (384-dim vectors, cosine similarity) - **Schema management:** Prisma (4 schemas: `public`, `github`, `ml`, `match`) - **Containerization:** Docker (3-stage build) @@ -51,4 +53,7 @@ The pipeline is fully automated and orchestrated by **Dagster**. Codebase directory layout + + API endpoints and MCP server integration + diff --git a/ai/rest-api.mdx b/ai/rest-api.mdx new file mode 100644 index 0000000..b62960b --- /dev/null +++ b/ai/rest-api.mdx @@ -0,0 +1,120 @@ +--- +title: "REST API & MCP" +description: "FastAPI service and MCP server for project discovery" +--- + +## Overview + +OST Linker exposes a **read-only REST API** (FastAPI) that serves project data to the **OST MCP server**. Together, they let developers discover and explore open-source projects directly from Claude Desktop, IDEs, and other MCP-compatible clients. + +``` +User (Claude Desktop / IDE) + -> MCP Server (stdio, TypeScript) + -> OST Linker REST API (HTTP, Python) + -> PostgreSQL (pgvector) +``` + +## REST API + +**Location:** `src/services/api/` + +The API is a lightweight FastAPI service with sync endpoints and a psycopg2 connection pool. It runs as a separate Docker container with minimal environment (only `DATABASE_URL`, no Dagster/LLM secrets). + +### Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/health` | Health check | +| `GET` | `/projects/search?q=...` | Search projects by keyword (supports `category`, `domain`, `techstack` filters) | +| `GET` | `/projects/{id}` | Get full project details (includes categories, domains, tech stacks) | +| `GET` | `/projects/{id}/similar` | Find similar projects via pgvector cosine similarity | +| `GET` | `/recommendations/trending` | Get globally trending projects | +| `GET` | `/categories` | List all categories | +| `GET` | `/domains` | List all domains | +| `GET` | `/techstacks` | List all tech stacks | + +### Rate Limiting + +All endpoints (except `/health`) are rate-limited to **60 requests/minute per IP** using [slowapi](https://github.com/laurentS/slowapi) decorators. Exceeding the limit returns `429 Too Many Requests`. + +### Running Locally + +```bash +# Requires DATABASE_URL in .env +uvicorn src.services.api.main:app --host 0.0.0.0 --port 8000 +``` + +### Docker + +The API runs as the `api` service in `docker-compose.yml` on port **8000**, with a health check at `/health`. + +```yaml +api: + environment: + DATABASE_URL: ${DATABASE_URL} + API_HOST: ${API_HOST:-0.0.0.0} + API_PORT: ${API_PORT:-8000} + API_RATE_LIMIT: ${API_RATE_LIMIT:-60} + DAGSTER_ROLE: api +``` + + +The API container does **not** receive Dagster, GitHub, or LLM secrets -- only `DATABASE_URL` and API-specific variables. + + +### Key Files + +| File | Purpose | +|------|---------| +| `main.py` | FastAPI app, lifespan (pool init/close), rate limit handler | +| `config.py` | `APIConfig` (pydantic-settings), reads env vars | +| `database.py` | `ConnectionPool` wrapper (psycopg2 `SimpleConnectionPool`) | +| `schemas.py` | Pydantic v2 response models | +| `rate_limit.py` | slowapi `Limiter` instance | +| `routes/` | One file per route group (health, projects, recommendations, references) | + +--- + +## MCP Server (ost-mcp) + +**Repository:** [opensource-together/ost-mcp](https://github.com/opensource-together/ost-mcp) + +A TypeScript MCP server that wraps the REST API into 7 tools for AI assistants. + +### Tools + +| Tool | API Endpoint | Description | +|------|-------------|-------------| +| `search_projects` | `GET /projects/search` | Search projects by keyword with optional filters | +| `get_project` | `GET /projects/{id}` | Get full project details | +| `find_similar` | `GET /projects/{id}/similar` | Find similar projects via AI embeddings | +| `get_trending` | `GET /recommendations/trending` | Get trending projects | +| `list_categories` | `GET /categories` | List all categories | +| `list_domains` | `GET /domains` | List all domains | +| `list_techstacks` | `GET /techstacks` | List all tech stacks | + +### Installation + +Add to your Claude Desktop or MCP client config: + +```json +{ + "mcpServers": { + "ost": { + "command": "npx", + "args": ["@opensource-together/mcp"], + "env": { + "OST_API_URL": "https://api.opensource-together.com" + } + } + } +} +``` + +### Architecture + +- `src/index.ts` -- MCP server entry point, registers all tools via `@modelcontextprotocol/sdk` +- `src/client.ts` -- `OSTClient` HTTP client (fetch with 10s timeout) +- `src/tools/` -- One file per tool group (search, project, similar, trending, references) +- `src/config.ts` -- Reads `OST_API_URL` from env (defaults to production) +- `src/types.ts` -- Shared TypeScript interfaces matching API response schemas diff --git a/ai/structure.mdx b/ai/structure.mdx index 5913035..ff292b5 100644 --- a/ai/structure.mdx +++ b/ai/structure.mdx @@ -36,6 +36,23 @@ src/linker/ utils/ # Language detection, serialization helpers ``` +## `src/services/api/` -- REST API (FastAPI) + +``` +src/services/api/ + main.py # FastAPI app, lifespan, rate limit handler + config.py # APIConfig (pydantic-settings) + database.py # ConnectionPool (psycopg2) + dependencies.py # FastAPI dependency injection + schemas.py # Pydantic v2 response models + rate_limit.py # slowapi Limiter (60 req/min/IP) + routes/ + health.py # GET /health + projects.py # search, detail, similar + recommendations.py # trending + references.py # categories, domains, techstacks +``` + ## `src/services/go/` -- Go Binaries ``` diff --git a/docs.json b/docs.json index 30a6218..f6547a3 100644 --- a/docs.json +++ b/docs.json @@ -78,7 +78,8 @@ "ai/pipeline", "ai/database", "ai/dbt", - "ai/go-services" + "ai/go-services", + "ai/rest-api" ] }, {