Skip to content

docs(ai): rewrite Linker documentation and add REST API & MCP page #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
spideystreet wants to merge 4 commits into
base: main
Choose a base branch
from
Open

docs(ai): rewrite Linker documentation and add REST API & MCP page #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2aa06e1
docs(ai): complete rewrite of AI Engine documentation
spideystreet Mar 7, 2026
260827e
docs(ai): simplify and condense AI Engine documentation
spideystreet Mar 7, 2026
cf8f701
docs: rename AI Engine tab to Linker
spideystreet Mar 7, 2026
f4f3bf3
docs(ai): add REST API & MCP documentation page
spideystreet Mar 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions ai/database.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
title: "Database Schema"
description: "PostgreSQL schemas and pgvector setup"
---

## Overview

Single PostgreSQL database with **pgvector**, organized into 4 schemas. Schema is defined in Prisma (`prisma/schema.prisma`) and shared with the backend.

## `public` -- User-Facing Data

| Table | Description |
|-------|-------------|
| `user` | User accounts with social links and GitHub/GitLab identities |
| `Project` | Published projects with metadata |
| `Category` | Project categories (Framework, Library, CLI Tool, etc.) |
| `Domain` | Project domains (Web Dev, DevOps, Data Science, etc.) |
| `tech_stack` | Technologies and languages |
| `user_tech_stack`, `user_categories`, `user_domain` | User preference junctions |
| `project_tech_stack`, `project_category`, `project_domain` | Project attribute junctions |
| `project_bookmark` | User bookmarks |
| `match_global_recommendation` | Top-N global recommendations (dbt-materialized) |
| `match_user_recommendation` | Per-user personalized recommendations (dbt-materialized) |

## `github` -- Ingestion Data

| Table | Description |
|-------|-------------|
| `raw_github_project` | Raw JSON from GitHub Search API (Go scraper) |
| `raw_github_readme`, `raw_github_languages`, `raw_github_topics` | Fetched enrichment data (Go fetcher) |
| `int_github_detection` | FastText language detection results (Python) |
| `stg_github__*` | Staged data (dbt) |
| `int_project_enriched` | Joined enriched project data (dbt) |
| `fct_github_project` | Final fact table (dbt) |

## `ml` -- Machine Learning

| Table | Description |
|-------|-------------|
| `embd_github_project` | 384-dim project embedding vectors (Python) |
| `embd_user` | 384-dim user embedding vectors (Python) |
| `stg_public__project`, `stg_public__user` | Staged data for ML (dbt) |
| `int_user_enriched`, `int_project_contextualized`, `int_project_embedding_candidate` | ML prep models (dbt) |
| `fct_public_user` | User fact table (dbt) |

## `match` -- Classification

| Table | Description |
|-------|-------------|
| `project_classification` | LLM-assigned category and domain with confidence scores |

## pgvector

The `vector` extension enables cosine similarity search on 384-dimensional embeddings (MiniLM-L6-v2). Similarity is computed as `1 - cosine_distance` directly in dbt SQL models. Minimum similarity threshold is 0.25 (configurable via dbt var).

Vectors are defined in Prisma as `Unsupported("vector")` since Prisma does not natively support the pgvector type.

## Schema Routing

dbt models are routed to PostgreSQL schemas via the `generate_schema_name` macro and `+schema` in `dbt_project.yml`. The macro uses the configured schema name directly without prefix.
68 changes: 68 additions & 0 deletions ai/dbt.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: "dbt Layer"
description: "Data transformation models, macros, and recommendation scoring"
---

## Models

All models are materialized as tables and routed to PostgreSQL schemas via `generate_schema_name`.

| Model | Layer | Schema | Description |
|-------|-------|--------|-------------|
| `stg_github__project` | staging | `github` | Flattens raw JSON into typed columns |
| `stg_github__readme` | staging | `github` | Stages README content |
| `stg_github__languages` | staging | `github` | Stages language breakdowns |
| `stg_github__topics` | staging | `github` | Stages topic arrays |
| `stg_github__detection` | staging | `github` | Stages FastText detection results |
| `stg_public__project` | staging | `ml` | Stages projects for ML |
| `stg_public__user` | staging | `ml` | Stages users for ML |
| `int_project_enriched` | intermediate | `github` | Joins project + readme + languages + topics + detection |
| `int_project_contextualized` | intermediate | `ml` | Builds context strings for embedding |
| `int_project_embedding_candidate` | intermediate | `ml` | Filters projects ready for embedding |
| `int_user_enriched` | intermediate | `ml` | Builds user context strings |
| `fct_github_project` | mart | `github` | Final project fact table |
| `fct_public_user` | mart | `ml` | Final user fact table |
| `match_global_recommendation` | mart | `public` | Top-N global recommendations |
| `match_user_recommendation` | mart | `public` | Per-user personalized recommendations |

## Macros

| Macro | Purpose |
|-------|---------|
| `clean_text(column)` | Strip HTML, normalize whitespace |
| `deduplicate(relation, partition_by, order_by)` | Window-function dedup |
| `jsonb_to_list(column)` | JSONB array to comma-separated string |
| `build_project_context(...)` | Concatenate project fields into embedding input |
| `build_user_context(...)` | Concatenate user fields into embedding input |
| `safe_divide(num, denom)` | Returns NULL on division by zero |
| `clamp(expr)` | Clamps value to 0-1 range |
| `generate_schema_name(...)` | Uses `+schema` directly, no prefix |

## Recommendation Scoring

**Global:** Top-N projects (default 20) that are trending or published, ordered by recency and stars.

**User (hybrid scoring):**

```
final_score = 0.40 * similarity
+ 0.35 * preference
+ 0.15 * freshness
+ 0.10 * popularity
```

- **Similarity** -- cosine similarity between user and project embeddings (pgvector)
- **Preference** -- weighted overlap: tech stacks (30%), categories (45%), domains (25%)
- **Freshness** -- linear decay over 90 days from last push
- **Popularity** -- log-normalized star count, clamped to 0-1

Minimum similarity threshold: 0.25. Max recommendations per user: 30. All params are dbt vars in `dbt_project.yml`.

## Profiles

| Profile | Host | Port |
|---------|------|------|
| `local` (default) | `localhost` | 5433 |
| `docker` | `db` | 5432 |

Switch with `export DBT_TARGET=docker`.
55 changes: 55 additions & 0 deletions ai/go-services.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
title: "Go Services"
description: "GitHub scraper and fetcher binaries for data ingestion"
---

## Overview

Two independent Go binaries handle GitHub data ingestion. Both are called by Dagster assets via `subprocess.run()` with a 600-second timeout.

## Scraper

**Location:** `src/services/go/scraper/`

Searches the GitHub Search API for open-source projects and upserts results into `github.raw_github_project`.

- Reads queries from `GITHUB_SCRAPING_QUERIES` env var (JSON array)
- One goroutine per query, paginating up to 1,000 results each
- Batch upserts via `pgx.Batch`
- Outputs JSON summary to stdout (parsed by Dagster)

**Search filters:** stars 300-5000, good first issues, recent pushes, public, not archived, not forks. Excludes awesome lists, roadmaps, cheatsheets, interview repos.

**Rate limiting:** 30 req/min budget, tracks `X-RateLimit-*` headers, 3 retries with exponential backoff.

## Fetcher

**Location:** `src/services/go/fetcher/`

Enriches scraped projects by fetching additional data. Operates in 3 modes:

| Mode | Endpoint | Output Table |
|------|----------|-------------|
| `readme` | `GET /repos/owner/repo/readme` | `github.raw_github_readme` |
| `languages` | `GET /repos/owner/repo/languages` | `github.raw_github_languages` |
| `topics` | `GET /repos/owner/repo/topics` | `github.raw_github_topics` |

### CLI Flags

| Flag | Default | Description |
|------|---------|-------------|
| `--mode` | (required) | `readme`, `languages`, or `topics` |
| `--concurrency` | 10 | Number of concurrent workers |
| `--limit` | 0 (unlimited) | Max projects to process |

**Incremental:** only fetches projects missing data in the target table.

**Rate limiting:** 5,000 req/hour budget, 3 retries, 30-min global timeout. READMEs truncated to 50KB.

## Dagster Integration

Both binaries are invoked via `subprocess.run()`. Environment is built by `build_scraper_env()` / `build_fetcher_env()` helpers in `cfg_resource.py`.

<Warning>
If the Go binary is not found at the configured path, the asset raises a `RuntimeError`. Make sure `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` point to compiled binaries.
</Warning>
167 changes: 66 additions & 101 deletions ai/installation.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,127 +1,92 @@
# OST AI Engine – Installation & Deployment Guide

This guide explains how to install, configure, and deploy the OST AI Engine platform, with a focus on centralized configuration, multi-language management, Dockerization, and Dagster orchestration.

## Requirements
- OS: Linux, macOS, Windows
- Python: 3.13+
- Go: ≥ 1.20
- Node.js (for Prisma)
- Docker

## Centralized Configuration Management

Project configuration is centralized in the `config/config.yaml` file, which is automatically generated by the Python module `config.py` using environment variables and business logic (e.g., calculation of `seven_days_ago`).

- **Why this choice?** YAML is accessible by all languages (Python, Go, Node.js), ensuring consistent parameters (tokens, queries, N projects, etc.) throughout the pipeline.
- **Workflow:**
1. Environment variables are defined in `.env`.
2. On startup or build, `config/config.py` reads these variables and generates/updates `config/config.yaml`.
3. All components (Dagster, Go scrapers, Prisma) read config from YAML.

**Example config.example.yaml:**
```yaml
DATABASE_URL: postgresql://postgres:postgres@ost-db:5432/ost_dev
GITHUB_ACCESS_TOKEN: ...
GITLAB_ACCESS_TOKEN: ...
GITHUB_SCRAPING_QUERY: stars:>100 stars:<500 created:>=2025-10-11 is:public archived:false
GITHUB_TOP_N: 30
---
title: "Installation & Setup"
description: "Get OST Linker running locally"
---

## Quick Start (Docker)

```bash
git clone https://github.com/opensource-together/ost-linker.git
cd ost-linker
cp .env.example .env
# Edit .env with your values (see below)
docker compose up --build -d
```

Dagster UI will be at `http://localhost:3000`.

## Environment Variables

Copy `.env.example` to `.env` and fill in the values:
```ini
OST_CONFIG_PATH=config/config.yaml
### Required

DATABASE_URL=postgresql://ai-engine:ai-engine@localhost:7777/ai-engine
POSTGRES_DB=ai-engine
POSTGRES_USER=ai-engine
POSTGRES_PASSWORD=ai-engine
| Variable | Description |
|----------|-------------|
| `DATABASE_URL` | PostgreSQL connection string |
| `GITHUB_ACCESS_TOKEN` | GitHub fine-grained personal access token |
| `OPENROUTER_API_KEY` | API key for LLM classification |
| `GO_SCRAPER_PATH` | Path to compiled Go scraper binary |
| `GO_FETCHER_PATH` | Path to compiled Go fetcher binary |

GITHUB_ACCESS_TOKEN=your_github_access_token_here
GITLAB_ACCESS_TOKEN=your_gitlab_access_token_here
```
### Optional

**Propagation:**
- Python loads variables via `dotenv`.
- `config.py` injects them into YAML.
- Go and Dagster read config from YAML (never directly from `.env`).

| Variable | Description | Example |
|-----------------------|------------------------------|---------|
| DATABASE_URL | PostgreSQL connection URL | postgresql://db_user:db_password@localhost:port/db_name |
| POSTGRES_DB | Database name | db_name |
| POSTGRES_USER | Database user | db_user |
| POSTGRES_PASSWORD | Database password | db_password |
| GITHUB_ACCESS_TOKEN | GitHub API token | your_github_access_token_here |
| GITLAB_ACCESS_TOKEN | GitLab API token | your_gitlab_access_token_here |

## Install Dependencies
```bash
poetry install
go mod tidy ./src/infrastructure/services/go/github
go mod tidy ./src/infrastructure/services/go/gitlab
cd prisma
npm install
```
| Variable | Default |
|----------|---------|
| `FASTTEXT_MODEL_PATH` | `models/lid.176.ftz` |
| `DBT_TARGET` | `local` (port 5433) |
| `DBT_PROJECT_DIR` | `./dbt` |
| `DAGSTER_HOME` | `./dagster_home` |

<Warning>
Never commit `.env` or hardcode secrets. The file is gitignored.
</Warning>

## Local Development Setup

### 1. Python dependencies

## Database Setup
Start PostgreSQL with Docker Compose:
```bash
docker compose up -d
uv sync
```

## Prisma Migrations
Apply database migrations:
### 2. Compile Go binaries

```bash
cd prisma
npx prisma migrate deploy
scripts/go_binary_gen.sh
```

## Dockerization & Multi-language Build

The Dockerfile builds the entire stack:
- Installs Python dependencies (Poetry), Go, Node.js
- Compiles Go scrapers
- Generates Prisma client
- Copies and generates centralized YAML config
- Sets `DAGSTER_HOME` for Dagster
- Entrypoint: launches Dagster daemon
Then set `GO_SCRAPER_PATH` and `GO_FETCHER_PATH` in `.env` to the absolute paths.

**Dockerfile excerpt:**
```dockerfile
FROM python:3.13-slim AS base
### 3. Download FastText model

WORKDIR /app
```bash
mkdir -p models
wget -O models/lid.176.ftz https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz
```

COPY pyproject.toml poetry.lock ./
RUN pip install poetry && poetry install --no-root --only main
### 4. Start the database

COPY src/ src/
COPY prisma/ prisma/
COPY .env .env
COPY config/ config/
```bash
docker compose up db -d
```

RUN poetry run python config/config.py
RUN poetry run prisma generate
ENV GOARCH=arm64
RUN cd src/infrastructure/services/go/github && go build -o /app/github-scraper main.go
ENV DAGSTER_HOME=/app/src/dagster
### 5. Initialize the database

EXPOSE 3000
CMD ["poetry", "run", "dagster-daemon", "run"]
```bash
npx prisma db push
npx ts-node prisma/seed/seed.ts
```

## Dagster Orchestration & Cron
### 6. Install dbt deps and run Dagster

```bash
cd dbt && dbt deps && cd ..
dagster dev -h 0.0.0.0 -p 3000
```

- The main job (`github_scraper_job`) is scheduled via a cron table (e.g., every 6 hours: `0 */6 * * *`).
- Dagster config (`dagster.yaml`) allows customization of storage, logs, etc.
- Dagster assets are modular: each pipeline step is an asset, including Go scrapers.
## Verify Setup

**Dagster local launch** :
```bash
export DAGSTER_HOME="$PWD/src/dagster"
poetry run dagster dev -m src.dagster.definitions --host 127.0.0.1 --port 3000
dagster definitions validate
cd dbt && dbt build && cd ..
pytest
```
Accès UI Dagster : [http://localhost:3000](http://localhost:3000)
Loading