Skip to content

starkyru/learn-ai

Repository files navigation

learn-ai — a hands-on course in LLMs, RAG, and agents

A personal, project-based curriculum that takes you from a vague sense of how AI apps work to building real ones: provider integration, embeddings, retrieval (RAG), and autonomous agents — in both TypeScript and Python.

You don't just read. Every module is code you run, break, and extend, with three depth levels woven through:

Depth What it means Where
🟢 App Build something that works using the ecosystem. every module
🟡 Balanced Build the app and implement one core piece by hand for intuition. 01, 03, 04, 05, 07
🔴 Deep Implement the machinery from scratch (tokenizer, attention, vector index, ReAct loop). 01, 04, 06

Pick your lane per module, or do all three. The 🔴 deep tasks are optional but they're where the real understanding lives.

Most lessons include runnable TypeScript and Python scaffolds. The explicitly labelled documentation-first production deep dives define capstone work that you apply in either language while their scaffolds are being built; see the maintenance policy for current parity status.


The map

How the tracks feed each other — foundations first, then two big tracks (retrieval and agents) that merge in production and the capstone:

flowchart LR
    F[Foundations<br>00-01f: setup, ML, DL,<br>LLMs and transformers] --> B[Building with LLMs<br>02 integration - 03 prompting]
    B --> R[Retrieval track<br>04 embeddings - 05/05b RAG<br>11 ingestion - 12 text-to-SQL]
    B --> A[Agents track<br>06-06d agents - 16 context<br>17 MCP - 18 computer use]
    B --> T[Training and inference<br>13/13b fine-tuning and alignment<br>14 local - 15 reasoning]
    R --> P[Production<br>07/07b service delivery<br>20/20b security + governance<br>21/21b eval - 22 UX]
    A --> P
    T --> P
    P --> C[23 Capstone]
Loading
# Module You'll build Core ideas
00 Setup & Providers "Hello LLM" across 6 providers API keys, the provider abstraction, OpenAI-compatible APIs
01 LLM Fundamentals A BPE tokenizer, cosine similarity, a toy attention head, samplers tokens, embeddings, attention, sampling, what a "model" even is
02 LLM Integration Streaming chat, JSON/structured output, tool calling, retries the request/response loop, function calling, cost & tokens
03 Prompting & Patterns A prompt library + evaluator few-shot, chain-of-thought, self-consistency, prompt eval
04 Embeddings & Vectors An in-memory vector index from scratch, then Chroma/Qdrant embeddings, ANN search, chunking (incl. semantic), hybrid (BM25 + dense) search
05 RAG A full retrieval-augmented Q&A pipeline + eval chunk→embed→retrieve→rerank→generate, citations, faithfulness, HyDE + reverse HyDE
06 Agents A ReAct agent from scratch, then with LangGraph tools, planning loops, memory, multi-agent
07 Advanced & Production Eval harness, tracing, caching, a served API LLM-as-judge, observability, cost control, deployment
07b Delivery & AI Service Operations A containerised, authenticated service with a safe release runbook CI, secrets, identity, tenant isolation, queues, rollout, rollback
08 Classification A text classifier 3 ways + a softmax/GD one from scratch LLM zero-shot vs embeddings+ML vs trained head, metrics (F1)
09 Computer Vision Image classification, CLIP zero-shot, multimodal-LLM vision, a convolution from scratch pixels→features, CNN/ViT, CLIP, vision LLMs
10 Image Generation Text-to-image (hosted Stable Diffusion), img2img/inpainting, a toy diffusion sampler diffusion process, latent space, U-Net, guidance
11 Document Ingestion A real RAG ingestion pipeline (PDF/HTML, cleaning, structure-aware chunking, incremental indexing, multimodal PDF) the messy-data front-end RAG actually needs
12 Text-to-SQL NL→SQL over a real DB, with safety + hybrid routing querying structured data, schema grounding, SQL guardrails
13 Fine-tuning Prompt vs RAG vs fine-tune; hosted SFT; LoRA from scratch; distillation SFT, LoRA/QLoRA, dataset prep, when to fine-tune
14 Local Inference & Optimization Quantization & throughput benchmarks, a KV cache from scratch, model routing/fallbacks quantization, KV cache, serving engines, load testing
15 Reasoning & Test-time Compute Reasoning vs standard models, self-consistency, best-of-N, self-refine extended thinking, spending compute at inference
16 Context Engineering Token budgeting, prompt caching, memory compaction, map-reduce, batch API using the context window as a scarce budget
17 MCP & Modern Agent APIs A course MCP server + an agent that uses it; Responses API Model Context Protocol, hosted tools, remote MCP
18 Computer Use A browser agent (vision + DOM grounded) with safety gates computer-use models, automation, action allowlists
19 Audio & Speech STT, TTS, a voice tutor; VAD/denoise; realtime Whisper, TTS, voice agents, audio preprocessing
20 AI Security Attack then harden your own RAG agent; a red-team harness OWASP LLM Top 10, prompt injection, excessive agency
20b Governance, Privacy & Responsible Practice A data map, rights workflow, model/data card, and accountable UX review privacy, retention, licences, accessibility, recourse
21 LLMOps & Eval Versioned eval sets, a CI regression gate, monitoring, feedback loop the eval lifecycle, regression gates, human review
21b Evaluation Science & Agent Reliability Gold-evidence retrieval and deterministic agent-safety suites MRR/NDCG, uncertainty, judge agreement, trajectory tests
22 AI Product UX A real mini app: streaming UI, citations drill-down, feedback, approval flow trust, failure states, "show sources", human-in-the-loop
23 Capstone A RAG-powered agent app, end to end everything above, integrated

Deep dive — modules/05b-advanced-rag/: extends module 05 with the named advanced RAG architectures (TS + Py): Contextual Retrieval, Corrective RAG (CRAG), Self-RAG, and GraphRAG (multi-hop, from scratch) — the feedback-loop and graph patterns that fix naive RAG's blind spots. docs/ADVANCED_RAG.md is the reference + pick-by-failure cheat-sheet.

Deep dive — modules/06b-langgraph/: extends module 06 with a full LangGraph lesson (TS + Py): state & reducers, conditional edges/ToolNode, persistence, human-in-the-loop interrupt(), streaming, subgraphs, supervisor handoff, and time travel — the production features interviewers ask about. docs/LANGGRAPH.md is the reference + interview cheat-sheet.

Foundations companions — modules/01b-ml-foundations/, modules/01c-deep-learning/, modules/01d-transformer/, modules/01e-trees-ensembles/, modules/01f-stats-foundations/: extend module 01 with the classic-ML / deep-learning / transformer theory interviews probe — pure from-scratch numpy + TS, no provider needed. 01b covers regression, bias–variance, ridge, cross-validation, ROC/AUC, and k-means; 01c builds a scalar autograd engine + MLP, optimizers (SGD/Momentum/Adam), initialisation, regularisation, and an RNN with BPTT; 01d assembles a GPT-style decoder — multi-head attention with causal masking, sinusoidal positional encoding, LayerNorm/GELU/FFN pre-LN blocks, a KV cache, and an encoder-vs-decoder (BERT vs GPT) comparison, plus interview notes on RoPE, GQA/MQA, FlashAttention, MoE, and scaling laws; 01e builds decision trees (CART), random forests, gradient boosting, and an empirical bias–variance decomposition; 01f covers Bayes & naive Bayes, MLE (and why cross-entropy is MLE), hypothesis testing / A/B tests, and PCA from scratch.

Deep dive — modules/13b-alignment/: extends module 13 with post-training & alignment — the "how is ChatGPT actually trained?" interview answer, built from scratch on toy models (TS + Py, fully offline): preference data + Elo/win rates, a Bradley–Terry reward model, RLHF with REINFORCE including a reward hacking (Goodhart) demo with and without the KL leash, and DPO from scratch with its derivation.

Deep dive — modules/06c-agent-frameworks/: extends modules 06/06b with the five framework names interviews drop — LangChain (LCEL prompt | model | parser, buffer memory, retrievers/RAG), CrewAI (role → task → sequential crew), AutoGen (round-robin group chat), LlamaIndex (Documents → index → query engine for RAG), and Semantic Kernel (skills/functions + a planner) — each reimplemented through llm_core then mapped back to the real library's API. Every task runs offline via a --stub deterministic model.

Deep dive — modules/06d-agent-memory/: extends modules 06/06b with agent memory engineering — the memory-augmented vs memory-aware distinction and the full taxonomy (episodic, semantic, procedural, entity, summary): the read-before / write-after lifecycle, relevance thresholds against noisy retrieval, entity extraction + merge, summary compaction with just-in-time expansion, and a composed MemoryManager with TTL eviction that keeps context under a token budget across turns. Offline via --stub, like 06c.

Production deep dives — modules/07b-delivery-operations/, modules/20b-governance-privacy/, and modules/21b-evaluation-reliability/: close the gap between a locally working AI feature and an accountable product: deployment and rollback, identity and tenant isolation, data lifecycle and recourse, gold-evidence retrieval metrics, uncertainty, and deterministic agent safety tests. These are documentation-first specifications to apply to a capstone; language-specific scaffolds are tracked in the maintenance policy.

Hosted-first (default to APIs, optional local heavy path documented): modules 09, 10 (vision/diffusion), 13 (fine-tuning), 19 (audio). Nothing multi-GB downloads unless you opt in.

Lost in the acronyms? Every abbreviation used in the course (BM25, RAG, HNSW, KV cache, LoRA, …) is expanded on first use in each lesson and collected in docs/GLOSSARY.md.

Applied projects (in projects/):

  • projects/news-agent — a Telegram bot that an agent drives to collect news on a topic you choose and post a daily digest. Build it after module 06; it pulls together integration, agents, tools, and scheduling into something you'll actually run. (uv run python -m news_agent --dry-run to try it with no Telegram setup.)
  • projects/tutor — your study buddy: a Q&A mode to ask about the project and how to proceed, and an exam mode that quizzes you on a module and grades you. Runs free on Ollama. Usable from day one. There are also Claude Code slash commands /tutor and /exam if you're in Claude Code — see docs/TUTOR_AND_EXAM.md.

Read CURRICULUM.md for the detailed week-by-week plan, prerequisites, and "you're done when…" checklists.


Repo layout

learn-ai/
├── packages/
│   ├── ts/llm-core/        # provider-agnostic LLM client (TypeScript)
│   └── py/llm_core/        # the same, in Python
├── modules/
│   ├── 00-setup/ … 23-capstone/   # 24 numbered modules + 13 companions/deep dives
│   │   ├── README.md       # the lesson + tasks + "done when" checklist
│   │   ├── ts/             # TypeScript exercises
│   │   └── py/             # Python exercises
├── projects/
│   ├── news-agent/         # the Telegram daily-news agent
│   └── tutor/              # Q&A + exam study CLI
├── docs/                   # tooling, learning references, and maintenance policy
├── data/                   # sample corpora for RAG exercises
├── scripts/                # helpers (smoke tests, etc.)
└── .claude/commands/       # /tutor and /exam slash commands

The two llm-core packages are the spine of the course: you write exercises against one small interface and swap OpenAI ↔ Claude ↔ Ollama ↔ LM Studio ↔ NVIDIA ↔ Gemini by changing a single env var. Understanding why that abstraction is possible (and where it leaks) is module 00–02. For the full tooling reference (uv, pnpm, pytest, jest, formatters), see docs/TOOLING.md. Maintainers should also follow the course maintenance and language-parity policy.

Prefer reading in a browser? uv run scripts/build_site.py renders every module lesson and reference doc (with the diagrams) to a static HTML site — open site/index.html.


Setup (do this once)

1. Clone & secrets

cp .env.example .env
# edit .env — you only need keys for the providers you'll use.
# Zero-cost path: install Ollama and leave LLM_PROVIDER=ollama.

2. Pick a free or paid path

Path Cost Setup
Ollama (recommended to start) free Install Ollama, then ollama pull llama3.2 && ollama pull nomic-embed-text
LM Studio free Install LM Studio, load a model, Start Server (port 1234) → LLM_PROVIDER=lmstudio + LMSTUDIO_CHAT_MODEL
NVIDIA NIM free tier get a key at build.nvidia.comNVIDIA_API_KEY in .env
Google Gemini free tier key at aistudio.google.com/apikeyGEMINI_API_KEY in .env, LLM_PROVIDER=gemini
OpenAI paid (~$5 covers the course) key at platform.openai.com
Anthropic paid key at console.anthropic.com; set ANTHROPIC_MODEL=claude-haiku-4-5 for cheap iteration

3. Python

# uv (https://docs.astral.sh/uv/) manages the env + the editable llm_core install.
uv sync
uv run python scripts/smoke_test.py        # verifies your provider works

Some modules need extras: uv sync --extra vectors (04, 05), agents (06), production (07, 22), ml (08), vision (09, local only), imagegen (10, local only), ingest (11), mcp (17), browser (18), finetune (13, local only), audio (19, local only), telegram (news-agent). The vision, imagegen, finetune, and audio extras are optional — those modules default to hosted APIs.

4. TypeScript

pnpm install
pnpm build:core
pnpm tsx modules/00-setup/ts/hello.ts        # run any exercise file directly

How to work through it

  1. cd modules/00-setup and read the README.md. Each module README is the lesson — concepts first, then numbered tasks with acceptance criteria.
  2. Do the 🟢 app task to get something working, then circle back for 🟡/🔴.
  3. Run the same exercise against two different providers and notice what changes.
  4. Don't skip module 01 even though it's the least "useful" — it's the load-bearing intuition for everything after.

Modules 04→05→06 are the heart; budget extra time there. Don't assume every module fits into one week — pick a route below.


Choose a route

There is no single "finish it" path. Pick one of four independently usable routes by the outcome you want. Every companion and deep dive is either scheduled inside a route or explicitly excluded with a pointer to the route that owns it — nothing is silently dropped. Full week-by-week schedules, prerequisite orders, and route decision guidance (goal, background, provider/hardware, portfolio artifact, next route) live in CURRICULUM.md.

Time budgets are split into four buckets that are summed separatelyexercise-only (writing the tasks), setup/debug (reading + installing + debugging), provider/cloud (hosted keys, deploy setup, heavy downloads beyond the free Ollama baseline), and capstone (module 23 only). Every number below is generated from and CI-verified against one source, scripts/curriculum/module_times.json, by scripts/curriculum/check_route_hours.py, so you never reconcile conflicting totals. Ranges are honest planning estimates, not guarantees, and none of this is legal or financial advice.

Totals in hours (min–max), each bucket summed separately:

Route Lessons Exercise-only Setup/debug Provider/cloud Capstone
Core app-builder 27 114–153 28–52.5 11–22 10–20
ML foundations 5 22–32 6–11 0 0
Agent systems 12 55–74 14.5–25.5 4.75–9.5 10–20
Model training 5 20–28 4.5–9 2.25–4.5 0
  • Core app-builder — the existing ~20-week main sequence, labelled: all 24 numbered modules (00–23) plus the production deep dives 07b / 20b / 21b. General programming assumed, no ML background; the free Ollama path completes most of it. Ends with a deployed, evaluated RAG-agent app (module 23). Intentionally excludes 01b–01f (→ ML foundations), 05b/06b/06c/06d (→ Agent systems), and 13b (→ Model training). Next: Agent systems or Model training.
  • ML foundations — the five from-scratch companions 01b–01f (regression, autograd/backprop, the transformer decoder, trees/ensembles, probability/PCA). Pure offline NumPy + TypeScript, no provider. Order: 01b → 01e/01f → 01c → 01d. Ends with a from-scratch ML repo. Next: Model training or Agent systems.
  • Agent systems05b, 06, 06b, 06c, 06d, 16, 17, 18, 21, 21b, 07b, 23: from a hand-rolled ReAct loop to a deployed, memory-aware, multi-framework agent service with an offline trajectory/safety gate. Assumes core modules 02–07; Ollama runs most tasks, paid keys help for MCP (17) and computer use (18). Next: Model training or Core app-builder.
  • Model training08, 13, 13b, 14, 15: decide when to train, run a hosted fine-tune, implement LoRA and DPO from scratch, and optimise local inference. Assumes the ML foundations route + modules 00–04; needs a hosted SFT key (optional GPU for local LoRA/serving). Ends with a fine-tuned, DPO-aligned model plus training evidence. Next: Agent systems or Core app-builder.

Conventions

  • Exercises that need a model use getProvider() / get_provider() from llm-core. Never hardcode a provider in exercise code.
  • Each task folder has its own short README / docstring explaining what and why — treat writing those notes as part of the exercise.
  • 🔴 "from scratch" tasks forbid the obvious library (no tiktoken for the tokenizer task, no chromadb for the vector-index task) — that's the point.

License

MIT — use it, fork it, learn from it. Contributions welcome.

About

Hands-on, project-based course in AI, LLMs, RAG, and agents — 24 modules in TypeScript + Python, provider-agnostic (OpenAI/Anthropic/Ollama/NVIDIA/LMStudio).

Resources

License

Stars

93 stars

Watchers

1 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors