YearBuildingWithLLMs
title: "What We Learned from a Year of Building with LLMs" radar_quadrant: Techniques radar_ring: Assess radar_position: inner
"What We Learned from a Year of Building with LLMs" is a practitioner retrospective published at applied-llms.org by Hamel Husain, Eugene Yan, Bryan Bischof, Charles Frye, Shreya Shankar, and Jason Liu. It documents tactical and strategic lessons from building production LLM applications, organised into prompting, RAG, evaluation, operations, and team/product guidance sections.
Prompting and retrieval: Hybrid RAG combining BM25 keyword search with vector embeddings outperforms embeddings alone. Complex tasks benefit from decomposition into sequential deterministic steps; agent-based approaches are less reliable than plannable workflows. Structured output (via Instructor for APIs, Outlines for self-hosted) eases downstream integration.
Evaluation: LLM-as-judge works for pairwise comparisons and directional improvement detection, but requires position bias mitigation, tie allowances, and chain-of-thought explanations. Hallucinations persist at a 5–10% baseline even on simple tasks. Start evaluation with unit tests built from real production input/output pairs before adopting evaluation frameworks.
Operations: Development-to-production skew — both structural (formatting, casing) and semantic — is a chronic failure mode; review actual samples daily. Pin specific model versions to avoid unexpected behavioural changes. The model is the least durable component; build moats through evals, guardrails, caching, and data flywheels.
Strategy: No GPUs before product-market fit. No finetuning until prompting is proven insufficient. Model pricing halves every ~6 months — applications uneconomical at $20/M tokens become viable at <$0.10/M within 18 months. Specialise in specific domains rather than building generic tools.
Placed in Techniques / Assess / inner.
This is the highest-signal LLM engineering practitioner piece published in 2024. It names specific techniques — hybrid RAG, LLM-as-judge evaluation, task decomposition, deterministic workflows over agents — with concrete implementation guidance derived from real production systems. The multi-author provenance (practitioners from Honeycomb, Parlance Labs, and independent consultants) provides breadth across different application types.
Inner position reflects zero tooling cost, direct applicability to any active LLM integration project, and strong complementarity with existing radar blips: Exploring Generative AI (Fowler Memos) covers conservative practitioner observation; this article covers tactical implementation patterns. AI Conversation Types provides the design taxonomy; this article provides the production engineering lessons.
Trial gate: hybrid RAG, LLM-as-judge evaluation, and at least one deterministic multi-step decomposition applied in a real production LLM application, with daily sample review established as an operational practice.
