Skip to content
This repository was archived by the owner on Apr 13, 2026. It is now read-only.
This repository was archived by the owner on Apr 13, 2026. It is now read-only.

feat(ai): evaluate lighter LLM backend for general-purpose hardware #14

@db-tycoon-stephen

Description

@db-tycoon-stephen

Background

The current `tycoon ai` stack is hard-wired to LM Studio's local server at `localhost:1234`. LM Studio is a heavy desktop app — it's fine for development on a high-end machine but is not a realistic dependency for general-purpose users.

Research findings

  • OpenAI Codex CLI — cloud-only, conflicts with local-first design. Not worth pursuing.
  • OpenAI gpt-oss-20b — Apache 2.0 open-weight model, 16GB RAM, available on Ollama. Viable locally, but adds Ollama as a dependency.
  • Ollama — lighter than LM Studio, broader hardware support, CLI-first, no GUI. But still a separate install.

Goal

Find the single shortest path to running a capable model for the specific tasks tycoon needs:

  • `TestFixer` — fix a failing dbt test
  • `ColumnDocumenter` — generate schema.yml descriptions
  • `StagingImprover` — refactor a staging model

These are focused, single-turn tasks with small context windows. They do not need a general-purpose chat model or a large context.

Design questions to answer

  1. What is the lightest runtime that can serve a small model locally without a GUI dependency? (Ollama, llama.cpp server, mlx, transformers pipeline?)
  2. Is there a model small enough to run on CPU-only hardware that is still useful for these tasks? (e.g. Qwen2.5-Coder-1.5B, Phi-3-mini)
  3. Should `tycoon ai` ship with a recommended model + one-line install command, rather than requiring the user to set up LM Studio separately?
  4. Can we reduce `tycoon.ai.client` to a single, minimal HTTP call with no backend abstraction layer?

Out of scope

  • Multiple LLM backend options / provider abstraction layer
  • Cloud API fallbacks
  • OpenAI Codex CLI integration

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions