CI Infrastructure and Test Health — backlog and dependency chain

[magnus919]

Summary

Cashew has a substantial test suite (38 test files, ~454 tests) and a strong testing philosophy documented in CLAUDE.md, but no automated CI pipeline. The only GitHub Actions workflow builds and publishes to PyPI — it never runs the tests. This means:

• PRs can merge with broken tests and nobody knows
• Test regressions go undetected until a human runs them locally
• The test suite has drifted stale in 8 places since the default embedding model changed
• Lint/type debt has grown substantially (207 ruff errors, 66 mypy errors)

This discussion captures the individual issues, their dependencies, and the order of operations to get from where we are to a working CI pipeline.

Current State (May 2026)

Metric	Value
Test files	38
Tests passing	441/454
Tests failing	13 (8 codebase rot + 5 environment-local)
Ruff errors	207
MyPy errors	66
CI workflows	1 (release only, no test step)

Issues (in dependency order)

Phase 1 — Fix the test suite

Issue #68 — test: fix stale embedding dimension assertions for gte-large default model

Status: PR #71 ready for review.

Upstream PR #46 changed the default embedding model from all-MiniLM-L6-v2 (384-dim) to thenlper/gte-large (1024-dim), but several tests still hardcode 384-dim or expect the old model name. These 8 tests fail on a fresh install:

• test_embeddings.py — 4 tests expect 384-dim vectors
• test_embedding_daemon_integration.py — 3 tests expect 384-dim vectors
• test_session.py — 1 test expects all-MiniLM-L6-v2 as default

Fix: update assertions to match the model in use (either parameterize by dim, or make the tests model-aware). This must merge before any CI pipeline is added, otherwise main will always be red.

Phase 2 — Test infrastructure

Tests in test_vec_table_dim.py require sqlite-vec loadable at runtime. This needs sqlite-vec installed and a Python that supports extension loading — which the actions/setup-python images do. Handled as part of the CI pipeline install step.

Phase 3 — Continuous Integration

Issue #69 — ci: add GitHub Actions workflow for continuous integration

Status: Depends on Phase 1 merging green.

A .github/workflows/ci.yml that runs on push and pull_request:

• pytest (hard gate — must pass)
• ruff (report-only / annotations, given 207 pre-existing errors)
• mypy (report-only, given 66 pre-existing errors)
• Codecov (upload coverage data)
• Python matrix: 3.10, 3.11, 3.12
• pip install -e ".[dev]" with sqlite-vec added

Phase 4 — Branch protection

Issue #70 — docs: enable branch protection for main

Status: Requires repository admin access (maintainer action).

Once CI exists and is green on main, enable branch protection to require CI passing before merge. This prevents untested PRs from landing. Cannot be implemented by PR — needs repository settings.

Phase 5 — Optional: Lint/type debt reduction

Status: Independent of the above, could be parallel work.

The lint and type debt has grown significantly. Separate issues can be filed for specific areas as needed.

Dependency Graph

flowchart LR
    subgraph MustMerge[Must merge first]
        A[Issue #68<br/>Fix stale tests] --> B[PR #71]
    end
    subgraph Pipeline[CI Pipeline]
        C[Issue #69<br/>Add CI workflow]
    end
    subgraph AdminAction[Requires admin]
        D[Issue #70<br/>Enable branch protection]
    end

    B -->|merges| C
    C -->|green on main| D

    E[Lint/type debt
       reduction]

    style A fill:#d4edda,stroke:#28a745
    style B fill:#ffeeba,stroke:#ffc107
    style C fill:#e8f4f8,stroke:#2980b9
    style D fill:#fef9e7,stroke:#f39c12
    style E fill:#f0f0f0,stroke:#95a5a6

Loading

Questions for discussion

1. Should the CI workflow gate on pytest passing, or start with informational status while the debt is worked down?
2. Should lint/type checks be hard gates with a baseline, or should they be separate repos (pre-commit, etc.)?
3. Is there interest in pre-commit hooks for ruff + mypy to catch issues before PR?

This discussion was filed on behalf of Magnus Hedemark by Jasper, his AI agent.

[magnus919]

Issues have been created referencing this discussion:

• #68 — test: fix stale embedding dimension assertions for gte-large default model (Phase 1)
• #69 — ci: add GitHub Actions workflow for continuous integration (Phase 3, blocked on test: fix stale embedding dimension assertions for gte-large default model #68)
• #70 — docs: enable branch protection for main branch (Phase 4, blocked on ci: add GitHub Actions workflow for continuous integration #69)

[magnus919]

PR #71 opened for Phase 1 (fix stale tests): #71

Dependency chain:

• Discussion CI Infrastructure and Test Health — backlog and dependency chain #67 ← this meta-discussion
• Issue test: fix stale embedding dimension assertions for gte-large default model #68 → PR test: fix stale embedding dimension assertions for gte-large default model #71 (Phase 1: fix stale tests) ← current state, ready for review
• Issue ci: add GitHub Actions workflow for continuous integration #69 → (Phase 3: CI pipeline, blocked on test: fix stale embedding dimension assertions for gte-large default model #68)
• Issue docs: enable branch protection for main branch #70 → (Phase 4: branch protection, blocked on ci: add GitHub Actions workflow for continuous integration #69)

[rajkripal]

Closing this out: CI is live. The matrix workflow (3.10 / 3.11 / 3.12) runs on every PR and main push, and the stale-test fixes from Phase 1 (#71) are in. Branch protection is the remaining nicety but the test-health goal of this thread is met.

Thanks for raising it, Magnus. The phased framing (fix the tests first, then wire CI, then protect the branch) was the right order and made it easy to land incrementally instead of in one big drop. Appreciated.