Goal. Stand up the package so every later experiment is a controlled, reproducible run. Nothing downstream is trusted unless this is in place.
Scope / what to build.
- Python package skeleton (
pyproject.toml, src/ layout, tests/), Apache-2.0 license, README.
- A configuration layer (reuse an established config manager, e.g. Hydra) that drives every stage from a single config file: which models, which dataset, which cue set, seeds, output paths.
- Run manifests: every run writes a manifest recording the exact model ids, prompt versions, seed, cue-set version, dataset revision, and library versions, so a result is always reproducible against a known dependency set.
- Deterministic replay: full turn-level transcripts are logged to JSONL; a run can be re-loaded and re-analyzed offline without re-calling any model. Analysis is pure and offline; only the model calls are online.
- Continuous integration: lint + unit tests on push and PR.
Reuse vs bespoke. Reuse the config manager, the logging/serialization, and the CI templates. The only bespoke piece is the manifest schema and the transcript format (keep both small and documented).
Definition of done. A one-command run on a tiny fixture produces a manifest + JSONL transcript; re-loading the transcript reproduces the same analysis bit-for-bit; CI is green; versions are pinned.
Model backend. For now every agent uses the Gemini API (multimodal). All model access goes through one gateway wrapper, so the roster can be extended to other model APIs later without changing any experiment code. Gemini is the default for now; no fine-tuning, models are used off-the-shelf.
Goal. Stand up the package so every later experiment is a controlled, reproducible run. Nothing downstream is trusted unless this is in place.
Scope / what to build.
pyproject.toml,src/layout,tests/), Apache-2.0 license, README.Reuse vs bespoke. Reuse the config manager, the logging/serialization, and the CI templates. The only bespoke piece is the manifest schema and the transcript format (keep both small and documented).
Definition of done. A one-command run on a tiny fixture produces a manifest + JSONL transcript; re-loading the transcript reproduces the same analysis bit-for-bit; CI is green; versions are pinned.
Model backend. For now every agent uses the Gemini API (multimodal). All model access goes through one gateway wrapper, so the roster can be extended to other model APIs later without changing any experiment code. Gemini is the default for now; no fine-tuning, models are used off-the-shelf.