Skip to content

RelyLoop v0.1.0 — MVP1 alpha

Choose a tag to compare

View all tags
@SoundMindsAI SoundMindsAI released this 13 May 10:59
· 371 commits to main since this release
v0.1.0
d099536
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

RelyLoop v0.1.0 — MVP1 alpha

What's in MVP1

The full Karpathy loop end-to-end on Elasticsearch and OpenSearch, single-tenant, no auth, Docker Compose:

  • Engine adapter — one SearchAdapter Protocol covering both ES 8.11+/9.x and OpenSearch 2.x/3.x. Cluster registration via UI or API.
  • Optuna optimizer — TPE sampler against a parametrized query template; up to N trials per study; per-trial budget guard; pytrec_eval metrics (ndcg@k, map, precision, recall, mrr, err).
  • LLM-as-judge — POST /api/v1/judgments/generate rates query-document pairs against a rubric. ~$0.01–$0.05 per query set with gpt-4o-mini. Provider-agnostic: works against any OpenAI-compatible endpoint (Ollama / LM Studio / vLLM / TGI) via OPENAI_BASE_URL.
  • Digest — LLM-generated narrative summary of each completed study, plus parameter-importance chart and recommended config.
  • GitHub PR worker — winning configs land as Pull Requests against a central search-config Git repo. Operator's CI deploys.
  • Chat agent — describe the problem in chat; the agent introspects the cluster, proposes a search-space, and queues the study after operator confirmation. 19-tool surface.
  • Operator tutorial + sample data — 1,000 curated Amazon ESCI products + 48 queries + canonical Jinja2 query template. docs/08_guides/tutorial-first-study.md walks git clone → Open PR in under 30 minutes on a fresh laptop.
  • CI smoke gate — every PR runs the full Karpathy loop end-to-end against a fresh stack with a budgeted OpenAI key. Same operator path as the tutorial; no degraded variants.

Full feature list: see docs/02_product/mvp1-user-stories.md.

Audience

Technical evaluators, Relevance Engineers, and search-platform teams considering an open-source query-tuning tool. Not yet production-deployable — see docs/01_architecture/deployment.md for the MVP1 → MVP3 → GA v1 deployment maturity ramp.

How to install

Follow the tutorial: docs/08_guides/tutorial-first-study.md.

Operators build images locally via make up. Pre-built GHCR images ship at MVP3 per the canonical release matrix; until then, make up triggers a local Docker build of relyloop/api and relyloop/ui on first run.

Known limitations

This is alpha. Three operator-visible issues are tracked but not blocking:

  • Long chat sessions silently drop context after 100 messages. The agent prompt-window cap is brute-force; smarter context management ships in MVP2. Tracked: bug_chat_long_conversation_truncation.
  • Query templates created via the API with declared params can't be used for LLM judgment generation. Workaround: use one template with declared_params={} for judgment generation, a separate template with declared params for the optimization study (this is what the tutorial does). Tracked: bug_judgment_template_default_params_contract.
  • Worker may need a manual restart after first-run make migrate. If you make up and immediately fire a study before make migrate completes, the Arq worker dies on Optuna schema init and stays down. Workaround: docker compose restart worker after make migrate. Tracked: bug_worker_optuna_init_race.

How to provide feedback

Roadmap

Release Theme Adds
MVP1 / v0.1.0 (you are here) "The Loop" ES + OpenSearch adapter, OpenAI-compatible LLM, GitHub provider, single-tenant, no auth, Docker Compose, 80% coverage gate
MVP2 / v0.2 "Observable" Langfuse + ClickHouse + SigNoz; canonical event catalog; audit_log + immutability trigger; lineage columns; PII redaction; trace propagation
MVP3 / v0.3 "Production Stacks" Lucidworks Fusion adapter; multi-Git-provider abstraction (GitLab, Bitbucket); production install (TLS via Caddy + Let's Encrypt, managed Postgres/Redis); AWS managed OpenSearch
MVP4 / v0.4 "Multi-tenant, Multi-LLM" tenants + tenant_memberships + users + api_keys; tenant_id columns + backfill; SSO via reverse proxy; native non-OpenAI provider SDKs
GA v1 "Production-ready" LangGraph orchestrator + PostgresSaver; full RFC 7807 errors; Idempotency-Key; Helm chart; container scanning; image signing

Canonical release matrix: docs/01_architecture/tech-stack.md.

Assets 2
Loading