docs(llms.txt): tell agents how Flox is actually installed - #90
Merged
Conversation
The preamble told agents there is no `curl | bash` installer, and that `install.flox.dev` and `flox.dev/install` do not exist. Every one of those claims was true when it was written and none is true now: `get.flox.dev` serves the install script, and llms-full.txt on this same site carries the command six times. The guardrail had become the thing agents were wrong about. Replace the prohibition with the command, the platform dispatch the script performs, and the FLOX_VERSION pin. Regenerate llms.txt, which check-llms-txt.yml requires to match the generator.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Review found three problems with the replacement bullet. It sent readers to flox.dev/download for "package-manager installs and the raw downloads". That page leads with the same curl script and one platform button; Homebrew, the APT and YUM repositories, and every raw .deb/.rpm/.pkg link are on the install docs page, which the bullet cited second. The two clauses were swapped relative to reality. `FLOX_VERSION=<version> pins a release` did not say where the assignment goes. It must follow the pipe; an agent writing `FLOX_VERSION=1.16.0 curl … | sh` sets it on curl and silently installs current stable. Preventing exactly that guess is what this preamble is for, so show the whole pipeline. Added the bounds an agent will otherwise walk into: the script installs but never upgrades, and it refuses on an existing Nix install, ostree distributions, openSUSE/SLES, and RPM systems without dnf or yum. Code-format apt, dnf and yum, per AGENTS.md, matching line 16 of this file.
Contributor
Author
|
Automated review: 2 independent seats (standards, spec). Fixed here:
Verified: generator re-run leaves an empty diff, so |
Review: the landing page is for the marketing site, and an agent reading this preamble has no use for it. The install docs page it already links carries Homebrew, the APT and YUM repositories, and every raw package, so the second URL added a hop rather than a destination.
imkarrer
added a commit
to flox/flox-skills
that referenced
this pull request
Sep 9, 2026
…pins Review (@stephenyeargin): we do not want an agent installing anything but the newest Flox if we can help it, and FLOX_CHANNEL has no business in agent-facing prose at all. Both are right, and the evidence for the version half is in this branch's own history: earlier drafts hardcoded 1.14.0 and then 1.16.0, and 1.14.0 was already two releases stale the day it was written. An agent that pins picks the version it remembers, which is exactly the failure mode. So FLOX_CHANNEL is gone everywhere, and FLOX_VERSION is gone from SKILL.md, ci.md and publish.md. It survives only as a named escape hatch in the two places a reader could otherwise think it was forbidden, each saying not to reach for it unasked. The after-the-pipe placement note goes with it — it existed only to serve a pin we no longer teach. `ghcr.io/flox/flox:latest` replaces the pinned tag, which also answers the review question directly: latest is maintained, and resolves to the same digest as v1.16.0, itself current stable. Pinning is now named as the exception for byte-identical rebuilds rather than the default. Carried the same reviewer's flox.dev/download point from flox/docs#90 through this repo: an agent does not care about the marketing landing page, and the raw packages are on the install docs page anyway. Docs links take the .md suffix, per the review suggestion and the rest of the file. The gated task asked "how do I pin a specific version?", so it moved too — otherwise the skill would start failing an eval for obeying the review. gate PASS judge 4/5 screening --reps 5 baseline 0/5, skills 5/5, gap 3.4, discriminator
stephenyeargin
approved these changes
Sep 9, 2026
imkarrer
added a commit
to flox/flox-skills
that referenced
this pull request
Sep 10, 2026
…answers that do (#113) The `flox` skill told agents that Flox has no `curl | bash` installer. One has existed since `get.flox.dev` shipped, and the eval suite enforced the retracted claim — failing the *correct* answer on 24 of 36 tasks. This fixes the claim, the check that enforced it, and the gap it left. **Review this first:** the two endpoint rules in `evals/flox/run.py`, and the Quick Reference entry in `SKILL.md`. Everything else follows from those two decisions. Linear: [DEV-315](https://linear.app/floxdotdev/issue/DEV-315/update-llm-documentation-for-new-install-capabilities). The `llms.txt` half is [flox/docs#90](flox/docs#90). Measured, `--reps 5` on the fixed harness: baseline **1/5**, skills **5/5**, judge gap 3.4, classified a discriminator. <details> <summary>What was wrong</summary> Three surfaces carried the claim, and each was correct when written: - `SKILL.md` — "Do NOT suggest `install.flox.dev`, `flox.dev/install`, or any `curl | bash` one-liner — none of these exist." - `references/ci.md`, in three places, and `references/publish.md`'s GitLab section. - `evals/flox/run.py` — `no_fake_install_url` banned the `curl … | sh` *shape*, on 24 tasks. Meanwhile `llms-full.txt` on flox.dev carried the command six times. The skill and the docs contradicted each other on the same site. A confidently wrong answer is worse than the hallucination it replaced: a model that invents a URL produces a 404 the user can see; a model that says the installer does not exist sends them off to hand-roll a `.deb` download. </details> <details> <summary>The eval check: a shape ban became an endpoint allowlist</summary> Two rules replace the shape ban: 1. **Only a host that serves the script may be piped into a shell.** Catches `curl https://flox.dev/install | sh` — that URL redirects to an HTML page, so it installs nothing. Measured: `sh` exits 127 without `-L`, 2 with it. Both loud, so this grades a *wrong* answer rather than guarding a dangerous one. 2. **No invented `*.flox.dev` host anywhere.** `releases.flox.dev` appears in a committed baseline un-piped, so rule 1 alone would miss it. Each URL is paired with the pipe it actually feeds — the span to the next URL — because scanning whole lines failed correct answers that carried both the one-liner and a docs link. `evals/floxify` keeps the old shape ban deliberately, now with a comment saying why: it greps the generated *manifest*, where installing Flox is wrong whatever URL it names. </details> <details> <summary>The skill fix had to move, and the eval is how we know</summary> Rewriting the "Installing Flox" section was **not enough**. With the prohibition deleted and the section leading with the installer, the model still answered: > "There is **no `curl | bash` one-liner** for Flox — ignore anything you may have seen suggesting `install.flox.dev` or `flox.dev/install`." It reproduced the retracted claim nearly verbatim while reading a skill that no longer contains it. That text was published on flox.dev long enough to be in the model's prior; deleting it from the skill does not delete it from the model. So the fact also lives in the Quick Reference — which `SKILL.md` defines as the home for specifics "easy to get wrong from memory", authoritative and used inline — and says plainly that recall to the contrary is stale. Not isolated: whether the *placement* or the *wording* carries it. The section-only variant has not been run on the fixed harness. </details> <details> <summary>The harness was grading models against their own answer key</summary> Found while screening this change. A **baseline** answer opened: > "I checked the eval's ground truth for the real Flox install URLs, so this is accurate." Both harnesses allow every arm the `Read` tool, which the skills arm needs for a skill's reference files, and neither passed a `cwd` — so `claude` inherited `evals/flox`, where `tasks/*.jsonl` hold the prompt, the rubric and `must_match`. The answer key was one relative path away for every run this suite has ever made. It inflates the baseline hardest, because what a skill contributes is what a rubric spells out — so it presents as "no signal", indistinguishable from a task that genuinely fails to discriminate. That is exactly how this task first screened. The agent now launches in an empty temp directory; three tests pin it. Closing it moved the same baseline from 5/5 to 1/5. **Every screening number predating this is suspect, `baselines/screen-*.json` included.** Failures are the exception: a run that failed with the answer key reachable failed for real. Re-baselining is bigger than this PR. </details> <details> <summary>Installing Flox into an image</summary> `containers.md` covered `flox containerize` — turning an environment *into* an image — and nothing covered getting the CLI *into* an image you build. That gap is why hand-rolled agent images reconstruct the download with an architecture case statement and a stale pin. Container-tested on aarch64 under Colima: | Case | Result | |---|---| | Root, no daemon, `NIX_REMOTE=auto` | `flox activate` exit 0 | | Non-root, no daemon | fails on `/nix/var/nix/db/big-lock` | | Non-root, daemon started | exit 0 | | `buildkite/agent:3-ubuntu` | runs as **root**; no `buildkite-agent` account | The first draft had this backwards, asserting the Buildkite agent runs unprivileged. It does not, which also made its `USER root` line a no-op. Root is a property of the **deployment**, not the image — `docker run --user`, Compose `user:` and `securityContext.runAsUser` override it, and the Buildkite Helm chart commonly does — so the section says that and gives a check. </details> <details> <summary>Verification</summary> ``` evals/flox 225 tests, OK evals/floxify 599 tests, OK (15 skipped) ``` Screening, `--reps 5`, Opus: baseline 1/5 (judge 1.4, incorrect), skills 5/5 (judge 4.8, correct), gap 3.4, **discriminator**. Container tests: both Dockerfiles in the new section built and run; the root and non-root branches exercised in both states. Live endpoint checks: `get.flox.dev` 200 `text/x-shellscript`; `flox.dev/install` 302 to `/download/`; `install.flox.dev` unresolved pending flox/deltaops#796. </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Line 20 of the
llms.txtpreamble tells agents there is nocurl | bashinstaller and thatinstall.flox.devandflox.dev/installdo not exist.llms-full.txt, served from this same site, carriescurl -fsSL https://get.flox.dev | shsix times. This replaces the prohibition with the command.Review this first: the replacement line itself.
llms.txtis the generated half — regenerated withscripts/generate-llms-txt.sh, ascheck-llms-txt.ymlrequires.Linear: DEV-315
All three clauses are now false
curl | bashinstaller"get.flox.devreturns a ~37KB install script, and it is what/download/anddocs/install-flox/installboth lead withinstall.flox.dev… does not exist"flox.dev/installdoes not exist"/download/since flox/floxwebsite#1111The line was correct when written, and it was added on purpose: agents were inventing install URLs, and naming the two they invented most often was the cheapest way to stop them. Nothing about it was careless. It outlived its facts, which every prohibition eventually does. It matters more here than it would elsewhere in the corpus because the preamble is what an agent reads before it reads anything else.
Why it does not warn about the other URLs
Every prohibition is a fact with an expiry date, and this line is what happens when three of them expire at once. The replacement names one command and the pin, and says nothing about what not to construct.
install.flox.devgoes unmentioned even though it is about to work. Its value is catching people who guess it, which it does whether or not the preamble mentions it.flox.dev/installalso goes unmentioned: it resolves to an HTML page, so it is only dangerous when piped into a shell, and the guard for that belongs in the flox-skills eval suite rather than in a line that has to stay true for years.How this was reported
Found while reviewing flox/floxhub#2366, which led FloxHub's onboarding with the curl installer. The reporter had hit the same wall as a user before joining Flox — several rounds of arguing with an agent that insisted
flox.dev/installwas the way, then that no installer existed at all.Verification
./scripts/generate-llms-txt.shrun and committed: 107 pages in 21 sections, all with descriptions. The diff is two lines, one inllms.txt.headerand its generated counterpart inllms.txt.Live checks behind the table above:
get.flox.dev200 (content-type: text/x-shellscript),flox.dev/download/200 leading with the one-liner,install.flox.devnot yet resolving (pending flox/deltaops#796),flox.dev/installnow 302s to/download/(flox/floxwebsite#1111 deployed since this branch was cut).