fix(llm): detect Lemonade on macOS; stop printing commands that don't work

[kovtcharov]

GAIA told macOS users their Lemonade Server was not installed while it was answering requests on the very port GAIA probes, and told everyone else to fix a stopped server by running lemonade-server serve — a CLI modern Lemonade removed. Both are the same defect: code reporting a state that reality contradicts, then handing the user a way out that doesn't work. That reads as "GAIA is broken" rather than "that advice is stale."

Detection. resolve_lemonade() probed canonical paths for Windows and Linux only. On macOS it fell through to the legacy lemonade-server probe, found nothing, and returned found=False. check_installation() delegates straight to it, so gaia init concluded Lemonade was absent on a working machine. There is now a Darwin branch probing /usr/local/bin/lemond (plus the Homebrew prefix, then lemond on PATH). It probes the daemon, not the client — the daemon is what gets started; the client only serves the version query.

Remedies. Two resolver-backed helpers replace six hand-rolled copies of the same resolve-then-fall-back-to-legacy logic: describe_start_hint() for starting the server and describe_client_hint(action, model) for pulling/loading a model. Both return prose with no command when the platform has no command a user would run (Windows tray, macOS app, no client on disk) rather than inventing one. lemonade-server is only ever named when a legacy install is actually detected.

Fixing the strings without fixing detection would just have swapped one wrong answer for another: "install Lemonade" for a user who already has it running.

build_start_command() starts the daemon directly on Darwin — macOS has no systemd, so the Linux systemctl form must not leak there. LEMONADE_SERVER_PATH still wins and is used verbatim for the server; it does not double as the client. lemonade-server stays in the legacy probe list: it is still the correct package name (the MSI and .deb are lemonade-server-*).

Four defect layers, each found by running the command rather than reading it

1. Command doesn't exist — lemonade-server serve on a modern install.
2. Command exists but rejects the argument — gaia download SD-Turbo exits 2; gaia download has no positional model argument.
3. Command runs and still doesn't fix it — the SD remedy hardcoded SD-Turbo while the mixin default is SDXL-Turbo, so a user pulled the wrong model successfully and stayed stuck. An exit-code check passes on this one.
4. The tool reports success while refusing to act — gaia kill with no target printed ❌, did nothing, exited 0, so gaia kill && next-step proceeded. Same for gaia cache clear. Both now exit 1, matching gaia install.

Layer 3 was hiding behind a stale docstring claiming the SD default was SD-Turbo, which made the literal look correct in review. That docstring is corrected too.

Why an installer fix touches eight files

Every non-test file is the same dead-command fix threaded through a caller — no unrelated cleanup:

File	Sites	What
llm/lemonade_launcher.py	—	The fix: Darwin detection + both remedy helpers
installer/init_command.py	3	The originally reported sites
llm/lemonade_client.py	3	Had its own copy of the legacy fallback (_start_command_hint), now delegating
llm/lemonade_manager.py	5	More copies of the dead start command in user-facing errors
cli.py	7	Remedies, plus the two exit-0-on-refusal bugs (layer 4)
code_index/sdk.py	2	lemonade-server load <model> and a gaia download <model> template
sd/mixin.py	1	lemonade-server pull, now naming the configured model

Audit: every surviving lemonade-server literal in src/gaia

Walked all 47. None is a command a message tells a user to run, with one known exception:

Category	Count	Verdict
Comments / docstrings describing the legacy CLI	25	✅ prose
Deliberate legacy probe (which, _classify_kind_from_name, legacy argv)	3	✅ intentional
Package name (apt/PPA install + remove, MSI/.deb filenames)	5	✅ correct — package name is unchanged
Diagnostics stating it was not found ("no lemonade-server in PATH")	6	✅ prose, remedy is gaia init
Log messages naming the process	7	✅ not instructions
providers/lemonade.py:117 — "Restart cleanly: gaia kill && lemonade-server serve"	1	❌ dead, eval-gated (below)

Test plan

• pytest tests/unit/ — 31 vs 31, identical failure names, Lemonade confirmed down across both runs
• macOS detection tests inject platform.system() and Path.exists(), so Windows/Linux/macOS answers are provable on any runner, and drive the real resolver (patching resolve_lemonade would only prove it was called). 10 fail without the fix.
• Remedies parsed against the real build_parser(), plus a static guard failing on any new gaia download <model> — verified by planting the regression
• SD remedy parameterised over all four models; 4 of 5 fail against the old literal (the SD-Turbo case passes, which is why the bug was invisible)
• Refusal exit codes tested by running the real CLI in a subprocess — an in-process call cannot observe an exit code. 3 of 4 fail against the old behaviour.
• python util/lint.py --all passes

Every command this PR emits, actually run:

Command	exit
LEMONADE_CTX_SIZE=32768 /usr/local/bin/lemond --help	0
lemonade pull --help / SD-Turbo registered in lemonade list	0 / 0
lemonade load --help	0
gaia init --help	0
gaia download --clear-cache (parse)	0
hash -r	0
gaia kill (no target) — was 0	1
gaia cache clear (no target) — was 0	1
gaia kill --port 1 (nothing listening) — must stay chainable	0
gaia download SD-Turbo — the removed regression	2
command -v lemonade-server — the removed legacy CLI	1

Detection, before vs after, on a Mac with lemond v10.10.0 healthy on 13305:

                      BEFORE                          AFTER
resolve_lemonade      found=False kind=none           found=True kind=modern server=/usr/local/bin/lemond
build_start_command   ValueError: no tooling found    StartSpec(argv=['/usr/local/bin/lemond'])
check_installation    installed=False                 installed=True version=10.10.0

• Reviewer on Windows/Linux: confirm the tray prose (Windows) and systemctl --user start lemond (modern Linux) read correctly in gaia init when auto-start fails

Deliberately out of scope

• gaia kill --port <n> with no process still exits 0. "Already stopped" is the desired end state, and docs/reference/troubleshooting.mdx documents gaia kill --lemonade && ..., which a non-zero exit would break for the common case. Telling "nothing to kill" apart from "kill failed" needs kill_process_by_port to return more than a bool — a deliberate semantics change, not a drive-by. It does still print ❌ for that benign outcome, which is worth fixing separately.
• gaia kill cannot target a stray agent sidecar by name or install path, which is why the daemon's uninstall guard naming it can dead-end. The refusal now says so explicitly rather than leaving the user to retry. Whether gaia kill should grow that capability is a product call for the daemon owner.
• 3 gaia download <model> sites in llm/providers/lemonade.py (×2) and agents/builder/agent.py — all LemonadeError.user_message text, returned verbatim as the agent's answer by agents/base/agent.py:3494, so CLAUDE.md's eval gate applies and they need a gaia eval agent run. Pinned in the guard's KNOWN_EVAL_GATED_SITES, with a second test that fails if an entry goes stale.
• cli.py:4025 runs subprocess(["lemonade-server", "stop"]) in gaia kill --lemonade. Modern lemonade has no stop subcommand, so this always falls back to the port kill — it degrades correctly, but the attempt is dead code on modern installs.
• 5 log warnings in agents/base/ say "start lemonade-server". Fixing them means importing gaia.llm.* into the agent base layer, which its documented invariant forbids.
• apps/webui/** (TypeScript) carries the same stale string but cannot call a Python resolver; it needs a field on the status API.
• tui/** — owned by a sibling change.

Unrelated observation: the 3 test_init_command.py install-path tests do a real health probe against localhost:13305, so they pass or fail purely on whether Lemonade happens to be running — the sole cause of the 31-vs-34 baseline drift seen locally.

[github-actions]

Verdict: Request changes (one blocking issue — a small, contained fix).

This PR fixes a genuinely user-hostile bug: when Lemonade auto-start fails, GAIA was telling users to run lemonade-server serve — a CLI modern Lemonade removed — leaving them stuck. It centralizes all "here's how to start it" advice into one describe_start_hint() helper that emits a command the host can actually run, or GUI prose (Windows tray / macOS app) when there's no shell command to give. The design is clean and the test coverage of the new helper is thorough.

The one thing to fix before merge: this change breaks an existing test that isn't in the PR. test_validate_context_size_insufficient in tests/test_lemonade_client.py asserts the restart hint mentions a context size — but the new helper (correctly) emits prose with no ctx-size when no Lemonade is installed, so the test now fails on any Lemonade-less machine, which is exactly the new-user / CI state this PR targets. The PR's own evidence bundle caught this and flagged it as a follow-up; per GAIA's rule that a behavior change updates every test describing it, it should be fixed here, not deferred. The fix is a one-line test change (mock the resolver to a modern install so the assertion stays meaningful).

Real-world evidence

An evidence bundle ran on ubuntu-latest (no Lemonade/GPU) and exercised the real CLI + library. It's strong and I'm relaying it as-is — it both supports the fix and surfaces the blocking issue above.

• Not-installed state (gaia llm "hello" with nothing installed): describe_start_hint() returns Lemonade Server is not installed. Run gaia init ... , command=None — no fabricated lemonade-server serve. ✅
• Modern install via LEMONADE_SERVER_PATH override, through the real gaia llm entry point:

  • Run: LEMONADE_CTX_SIZE=65536 /tmp/fake_lemonade/lemonade
  

  Previously printed lemonade-server serve --ctx-size .... ✅
• Legacy install (real shutil.which probe): Run: /tmp/fake_legacy/lemonade-server serve --ctx-size 8192 — the CLI is still named when it genuinely exists. ✅
• Spot regression: gaia --help, gaia init --help, gaia download --help all exit 0; all six files with new imports import cleanly.
• Screenshot: N/A — CLI/library stdout+stderr change, no Agent UI route touched.
• ⚠️ Net-new failure the bundle found (confirmed with a before/after worktree diff):

  FAILED tests/test_lemonade_client.py::TestLemonadeClientMock::test_validate_context_size_insufficient
  

  Parent d6c02c2f: 4 failed. This PR 015dd027: 5 failed — this one test flips from pass→fail. This is the blocking issue above.

🔍 Technical details

🟡 Important

Introduced test failure — test_validate_context_size_insufficient not updated for the new behavior (tests/test_lemonade_client.py:1509)

The assertion:

self.assertTrue(
    "--ctx-size" in error or "LEMONADE_CTX_SIZE" in error,
    f"restart hint missing ctx-size guidance: {error}",
)

relied on the old _start_command_hint no-tooling fallback (f"lemonade-server serve --ctx-size {ctx_size}"), which always contained --ctx-size. The new describe_start_hint() intentionally returns prose ("Lemonade Server is not installed. Run \gaia init`...") when resolve_lemonade()finds nothing — so on any Lemonade-less host (CI runners included)error` no longer contains a ctx token and the test fails. This is not flaky ordering; it fails deterministically wherever Lemonade isn't installed. The file isn't in the PR diff, so the PR ships red.

Fix by mocking the resolver so the assertion tests what it means to test — that a resolved install's hint carries the ctx size — rather than depending on the host:

        from gaia.llm.lemonade_launcher import LemonadeTooling

        with patch(
            "gaia.llm.lemonade_launcher.resolve_lemonade",
            return_value=LemonadeTooling(
                found=True,
                kind="modern",
                client_path="/usr/bin/lemonade",
                server_launcher="/usr/bin/lemond",
            ),
        ):
            valid, error = self.client.validate_context_size(
                required_tokens=32768, quiet=True
            )

        self.assertFalse(valid)
        self.assertIsNotNone(error)
        self.assertIn("4096", error)
        self.assertIn("32768", error)
        self.assertIn("LEMONADE_CTX_SIZE", error)

(Add patch platform.system to "Linux" too if you want it deterministic across dev machines.)

Strengths

• Single source of truth. Collapsing four hand-rolled resolve-then-fall-back copies (including one inside LemonadeClient._start_command_hint, now a thin delegator) into describe_start_hint() is the right call — the same stale literal was duplicated across CLI, manager, client, and code index.
• Correct restraint on the string swap. The PR keeps lemonade-server serve where a legacy install is actually resolved and keeps lemonade-server as the package name — this isn't a blanket find-replace, and the tests pin both directions.
• Windows/POSIX rendering care. _render_command avoiding shlex.join's POSIX single-quoting on Windows (and rendering env as set VAR=... &&) is a real correctness detail, and test_start_hint_legacy_windows_path_is_joined_for_cmd_not_posix / ..._renders_env_the_cmd_way_never_drops_it lock it in.
• Honest, self-critical evidence bundle — it found and reported the regression above rather than glossing it.
• The two deferred follow-ups (providers/lemonade.py under the eval gate; the missing macOS probe in resolve_lemonade()) are correctly scoped out with reasons.

[kovtcharov-amd]

Nice cleanup, and CI is green. The macOS Darwin detection correctly probes the daemon (lemond) rather than the client, build_start_command fails loudly instead of guessing when a modern macOS launcher is missing, and the sys.exit(1) refusals aren't swallowed by the surrounding except Exception (SystemExit is a BaseException).

One follow-up worth doing: tests/test_lemonade_client.py::test_validate_context_size_insufficient (not in this diff) becomes host-dependent after the change. It asserts the restart hint contains --ctx-size or LEMONADE_CTX_SIZE, but describe_start_hint(ctx_size).instruction now returns a ctx-free string whenever the resolver finds no runnable command (no Lemonade installed, or a modern Windows-tray / macOS-app install). It passes on CI because the runner has a modern non-.exe install, but a contributor on Windows-tray or a Lemonade-less box will see it go red. Make it hermetic by patching resolve_lemonade to a modern-Linux install so the assertion tests the intended path.

(The remaining docs/ references to lemonade-server serve are the same dead command, correctly scoped out to a follow-up per the PR body.)