-
Notifications
You must be signed in to change notification settings - Fork 479
Meeting Notes
Jeremy Fowers edited this page Jul 23, 2026
·
7 revisions
Attendees: (super)Mario, anditherobot, Anthony Mikinka (AMD), Bone, Daniel Holanda (AMD), fl0rianr, gavinbarnard, geramyl (mod), Jeremy Fowers (AMD), Kalin Ovtcharov (AMD), kenvandine, Krishna (AMD), mikkoph, Occam, Phoenix, primaL- (mod), sawan, TG95, Victoria (AMD)
- LMX omni models exposed to external OpenAI-compatible apps (Open WebUI, etc.), not just the Lemonade app
- Benchmarking tool ships with 10.7; early testers already finding non-optimal params on their machines
- Action: Finalize external app exposure for LMX models before tag
- Hundreds of tests run on every PR; head of main is generally release-ready
- AMD-sponsored hardware covers Strix Halo on Windows and Linux
- Gap: no equivalent CI for Arm, Intel, CUDA
- Robust CI is the gating factor for faster release cadence
- Action: Ken to explore vendor donations of CI hardware via industry contacts
- Strong consensus to move faster with more predictability
- Direction: weekly or bi-weekly stable tags on a fixed weekday (Wednesday preferred, avoid Friday)
- Cherry-picked point releases for urgent fixes
- Add a continuously refreshed "tip" channel (Ghostty-style) for developers wanting main builds
- Snap already does this via edge channel; Debian PPAs similar
- Windows signing capped daily, so signed Windows builds stay on stable cadence
- Unsigned Windows, Docker, Fedora could update daily
- Action: Continue discussion in Discord thread; Jeremy to share Ghostty link
- New structure: six working groups, each led by a maintainer with autonomy to merge PRs and drive roadmap within scope
- Designed to unblock the ~80 open PR backlog and concentrate reviewer enthusiasm
- Some groups have a discrete end state (cross vendor), others are open-ended (omni)
- Separate maintainer table handles areas outside working groups (security, quality, distros, ROCm, etc.)
- Open question: possible new working group for backend / plugin system; Geramyl volunteered to lead
- Action: Leads to draft charters with scope and, where possible, completion criteria
- Led by Jeremy
- Virtual omni via LMX router behind an OpenAI-style interface
- Already shipped: LMX representation, image-gen perf improvements, GUI import/export
- Next: expose to any OpenAI-compatible client (in 10.7)
- Halo Tales reference RPG app is coded, pending release
- Occam noted naming overlap with true omni research models; current design is intentional and gives per-model customization that single omni models lack
- Action: Jeremy to release Halo Tales after 10.7 ships
- Led by Michele
- Splits machine-specific args (global to server) from model-specific args
- Benchmarking tool layered on top to find optimized params
- Goal: community-shared profiles for common archetypes (Strix Halo, dGPU configs)
- Ships in 10.7, CLI only for now
- Not using llama-bench (needs to work with FLM, vLLM and other backends; uses normal inference endpoint instead)
- Custom scenarios supported: short, multiturn, long-context (32K, 64K), coding vs non-coding (relevant for MTP)
- Action: Community help wanted on more benchmark scenarios; UI button for benchmarking to come later
- Led by Ken Van Dine
- Goal: Lemonade runs on all mass-market hardware so app developers can target Lemonade regardless of backend (AMD, NVIDIA RTX/Spark, Intel, Qualcomm, Arm64)
- Discrete checkbox-style scope; dissolves when complete
- Ken to leverage vendor relationships through day job, including potentially sourcing CI hardware
- IoT and embedded (Canonical core devices) in scope; mobile may need Geramy
- Action: Ken to draft the hardware matrix and pursue vendor partnerships
- Led by Krishna
- Open PR adds cloud backend: bring your own OpenAI-compatible API key, provider models appear in Lemonade, chat works seamlessly
- Not about upselling to cloud
- Intermediate step toward intelligent routing between cloud and local based on user-defined policy
- Lemonade owns the platform and interface, not the policy
- Policy format TBD (markdown file, separate HTTP service, etc.)
- AMD research teams looking at this, plus alignment with vLLM Semantic Router
- Geramyl and Sawan asked for an API to retrieve, manage, and secure policies; Krishna agreed
- Action: Land scaffolding PR, then add basic router, then policy API
- Led by Kyle
- Stack stays React plus Tauri to avoid framework-change debate
- New capabilities:
- Self-configuration (LLM loads model and builds recipe)
- Pulls recommended settings from HuggingFace
- Multiple-choice question tool for agent interactions
- Markdown rendering
- Roughly a month from release-ready
- Florian helping; more contributors needed
- Time to market is the differentiator now, not the features themselves
- Action: Kyle to bring a roadmap to next Friday's meeting; testers and UI contributors wanted
- Led by Geramy, with Nathan (kg95) on attestation
- Mesh network vision: install Lemonade and reach your nodes from anywhere; add servers and mobile devices to a private mesh
- Tier 1 nodes require TPM 2.0 hardware attestation
- VMs require TEE; otherwise capped at Tier 2 or lower
- Security model includes ban plus master key rotation on detected abuse (self-healing)
- Router demoed: token-based API, delegates by occupancy and TPS across two Strix Halos plus two Thunderbolt 5 R9700s (over 110 TPS)
- Pulls existing Lemonade collections; supports new ones
- Open proposal: subscription or credits model so mobile users without high-end hardware can use community-donated inference, with a percentage flowing back to Lemonade
- Raises broader question about commercial ventures built on Lemonade and how close they sit to the project itself
- Action: Dedicated discussion next week or later; Geramy to organize separate call for interested parties
- Non-Apple device support landed (Ken, Mario, Florian); big platform expansion
- New website with embedded SDK docs published; improves discoverability and adoption
- Squash-only workflow breaks release-branch tagging; enabling rebase merges alongside squash as a fix
- Default stays squash; rebase reserved for clean, logically-split commits
- Action: Enable rebase merges as a trial next cycle (Mario has admin); maintainers check squash-vs-rebase before merging, don't blindly click
- Proposal to move from semver to date-based versioning (e.g. 2026.07.01) to avoid confusion and breaking-change flak
- Trade-off: loses marketing pop and semver signal, but less mental load given weekly releases
- Action: Mario to post a Discord poll (date-based vs semver), asking voters to justify their choice
- Open PRs up to 90–100; limited reviewer bandwidth, plus duplicate issues/PRs, missing issue-links, and stale PRs (oldest ~4 months)
- Direction: automate cross-linking, auto-close on merge, flag large/new-contributor PRs, auto-tag maintainers, "stand down" tag for intentionally-open PRs; direct complex feature PRs to Discord
- Action: Krishna to expand the issue-labeling framework to handle these cases; issue-linking becomes part of review criteria
- Merged ~7000-line refactor cutting backend touch points from ~30 to ~5; enables adding ~10 more backends quickly
- Inclusion criteria now needed: value, usability, reachability, maintainability
- Action: Apply value/reach/maintainability basis for backend inclusion; formalize later
- Consensus on one source tree with a beta toggle (disabled by default, own folder) instead of forks, which tend to die
- New backends land as beta, get bumped to stable later
- Action: Jeremy to open a PR integrating MLX behind a beta toggle; coordinate with Florian's existing MLX PR
- Kyle wiring stable CDNA on Hot Aisle so Lemonade scales Strix Halo → CDNA and runs large models (GLM 5.2)
- Exploring Lemonade as a LiteRT stand-in and an Android backend for edge/telecom developers
- Action: Kyle to hand CDNA/Hot Aisle wiring to agents (targeting Thursday); chat offline with Krishna on VLLM + CDNA
- Building terminal UI for launch/run/pull; wants design feedback; heavy config should go through a dedicated
onboardflow, not launch - Mesh integration into the TUI requested as first-class
- Action: Open draft PRs for run and pull; gather feedback; coordinate with networking on mesh onboarding
- Strong baseline but scope creep risk (database, user accounts, auto-optimization)
- Strategy: define core launch feature set, disable/hide immature features rather than delay, land MVP sooner; defer database until mesh lands (local JSON for now)
- User accounts debated as arguably Open Web UI's lane; not freezing existing UI yet
- Action: Race GUI 3 to merge readiness; Florian to hide auto-optimization work until it functions; circle back on landing the MVP
- ~75% complete on the current milestone; remaining work is polish, docs, and tests.
- UI design still under discussion: keep the default experience simple for new users, while allowing advanced setups (classifier models, embedding models, multi-source routing) for power users.
- Ken demoed a working prototype, "Nimbus" — a personal cloud appliance (LXD containers) that runs agents (e.g., pico claw) and routes their queries through Lemonade based on policy (image content, message length, or keyword → cloud; otherwise local). Shown live: local-vs-cloud query stats, automatic image-query routing to a GPU-equipped node, zero-config agent onboarding.
- Router policies will ship as downloadable JSON checkpoints on Hugging Face (e.g., cost-saving, privacy-first) that users can use as-is or fine-tune.
- Action: Close out remaining docs/tests/polish for the milestone. Ken to continue iterating on Nimbus and explore a "privacy-sensitive" routing recipe.
- Decision: freeze GUI 2 feature scope — bug-fixing and polish only from here.
- Plan: ship a GUI 2 beta (target Monday) via a new branch + beta tag, then run a 1–2 week feedback/bug-bash window before merging.
- Nexus's UI won't be ready for the beta; the Nexus "bundle" will instead be integrated directly into Lemonade around the same time so it can be tested as its own workstream.
- Noted for awareness only: LM Studio shipped a cloud-integration update this week ("LM Studio Bionic").
- Action: Florian to stand up the beta branch/tag. Team to block time for a bug-bash once the beta ships (Jeremy to review when back from vacation next week).
- One remaining blocker: support for manually created certificates.
- Adding a file-transfer helper library so chat history/files can move between nodes.
- Auth model uses hierarchical, tree-based group permissions (no fine-grained ACLs planned).
- Status: close to functionally done ("barely a day away," per Nate) once the cert blocker clears.
- Practical blocker: needs a TEE-capable machine to host another tier-one node; Jeremy's home server is offline while he's traveling.
- Action: Resolve the manual-certificate gap. Find/allocate a TEE-capable machine for the next tier-one node.
- Article in progress on the new CDNA integration story — prototype locally at low power, then move the same package/OS to CDNA in the cloud.
- Jon from Hot Aisle is reviewing/editing content and related PRs.
- Publish timing likely tied to when the CDNA bundle lands in a Lemonade release (release date still pending).
- Action: Ian/Jon to finish the article; align publish timing with the CDNA bundle's release.
- llama.cpp ROCm backend bump (713 → 714) targeted for the next release (~v11.1);
- Requires the updated llama.cpp revision plus a matching stable-diffusion.cpp update; ABI expected compatible across 713/714, so SD may be tested against both.
- VLLM and Moonshot backend updates are separate items;
- 714 adds hardware support for Gorgon Point devices; other optimizations TBD pending changelog review.
- Problem: PR volume is outpacing maintainer review bandwidth.
- Ideas discussed:
- Stale-bot to auto-close inactive PRs/issues (threshold TBD).
- Cap open PRs per contributor tier — likely 1 (maybe 2) for contributors without write access; no cap for maintainers. GitHub's native PR-limit feature enforces a flat cap + bypass list but can't natively distinguish first-time vs. repeat contributors.
- A bot to auto-close PRs beyond the limit was considered, with concern about how that lands with contributors.
- Contributors could graduate off the limit after a track record of merged, good-quality PRs (e.g., 5–10).
- Ken to check how the Ghostty project handles this — they use a "vouch" system to gate contributions.
- No final policy decided.
- Action: Ken to research the "vouch" model. Team to finalize PR-limit + stale thresholds next week (Mario won't attend — will weigh in async).
- An open PR adding a VTE backend raised the broader question of acceptance criteria for new backends.
- Direction agreed: new backends land as "experimental" (preferred over "beta" — implies less permanence), graduating to first-class/production status only with demonstrated usage and active maintenance.
- UI: add a filter/toggle in the Backend Manager to show/hide experimental backends — hidden by default for new users; prompt users on backend updates about whether to keep something flagged experimental.
- Criteria to weigh: project maturity/maintainer activity, whether it fills a real gap, and security exposure (backends run as subprocesses with access to the Lemonade user's directory/service).
- Action: Florian to file an issue for the experimental-backend UI filter (ties into GUI 3). Team to draft a formal backend-inclusion policy; VTE PR decision pending that policy. Geramy to work on policy doc for experimental category.
- Project is ~30 stars from 5,000 on GitHub — more visibility, more attack surface.
- Recent incident: a GitHub issue drew a convincing comment with a zip attachment from an account created about an hour earlier; Krishna deleted the comment before anyone opened the file.
- Discussed restricting issue/PR comment attachments to safe file types (e.g., images only) — unclear if GitHub allows this to be enforced.
- Discussed a security audit of the Lemonade codebase and the backends it pulls in, plus automation to flag risky changes (noting automation itself can be an abuse vector).
- Noted: an earlier internal scan of the codebase with an early-access Claude model had already surfaced several security items still being worked through.
- Action: Assign an owner for a security audit + scanning policy, ideally before the 5,000-star mark. Revisit alongside the PR-limit/stale-bot policy next week.
- Confirmed: sticking with semantic versioning (per an earlier team poll).
- Current release is v11; next release with GUI3 will be v12 rather than v11.5, since GUI 3 counts as a breaking/major UI change.
- Florian integrated ModelScope as a second model source alongside Hugging Face in GUI 3; users can toggle which source(s) to search.
- Rationale: serves AMD-hardware users in China (no Hugging Face access) and aligns with other silicon partners (Qualcomm, MediaTek, Broadcom) who also use ModelScope.
- ModelScope team is pleased with the integration; a joint promotion is being discussed, possibly including a homepage feature. ModelScope has roughly 200K daily active users, similar to Hugging Face, but is the only accessible option for users in China.
- Raised, not yet started: whether GUI 3 should support localization (e.g., Simplified Chinese) given the growing China-based audience. Flagged as a future discussion item.
- Interest in wiring in LiteRT as a backend, potentially extending Lemonade to more devices (mobile, home routers, edge/telco) — feasible, not yet scoped.
We were fortunate to have the opportunity for a Lemonade meet up at AMD Advancing AI Day yesteday! It has been wonderful to see people here at the conference. Here are some notes:
Attendees: Jeremy, @mikkoph @geramyl @sofiageo @fl0rianr_89165 @clemperorpenguin @.supermario1 @danielhn4744 @ramkrishna2910 @kovtcharov_amd_74552 @avatarh @kenvandine @.sawan. @sreeram.20 Adrian Macias, Kevin Cazabon, Logan Wemyss
We discussed a range of topics from the user personas we serve, to the way we maintain the project, to the future roadmap and working groups.
Lemonade today serves (at least!) 3 personas, with 3 more emergent personas on the horizon.
Current personas:
- Exploring local AI: people getting their first great experience with local LLMs, image gen, etc. Hopefully getting inspired and becoming a builder!
- Personal AI server: connected Lemonade to great apps for productivity, coding, and creativity as part of daily workflows.
- Software vendors: building Lemonade support into their own apps for first-class local AI support.
Emergent personas:
- Personal agent appliance: plug-and-play device that works without a keyboard or monitor, and serves agents over wifi right away.
- Enterprise token server: use local AI to replace cloud AI for the employees of a company.
- Embedded: robotics, automotive, etc. use cases that rely on vision, VLAs, and Lemonade's current capability set.
We have a consensus that the fast pace of the project has allowed bugs, issues, and PRs to pile up. We also want to take a look at future feature scope increases. Our goal is to ensure Lemonade and the core lemond SDK are a consistent experience for contributors, users, and downstream projects.
The GUI3 project is nearly ready to merge! This is a complete overhaul of the Lemonade GUI. It adds many features for users and developers alike, and provides a new foundation for future work.
GUI3 also needs to do an even better job of referring users to state of the art apps like AnythingLLM, Hermes, and OpenCode that use Lemonade as an inference runtime.
We’re going to need everyone’s help with the beta!
AMD has launched agent skills into the official skills marketplaces for Claude, OpenAI, and Cursor. This includes Lemonade-based skills! We have an opportunity to expand this catalogue, which can reach the massive user bases of those companies.
- Improve the marketplace with metadata, mcp, install buttons, and links to more AMD playbooks.
- Extend the mcp to let people completely explore lemonade in natural language.
- Pursue first class support in third party apps (Hermes, OpenCode, etc.).
- Daniel and Sawan to co-found a new Skills working group.
- Daniel to investigate additional dGPU runners for CI.
- Jeremy to found a QA working group.
- Upgraded bug auditing: track amount of live bugs, and trace them back to the PR that introduced them, see if repo-manager caught it or not.
- Code deduplication bot to combat the way agentic coding repeating functionality.
- Geramy: Stress testing long running agents and finding graceful failure scenarios (telling the agent it ran out of memory, ran out of context, etc).
- Add Radeon emulators to CI.
- Declaring victory on the Omni Models working group! The foundation has been built, and we will continue maintaining and extending it over time.
- Daniel and Florian to analyze the apparent overlap between the skills and MCP.
- Krishna to recruit Ken, Sawan, and Geramy to explore the router functionality.
- Florian to launch the GUI3 beta after we return from the conference!