Skip to content

feat(plugins): plugin system v1 + voice-driven meme generator (10-iteration spike, productionized) - #244

Merged
initcore0 merged 3 commits into
mainfrom
feat/plugin-system-v1
Aug 4, 2026
Merged

feat(plugins): plugin system v1 + voice-driven meme generator (10-iteration spike, productionized)#244
initcore0 merged 3 commits into
mainfrom
feat/plugin-system-v1

Conversation

@initcore0

Copy link
Copy Markdown
Owner

Ships the plugin system and the voice-driven meme generator from the
10-iteration spike in #243, brought to production quality.

Every behavior the owner tested in #243 is carried byte-identical. What
changed is everything around it: the build defaults, the developer-only
surfaces, the manifest contracts, CI, and the docs. #243 stays open for you to
close; nothing here is marked do-not-merge.


Provenance

The spike ran ten owner-tested rounds, each fixing something reported against a
running build — the confident-wrong-template fallback, the stuck spinner, the
day-long-uptime download failure, the four-items-became-two-captions bug, and
finally the voice-command route. The full iteration log, including the root
cause of each defect and what was deliberately not done, lives in #243 and
is worth keeping as the design record.

The spike branched from what is still main today, so this is a clean
carry
— no rebase conflicts, and nothing had to be dropped or reworked to
land it.


What's in it

Plugin system

Pure rules in OpenWhispCore (all swift test-covered), IO and AppKit in the
app.

  • PluginManifest — the host/plugin contract. Id validation is strict because
    the id becomes a path component under Application Support.
  • PluginDiscovery — merges an ordered provider list; providers are passed
    in descending trust order and earlier wins, so a user-writable directory
    can never shadow a reviewed plugin. The compile-time registry is one entry in
    that list — the seam a real loader plugs into rather than replaces.
  • PluginEnablement — default-off, with pruning.
  • PluginHost / PluginsPane / menu-bar submenu — none of it on AppState.
  • Three window seams: PluginWindowLifecycle, PluginDictationSink,
    PluginVoiceCommandSink.

The meme plugin

Three merged template providers (imgflip, memegen.link, and the user's own
imported library — the only one that works offline and in any language), scored
search over names and keywords, per-template caption slots, schema-constrained
decoding, a normalized box model where the preview IS the export, an
idempotent ticket-guarded state machine, a rebuildable URLSession, and a
bounded learned affinity.

Voice-command routing

With refine armed, an instruction starting with a manifest-declared phrase goes
to the plugin instead of the refine LLM. Both flows work: selection-as-material
and spoken-remainder-as-material. Matching is prefix-only, word-boundary, exact,
no fuzzy — a match redirects a dictation away from the user's editor, so a
false positive costs them text. Non-match, disabled plugin, or a window that
can't take it all fall through to the untouched normal refine.


What productionizing changed vs the spike

PLUGINS ships on by default

PLUGINS now defaults to 1, matching WHISPERKIT / PARAKEET / SPARKLE;
PLUGINS=0 is the lean escape hatch. Compiled in != enabled — plugins stay
disabled at runtime until you enable one per-plugin in the pane.

The source list moved into scripts/plugin-source-args.sh, shared by build.sh
and build-dmg.sh. build-dmg.sh is the release path and had no plugin
support at all in the spike — left alone, the shipped DMG would have been the
only build with an empty Plugins pane.

scripts/verify-plugins-binary.sh guards package.sh and build-dmg.sh the
way the WhisperKit/Parakeet guards do. Plugins live outside build.sh's
OpenWhisp/ glob, so a broken source list drops every one of them with no
compile error
— the app builds and runs fine with an empty pane. Verified
non-vacuous: it fails on a PLUGINS=0 binary and skips when that was
intentional.

Developer surfaces compiled out of consumer builds

MemeTrace's emission and the launch-gated probes are now behind
OPENWHISP_INSTRUMENTATION — the gate LLMBenchRunner and LLMLabView already
use. An env var that opens a window and drives a generate is a fine debugging
tool and a poor thing to leave reachable in a signed, notarized app holding
Accessibility and microphone grants.

Verified in both directions rather than asserted: [MemeGen] appears 0 times
in a default binary's strings, 1 time with INSTRUMENTATION=1.

The pure line-builders stay compiled — they're swift test-covered and the core
target has no instrumentation define. Only the side effect is conditional.

scripts/meme-runtime-proof.sh was referenced by a doc comment but never
existed. It does now, mirroring the voice-command harness.

MAK-100 contracts — all additive, all forward-compatible

Contract Status
clipboardAccess (bool, default false) Wired, not just declared
destination (enum, default ownWindow) cursor / outputTarget reserved + validated
appAffinity ([String]) Reserved router metadata
voiceTriggers Already shipped in the spike

Clipboard is genuinely gated. The host reads the pasteboard only for a
plugin that declared it, and the check runs before the read — an undeclared
plugin causes no NSPasteboard access at all. The rule lives in the pure,
tested PluginInvocationContext rather than at the AppKit call site where a
forgotten if would be invisible. The pane discloses it beside the network
hosts. The meme plugin declines it deliberately (its Cmd-V import is for
template images, an explicit user action).

Reserved routes are refused honestly. A manifest declaring .cursor reports
.unsupportedDestination, stays valid and runnable, and effectiveDestination
falls back in one place — rather than silently doing something the author didn't
ask for. An unknown destination decodes to the default instead of throwing,
so a manifest written for a future host degrades rather than vanishing.

appAffinity is a hint, never a priority. A test pins that declaring it buys
no ranking advantage, because MAK-100's ~15-tool cap makes trigger surface a
host-arbitrated resource — a manifest must never self-assign priority, for the
same reason networkHosts is a label.

Docs

New docs/PLUGINS.md: architecture, full manifest schema reference, how to
write an in-repo plugin, the security/trust model, and — as a committed
roadmap rather than an open question
— the path to hot-swappable.

Hot-swap roadmap. The shipping requirement is installing a plugin without
rebuilding. In order: (1) manifest/script-driven plugins first, because it's
cheapest and delivers install-without-rebuild on its own (ScriptPostProcessor
and ConfigPack are already this shape); (2) out-of-process executables
next, the only option with real isolation, reusing two precedents this app
already ships — helper binaries at Contents/Helpers/ and the Agent Bridge's
local-socket protocol; (3) WKWebView as an optional UI layer; (4) dylibs
permanently ruled out
, per ROADMAP section 6, which this PR leaves exactly as
it was.

README gains a plugin line and a docs link, ARCHITECTURE gains the plugins/
row, and ROADMAP section 6 gains a status banner plus two table rows.

CI

Two jobs pin opposite sides of the same flag, which is the point:

  • build-app is now pinned to PLUGINS=0. Without the flag it would have
    silently started building with plugins when the default flipped, and the
    lean escape hatch would have rotted untested.
  • build-app-plugins (new) builds the default configuration users actually
    get, then runs the symbol verify. A green build is not evidence the plugins
    linked.

The nightly full-fat package.sh job picks up plugins and the new guard
automatically.


Testing

plugins/ compiles only into the app and sits outside the swift test
target, so the split matters. Which pure core covers what:

Behavior Covered by
Manifest schema, validation, forward-compat decode PluginManifest
Provider merge, precedence, trust ordering PluginDiscovery
Enablement + pruning PluginEnablement
Shortcut assignment + collisions PluginKeyEquivalent
Voice trigger matching, negatives, enablement gating PluginVoiceCommandRouter
Clipboard gate, reserved destinations, affinity-is-not-priority PluginInvocationContext + PluginManifest
The captions-to-boxes decision MemeCaptionSeeding
List extraction from a dictation MemeCaptionExtraction
Ranked parsing, slot enforcement, schemas MemeAI
Search scoring, prefilter, merge, dedup MemeTemplateCatalog / MemeTemplateMatcher
Busy state, tickets, timeouts, transport-failure detection MemeGenerationState
Cache staleness policy MemeCatalogCache
Box geometry, slot layouts, shrink-to-fit MemeCaptionLayout
Learned affinity + its caps MemeTemplateAffinity

Verified none regressed to the app layer during the carry. The decision
logic the spike moved into core is still in core: MemeCaptionSeeding.resolve
still owns the whole captions-to-boxes decision and applyRanked is still a
call to it plus UI glue.

What remains app-layer and untestable by swift test: window lifecycle, the
network round-trip, AppKit rendering, and the async refit. Those are what the
two runtime-proof harnesses exist for.

Gates

  • swift test2685 passing, 0 failures (2670 carried + 15 new contract tests)
  • AppState ratchet — OK at 7024 / 7024
  • ./build.sh (default, now with plugins) — pass
  • PLUGINS=0 ./build.sh — pass
  • lean WHISPERKIT=0 PARAKEET=0 SPARKLE=0 — pass
  • lean + PLUGINS=0 (the CI lean job's exact config) — pass
  • INSTRUMENTATION=1 ./build.sh — pass
  • Zero warnings in any plugin-owned file (113 total on the PLUGINS=0 build vs 112 with plugins — the plugin sources add none)
  • Instrumentation gate verified in both directions on real binaries
  • Plugins verify guard proven non-vacuous

Not verified

Same honest caveat the spike carried: the Mac app can't be screenshotted in
this environment
(the name resolves to the iOS bundle), so no UI was visually
confirmed, and no live LLM round-trip was run against this build. The runtime
proofs in #243 were run against the spike binary; the code they exercise is
carried unchanged here, but they were not re-run after the instrumentation
gating — re-running them now requires INSTRUMENTATION=1 ./build.sh, which is
the one behavioral change to that path.

🤖 Generated with Claude Code

initcore0 and others added 3 commits August 3, 2026 18:26
Carries the owner-tested spike (PR #243, 10 iterations) onto main. This commit
is the spike content verbatim; the production hygiene, the shipping build
defaults, the MAK-100 manifest contracts, CI, and the docs follow as their own
commits on top, so the diff between "what was tested" and "what productionizing
changed" stays readable.

THE SYSTEM

- `PluginManifest` — the host/plugin contract: id, name, version, SF Symbol,
  entry kind, `networkHosts` disclosure, `keyEquivalent` request, and
  `voiceTriggers`. Id validation is strict because the id becomes a path
  component under Application Support, so traversal-shaped ids are refused
  before they are ever joined onto a URL.
- `PluginDiscovery` — merges an ORDERED provider list. Providers are passed in
  DESCENDING trust order and earlier wins, so a writable directory can never
  shadow a reviewed plugin. The compile-time registry is one entry in that list,
  which is the seam a real loader plugs into rather than replaces.
- `PluginEnablement` — the enabled set, DEFAULT-OFF, with pruning.
- `PluginRegistry` — the compile-time list of in-repo plugins.
- `PluginHost` / `PluginsPane` / menu-bar submenu — the app-side surface, none of
  it on AppState (the MAK-32 ratchet is at zero headroom).

THE PLUGIN

A voice-driven meme generator: dictate a description, and it picks a template,
writes the captions, renders them locally, and lets you edit, export, or share.

Three merged template providers (imgflip, memegen.link, and the user's own
imported library, which is the only one that works offline and in any language),
scored search over names + keywords, per-template caption slots, schema-
constrained decoding, a normalized box model whose preview IS the export, a
busy-state machine whose `finish` is idempotent and ticket-guarded, a rebuildable
URLSession, and a learned template affinity that is bounded so it can never
resurrect the confident-wrong-template bug this plugin was built to kill.

Every DECISION lives in `OpenWhisp/Services` and is covered by `swift test`;
`plugins/MemeGenerator/` holds only AppKit/SwiftUI and IO.

Co-Authored-By: Claude <noreply@anthropic.com>
…ion-only probes, MAK-100 manifest contracts

Productionizes the carried spike. Three groups of change, all verified against
real builds rather than by reading.

1. PLUGINS SHIPS BY DEFAULT

`PLUGINS` now defaults to 1, matching WHISPERKIT / PARAKEET / SPARKLE;
`PLUGINS=0` stays the lean escape hatch. Two things this deliberately does NOT
change: plugins remain DISABLED at runtime until the user enables one per-plugin
in Settings → Plugins (`PluginEnablement` defaults to the empty set), and the
pure plugin core is compiled and tested either way.

The source list moved into `scripts/plugin-source-args.sh`, sourced by BOTH
build.sh and build-dmg.sh. build-dmg.sh is the RELEASE path and previously had
no plugin support at all — left as it was, the shipped DMG would have been the
only build with an empty Plugins pane. Sharing the helper is the same pattern
whisperkit-link-args.sh / sparkle-link-args.sh already use, for the same reason.

`scripts/verify-plugins-binary.sh` guards package.sh and build-dmg.sh the way
verify-whisperkit-binary.sh does. Plugins live OUTSIDE build.sh's OpenWhisp/
glob, so a broken source list drops every one of them with NO compile error —
the app builds and runs perfectly with an empty pane. Verified non-vacuous: the
guard fails on a PLUGINS=0 binary and skips when PLUGINS=0 is intentional.

2. DEVELOPER SURFACES ARE COMPILED OUT OF CONSUMER BUILDS

`MemeTrace`'s emission and the two launch-gated probes in AppMain (plus the
window controller's probe hooks) are now behind `OPENWHISP_INSTRUMENTATION`, the
gate `LLMBenchRunner` and `LLMLabView` already use. An env var that opens a
window and drives a generate is a fine debugging tool and a poor thing to leave
reachable in a signed, notarized app holding Accessibility and mic grants.

Verified in both directions rather than asserted: `[MemeGen]` appears 0 times in
a default binary's strings and 1 time in an INSTRUMENTATION=1 one.

The pure LINE BUILDERS stay compiled — they are `swift test`-covered functions
and the core target has no instrumentation define. Only the side effect is
conditional. `scripts/meme-runtime-proof.sh` was referenced by a doc comment but
never existed; it now does, alongside the voice-command harness it mirrors.

3. MAK-100 CONTRACTS (all additive, all forward-compatible)

- `clipboardAccess` (bool, default false) — WIRED, not just declared. The host
  reads the pasteboard only for a plugin that declared it, guarded by
  `needsPasteboard` BEFORE the read, so an undeclared plugin causes no
  NSPasteboard access at all. The rule lives in the pure, tested
  `PluginInvocationContext`; the pane discloses it beside the network hosts. The
  meme plugin declines it deliberately.
- `destination` (enum, default ownWindow) — `cursor` / `outputTarget` are
  RESERVED: validated and reported, never fatal, and `effectiveDestination`
  falls back in one place so a declared-but-unimplemented route is refused
  honestly instead of silently doing something else.
- `appAffinity` ([String]) — reserved router metadata. A HINT, never a priority:
  a test pins that declaring it buys a plugin no ranking advantage, because
  MAK-100's ~15-tool cap makes trigger surface a host-arbitrated resource.

An unknown `destination` decodes to the default rather than throwing, so a
manifest written for a future host degrades instead of vanishing from the list.

Also swept spike-era provenance markers from doc comments and the user-facing
"this prototype" strings, and corrected the `PLUGINS=1` references that inverted
meaning when the flag became default-on.

Gates: `swift test` 2685 passing (2670 carried + 15 new contract tests) ·
ratchet 7024/7024 · default / PLUGINS=0 / lean / INSTRUMENTATION=1 builds all
compile · zero warnings in any plugin-owned file.

Co-Authored-By: Claude <noreply@anthropic.com>
DOCS

`docs/PLUGINS.md` — the reference the plugin system didn't have: architecture
(core/app split, the provider seam, the three window seams), a field-by-field
manifest schema including the new contract fields, how to write an in-repo
plugin today, the security and trust model, and the path to hot-swappable.

Two sections carry decisions rather than description:

- **Security and trust.** Why plugins are runtime-opt-in, why providers are
  ordered by descending trust (a user-writable directory must never shadow a
  reviewed plugin in an app holding Accessibility + mic + clipboard rights), and
  the honest limits — `networkHosts` is disclosure not a sandbox, and the
  clipboard gate is real at the host but cannot bind an in-process plugin.
  Enforcement only becomes meaningful at a process boundary.
- **Path to hot-swappable.** The owner's shipping requirement is installing a
  plugin without a rebuild, so this is written as a committed roadmap rather than
  an open question: manifest/script-driven plugins FIRST (cheapest, and delivers
  install-without-rebuild on its own — `ScriptPostProcessor` and `ConfigPack` are
  already this shape), out-of-process executables NEXT (real isolation, and it
  reuses two precedents this app already ships: helper binaries at
  Contents/Helpers and the Agent Bridge's local-socket protocol), WKWebView as an
  optional UI layer, and dylibs permanently rejected per ROADMAP §6.

Also documents the testing trap the system is shaped around: `plugins/` is
outside the `swift test` target, so a test that re-implements the app's sequence
proves nothing about the chain — extract the decision into one core function and
have both the app and the test call it.

README gains a plugin line and a docs link; ARCHITECTURE gains the `plugins/`
row; ROADMAP §6 gets a status banner and two table rows (v1 shipped,
out-of-process next), with the dylib rejection left exactly as it was.

CI

Two jobs now pin OPPOSITE sides of the PLUGINS flag, which is the point:

- `build-app` is pinned to `PLUGINS=0`. Without the flag it would have silently
  started building WITH plugins when the default flipped, and the lean escape
  hatch would have rotted untested.
- `build-app-plugins` (new) builds the DEFAULT configuration users actually get,
  then runs `verify-plugins-binary.sh`. The build succeeding is not evidence the
  plugins linked — a broken source list yields a working app with an empty
  Plugins pane and no error anywhere — so CI asserts the symbols instead.

The nightly full-fat `package.sh` job picks up the plugins and the new verify
guard automatically; its comment now says so.

Co-Authored-By: Claude <noreply@anthropic.com>
Comment thread .github/workflows/ci.yml
Comment on lines 67 to +95
@@ -82,8 +89,51 @@ jobs:
with:
path: build/sparkle
key: sparkle-${{ runner.os }}-${{ hashFiles('scripts/fetch-sparkle.sh') }}
- name: Build mac app (lean, WHISPERKIT=0 PARAKEET=0)
- name: Build mac app (lean, WHISPERKIT=0 PARAKEET=0 PLUGINS=0)
run: WHISPERKIT=0 PARAKEET=0 PLUGINS=0 ./build.sh

build-app-plugins:
# Compiles the app WITH the in-repo plugins — the DEFAULT configuration, and
# therefore the one users actually get (docs/PLUGINS.md).
#
# Separate from `build-app` rather than folded into it because the two jobs
# pin opposite sides of the same flag: that job proves PLUGINS=0 still
# compiles (the lean escape hatch), this one proves the shipped default does.
# Collapsing them into one build would leave whichever side wasn't chosen
# untested, and the plugin surface is exactly the kind of code that breaks
# without anyone noticing — it lives outside build.sh's OpenWhisp/ glob, so
# dropping it is not a compile error.
#
# Stays lean on the ENGINES (WHISPERKIT=0 PARAKEET=0) for the same reason
# build-app does: the engines are irrelevant to whether the plugin layer
# type-checks, and skipping them keeps the job to minutes instead of an hour.
name: build mac app (with plugins)
runs-on: macos-14
timeout-minutes: 15
@initcore0
initcore0 merged commit 65e868a into main Aug 4, 2026
11 checks passed
initcore0 added a commit that referenced this pull request Aug 4, 2026
…cratchpad editor (#245)

Adds the 2026-08 release entry covering everything shipped after the
2026-07 entry (v1.0.19 → v1.1.0+177), and re-renders changelog.html from
the JSON.

18 entries — 9 features, 6 fixes, 3 improvements — verified against
`git log v1.0.18+173..v1.1.0+177`, docs/PLUGINS.md, and the manifest in
PluginRegistry.swift rather than from PR intent.

The lead story is the plugin system (#244 / MAK-100) split into three
entries, because "there are plugins now", "the first one makes memes",
and "you can drive it hands-free from refine" are three different things
a reader needs:

- Plugins are opt-in twice over — the pane defaults to nothing enabled,
  and the entry says so, along with the networkHosts disclosure being a
  label rather than a sandbox (the honest limit docs/PLUGINS.md states).
- The meme entry discloses the only network use: api.imgflip.com,
  i.imgflip.com and api.memegen.link serve the blank template image;
  captioning and rendering are local, and imported templates need no
  network at all. memegen.link's server-side captioning is deliberately
  unused, and the entry doesn't claim capability the plugin declines.
- The voice-command entry spells out both flows (selection vs. spoken
  remainder), the EN + RU triggers, and the strict prefix/word-boundary
  matching — the reason "create a memo about Q3" is never eaten.

Remaining entries cover the Scratchpad quality editor (MAK-95/96/97),
in-note find (#240), AI actions + file-transcript import (MAK-98/99),
model-readiness UX (MAK-94), the stream-overlay counter (#232), and the
fixes: vocabulary substitutions before translation (#238), ru/uk
detection (#235/#238), app-wide ⌘V (#242), per-keystroke Scratchpad
saves, "Model: Not checked", and the "in 0s" timestamp.

Every `howTo.availability` is "live" — each was checked against the
shipping code path, per the truth contract in docs/changelog/README.md.

stats.testsPassing = 2685 is the measured `swift test` count on this
tree, not a carried-forward number.

changelog.html renders only the latest release (its established
behavior), so July's entries move out of the reference render and stay
canonical in the JSON. The <style> block and page skeleton are
byte-identical; only the content under div.wrap is regenerated.

Gates: swift test 2685 passing, 0 failures (docs-only change — untouched).

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants