Skip to content

chore(terminal): bundle static terminal.so into release artifacts#2205

Merged
theangelperalta merged 3 commits into
lem-project:mainfrom
theangelperalta:chore/terminal-bundle-in-releases
Jun 3, 2026
Merged

chore(terminal): bundle static terminal.so into release artifacts#2205
theangelperalta merged 3 commits into
lem-project:mainfrom
theangelperalta:chore/terminal-bundle-in-releases

Conversation

@theangelperalta
Copy link
Copy Markdown
Collaborator

Summary

Makes the terminal extension actually work from a downloaded AppImage and macOS zip by shipping a self-contained, statically-linked terminal.so in both release bundles.

Stacked on #2204 ("remove committed terminal.so binaries"). Until that merges, this PR's diff includes its commits; review the top commit (bundle static terminal.so into release artifacts) here. Please merge #2204 first.

Why

Release bundles never shipped or loaded terminal.so:

  • ffi.lisp resolves the lib via asdf:system-relative-pathname — the build machine's source tree, absent on a user's machine.
  • The macOS build uses Shinmera's deploy, which only bundles a foreign library that is loaded at build time. The CI runner installs sbcl tree-sitter but not libvterm, so the dlopen failed (swallowed by ignore-errors) and deploy never recorded it.
  • Even when bundled, a dynamic terminal.so referenced /opt/homebrew/.../libvterm.0.dylib by absolute path (verified via otool -L) and deploy does not follow the transitive libvterm dependency.

Approach — static libvterm

Compile terminal.so for releases with libvterm statically linked, producing one self-contained .so with no external libvterm dependency. Then bundling is trivial and identical in spirit on both platforms; no install_name_tool / patchelf / linuxdeploy --library relinking. libvterm is MIT, so static linking is license-clean. The existing extensions/terminal/Dockerfile{,.musl} already document the Linux static recipe.

Key mechanic: once terminal.so is loaded at build time, deploy bundles it next to the executable and reloads it by filename at boot — so a self-contained static .so "just works". No ffi.lisp / core changes are needed.

Changes

  • scripts/build-terminal.lispLEM_TERMINAL_STATIC mode. macOS links libvterm.a directly (prefix from LIBVTERM_PREFIX, else brew --prefix libvterm); Linux wraps -lvterm in -Wl,-Bstatic. Default stays dynamic for source installs.
  • macOS (scripts/macos-deploy.bash, .github/workflows/nightly-builds.yml) — brew install … libvterm; build the static helper before asdf:make :lem; assert terminal.so landed in Lem.app.
  • Linux AppImage (docker/Dockerfile-AppImage, docker/make_appdir.sh) — apt-get install … libvterm-dev; build the static helper before asdf:make; AppRun already sets LD_LIBRARY_PATH=usr/lib; make_appdir.sh now asserts terminal.so reached usr/lib.

Test plan

  • macOS arm64: LEM_TERMINAL_STATIC=1 make terminal-libotool -L shows no libvterm dependency (self-contained, 93 KB). Default make terminal-lib still dynamic-links libvterm.0.dylib.
  • CI nightly-builds.yml (workflow_dispatch) builds both Linux + macOS green; the bundle assertions pass.
  • Confirm Ubuntu 22.04 libvterm-dev provides libvterm.a for the -Wl,-Bstatic link (fallback: build libvterm from source as extensions/terminal/Dockerfile does).
  • Download the nightly AppImage / macOS zip on a clean machine, open a *terminal* buffer, confirm a shell runs.
  • make sdl2 / ncurses / webview (dynamic, source install) unaffected.

Refs #1964, #2060.

…urce

Removes the three prebuilt terminal.so shared objects from version control
and replaces them with a from-source build path, addressing lem-project#1964 ("Open
source software should, by definition, not be distributing binary files").

The build script and `make terminal-lib` target landed in lem-project#2182; this wires
them into the editor build and removes the binaries:

- git rm the linux/arm64, linux/x64 and macosx/arm64 terminal.so files.
- Invoke `terminal-lib` best-effort (`-$(MAKE) terminal-lib`) from the
  ncurses, sdl2, sdl2-ncurses, webview and webview-ncurses targets so source
  installs still get a terminal. A missing libvterm/compiler is non-fatal:
  lem-terminal/ffi.lisp already wraps use-foreign-library in ignore-errors
  and silently disables itself when the library is absent.
- gitignore extensions/terminal/lib/**/*.so so locally-built helpers are
  never re-committed.
- Convert .github/workflows/build-terminal-shared-object.yml from a workflow
  that recompiled and re-committed terminal.so on every push (which would
  reintroduce exactly the binaries this removes) into a verify-only CI check
  that builds terminal.so from source on Linux and macOS and fails if the
  build breaks.

Refs lem-project#1964, lem-project#2060, lem-project#2182.
build-terminal.lisp dropped into the SBCL REPL after a successful build
(exiting only on stdin EOF) and trusted the compiler's exit code without
checking that terminal.so was actually written.

- Quit 0 after a successful build so `sbcl --load` never blocks on stdin.
- Delete any stale .so before compiling, then assert the artifact exists
  afterwards, so a compiler that exits 0 without producing the file is
  treated as a failure instead of a silent no-op.
- Factor the failure path into build-failed (stderr + exit 1).
Release bundles (Linux AppImage, macOS zip) never shipped or loaded
terminal.so: ffi.lisp resolves it from the build machine's source tree via
asdf:system-relative-pathname, and the macOS deploy library only bundles a
foreign library that is loaded at build time -- which failed because the CI
runner lacked libvterm. Even when bundled, a dynamic terminal.so referenced
Homebrew's libvterm by absolute path and would break on a clean machine.

Build a self-contained terminal.so with libvterm statically linked for
releases, so the deploy library (macOS) and LD_LIBRARY_PATH=usr/lib
(AppImage) can load it with no external libvterm dependency.

- build-terminal.lisp: add LEM_TERMINAL_STATIC mode. macOS links
  libvterm.a directly (prefix via LIBVTERM_PREFIX / brew --prefix); Linux
  wraps -lvterm in -Wl,-Bstatic. Default stays dynamic for source installs.
- macos-deploy.bash + nightly-builds.yml: install libvterm and build the
  static helper before asdf:make so deploy bundles it; assert it landed in
  the .app.
- Dockerfile-AppImage: install libvterm-dev and build the static helper
  before asdf:make; make_appdir.sh asserts terminal.so reached usr/lib.

No ffi.lisp/core changes: deploy + LD_LIBRARY_PATH handle runtime resolution
once a self-contained .so is bundled.

Refs lem-project#1964, lem-project#2060.
@code-contractor-app
Copy link
Copy Markdown
Contributor

code-contractor-app Bot commented Jun 3, 2026

✅ Code Contractor Validation: PASSED

=== Contract: contract ===

✓ Code Contractor Validation Result
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📋 Contract Source: Repository

📊 Statistics:
  Files Changed:    3
  Lines Added:      0
  Lines Deleted:    0
  Total Changed:    0
  Delete Ratio:     0.00 (0%)

Status: PASSED ✅

🤖 AI Providers:
  - codex — model: gpt-5.4

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🎉 No violations detected. Great job!
📋 Contract Configuration: contract (Source: Repository)
version: 2

trigger:
  paths:
    - "extensions/**"
    - "frontends/**/*.lisp"
    - "src/**"
    - "tests/**"
    - "contrib/**"
    - "**/*.asd"
  head_branches:
    exclude:
      - 'revert-*'

validation:
  limits:
    max_total_changed_lines: 400
    max_delete_ratio: 0.5
    max_files_changed: 10
    severity: warning

  ai:
    system_prompt: |
      You are a senior Common Lisp engineer reviewing code for Lem editor.
      Lem is a text editor with multiple frontends (ncurses, SDL2, webview).
      Focus on maintainability, consistency with existing code, and Lem-specific conventions.
    rules:
      # === File Structure ===
      - name: defpackage_rule
        prompt: |
          First form must be `defpackage` or `uiop:define-package`.
          Package name should match filename (e.g., `foo.lisp` → `:lem-ext/foo` or `:lem-foo`).
          Extensions must use `lem-` prefix (e.g., `:lem-python-mode`).

      - name: file_structure_rule
        prompt: |
          File organization (top to bottom):
          1. defpackage
          2. defvar/defparameter declarations
          3. Key bindings (define-key, define-keys)
          4. Class/struct definitions
          5. Functions and commands

      # === Style ===
      - name: loop_keywords_rule
        prompt: |
          Loop keywords must use colons: `(loop :for x :in list :do ...)`
          NOT: `(loop for x in list do ...)`

      - name: naming_conventions_rule
        prompt: |
          Naming conventions:
          - Functions/variables: kebab-case (e.g., `find-buffer`)
          - Special variables: *earmuffs* (e.g., `*global-keymap*`)
          - Constants: +plus-signs+ (e.g., `+default-tab-size+`)
          - Predicates: -p suffix for functions (e.g., `buffer-modified-p`)
          - Do NOT use -p suffix for user-configurable variables

      # === Documentation ===
      - name: docstring_rule
        prompt: |
          Required docstrings for:
          - Exported functions, methods, classes
          - `define-command` (explain what the command does)
          - Generic functions (`:documentation` option)
          Important functions should explain "why", not just "what".
        severity: warning

      # === Lem-Specific ===
      - name: internal_symbol_rule
        prompt: |
          Use exported symbols from `lem` or `lem-core` package.
          Avoid `lem::internal-symbol` access.
          If internal access is necessary, document why.

      - name: error_handling_rule
        prompt: |
          - `error`: Internal/programming errors
          - `editor-error`: User-facing errors (displayed in echo area)
          Always use `editor-error` for messages shown to users.

      - name: frontend_interface_rule
        prompt: |
          Frontend-specific code must use `lem-if:*` protocol.
          Do not call frontend implementation directly from core.
        severity: warning

      # === Functional Style ===
      - name: functional_style_rule
        prompt: |
          Prefer explicit function arguments over dynamic variables.
          Avoid using `defvar` for state passed between functions.
          Exception: Well-documented cases like `*current-buffer*`.

      - name: dynamic_symbol_call_rule
        prompt: |
          Avoid `uiop:symbol-call`. Rethink architecture instead.
          If unavoidable, document the reason.

      # === Libraries ===
      - name: alexandria_usage_rule
        prompt: |
          Alexandria utilities allowed: `if-let`, `when-let`, `with-gensyms`, etc.
          Avoid: `alexandria:curry` (use explicit lambdas)
          Avoid: `alexandria-2:*` functions not yet used in codebase

      # === Macros ===
      - name: macro_style_rule
        prompt: |
          Keep macros small. For complex logic, use `call-with-*` pattern:
          ```lisp
          (defmacro with-foo (() &body body)
            `(call-with-foo (lambda () ,@body)))
          ```
          Prefer `list` over backquote outside macros.
📚 About Code Contractor

Declarative Code Standards That Learn and Improve

Define domain-specific validation rules in YAML.
Your contracts document team knowledge and evolve into more accurate AI enforcement.

Want this for your repo?
Install Code Contractor

Copy link
Copy Markdown
Contributor

@code-contractor-app code-contractor-app Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Contractor validation passed ✅ — see the sticky comment for full results.

@theangelperalta theangelperalta merged commit 7101d0b into lem-project:main Jun 3, 2026
12 checks passed
@theangelperalta theangelperalta deleted the chore/terminal-bundle-in-releases branch June 3, 2026 01:30
theangelperalta added a commit that referenced this pull request Jun 3, 2026
PRs #2204 and #2205 merged in the reverse of their stacking order (#2205,
which contained #2204's commits, merged first; #2204 merged second). Because
#2205 had added static-p/vterm-flags just above build-failed, the squash of
#2204 re-applied the build-failed block instead of recognizing it as already
present, leaving two identical definitions in build-terminal.lisp.

Remove the duplicate. No behavior change (the copies were identical); this
just silences the SBCL redefinition warning.
theangelperalta added a commit that referenced this pull request Jun 3, 2026
* feat(nix): build and bundle terminal.so in the flake

The Nix flake never built the lem-terminal native helper: it has no
libvterm input and no terminal.c build step, so on Nix the terminal
extension silently disabled itself (ffi.lisp's use-foreign-library is
wrapped in ignore-errors). Removing the committed binaries in #2204 did not
regress this -- the prebuilt .so was dynamically linked against a libvterm
that was never present in the Nix sandbox -- but it also left Nix without a
working terminal.

Add a terminal-so derivation that compiles extensions/terminal/terminal.c
against pkgs.libvterm, mirroring the existing ts-wrapper / c-webview C
derivations, and add it to the nativeLibs of lem-ncurses, lem-sdl2 and
lem-webview.

Dynamic linking is the idiomatic choice on Nix (unlike the AppImage/macOS
bundles, which static-link in #2205): the stdenv records an RPATH to the
pinned libvterm store path, so terminal.so resolves libvterm at runtime with
no bundling or relinking, and nativeLibs puts terminal.so itself on the
library path that ffi.lisp's "terminal.so" lookup uses at build and run time.

* fix(nix): gate terminal-so on Linux (libvterm is Linux-only in nixpkgs)

nixpkgs' libvterm (0.99.7) has meta.platforms = Linux only, so referencing
pkgs.libvterm unconditionally made `nix flake check --all-systems` fail at
evaluation on the aarch64-darwin / x86_64-darwin systems, breaking every
build job.

Add terminal-so to nativeLibs only when stdenv.isLinux (via lib.optionals,
which doesn't force its list when the condition is false). Terminal works on
Linux Nix builds; Darwin Nix is unchanged (no terminal, as before, since
nixpkgs doesn't package libvterm for Darwin).

* fix(nix): add glib cflags for terminal.so (nixpkgs libvterm is the neovim fork)

nixpkgs' libvterm (0.99.7) is the Neovim fork, whose vterm.h #includes
<glib.h>. terminal.c includes vterm.h, so the build failed with
"fatal error: glib.h: No such file or directory".

Add pkg-config + glib and pull GLib's compile/link flags via
`pkg-config --cflags --libs glib-2.0`.

* fix(nix): use libvterm-neovim, not libvterm (wrong library)

The compile failed on glib.h then curses.h from libvterm-0.99.7's vterm.h
because pkgs.libvterm is the old, abandoned glib/curses-based "libvterm" with
an API incompatible with terminal.c. Leonerd's modern libvterm -- the one
terminal.c targets -- is packaged as pkgs.libvterm-neovim.

Switch to pkgs.libvterm-neovim and drop the glib/pkg-config workaround.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant