Skip to content

The Screen

Jump to bottom
Elliot Boney edited this page Jun 24, 2026 · 4 revisions

The Screen

shelldon has a face. It lives on a 2.13" E-Ink panel wired to a Raspberry Pi Zero 2W, and it shows three things at once: a battery readout, an expression, and a one-line thought. This page documents the whole display path — from the bus snapshots core pushes, through the renderer that paints the glass, down to the way the model deliberately picks its own face and what it's thinking.

If you haven't yet, skim Architecture for the actor/bus model and How a Turn Works for where the lifecycle faces come from. The expressions are driven by mood, which is covered in Personality & Autonomy.

The display is a long-lived bus receiver

The screen is its own process — an actor named DISPLAY that connects to the message bus and never sends anything back. It is the first (and still the canonical) pure receiver in the system: it consumes state snapshots and emits nothing.

Core is the sole writer of everything on the glass. It never shares memory with the display; instead it pushes a snapshot every time something should change, and the display renders what arrives. Each snapshot is a frozen StateSnapshot carrying a region id, a monotonic seq, and a string token:

# shelldon/contracts/__init__.py
class StateSnapshot(...):
    region: Region   # FACE | CAPTION | BATTERY | STATUS_BAR
    seq: int         # per-region monotonic sequence
    face: str        # the token / text for that region

Region is a closed enum, not a free string — a typo can't silently mint a new region:

# shelldon/contracts/__init__.py
class Region(StrEnum):
    FACE = "face"           # core-owned: the expression
    STATUS_BAR = "status-bar"  # plugin-claimed widget (e.g. the XP counter)
    CAPTION = "caption"     # core-owned: the bottom thought strip
    BATTERY = "battery"     # plugin-claimed: PiSugar charge widget

Latest-wins per region, coalescing under a slow panel

E-Ink is slow. A full refresh on this panel is roughly two seconds. If the display naively queued every snapshot and drew them in order, a burst of mood drift during a turn would build a backlog and the screen would lag the soul by many seconds.

The display service avoids that with a single pending slot per region, not a queue. The logic lives in shelldon/display/service.py (run_display) and runs two concurrent loops:

  1. Intake loop — reads frames off the bus. For each snapshot it applies latest-wins by seq, per region: it tracks latest_seq[region] (the highest seq ever accepted, drawn or not). A snapshot whose seq is not strictly greater than that is dropped at the door (stale or duplicate). Otherwise it overwrites pending[region] with the new snapshot and signals the render loop. A malformed frame is skipped, never fatal — a display crash kills the screen, not the soul.
  2. Render loop — waits for the signal, atomically takes and clears all pending regions, then awaits the slow renderer.render(...) for each. This is the coalescing seam: while a draw is in flight, the intake loop keeps overwriting the (now-cleared) pending slot with newer snapshots. When the draw finishes, the render loop picks up only the latest per region. Intermediate frames never reach the glass — there is no backlog, and memory stays bounded to one frame per region.

The net effect: the panel always shows the most recent state core wanted, and rapid reflex churn coalesces away instead of flickering.

The panel and the renderer

The physical device is a Waveshare 2.13" V4 E-Ink HAT — 122×250 pixels, one-bit mono (black/white). The production Renderer that drives it is WaveshareRenderer in shelldon/display/waveshare.py. The vendored Waveshare driver (lifted from shelldon's v1, MIT-licensed) lives in shelldon/display/drivers/epd2in13_V4.py and epdconfig.py.

Hardware is lazily imported and fails soft

The renderer is built so the module imports cleanly on a laptop with no display hardware. Everything Pi-specific — pillow, spidev, gpiozero + lgpio, and the vendored driver — is imported inside the methods that touch hardware, never at module load. Those are component-local install-time deps (uv pip install pillow spidev gpiozero lgpio rpi-lgpio on the Pi), not part of the locked dependency set; uv sync --locked stays at zero new deps. The app.py gate _default_renderer(env) picks WaveshareRenderer when SHELLDON_DISPLAY=waveshare, else the recording StubRenderer used for tests and headless runs.

The panel is initialised lazily on the first render (the init() is slow), and each draw runs in a worker thread via asyncio.to_thread so the ~2s E-Ink refresh never stalls the event loop. A draw that throws is logged and the frame is skipped — a display failure must never take down the soul:

# shelldon/display/waveshare.py — _draw_blocking
try:
    epd = self._ensure_panel()
    image = self._render_image()
    epd.display(epd.getbuffer(image))
except Exception as exc:
    log.warning("waveshare draw failed (%s); skipping frame", exc)

Text-as-faces with GNU Unifont

The faces are chunky Unicode expressions drawn as text, not bitmap sprites. Each token maps to an expression string in FACE_ART, and they're rendered with GNU Unifont — chosen because it has full Basic-Multilingual-Plane coverage, so every glyph (flowers, gears, combining diacritics, Thai marks) renders with no missing-glyph "tofu", and its bitmap look suits the panel. The font path is overridable via SHELLDON_FACE_FONT (default /usr/share/fonts/opentype/unifont/unifont.otf).

# shelldon/display/waveshare.py
FACE_ART = {
    "content":     "(•‿•)",
    "happy":       "(◠‿◠✿)",
    "excited":     "٩(⚙ᴗ⚙)۶",
    "curious":     "٩(๏̯๏)۶",
    "thinking":    "Σ(-᷅_-᷄ ๑)",
    "sleepy":      "(_ _ ) Zzz z",
    "grumpy":      "(>_<)",
    "cant-think":  "(⊙_◎)",
    "low-battery": "(u_u)",
}

An unknown token renders as its own text (face_for(token) falls back to the token string), so a self-modified face the owner added by chat still shows something legible even before its art is hand-tuned.

The face system

A face token comes from one of two places: the mood the pet is drifting through, or a deliberate reaction the model picked for a specific message.

Mood-derived faces from a self-modifiable registry

Between turns, core maps the pet's drifting mood to a face token. The registry lives in shelldon/core/faces.py and is backed by an editable ~/.shelldon/faces.toml — faces are data, not a hardcoded enum. The owner can tune them by hand, and the bot can extend them by chat (see below). On first run the file is seeded with six starter emotions; a corrupt or invalid file falls back to the built-in defaults and logs a warning, never crashing.

Each entry is a name plus the mood region that selects it (valence / arousal / energy ranges) plus an optional render token:

# shelldon/core/faces.py — DEFAULT_FACES (order = selection priority, first match wins)
Face("low-battery", (-1, 1), (-1, 1),    (0.0, 0.15)),
Face("sleepy",      (-1, 1), (-1, -0.3), (0.15, 0.45)),
Face("grumpy",      (-1, -0.2), (-1, 1), (0.15, 1.0)),
Face("excited",     (0.4, 1), (0.4, 1),  (0.5, 1.0)),
Face("curious",     (0.0, 1), (0.1, 1),  (0.15, 1.0)),
Face("content",     (-1, 1), (-1, 1),    (0.15, 1.0)),  # broad catch-all, MUST stay last

select_face(faces, valence, arousal, energy) is a pure function: it returns the token of the first entry whose ranges all contain the current mood, defaulting to content. Core resolves this between turns and pushes the token through _push_face. The content face is a deliberate catch-all kept last so selection always resolves.

Self-modifying faces by chat

The pet can add or tweak its own expressions. When the owner asks for a new face, the worker proposes a structured add_face op on its Result (workers never write to disk — only core does). Core validates it against the closed face schema (non-empty name, well-ordered in-range tuples, no duplicate unless replace=True) via FaceRegistry.add_face and, on success, atomically rewrites faces.toml preserving the owner's comments (using tomlkit for a read-modify-write). A malformed proposal is rejected without mutating RAM or disk, the turn survives, and the reply is unaffected. The new face is selectable on the next mood match.

The reaction face: the model picks its own expression

The face on the screen after a reply is not the ambient mood — it's the expression the model deliberately chose for that message. The worker prompt asks the model to emit a hidden FACE: line; the worker pulls it out (and strips it from the chat reply) into Result.face.

Core then validates that pick against a closed palette before drawing it — arbitrary model text never reaches the panel as a "face":

# shelldon/core/runtime.py
_REACTION_FACES = frozenset({"happy", "excited", "curious", "content", "grumpy", "sleepy"})

def _reaction_face(self, face: str) -> str:
    token = face.strip().lower()
    return token if token in _REACTION_FACES else FACE_REPLY  # FACE_REPLY = "happy"

The lifecycle/system tokens (thinking, cant-think, low-battery) are not in the palette — those are core-driven states, not reactions the model gets to choose. An unknown or absent pick falls back to the default reply face (happy).

The status bar

The panel is a single framebuffer — every epd.display() call repaints the whole screen, so drawing one zone alone would erase the others. To fit a battery readout, the face, and a thought caption onto that one panel, WaveshareRenderer is a stateful compositor that carves the glass into three stacked, non-overlapping zones and recomposites the full canvas on every snapshot.

Three core/plugin regions feed those zones; a snapshot for any other region (e.g. the XP STATUS_BAR widget) is ignored rather than fighting the face for the framebuffer:

# shelldon/display/waveshare.py
_BATTERY_H = 18    # top strip
_CAPTION_H = 22    # bottom strip
_ZONES = (Region.FACE, Region.BATTERY, Region.CAPTION)

The renderer keeps the latest text per zone in self._zones. When a snapshot updates one slot, _render_image() repaints all known zones onto one white canvas: the face big and centered in the middle band, the battery small and right-aligned in the top strip, the caption shrunk-to-fit and centered in the bottom strip. A zone with no snapshot yet (e.g. no battery on a panel without the HAT) is simply not drawn.

The panel layout

The driver rotates a landscape image onto the portrait panel, so faces are drawn wide (250×122 canvas):

 ┌──────────────────────────────────────────────────────┐  ← y=0
 │                                         87%⚡          │  BATTERY  (18px, top-right)
 ├──────────────────────────────────────────────────────┤  ← y=18
 │                                                        │
 │                                                        │
 │                  ٩(๏̯๏)۶                               │  FACE  (centered, auto-sized)
 │                                                        │
 │                                                        │
 ├──────────────────────────────────────────────────────┤  ← y=100
 │              huh, that's interesting…                  │  CAPTION  (22px, centered, fit)
 └──────────────────────────────────────────────────────┘  ← y=122
   0                                                  250

The battery widget

The top-right zone is the PiSugar battery widget, owned by the battery plugin (shelldon/plugins/battery.py). Like every plugin it imports only contracts + the plugin manifest — never core/ — and does its I/O in a host-owned background loop.

It reads the PiSugar2 UPS charge over a local socket. The PiSugar power server speaks a tiny line protocol on 127.0.0.1:8423 (get battery → battery: 100), which the plugin queries with a plain asyncio socket — zero new deps, no nc/echo subprocess. The charge moves slowly and a full refresh is ~2s, so it polls lazily at 60s intervals; a frequent poll would flash the panel for no new information.

The widget text is the charge percent plus a bolt glyph when on external power (_format → "87%⚡"). The plugin claims the BATTERY region and the pisugar:8423 resource — the host rejects a second claimant of either at load (single-writer). On a box with no PiSugar (a laptop, an unplugged HAT) the connect fails, the tick is skipped, and the widget just stays blank — nothing crashes.

The caption strip

The bottom zone is the caption — the short "what I'm doing / feeling / just said" line that rides alongside the face. It's core-owned (sole writer = core) and pushed via _push_caption on the Region.CAPTION stream. It updates on every reply, dream, heartbeat, degrade, and mood drift, giving the v1 desk-pet feel of a creature with an inner monologue.

On a real reply the caption shows the model's hidden THOUGHT: line — a short distilled inner thought, parsed by the worker into Result.blurb, separate from the chat reply the owner reads. If the model didn't write a thought, core falls back to a truncated first line of the actual reply (_caption_for, capped at 48 chars with an ellipsis, font auto-shrunk to fit). Pushes are deduped — an identical caption is skipped so the panel doesn't re-flash. Because the caption is purely cosmetic, _push_caption is internally guarded: a bus hiccup there can never abort a turn.

Reaction dwell: faces and thoughts linger

There's a timing problem the dwell mechanism solves. After a reply, core pushes the reaction face and the thought — but the very next reflex tick would resolve the ambient mood and immediately overwrite both. Without protection, the deliberate expression and thought would only flash before settling back to mood.

So a reaction holds the screen for ~60 seconds before the at-rest mood is allowed to replace it. When core pushes a reply caption it passes a dwell, which sets a hold deadline:

# shelldon/core/runtime.py
_REACTION_DWELL_S = 60.0

# in _handle_result, after a successful reply:
await self._push_face(self._reaction_face(result.face))
caption = _caption_for(blurb) if blurb else _caption_for(result.payload)
await self._push_caption(caption, dwell=_REACTION_DWELL_S)

The between-turns mood push respects that hold. _maybe_push_mood_face only runs when the fence and arbiter are idle (a turn's lifecycle face owns the screen otherwise), and it bails while the dwell is active:

# shelldon/core/runtime.py — _maybe_push_mood_face
if not (self.fence.is_idle and self.arbiter.is_idle):
    return
if self._monotonic() < self._reaction_hold_until:
    return  # the reaction face + thought still hold the screen
token = self.faces.select(m.mood.valence, m.mood.arousal, m.energy)
if token != self._last_face:
    await self._push_face(token)
await self._push_caption(token)

Because both the face and the caption are held by the same deadline, the deliberate expression and its thought settle back to the ambient mood together, as one — not on split cadences. A repeated reply re-arms the hold even when the text is unchanged.

Reasoning-tag stripping

GLM (the model shelldon runs on) wraps its chain-of-thought in <think>…</think>. That private reasoning must never leak into the owner's reply or onto the screen. The worker strips whole reasoning blocks and any orphan tag before anything is parsed out:

# shelldon/worker/worker.py
_REASONING_RE = re.compile(r"<think>.*?</think>", re.DOTALL | re.IGNORECASE)
_ORPHAN_THINK_RE = re.compile(r"</?think>", re.IGNORECASE)

def _strip_reasoning(text: str) -> str:
    cleaned = _ORPHAN_THINK_RE.sub("", _REASONING_RE.sub("", text))
    return cleaned.strip() if cleaned != text else text

parse_reply strips reasoning first, then extracts the hidden THOUGHT: and FACE: directive lines, then pulls out any fenced ```ops blocks — returning (payload, ops, thought, face). The THOUGHT: and FACE: lines are removed from the payload so they never appear in the chat reply, exactly the way the ops block is stripped. A tag-free reply is preserved byte-for-byte.

Summary of the data flow

What Region Source Pushed via
Expression (mood) FACE select_face over the mood registry, between turns _push_face
Expression (reaction) FACE model's FACE: line, validated against the palette _push_face (held ~60s)
Lifecycle face FACE core state (thinking / happy / cant-think) _push_face
Thought / caption CAPTION model's THOUGHT: line, else truncated reply _push_caption (held ~60s)
Battery BATTERY PiSugar2 charge over 127.0.0.1:8423 BatteryPlugin._poll_loop → host.draw

All four ride the same bus as latest-wins StateSnapshots; the display coalesces them under E-Ink's slow refresh and the WaveshareRenderer composites the three on-panel zones into one framebuffer repaint.

Key files

See also

shelldon — an E-Ink AI desk pet · docs generated from the project's design + implementation notes

shelldon

🏠 Home

How it works

Run & build

Clone this wiki locally