Skip to content

Hermes Gadget 0.6.0

Latest

Choose a tag to compare

@github-actions github-actions released this 17 Nov 21:37
· 29 commits to main since this release
5160f50

Hermes Gadget firmware 0.6.0.

Hermes Gadget 0.6.0 gives the agent more to do with a gadget. It can play a sound file or a radio stream on the speaker (gadget_play), show lists, progress bars and QR codes, ask a question with up to six answers on the screen (gadget_ask), set the face's expression, and fill the idle screen with widgets such as a disk's fill level or the room temperature. Interpreter mode turns a gadget into a translator that two people pass back and forth. Messages and cron jobs sent to a gadget outside a turn are spoken, and hermes gadget bot-chat lets a gadget talk to Hermes Bot Mode. Profile switching works on a stock Hermes with profile_mode: virtual, and a board's diagnostics report comes over the network, with the reason for a hang or a restart. The M5Stack CoreS3 gains its camera, as a camera.capture action and a camera mode with a live view. Five experimental board profiles arrive: the M5Stack StopWatch, the M5Stack Cardputer ADV, the Seeed Studio SenseCAP Watcher W1-A, the Waveshare ESP32-S3-Touch-LCD-1.47 and the LCDWIKI ES3C28P. The Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C boots on esp-hosted 2.12, and its speaker pops less when it turns on and off. Details by area follow.

Firmware

  • Add the experimental sensecap-watcher profile for Seeed Studio SenseCAP Watcher W1-A revision v0.2: round 412×412 SPD2010 display and touch, wheel scrolling with its button as CANCEL, ES8311 speaker, ES7243E microphone (or an older ES7243), PCA9535 power rails, battery voltage and Power off. Hold the screen to talk. The board keeps Seeed's per-device nvsfactory partition: it gets its own partition table, and the browser installer now never writes a partition a release marks as factory data (a new keep list in manifest.json); writing the full image with esptool still erases it, and the release notes say so. The PCA9535 latches the battery supply before Wi-Fi starts, a key held when the board starts is ignored until it is let go, and the console stays on UART0, which the data USB-C port reaches through a USB-to-UART bridge. Camera/HX6538 and microSD remain unsupported and unpowered. Touch is read only while its INT (PCA9535 P0.5) is low, since the SPD2010 answers "no points" between reports and a held touch kept ending; the wheel counts a detent every two quadrature steps, and diag reports its edges and detents. The sim-412x412-round simulator profile has the same screen. The pin map was checked against the official schematic, Seeed's BSP and xiaozhi-esp32's Watcher board; on one unit, hold-to-talk on the screen, wheel scrolling and its button as CANCEL, Power off and starting again from the wheel, and a complete voice turn through a real Hermes gateway work; the full hardware checklist is not done. Hardware Validation

  • A board that starts on battery takes light sleep while dozing. With no USB host, ESP-IDF's console read fails at once and its console task asks again every 10 ms, which never leaves the idle task the three ticks tickless idle needs: an M5Stack StopWatch started without a cable logged 0 s of light sleep in 45 minutes of doze and woke to a tap only after several, and slept 68 to 70 % from its first plug-in on. On boards with the USB Serial/JTAG console, the console now starts once a host is connected. Run on an M5Stack StopWatch: started on battery, 71 % and 70 % light sleep in two dozes of 22 and 9 minutes, the first tap wakes it, and the console answers after plugging in.

  • The ws-p4-touch-lcd-7b firmware builds again. With the face's expressions its image was 2,640 bytes larger than the 1.9 MB app slot. The board's build now leaves the expression, file and line out of a failed assert's message; diag still reports the crashed task from the core dump, with its pc, mcause, mtval, ra and sp, on a board whose partition table has the coredump partition. This chip's report has no backtrace.

  • The face has expressions. Hermes can make the face on the screen look happy, confused, alert or sleepy with the new face.set device action, for 1 to 3600 seconds or until your next turn (a recording, a typed message or a new session); neutral clears it. The device draws each one from the frames a face already has (sleepy shuts the eyes, alert opens the mouth and stops blinking) and a small mark beside the head: sparkles, ?, ! or zZ. It adds no bitmaps, custom faces and colour avatars show it too, and only the rows it touches are redrawn. The mark moves for a minute, then stands still. It shows on the idle, thinking and speaking screens. The serial console's face <expression> [seconds] sets one by hand, on a board and in the simulator, and examples/scripts/faces.txt runs through them. Tested in the simulator and the core tests only, not on a board. See the mascot and the action.

  • Cards can show a list, a progress bar and a QR code, and a new screen asks a question with up to six answers. On a touchscreen a tap on an answer picks it, a swipe up shows more and a swipe down cancels. Up and Down, or a knob, move a highlight that TALK picks. With TALK and CANCEL alone, CANCEL moves to the next answer and a last row cancels. The firmware announces both in hello (caps.card, caps.choice), with the largest QR code its screen fits: from 84 bytes on 240x135 to 287 on 320x240 and larger. A board with one button and no touchscreen takes no choices. A question too long for the title wraps above the answers. A question or a choice that arrives while the profile grid is open closes the grid, so a tap answers it. Checked in the simulator on rectangular, small, round and touch panels; not yet on a physical board, so whether a phone reads a code off each panel is untested. Lists, progress and QR codes, pick an answer.

  • A gadget's idle time can show glanceable data. Widgets are full-screen pages set on the Hermes computer: a big value with a unit, a progress bar, a list of up to four lines, or a two-line headline, each with a small label and an optional ok, warn or bad colour. hermes gadget widget set office --value 21.5 --unit C sets one for every gadget, --device kitchen for one gadget, --ttl makes it expire, and widget list and widget clear manage them; a gadget shows up to eight. The agent sets them with the new gadget_widget tool, and examples/widgets/system_widgets.py feeds a disk's fill level, the load average and the host name from cron. On the gadget, Down or a swipe from left to right on the idle screen opens the first widget and each further one, Up or a swipe from right to left goes back, and CANCEL or a swipe down closes them; the sessions list, the profile grid and talking keep their gestures. A widget left open rests with the resting clock: the clock face and the widgets take turns each minute, dimmed and shifted like the clock. Widgets travel in the dashboard message as layout 2 (caps.dashboard.v), so they are data that never takes the screen and they follow the screen-sleep holding. A turn or a reply from Hermes closes an open widget so that it is shown. Tiles and widgets are changed under a lock, so the CLI, cron jobs, the tool and the gateway don't lose each other's changes. Firmware that draws only layout 1, and boards with neither a touchscreen nor scroll buttons, get each widget as a tile on the sessions list. Core tests, plugin tests and the simulator (round 466x466, 240x135, 320x240 and 800x480); not yet run on a board. Glance at widgets.

  • Interpreter mode on the screen: the idle screen and the reply show who speaks next and into which language (ES > EN), the translation is drawn one text size larger when it fits, with what was said above it, a short CANCEL press switches who speaks next, and holding CANCEL leaves the mode. The device keeps the mode through a lost connection, marks what is typed or recorded in it, and asks to leave it with interpret.stop. It announces caps.interpret and takes the new interpret message (protocol). hermes-gadget devserver has an interpret command to try it without Hermes.

  • A board report now shows why a board hangs, stops responding to touch, or restarts. diag adds touch, with the reads, failed reads, the age of the last good read, INT edges and dropped touch events. A task_ms_ago shows whether the touch task still runs. events shows the app's event queue: waiting, peak and dropped events. Each task in stacks gains its state and cpu_pct, its share of one core since the previous diag. crash reports the last crash from a core dump: the task, program counter and backtrace. The core dump is saved in a new 56 KB coredump partition in unused flash before the first app slot. A board gets that partition the next time the browser installer installs it, which keeps its settings; writing the full release image with esptool at 0x0 erases them. The log copy that diag log reads now survives a panic, a watchdog or a software restart: the next boot's log starts with the lines from before the restart. That copy moved to internal RAM, 4 KB on boards with PSRAM (it was 8 KB in PSRAM) and 2 KB without. If the app task stops answering, the console answers diag with the port's part, marked "app": "not answering", and diag log still works. A new log <tag | *> <none | error | warn | info | debug> console command changes a tag's log level until restart; the firmware's own debug lines are now in the build for it. Release runs keep every board's firmware.elf for 90 days so a backtrace can be read. Host tests and two board builds; not yet run on a board. Save a board report.

  • Hold mode can drop a recording nobody spoke in, on boards that ask for it (codec.drop_silent_recordings in the board profile, for boards whose speech floor codec.vad_min_rms was measured on the board). Hold mode sent every recording of 350 ms or more, a speech-to-text model makes words out of silence (Whisper wrote "Vielen Dank." for a silent one), and Hermes answered them. The voice detector now listens in hold mode too, without ending anything: releasing TALK or reaching the recording limit after audio with less than 200 ms of speech over the floor cancels with reason no speech and shows "Didn't hear anything". The M5Stack StopWatch sets it; nothing changes on the other boards, and a recording with less than 200 ms of audio (a port that delivered little or none) is sent as before. Measured on recordings from an M5Stack StopWatch at its floor of 180: of 10 silent recordings that half the floor let through, 2 still pass, both with real noise in the room, and a long silent recording in a loud room can gather its 200 ms and be sent. On that watch, silence was dropped and speech sent in everyday use.

  • Add the experimental text-only esp32-s3-touch-lcd-147 profile for the Waveshare ESP32-S3-Touch-LCD-1.47: a 172×320 panel on the ST7789 SPI path (column offset 34) and an AXS5106L touchscreen. There is no microphone or speaker, so cards, prompts and notices stay on the glass, a timer alarm is silent, and holding TALK reports that no microphone is available. The battery ADC and the microSD slot are unused. On one ESP32-S3 rev v0.2 the boot log reports st7789 172x320 ready, touch ready and I2C 0x63. The setup text read backwards until the panel and the touch report were mirrored in X. On this unit the picture orientation is correct, Wi-Fi setup and pairing succeeded, and one card was delivered. Taps did not register while the touch report was read with a repeated start; the report is now a stopped register write followed by a read, and a swipe down then cleared that card. The chip id read was 51 06 01. Colour, a tap, a prompt and a silent timer were not checked. Text on this glass is drawn at twice the automatic size, and the mascot uses the 64 px bitmap instead of the 144 px one. The UI is landscape (320×172): the panel axes are swapped and the column offset of 34 is applied on the short side, so a doubled line has room. The portrait X mirror reads backwards once the axes are swapped, so landscape leaves mirror off. Not verified. Contributed by @james-see.

  • A board no longer restarts in a loop after esptool has talked to it. The USB Serial/JTAG block keeps its interrupt enables through a software reset, and ESP-IDF's driver adds its own two without clearing the rest and acknowledges only those. An M5Stack StopWatch came up with the bus-reset interrupt enabled as well (int_ena 0x0204) after esptool had used the port. Plugged into USB while running on battery, it went silent within seconds and then restarted every 3.8 s from the interrupt watchdog inside usb_serial_jtag_driver_install, until a reset over USB. Before the console installs the driver, every other enable is now cleared and its pending status acknowledged, with a warning that names what was found (usb serial/jtag had interrupts enabled that nothing handles). The cause was read from one unit's backtrace and register dump; the loop was not reproduced on purpose, so all that speaks for the fix is that this unit, replugged several times since, has not looped again. Contributed by @nsyring.

  • New board profile m5stack-stopwatch, experimental, for the M5Stack StopWatch (SKU C152): a round 1.75" 466×466 CO5300 AMOLED over QSPI, CST820 touch, one ES8311 for the speaker and the MEMS microphone, and a 450 mAh battery. The blue key or the screen talks, the yellow key cancels, and a short press of the red key turns the screen off and on. The panel, the touch controller, the codec and the amplifier have no ESP32 pin for power or reset: the new drivers/stopwatch.cpp runs the power sequence on the M5IOE1 expander, switches the AW8737A amplifier with the audio, and reads the M5PM1 for the cell and USB voltage, the charger's status, shutdown and the red key. The battery percentage follows discharge and charge curves measured on the cell, a BMI270 gives face-down sleep and an RX8130CE keeps the time; the touch interrupt and the vibration motor are not used. Its microphone is quiet, so tap mode's speech floor is 180. On USB power with a nearly full cell its recordings carry a 10.4 kHz tone in bursts, which does not change the transcription. Ran on one unit; see hardware validation. Hardware. Contributed by @nsyring, based on the port by @allinnb and heyia.

  • A recording runs up to 60 s instead of 30 s, the limit the plugin already allows by default (max_utterance_s). The Listening arc fills over 60 s and turns amber for the last 10. On a board with PSRAM the recording buffer grows with the limit, to 2 MB at 16 kHz and 3 MB at 24 kHz, and is still taken only when about 1 MB of PSRAM stays free after it: a board that had room for the 30 s buffer but not for this one streams without a buffer, as boards without PSRAM do, and says so at start-up. On an M5Stack StopWatch (8 MB PSRAM, 24 kHz) the 2,859 KB buffer was reserved with 5.9 MB of PSRAM free and held recordings of 45 s were sent; one held to the limit stopped at 60 s with no frame lost and was transcribed in full. No other board has run it. Contributed by @nsyring.

  • New board profile lcdwiki-es3c28p, experimental, for the LCDWIKI ES3C28P (also sold as Freenove's ESP32-S3 2.8 inch display): a 2.8 inch 240×320 ILI9341V IPS panel over SPI in portrait (X mirrored), FT6336G touch, and one ES8311 for the onboard MEMS microphone and the speaker output. 16 MB flash and 8 MB octal PSRAM. Hold the screen or BOOT to talk; a swipe down cancels. The amplifier enable on GPIO 1 is active low, so it is held on from boot as a power rail. GPIO 9 reads the cell through a divider of two. There is no charge pin and no software power-off. The addressable RGB LED, the microSD slot and the extension pins are unused. Adds a sim-240x320 simulator profile. Physical report on one unit, from a local build of this profile with an extra display overlay and 4 MB OTA slots. Hardware. Contributed by @mathbatistela.

  • Tap mode's voice detector measures levels in the voice band: each block goes through a second-order low-pass at 3.4 kHz before its level is taken. An M5Stack StopWatch records a 10.4 kHz tone from its own electronics in bursts, about 300 RMS and as loud as speech through its microphone; the detector took it for speech, so a pause did not end a tap-mode recording. On every board the level of speech now leaves out what lies above 3.4 kHz (sibilants mostly), so it reads a little lower against the same floor (codec.vad_min_rms, 500 by default). Replayed through the detector, 7 of 7 speech recordings from that watch still count as speech and the tone alone no longer does; a host test feeds it a 10.42 kHz tone. Not tried on any other board's microphone. Contributed by @nsyring.

  • The CoreS3 no longer rejects its display solely because the touch firmware ID is unknown (such as 0x05). Known IDs 0x10/0x12 retain their C/E selection. Other IDs require a direct LCD register signature over 3-wire SPI; no response or a reported GPIO error leaves the display unavailable. The profile also selects the BSP's BGR color order instead of inheriting RGB, correcting swapped red and blue. Native tests cover the complete C/E GPIO probe, failure paths and the real board configuration. On one C-panel unit with touch firmware 0x05, the owner confirmed correct color bands and pink accent, touch/menu operation and a displayed/spoken Hermes reply after an app-only update (report and limits). Contributed by @f3rs3n.

  • The M5Stack CoreS3's GC0308 camera is also a local camera mode: the settings menu's new Camera item (shown only on boards whose port provides a camera) opens the device's own camera screen with the live view drawn straight into the panel at about 10 fps, mirrored left-to-right like a phone's front camera (the saved photo keeps the sensor's handedness). A tap starts a three-second countdown and captures; the photo, scaled and centred, stays on screen until the next tap (retake: the live view comes back first) or a swipe down; the screen never dims while the camera is out; and 60 s without a touch leaves, as do camera/camera close on the console. The capture pipeline (rails, CAM_RST, esp32-camera) starts when the mode opens or the first capture arrives and stops when it leaves or 60 s after the last capture — the driver's frame buffers go back to PSRAM and the ribbon's supplies go off — and the preview is fed without blocking the app. The photo stays on the device — sending it to Hermes is the next part of the plan in #294. On a CoreS3: the entry, countdown capture, retake, swipe-down leave, 60 s idle exit and menu gating verified by hand; the preview measured at 10.0 renders/s; the photo's 614,400 bytes returned on both retake and leave (camera mode check, hardware). First part of the camera mode in #294. By @james-see.

  • The M5Stack CoreS3's GC0308 camera is available as a camera.capture action, and the gateway saves each still as a PNG for the agent to look at. The sensor runs from its on-board 20 MHz crystal (the port never generates XCLK), shares the internal I2C bus for SCCB, takes its power and reset from AW9523B P1_0, and captures 640×480 RGB565 into a PSRAM frame that is byte-swapped on device to the little-endian layout the protocol already documents. New dependency: esp32-camera 2.1.8, fetched only for ESP32-S3 targets and compiled only for the CoreS3, so no other board's image grows. Verified on a CoreS3: the gateway saved a 640×480 PNG, upright and with correct colours, and repeat captures reused the running pipeline (598 host tests pass, camera and byte-order paths covered). Closes #291 (hardware, physical check). Reported by @james-see.

  • A new SD card logging setting, off by default (set sd_log on): with it on, the device log is copied to hermes.log on a FAT32 microSD card when one is present, rotating past 512 KB. Cards that are inserted later start logging without a reboot, and a missing or unreadable card never blocks boot. The Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C profile wires its SDIO 3.0 slot (CLK 43, CMD 44, D0-D3 39-42) on SDMMC slot 0, leaving slot 1 to the C6 companion's SDIO link. Run on a 3.4C rev 1.3 from an ESP-IDF 5.5.4 build with an 8 GB FAT32 card: nothing is written while it is off, a boot without a card shows no errors, a card inserted while it runs starts logging, turning it off and on resumes logging, and each boot appends its own header; Wi-Fi stays up throughout. The ESP-IDF 6.1 image builds in CI but has not run on the board. Contributed by @rahlquist.

  • The M5Stack CoreS3's speaker plays at full volume. The port reserved 15 dB of the AW88298's gain, which the codec driver subtracted from every volume, so 100 played near −11 dB, about a fifth of full volume by ear. The reserve is gone, so 100 reaches the amplifier's 0 dB, the level M5Stack's own driver uses on the CoreS3. Checked by ear on a CoreS3: loud at 100, with no audible distortion over several replies (speaker volume check). Fixes #289. Reported and tested by @james-see.

  • On the tailnet, the gadget connects to Hermes as soon as its WireGuard session with the Hermes computer is up, instead of waiting until TailTiny also reports itself connected. TailTiny does that only after its DERP relay connects, whose TLS handshake alone took about 2.7 s on an AMOLED-1.75C. After a Wi-Fi change it still waits for TailTiny to be registered again on the new network, since WireGuard keeps the old network's session. A Hermes computer that isn't at a tailnet address waits as before. Contributed by Antonio Lourenco (@tozes).

  • The Waveshare AMOLED-2.16 (SKU 33969) has a partial physical report on the 0.5.0 release image, from a second unit: pairing, a real Wi-Fi network, a complete voice turn with a spoken reply and the speaker.volume action ran on the board; touch accuracy, the PWR key, OTA, persistence and battery were not checked, so the port stays experimental. A board that shows nothing after an esptool write needs a power cycle to start (hardware, report). Contributed by @swissly.

  • A connection is no longer closed as it opens. The main loop reads the clock once per pass, and opening a connection stamps its start a few milliseconds later in the same pass. The connect timeout subtracted the two unsigned, so the difference wrapped to about 49 days, and the connection was closed at once as connect timeout (logged 10 ms after connecting to ...). The device then waited out a backoff of 1, 2, then 4 s before trying again. On an AMOLED-1.75C reaching Hermes over the tailnet, 1 or 2 attempts failed this way on most reboots. The timeouts in the main loop now treat a stamp later than the loop's clock as no time passed. Contributed by Antonio Lourenco (@tozes).

  • The tailnet's client is now TailTiny 3.1.0, which includes the fixes this firmware used to patch in (ESP-IDF 6 support, the relay certificate check and the WireGuard lock), all contributed upstream from here. Five gadget-only patches remain (TAILTINY.md). New from upstream: TailTiny's relay and WireGuard tasks wait for work instead of polling, so a board with automatic light sleep can sleep while the tailnet is idle. The host test of the relay's TLS check is replaced by a source check, because upstream checks the certificate inline in its relay code, which a host test can't build on its own. Contributed by Antonio Lourenco (@tozes).

  • On battery with the screen dark and nothing going on (the doze that lets a light-sleep board sleep), Wi-Fi switches from minimum to maximum modem sleep once Hermes has sent nothing but its heartbeat ping for 10 s: the radio wakes every third beacon, about every 300 ms, instead of at every DTIM. A message from Hermes then waits at most that long at the access point, well inside the 20 s heartbeat and its 60 s timeout. The screen lighting, a turn, or any other message from Hermes goes back to minimum modem sleep at once, and a turn or USB keeps the radio fully awake as before. The station asks the access point for a listen interval of 3 when it joins, and each switch is logged (hg.wifi: power save: max modem). A Deep Wi-Fi sleep settings item, or set wifi_deep_sleep off, keeps minimum modem sleep on battery as before, for an access point that drops or delays clients in deeper sleep; it is on by default and shows on ESP32 boards that read their battery. On a Waveshare AMOLED-2.06 under light use, discharges of the same firmware with the setting on and off reached 15% within 10 minutes of each other (6 h 22 min to 6 h 32 min), so no clear battery gain was measured on that board. Hardware verification.

  • The AMOLED-2.06 watch shows its battery percentage from a discharge curve measured on a real AMOLED-2.06, smoothed over a minute and never rising on battery, instead of the AXP2101 gauge. Over the measured discharge (8 h 14 min, light use, screen mostly dark) the gauge read 25% with 42% left, raised the 15% alert with 2 h 14 min left and reached 0% 11 minutes before shutdown; replayed through the curve, a full watch shows 100 right after unplugging, the percentage stays within 2 points of the time left on average, the 15% alert comes with 74 minutes left and 0% under two minutes before shutdown. A minute-long load dip can still read up to 8 points low until the charge catches up, and a heavier load reads a few points low. While charging it still shows the gauge, then 100 once charging is done: the cell reaches 4.18 V less than halfway through a charge, so voltage can't track it. Host tests replay the measured discharge; a build with the curve has not run on the watch yet. Hardware. Contributed by @BrOrlandi.

  • Add the experimental cardputer-adv profile for M5Stack Cardputer ADV / Stamp-S3A: ST7789 display, ES8311 voice, USB provisioning, battery voltage and TCA8418 keyboard with a local text editor. OTA reception on this no-PSRAM profile limits queued firmware chunks to avoid exhausting internal RAM during flash writes and owns each completed receive payload across the admission wait. No SD card is required. Original Cardputer models are not covered.

  • The Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C drives its speaker amplifier from the port, and every board whose port drives the amplifier now turns it on while the DAC is still muted and mutes before turning it off, with 20 ms to settle each way: the 3.4C, the AMOLED-1.75 and 1.75C, the AMOLED-2.06, the AMOLED-2.16, the LCD-1.85C V2 and the P4 7B. On the 3.4C only, the ES8311's DAC also ramps its volume on mute and unmute. Both aim at the pop when playback starts and stops. Built; not yet heard on any board, including the 3.4C. Contributed by @rahlquist.

  • The Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C now boots on esp-hosted 2.12. Its mempool pre-allocates the SDIO and common TX buffers at esp_hosted_init — a constructor that runs before the main task exists — taking about 88 KB of the board's roughly 97 KB low heap and leaving too little for the 8.7 KB main-task stack, so startup panicked at esp_startup_start_app with nothing on screen. The 3.4C profile now turns ESP_HOSTED_USE_MEMPOOL off, so transport buffers allocate on demand as they did on esp-hosted 2.7. The same profile turns off REGDMA sleep-clock retention (the board never light-sleeps) and draws with CPU copies instead of DMA2D, the configuration it was run with. Run on a Waveshare ESP32-P4-WIFI6-Touch-LCD-3.4C rev 1.3 from an ESP-IDF 5.5.4 build of this change: it boots, the 800x800 DSI panel draws with CPU copies, both codecs and the touch start, and the device pairs and comes online. The ESP-IDF 6.1 image that releases use builds in CI but has not been booted on the board. Contributed by @rahlquist.

  • A new Animation setting, on by default (set animation off): off, the edge ring's comet no longer runs while Hermes thinks, speaks, boots or connects, and the Updating screen stands still while a firmware update installs instead of blinking its spinner and dots. The timer arc and the update progress ring still show, because they carry information. Contributed by @rahlquist.

Hermes plugin

  • Hermes can play audio on a gadget's speaker. A new agent tool, gadget_play, plays a sound file from the Hermes computer or an http(s) URL (an audio file, a podcast episode, an internet radio stream), and stops it. The plugin decodes on the host, a chunk at a time, into the stream spoken replies use, so no firmware change is needed and a long stream is never held in memory. Playback starts after the current reply and ends at the end, after play_max_minutes (60), on stop, on CANCEL, when you talk to the gadget or when Hermes speaks. Formats other than PCM WAV, and all URLs, need ffmpeg. Local files are read only from Hermes's audio cache and the folders in the new play_dirs setting; URLs go through Hermes's URL policy, which refuses private and internal addresses, and ffmpeg is given no network or file access of its own and reads audio containers only, not playlists. A source that sends nothing playable within 15 seconds, or a stream that goes silent for 20, is given up on. See Playing audio. The dev server's console gains play <path> and stop.

  • The gateway stops the audio it is sending when the device starts a recording. The device had already stopped playing it; the rest of a reply was still sent and dropped.

  • The agent can put more on a gadget's screen and ask a question there. gadget_display takes items (a list), progress (a bar from 0 to 100) and qr (text shown as a QR code), and a card sent again with the same id replaces the one on screen in place, so a progress bar moves. A new tool, gadget_ask, shows a question with 2 to 6 answers and returns the one the user picked, or that they cancelled. A gadget whose firmware predates this gets the same content as a plain card, and gadget_ask shows it the options as a numbered card and tells the agent to ask by voice. Hermes's own yes/no confirmations, and the rules for approving dangerous commands, are unchanged. hermes-gadget devserver has show and choose to try them. Protocol: structured cards, choices.

  • Interpreter mode. Two people who speak different languages pass a gadget back and forth, and it says and shows each thing in the other language. Start it with hermes gadget interpret <device> <language> <language> or by asking Hermes (the new gadget_interpret tool); leave with off, or by holding CANCEL on the gadget. What is said in the mode never reaches the Hermes agent: the plugin transcribes it, asks the model for the translation through Hermes's completion API for plugins, which carries no tools, and speaks the result, so the other person's words cannot trigger a tool or an approval, and nothing is added to a session or to memory. The speakers take turns and each recording is transcribed in the language whose turn it is; Hermes's speech-to-text cannot be asked to detect the language. A translation in a script the screen can't draw is still spoken, and the screen says so. While the mode is on the gadget shows no Hermes question and takes no answer to one, messages sent to it wait, and what it sends is marked, so a mode that ended meanwhile never turns a stranger's words into a Hermes turn; the mode outlives a gateway restart. It ends by itself after 30 minutes unused. Tested in the simulator and against a Hermes gateway with stand-in providers, not on a board. See Interpreter mode.

  • A connected gadget's diagnostics report no longer needs a USB cable. hermes gadget diag <name> on the Hermes computer, or Save diagnostics report in the dashboard's Gadgets tab, asks the gadget through the gateway and saves the file hermes-gadget diag saves, plus the gateway's last 500 Gadget log lines, so a bug report has both sides of the same minutes. It works before pairing, too. The gateway sends a new diag message and the device answers with the report and its recent log (up to 16 KB); the protocol stays at version 1, and older firmware doesn't answer, so its report still comes over USB. Host and simulator tests only; not yet run on a board. Save a board report.

  • Messages sent to a gadget outside a turn are spoken. Hermes reads aloud only its replies to a voice turn, so a send_message or a cron delivery to gadget:<device id> reached a device with a speaker as text alone, and a speaker without a screen, such as an Echo Dot running LibreEcho, stayed silent. The adapter now also speaks such a message with Hermes's text-to-speech when the device speaks replies, with Markdown removed and, for cron output, Hermes's header, footer and truncation note. Messages play one at a time per device: each waits for audio already playing, and cron output that arrives during a turn waits for the turn; one that has waited two minutes stays text only. Before, a second clip cut off the first, which also happened between the files of one long clarify question. A caption sent with a voice file and the text of a clarify question are not spoken a second time. /voice off in the device's chat, or speak_replies: false, keeps them silent. Tried against Hermes v0.22.5 with Kokoro text-to-speech on two Echo Dot 2nd Gen speakers running LibreEcho and a Python gadget client; new adapter tests cover the spoken message, the cron wrapper, three messages in a row with no audio.abort, cron output during a turn, and the messages that stay silent. Hermes integration. First part of #304. Contributed by @NickJLange.

  • New and deleted profiles reach a connected gadget. A profile created in Hermes reached a gadget's profile grid only at its next welcome, so it stayed missing until a reconnect or a gateway restart. The plugin now compares each switching device's offered profiles with Hermes's at every pairing check (every 2 s) and, when the list changed, sends the new profiles message (protocol); thumbnails of new profiles follow. A list Hermes can't read sends nothing, so a failed read doesn't empty the devices' grids. Firmware that knows profiles replaces its list and redraws an open grid in place, keeping the page or moving to the last one; older firmware ignores it and gets the list at its next welcome. Tried on a Waveshare ESP32-S3-Touch-LCD-1.54 against Hermes v0.23.1 with the profile grid open: a profile made with hermes profile create was pushed one second later and appeared on the grid, and after hermes profile delete it left it the same way, with no reconnect; two Spotpear boards on the earlier firmware got the message and carried on. Contributed by Luiz Gustavo Saraiva (@luizgustavosaraiva).

  • A device that reconnects is welcomed at once. After a device rebooted or changed networks, its old connection still looked open. The new connection closed the old one and waited for that close before sending the welcome, and the old connection's peer is gone, so the wait lasted until TCP gave up or the device's new network stack reset it. That took 0.07 to 3 s per reconnect on an AMOLED-1.75C over the tailnet, and in the meantime the pairing check sent paired and the dashboard before the welcome. The old connection now closes in the background. Contributed by Antonio Lourenco (@tozes).

  • Profile switching without the Hermes hook: profile_mode: virtual. On Hermes versions with native session admission and without validate_native_profile_route (v0.22.5 to v0.23.1 and main today), every message to a picked profile failed with profile_mismatch (#232). With platforms.gadget.extra.profile_mode: virtual, a device talks to a picked profile in a chat of its own, <device id>~<profile>, and the plugin adds a gateway.profile_routes entry for that chat in the running gateway, so Hermes admits it as a routed chat. Each profile keeps its own sessions; routes of profiles deleted, parked or taken off the list are dropped at the next pairing check. Off by default. On Hermes v0.23.1 the gateway end-to-end switch passes in virtual mode and still fails with profile_mismatch without it; tried on a Waveshare ESP32-S3-Touch-LCD-1.54 switching between three profiles. Hermes integration. Contributed by Luiz Gustavo Saraiva (@luizgustavosaraiva).

  • hermes gadget bot-chat <device> [profile]: a gadget that talks to the other Bots. Hermes Bot Mode gives a profile's canonical Bot Chat the message_agent tool, but a gadget's conversation is a session of its own, and moving the Bot Chat onto it took typing /resume --all Bot Chat on the device's USB console. The new command asks the gateway to switch the device to the profile and send that command from it, then waits until the profile's Bot Chat is on the device's chat. It checks first, and says what to change, when the profile has no Bot Chat, the gadget is not an admin of the gadget platform (allow_admin_from, which /resume --all needs) or the profile isn't offered. Tried on a Waveshare ESP32-S3-Touch-LCD-1.54 on Hermes v0.23.1: the command moved profile nick's Bot Chat onto the gadget over the network; with the same /resume typed on the console, the gadget on profile nina asked two other Bots through message_agent and spoke their answers. Hermes integration. Contributed by Luiz Gustavo Saraiva (@luizgustavosaraiva).

  • The model no longer takes the gadget's name as its own. The per-device context now says the gadget's name is the device's, not the assistant's, so a profile answering through a gadget named after another profile keeps its own name. Contributed by Luiz Gustavo Saraiva (@luizgustavosaraiva).

Hardware and setup

  • The hardware guide says once how to run the boards' factory-backup lines when PlatformIO is the only install (pio pkg exec -p tool-esptoolpy -- esptool.py ..., or pip install esptool), to check that the backup has the full flash size before flashing, and that -b 921600 makes the read about seven times faster. Back up the factory firmware. Contributed by @mathbatistela.

Upgrade and verification

Update the plugin first, with the commit-pinned command in the release notes, in every Hermes profile that has a copy (see One profile per gadget). Then update devices with hermes gadget update <device> --latest, or use the browser installer for USB installation. Lists, progress bars, QR codes, questions, widgets, face expressions, interpreter mode and the diagnostics report over the network need both the new plugin and the new firmware. The new plugin still works with older firmware: a card that firmware can't draw arrives as a plain card, a question becomes a numbered card the user answers by voice, and a widget becomes a tile on the sessions list.

On Hermes versions with native session admission (v0.22.5 and later, including the current stable), a device that picks another profile is refused with profile_mismatch until NousResearch/hermes-agent#147690 lands. Set profile_mode: virtual to switch profiles without that change (see #232).

Hardware ports remain experimental until a complete physical report is recorded for the exact revision; see hardware capabilities and verification. Several entries above say "host tests only", "tested in the simulator" or "not yet run on a board": software tests and successful builds do not establish physical hardware verification.

Contributors

Thanks to @nsyring for the M5Stack StopWatch, 60-second recordings, hold mode that drops silent recordings, a voice detector that listens in the voice band, and the two USB Serial/JTAG console fixes that stop a restart loop and let a battery board sleep; Luiz Gustavo Saraiva (@luizgustavosaraiva) for profile_mode: virtual, hermes gadget bot-chat and profile lists that reach a connected gadget; Richard Ahlquist (@rahlquist) for SD card logging, the Animation setting and the ESP32-P4-WIFI6-Touch-LCD-3.4C boot and speaker fixes; @james-see for the CoreS3 camera and its camera mode, and the Waveshare ESP32-S3-Touch-LCD-1.47; Antonio Lourenco (@tozes) for TailTiny 3.1.0, a tailnet that is up as soon as the session is, and reconnects that are welcomed at once; @BrOrlandi for deeper Wi-Fi sleep on battery and the AMOLED-2.06 battery curve; Alexander Shalin (@alexandershalin) for the M5Stack Cardputer ADV; @binghan1227 for the SenseCAP Watcher W1-A; Matheus Batistela (@mathbatistela) for the LCDWIKI ES3C28P and the factory-backup instructions; @f3rs3n for CoreS3 displays with unknown touch firmware; Nick J Lange (@NickJLange) for speaking messages sent outside a turn; and @swissly for the AMOLED-2.16 physical report.

Install from your browser: https://adolanium.github.io/hermes-gadget-sdk/ (Chrome or Edge, over USB). It checks the board before writing, and a device you reinstall keeps its settings.

Board Full image Over-the-air update
ESP32-S3 breadboard build hermes-gadget-esp32s3-breadboard-0.6.0.bin hermes-gadget-esp32s3-breadboard-0.6.0-app.bin
M5Stack Cardputer ADV hermes-gadget-cardputer-adv-0.6.0.bin hermes-gadget-cardputer-adv-0.6.0-app.bin
Waveshare ESP32-S3-Touch-AMOLED-1.75 hermes-gadget-esp32s3-touch-amoled-175-0.6.0.bin hermes-gadget-esp32s3-touch-amoled-175-0.6.0-app.bin
Waveshare ESP32-S3-LCD-1.54 hermes-gadget-esp32s3-lcd-154-0.6.0.bin hermes-gadget-esp32s3-lcd-154-0.6.0-app.bin
Waveshare ESP32-S3-Touch-LCD-1.47 hermes-gadget-esp32-s3-touch-lcd-147-0.6.0.bin hermes-gadget-esp32-s3-touch-lcd-147-0.6.0-app.bin
Seeed Studio SenseCAP Watcher W1-A hermes-gadget-sensecap-watcher-0.6.0.bin hermes-gadget-sensecap-watcher-0.6.0-app.bin
LilyGO T-Display-S3 hermes-gadget-tdisplay-s3-0.6.0.bin hermes-gadget-tdisplay-s3-0.6.0-app.bin
Waveshare ESP32-S3-Touch-AMOLED-1.75C hermes-gadget-esp32s3-touch-amoled-175c-0.6.0.bin hermes-gadget-esp32s3-touch-amoled-175c-0.6.0-app.bin
Waveshare ESP32-S3-Touch-AMOLED-1.32 hermes-gadget-waveshare-touch-amoled-132-0.6.0.bin hermes-gadget-waveshare-touch-amoled-132-0.6.0-app.bin
Waveshare ESP32-S3-Touch-AMOLED-1.43C hermes-gadget-waveshare-touch-amoled-143c-0.6.0.bin hermes-gadget-waveshare-touch-amoled-143c-0.6.0-app.bin
Waveshare ESP32-S3-Touch-AMOLED-1.8 (V2) hermes-gadget-esp32s3-touch-amoled-18-0.6.0.bin hermes-gadget-esp32s3-touch-amoled-18-0.6.0-app.bin
Waveshare ESP32-S3-Touch-AMOLED-2.06 (watch) hermes-gadget-esp32s3-touch-amoled-206-0.6.0.bin hermes-gadget-esp32s3-touch-amoled-206-0.6.0-app.bin
Waveshare ESP32-S3-Touch-AMOLED-2.16 hermes-gadget-esp32s3-touch-amoled-216-0.6.0.bin hermes-gadget-esp32s3-touch-amoled-216-0.6.0-app.bin
Espressif ESP32-S3-BOX-3 hermes-gadget-esp32-s3-box-3-0.6.0.bin hermes-gadget-esp32-s3-box-3-0.6.0-app.bin
Espressif ESP32-S3-BOX hermes-gadget-esp32-s3-box-0.6.0.bin hermes-gadget-esp32-s3-box-0.6.0-app.bin
Espressif ESP32-S3-Korvo-2 hermes-gadget-esp32-s3-korvo-2-0.6.0.bin hermes-gadget-esp32-s3-korvo-2-0.6.0-app.bin
LCDWIKI ES3C28P hermes-gadget-lcdwiki-es3c28p-0.6.0.bin hermes-gadget-lcdwiki-es3c28p-0.6.0-app.bin
Waveshare ESP32-S3-Touch-LCD-1.83 hermes-gadget-waveshare-touch-lcd-183-0.6.0.bin hermes-gadget-waveshare-touch-lcd-183-0.6.0-app.bin
Waveshare ESP32-S3-Touch-LCD-1.85C V2 (experimental) hermes-gadget-esp32s3-touch-lcd-185c-v2-0.6.0.bin hermes-gadget-esp32s3-touch-lcd-185c-v2-0.6.0-app.bin
Waveshare ESP32-S3-Touch-LCD-1.46 hermes-gadget-esp32-s3-touch-lcd-146-0.6.0.bin hermes-gadget-esp32-s3-touch-lcd-146-0.6.0-app.bin
Waveshare ESP32-S3-Knob-Touch-LCD-1.8 (experimental) hermes-gadget-esp32s3-knob-touch-lcd-18-0.6.0.bin hermes-gadget-esp32s3-knob-touch-lcd-18-0.6.0-app.bin
M5Stack CoreS3 hermes-gadget-m5stack-cores3-0.6.0.bin hermes-gadget-m5stack-cores3-0.6.0-app.bin
M5Stack StickS3 hermes-gadget-m5stack-sticks3-0.6.0.bin hermes-gadget-m5stack-sticks3-0.6.0-app.bin
M5Stack StopWatch hermes-gadget-m5stack-stopwatch-0.6.0.bin hermes-gadget-m5stack-stopwatch-0.6.0-app.bin
Xorigin AIPI Lite hermes-gadget-aipi-lite-0.6.0.bin hermes-gadget-aipi-lite-0.6.0-app.bin
Espressif ESP-VoCat v1.2 (experimental) hermes-gadget-esp-vocat-0.6.0.bin hermes-gadget-esp-vocat-0.6.0-app.bin
Spotpear ESP32-S3-1.54 muma hermes-gadget-spotpear-muma-0.6.0.bin hermes-gadget-spotpear-muma-0.6.0-app.bin
Spotpear ESP32-S3-1.28-BOX hermes-gadget-spotpear-box128-0.6.0.bin hermes-gadget-spotpear-box128-0.6.0-app.bin
Elecrow CrowPanel 2.1-inch HMI hermes-gadget-crowpanel-21-0.6.0.bin hermes-gadget-crowpanel-21-0.6.0-app.bin
Elecrow CrowPanel 5.0-inch HMI hermes-gadget-crowpanel-5-0.6.0.bin hermes-gadget-crowpanel-5-0.6.0-app.bin
ESP32-2432S028 CYD (Cheap Yellow Display), two-USB revision hermes-gadget-esp32-2432s028-cyd-0.6.0.bin hermes-gadget-esp32-2432s028-cyd-0.6.0-app.bin
Elecrow CrowPanel Advanced 10.1-inch ESP32-P4 hermes-gadget-crowpanel-p4-101-0.6.0.bin hermes-gadget-crowpanel-p4-101-0.6.0-app.bin
Waveshare ESP32-P4-WIFI6-Touch-LCD-7B hermes-gadget-ws-p4-touch-lcd-7b-0.6.0.bin hermes-gadget-ws-p4-touch-lcd-7b-0.6.0-app.bin

Flash with esptool: esptool.py --chip <chip> write_flash 0x0 hermes-gadget-<board>-0.6.0.bin. This also erases the device's settings, so it pairs again as a new device.

Don't write the full image of Seeed Studio SenseCAP Watcher W1-A with esptool: it would erase the board's factory data. Use the browser installer, which writes around it.

Update a running device: hermes gadget update <device> hermes-gadget-<board>-0.6.0-app.bin.

The Hermes plugin from the same commit:

hermes plugins install https://github.com/Adolanium/hermes-gadget-sdk.git#plugin --ref 5160f509b285821a8b999ec365fa49f158d4a5e9 --enable

Checksums are in SHA256SUMS. License texts and notices are in hermes-gadget-0.6.0-licenses.zip.
Every file has a GitHub build provenance attestation. Check a download with gh attestation verify <file> --repo Adolanium/hermes-gadget-sdk.