Repository navigation
Roadmap
Summary of docs/ROADMAP.md, which has the full detail and test results. Open work is tracked as issues by milestone.
The bridge runs on Pepper, every endpoint answers, one AI conversation works end to end. Done 2026-09-22, including base moves checked with odometry. Five bugs found only on hardware were fixed (deploy path, emergency-stop rest, looping animations, speech stop, microphone frame size).
Pepper looks alive while the model thinks, without a model call: "stop" / "be quiet" / "look at me" handled locally, eye colours for listening / thinking / speaking, spoken fillers, head tracking, spoken touch reactions, the head turning to newcomers. Command router (0.5, #21): a local decision model picks the first physical action from a spoken command and it starts about 0.25 s after the transcript while Claude prepares the words; on the robot 2026-10-08, 32 actions with one wrong. Still listed: chest LED and idle gesture, tuning on the robot.
People talk to Pepper through its own microphone; Pepper does not hear itself. NVIDIA Nemotron streaming recogniser chosen (6 % word errors in a noisy room against 37 % before); Sonnet 5.5 at medium effort chosen for spoken turns (2026-09-29). Addressee gate (0.5): side talk gets no reply; alone with Pepper, everything in a conversation is answered. Built in 0.7.0 and on the robot since 2026-10-20: waiting up to 2.5 s for the rest of a clearly unfinished sentence (#26; 10 of 11 recorded cut-offs caught, no complete utterance held) and shorter replies (#19; replies of four sentences or more from about 42 % to about 20 % in a replay; on the robot 16 %, and none in free conversation). The end-of-turn pause is now 1.2 s: 1.0 s cut people off (2026-10-20) and 1.4 s felt a little long (2026-10-22). Still listed: side talk with two people (#25; on the robot 2026-10-22 Pepper joined in, because both people stood out of view), under 2 s from end of speech, barge-in, fine-tuning on the lab's voices.
Act on what Pepper sees: turn to a point in a photo, check the result of an action with a photo, depth for distances. Sharp photos done 2026-10-01 (verified on the robot); handshake that holds and notices being taken (2026-10-08). Built in 0.7.0 and working on the robot (2026-10-20): pointing at a spot in a photo, the person or a direction (#28), looking at a spot in a photo (#6), and a camera fallback for come to me (#24), still to test in the open space. Open: follow me (#18). Open: come to me in the open space, the open hand when pointing, follow me, verify-after-act, depth.
Pepper knows what is around it between turns. Done: debounced people events with distance and gaze, and an "Around you" sentence in the model's context every turn. Greeting newcomers done 2026-09-29 (tested on the robot; head turns to newcomers, face tracking while someone is in view). The perception host (Alien3), scene notes and the recall tool are done. Open: voice direction behind Pepper, things seen as people, expect then check, the gentle handshake. Local decision models (Ollama's Nimble and Clef Flash on the Creative AI Hub) judge the situation before Claude is called (#20): built in 0.5 and verified on the robot 2026-10-08 (addressee gate, command router, camera judgements about once a second; a wave gets a reply about 3.6 s later). Memory, built in 0.6.0 (design, #29), on the robot since 2026-10-20 (remembered with consent, recalled, forgotten on request):
- Pepper's pose and people remembered out of view (#27); replayed on the 2026-10-08 sessions, the direction was predicted within a median of 7°.
- Where a voice came from (#12, #24): good in front, wrong behind Pepper (#31).
- Scene notes (#11).
- A recall tool over a timeline of what just happened.
Initiative, a first slice in 0.7.0 (off by default): Pepper may say one short thing unprompted when someone lingers quietly or a question goes unanswered, at most every 3 minutes (90 s for a follow-up question). Replayed on the 58 minutes recorded on 2026-10-08, it fired three times, none during real conversations. On the robot (2026-10-20) it fired three times and felt natural.
Next idea: expect, then check (#30). Each action notes what it should change (after turning to face someone, they should be ahead), and plain code compares that with what Pepper perceives next; a mismatch becomes something the mind can act on. It borrows the idea behind world action models without the models, which do not fit Pepper.
Built in 0.8.0, robot test next (#33): a steady camera stream that keeps looking while nobody is in view and looks closer when the view changes, and glances towards voices from where nobody is seen. Every frame can be kept for review (RECORD_CAMERA): the camera records the whole session.
Built in 0.8.0, robot test next (#36): a map of the office and the open space made with NAOqi's own exploration (started by an operator, never by Pepper itself), named places on it kept across restarts, "where you are" on every turn, and driving to a named place.
Remember who Pepper met and what was said, across sessions; recognise returning people. Long-term store built in 0.6.0 (#13): people who agreed to be remembered (consent checked against what Pepper heard), facts, session episodes, 90-day expiry, "forget me". Identity, opt-in, comes next (#14). On the robot since 2026-10-20. Open: identity across encounters.
Bridge autostart and watchdog, per-turn latency budget, session recording. Bridge autostart at boot done 2026-10-01 (verified after a reboot). Watchdog built in 0.7.0 (#15): restarts a bridge that exits or stops answering; on the robot since 2026-10-20, where a killed bridge was back in 11 s. Every robot session is recorded and reviewed; real interactions are the main source of fixes. Done in 0.5 (#16): one folder per session with logs, audio, photos and a per-turn record with timings and decisions, and a review script. Open: a cold boot with the watchdog.
Pepper can be left running in the lab as a presence: M4 and M5 done and a week of unattended daily use without a safety incident or restart.
PepperEvolution