Second Screen How It Works

Second screen: how it works

Part of Second Screen (Raspberry Pi) ‐ experimental. For developers and the curious.

The two outputs

                         ┌──────────── head unit (Dashwheel) ────────────┐
 cluster widgets ──► ClusterPresentation ─┐                              │
 an app's stack ───► am display move-stack ┤► private VirtualDisplay ──► H.264 encoder (MediaCodec)
                                           │      (StreamDisplay)         │
 OBD / media / nav ──► ClusterFeed ────────┤                              │
 position + route ──► ClusterRouteFeed ────┼──────────────────────────────┤
                         └─────────────── DisplayLink (SecureChannel) ────┘
                                              │ TCP 47811 over the phone's hotspot,
                                              │ or a USB Ethernet cable to the Pi
                         ┌──────────── Raspberry Pi (display/) ──────────┐
                         │ DisplayServer ─► video: gst-launch fdsrc ! h264parse ! v4l2h264dec ! kmssink
                         │              └► data:  Painter draws the pages ─► rawvideoparse ! kmssink
                         │                        (map tiles: TileCache)
                         └───────────────────────────────────────────────┘

• Copies of tile apps. An app inside a dashboard tile draws into a PictureRelay, which draws each frame (on the GPU, as an external texture) both on the tile and on every copy, such as a spot on the cluster. The copy shows the whole window, with thin black bars where the shapes differ, so Google Maps' next turn at the top and arrival time at the bottom stay in sight. Android gives a virtual display only one picture; this is how the same Google Maps shows on the head unit and on the Pi. The 3D map page uses the dashboard's MapLibre map with the route and the next turn.
• Video (full picture, Simple display off). StreamDisplay creates a private virtual display whose surface is the input of a hardware H.264 encoder (MediaCodec, surface input, key frame every 2 s). The cluster is an Android Presentation on that display (ClusterPresentation, the same widgets as the dashboard); in An app mode the app's task stack is moved there with am display move-stack (root shell), the same way tiles park windows. The Pi pipes the stream into GStreamer, which decodes in hardware on the VideoCore and shows it with KMS, no desktop or browser.
• Data (Simple display on, or no encoder). When the encoder can't be had (another app holds it) or Simple display is on, the head unit sends only readings (ClusterFeed, at most one message per 200 ms when something shown changes, with a 5 s heartbeat). The Pi draws the pages itself (Painter) and feeds raw frames to the screen at 5 fps.

Simple display

• The board, drawn by the Pi. Each reading carries the page's arrangement from the page board (layout and slot kinds), so the Pi draws what the board says, in the chosen design: Cards, Dials, Large or Retro. It knows speed, clock, fuel, range, engine, media, directions, readings, doors and the map; a slot holding a copy of Maps or Waze becomes the Pi's own map.
• Newest reading only. The Pi paints on its own thread and always draws the newest state, so it no longer runs behind when a paint is slower than the feed.
• Words once. The labels go over once per language (DisplayWords) and the Pi keeps them, so its own lines (waiting, no signal, the pairing hint) are in Dashwheel's language even before a head unit links.
• The turn sign comes from the in-app navigation's maneuver; for other navigation apps it is guessed from the instruction's words, in all nine languages.
• Alerts go as a title, a detail and a level, drawn full screen with a warning sign.

The Pi's own map

Car-verified 2026-10-10 (tiles shown). With Simple display on, a Map slot makes the Pi draw a flat north-up map itself.

• The head unit adds the car's position and heading to each reading, and sends the route (ClusterRoute, up to 2000 points) when it changes and again when a display links: a few hundred bytes a second, no video, no decoder.
• The Pi fetches 256 px raster tiles from tile_url (tile_url_night for a dark style, else the day tiles are dimmed), with two fetch threads, a memory cache and a disk cache under tile_cache, and fetches ahead along the route. A server that refuses is asked again after a minute. There is no default tile server: the public OpenStreetMap servers are not meant for apps.
• Where the route comes from: the in-app navigation's own route. Google Maps and Waze never give theirs away, so when Dashwheel handed them the destination (a favourite, the agenda, the phone, a voice request), ShadowRoute works the same route out with the in-app navigation's free router, follows the car on it, and routes again when the car is more than 60 m off it for 8 s. Its next turn stands in when the navigation app's own turn can't be read. A destination typed into Maps or Waze themselves is read off their route preview (the tile's picture by on-device text recognition, or the screen through the accessibility service), looked up near the car, and routed the same way; with root, Waze's exact destination comes from its own files (WazeDestination) and wins.

Keeping it live

• The Pi confirms each picture it receives; the head unit never runs more than a third of a second ahead and drops frames the link can't take, instead of letting them pile up.
• Bit rate that settles: the stream remembers the rate that last lost frames and holds just under it. Only after a clean minute does it try a little more, in small steps. A new stream starts just under the known limit; a new connection forgets it. The first seconds of a stream, while the Pi waits for its first full picture, don't count, and one or two lost frames of a still page hold the rate instead of lowering it.
• A slow Pi is not a slow network. The Pi reports frames it dropped because its decoder fell behind apart from frames the link lost, plus its throttle flags and temperature (BoardHealth). Only network losses lower the bit rate; a Pi that can't keep up gets a lighter picture instead: half the frames, then one picture height down. Settings shows under-powered, hot or slowed down by its firmware.
• Each frame goes to the screen as soon as it arrives. Pages with a live map or an app copy stream at 30 fps, others at 15; the cluster keeps sending at full rate so a page change shows at once (150 ms fade).
• Lost links: both sides ping every 5 s; the head unit gives up after 15 s of silence, the Pi after 20 s. The Pi sends its stats in every mode, so they double as a heartbeat. Control messages (mode, brightness, words) go ahead of video frames, and a send stuck for 3 s closes the link rather than waiting.
• Turning the picture upside down is done by the display hardware, not per frame.
• A decoder that keeps failing is rested.

Picture size and bit rate

• The stream size follows the monitor's own size (read from DRM on the Pi), capped by the chosen height (480 / 720 / 1080) and by the Pi's decoder (1920×1080), rounded to multiples of 16.
• Default 2.5 Mbit/s at 720p. The Pi reports shown and dropped frames; the head unit lowers the bit rate when the network loses frames, down to 600 kbit/s (still a legible cluster at 720p), and raises it back as above when the link recovers.

Finding the Pi

• Beacon: until a head unit links, the Pi calls out on every network once a second (UDP broadcast on port 47812). The head unit dials the address it just heard first, so it connects about a second after the Pi is ready, even on a new network.
• DNS-SD: the Pi advertises _dashwheel-display._tcp through Avahi.
• The last address it answered on, tried every 2 s while not connected.
• A scan of the whole subnet every 15 s (Android 11+ randomises the hotspot range). A head unit that is itself the hotspot (a phone running Dashwheel) scans its own hotspot network.
• Cable: the Pi's Ethernet port runs a small DHCP server (systemd-networkd, 10.47.0.1/24, no gateway, no DNS), so a head unit with a USB Ethernet adapter gets an address there and keeps the phone's Wi-Fi for the internet. The head unit looks for the Pi on Ethernet as well as Wi-Fi.
• No Wi-Fi hopping while linked: the Pi's check that moves it to a preferred network (the phone's hotspot) stops while a head unit is linked, since its scans made it roam between access points and drop the picture.

Start-up

1. Power-up: the boot logo (Plymouth, tools/pi/splash/, or your splash.png).
2. The service starts without waiting for the Wi-Fi and keeps the logo up, with a small waiting line and the time in the top right corner, until its first picture.
3. It accepts the head unit's connection before finishing its first drawing, and starts the video player as soon as video is asked for.
4. Between pictures (logo to cluster, video starting) the last picture stays on screen instead of black.

About 17 s from power to picture on a Pi 3B.

Time, language and versions

• Time: a Pi has no clock of its own, and a card written from the image doesn't know the driver's time zone. The head unit sends its clock and zone when it links and with every ping (DisplayTime). The Pi draws with that offset at once, keeps the zone in zone.txt on the card for the screens shown before the next link, and sets its own clock only when it is more than a minute off, so a Pi on network time isn't stepped back and forth.
• Language: the labels and the Pi's own lines follow the language picked in Dashwheel, not only the unit's.
• Versions: the head unit and the card are updated apart. Both sides send their protocol version (now 2) and the Pi its program version; Settings shows the program version and says when the card is older or newer than the app. An older Pi still gets the labels with each reading.

Ready-made card

tools/pi/build-image.sh takes the stock Raspberry Pi OS Lite arm64 image, runs the installer inside it (install.sh --image: no pairing baked in, hostname dashwheel-display, the read-only overlay preinstalled so it needs no internet in the car) and compresses it. Each card makes its own pairing code at its first start. Wi-Fi comes from dashwheel/wifi.txt on the boot partition, read at every start (wifi-from-card.sh). GitHub Actions (.github/workflows/pi-image.yml) rebuilds it on an arm64 runner after each change to display/, link/ or tools/pi/, and publishes it to the Second screen card pre-release, which the app's updater ignores.

Pairing and security

• The Pi creates a random secret on first start and shows it as a QR code dashwheel://display?… (also saved as dashwheel/pairing.txt on the boot partition).
• The companion app scans it and sends it to the head unit over the phone link, or the head unit imports the file from USB.
• Every connection uses the same SecureChannel as the phone link (module link/): both sides prove they hold the secret (HMAC over the handshake), agree fresh keys with ephemeral ECDH P-256, and encrypt every frame with AES-256-GCM. The Pi is the server.

Rules

• Video apps while moving: an app classed as video (category or a known list) is blocked while the car moves, unless the screen is marked Rear-seat screen.
• Page keys: learnt key codes (ClusterKeys) turn the cluster's page, through the accessibility service's key filter when another app is in front. Optionally a long press on media next / previous does it too.
• OBD: the OBD link stays connected while the second screen shows readings, even with the dashboard in the background. Speed and revs are read about every 0.7 s. The speed correction is applied once, on the head unit.
• Alerts full screen: the most urgent alert up on the dashboard covers the second screen while it shows (SecondScreenAlerts), such as doors, belt, tyres, radar or climate, and a call only while it rings; never the volume bar or Gemini Live. Streamed, it is drawn in the dashboard's own design; in Simple display the Pi draws it.
• Rear view: in reverse gear the streamed cluster shows the camera, the radar or both over its page (ReverseSecondScreen), set in Settings → Car → Reverse view. The camera is a copy of Dashwheel's own reverse picture; with the car app's picture there is none, and the radar shows. Not in Simple display.

Files

Where	What
app/.../SecondScreenController.kt	Runs the link, picks video or data, moves app stacks
app/.../SecondScreenModel.kt	Pure rules: output choice, page order, stream size, bit-rate adaptation, lighter picture, board health, version gap, subnet scan (unit-tested in SecondScreenRulesTest)
app/.../SecondScreenPane.kt, SecondScreenStore.kt	Settings → Connections → Second screen
app/.../SecondScreenBoard.kt	The page board: layouts, widgets, designs per page (tested in ClusterLayoutsTest)
app/.../SecondScreenAlerts.kt	Which alert covers the second screen
app/.../PictureRelay.kt	One app picture drawn on its tile and on cluster copies
app/.../StreamDisplay.kt	Virtual display → H.264 encoder
app/.../ClusterPresentation.kt, ClusterScreen.kt, ClusterFeed.kt, ClusterKeys.kt	The cluster, its readings and keys
app/.../ClusterRouteFeed.kt, ShadowRoute.kt	The route for the Pi's map: the in-app navigation's, or the shadow of Maps' or Waze's
app/.../DisplayLink.kt	Discovery, connection, framing
display/	The Pi's JVM service: DisplayServer, Beacon, DisplayPairing, DisplayConfig, ScreenMode, Pipelines, GstProcess, VideoSink, Painter, Screen, TileCache and WebMercator (the map), BoardHealth, LocalClock, Words, ReadOnlyCard, BrightnessButtons, PixelShape, Rotation, BootLogo
tools/pi/	install.sh, build-image.sh, add-wifi.sh, wifi-from-card.sh, prefer-wifi.sh, systemd units, Avahi service, boot splash, display.conf.example, wifi.txt.example, the setup README
.github/workflows/pi-image.yml	Builds and publishes the ready-made card
companion/.../MainActivity.kt	Pair a car: scans the Pi's code and hands it to the head unit

Roadmap ideas

• Drive with the 7" dash panel fitted, and try the Ethernet cable on the K706.
• Shorter time to picture at each ignition, and no black gap when the mode changes.
• Automatic brightness (light sensor, OSD key presses).
• More cluster widgets (car care, tyres).