RaftCamera

Camera SysMod (RaftCamera)

Optional library that adds a camera to a Raft app: still JPEG captures to the local file system or an SD card, timed interval-capture sessions, and frames published to subscribers. Built on Espressif's esp32-camera driver for the ESP32-S3 (the sensor, e.g. an OV2640, encodes the JPEG).

Source: RaftCamera

Adding it to an app

1. Add RaftCamera@main to RAFT_COMPONENTS in the SysType's features.cmake.

2. Register the SysMod in main.cpp:

   #include "RegisterCamera.h"
   RegisterCamera::registerCamera(raftCoreApp.getSysManager());

3. Add the camera driver's Kconfig defaults to the SysType's sdkconfig.defaults:

   CONFIG_CAMERA_TASK_STACK_SIZE=8192
   CONFIG_CAMERA_CORE_0=y
   CONFIG_CAMERA_DMA_BUFFER_SIZE_MAX=32768
   

4. Add a Camera section to the SysType (below). PSRAM is needed for frames larger than CIF.

For captures to an SD card, enable RaftCore's SD support (set(ENV{RAFT_ENABLE_SD} "1") in features.cmake) and configure the SD pins in the FileManager section.

SysType configuration

"Camera": {
    "enable": 1,
    "imageDir": "/images",
    "imageFileSystem": "local",
    "publishTopic": "Camera",
    "camera": {
        "pinSCL": 41, "pinSDA": 42, "pinRST": 40, "pinPWDN": -1,
        "pinMCLK": 37, "pinVSYN": 39, "pinHREF": 38, "pinPCLK": 36,
        "pinD0": 12, "pinD1": 13, "pinD2": 14, "pinD3": 15,
        "pinD4": 16, "pinD5": 17, "pinD6": 18, "pinD7": 11,
        "xClkFreqHz": 20000000,
        "ledcTimer": 1,
        "ledcChannel": 4,
        "size": "QVGA",
        "maxSize": "UXGA",
        "quality": 15,
        "capturefps": 0,
        "fbCount": 1
    }
}

Key	Default	Meaning
enable	1	Enable the SysMod
imageDir	/images	Default folder for captures
imageFileSystem	local	Default file system for captures (local or sd)
publishTopic	Camera	Name of the publish data source
captureTaskStack	8192	Worker task stack (bytes)
captureTaskPriority	1	Worker task priority
captureTaskCore	the core the main task doesn't use	Worker task core
captureTaskPinToCore	1	Pin the worker task to that core

The camera object holds the pins and sensor settings:

Key	Meaning
pin..., xClkFreqHz, ledcTimer, ledcChannel	The camera's pins, clock and the LEDC timer/channel that generates the clock
size	Frame size at startup (96X96, QQVGA, QCIF, HQVGA, 240X240, QVGA, CIF, HVGA, VGA, SVGA, XGA, HD, SXGA, UXGA)
maxSize	The largest frame size set and start may change to (default size). The frame buffers are allocated for it at setup, in PSRAM - 384 KB for UXGA. A larger size is refused (without it, a larger frame overflows the buffer)
quality	JPEG quality 0-63 (lower is better quality, larger files)
capturefps	Throttle on captures for publishing (0: none)
fbCount	Frame buffers. With one, a capture discards the frame waiting in the buffer (taken when the last one was returned) and two frames after a size change, so the image is current - adding up to a frame time

REST API

One camera endpoint, with an action parameter:

Request	Does
camera?action=status	Enabled/ready, size and maxSize, quality, image count, queued jobs, and the interval-capture session's statistics
camera?action=capture[&fs=local|sd][&folder=/images][&filename=cam.jpg]	Capture a JPEG to a file (a timestamped name if no filename)
camera?action=set[&size=VGA][&quality=16]	Change the frame size and/or quality
camera?action=start[&fs=..][&folder=..][&intervalMs=..|fps=..][&maxImages=..][&size=..][&quality=..]	Start an interval-capture session (default one image a minute; maxImages 0 is unlimited)
camera?action=stop	Stop the session

The folder is created if needed, and paths with .. are refused.

Keeping the main loop free

All camera work runs on the SysMod's worker task, not the main loop: captures, frame size and quality changes (which talk to the sensor), interval captures, frames for publishing, and writing the image files (in 4 KB pieces, respecting the file system's write pacing).

• capture and set are queued for the worker, and the REST response is deferred until the job is done. Over a transport that can't defer, the response comes at once with "queued":true (and the filename), and status shows the result. At most 4 jobs are queued (busy otherwise).
• status, publishing and the REST handlers only take a lock that is never held while the camera or the file system is in use.

Saving to the local (flash) file system still stalls the whole chip for each flash erase, unless the app runs from PSRAM - see File System. Captures to an SD card don't.

Publishing frames

While the publish topic has a subscriber (StatePublisher), the worker captures frames (throttled by capturefps) and each frame is published once, at the subscribed rate, at most about 1 s old. The frame is handed to the message without copying. Each message is a 19-byte little-endian header and the JPEG:

Offset	Size	Field
0	1	Version (1)
1	4	Image count
5	4	Timestamp (ms)
9	2	Width
11	2	Height
13	1	Pixel format
14	1	JPEG quality
15	4	JPEG length
19	n	JPEG data

Sending a frame (encoding it and writing it to the connection) is done by the comms channel in the main loop, and takes time in proportion to its size. Measured on an ESP32-S3 with quad PSRAM over a WebSocket: about 5-6 ms for a QVGA frame (7 KB), 14 ms for VGA (18 KB) and 22 ms for SVGA (29 KB). Keep published frames small where the main loop has to stay responsive.

Over BLE, a frame is queued for the BLE stack to send in packets (without being copied again), and the main loop is held only while it is encoded: about 5.5 ms for a 10 KB frame. The link carries about 20-30 KB/s with notifications (the default for publishes), so expect one or two QVGA frames a second, and fewer at larger sizes; a publish waits while the BLE publish queue holds pubQMaxBytes (32 KB by default), so frames are at most about a second old - see BLE Manager: large messages. With RaftSysMods 1.27.0 and earlier, frames over BLE never arrived: a packet was lost from almost every frame (see Notifications and NimBLE's buffers).