This is part of I2DS. Utilizes tflite-micro person detector model to detect the presence of people, as well as capture photos when requested by other sensors within the network.
This device performs a few main functions:
- Uses person detection to detect presence of humans + visualize activity over time
- Takes pictures when intrusion is detected by the internal sensor network and uploads to Firebase storage to be viewed through app
- ESP32-CAM board with 4Mb PSRAM (mounted on 2.54mm sockets above mainboard)
- EFR32FG23 microcontroller for receiving warnings from the I2DS sensor network
- VL53L1X ToF sensor
- Array of 10 WS2812-2020 RGB LEDs
- LIS2DH12TR tri-axis accelerometer
- USB bridge and power management circuits
In this project, the main constraints were on system memory.
- Maximum statically allocated RAM is 160kB
- Large amounts of dynamically allocated memory, i.e SSL/TLS, frame buffer
The issues were resolved by utilizing the PSRAM to store most of the Firebase-related protocol overheads + tf arena.
In addition, the tflite person detection model was found to run too slowly for any useful real-time inference. The ESP-NN kernel optimizations were used to bring down the inference time from ca. 5000ms to ca. 400ms, allowing for a practical implementation of this model in the ESP32.
The build flags in platformio.ini must be used or the code will not compile with the kernel optimizations.
The OV2640 camera was not performant upon restart to JPEG mode, often lacking brightness and appearing tinted. This appears to be a known issue (espressif/esp32-camera#314), (espressif/esp32-camera#383) related to the camera AWB.
The workaround used was to wait for min. 150ms after restarting the camera in order to get a vaguely usable frame, then to persist in JPEG mode until it is no longer needed (implemented in commit #b9b3d20).
Note: Due to limited IO, io12 was used for UART rx. ESP32 must be configured in efuse to disregard io12 as strapping pin for VDD_SDIO regulator, or it will fail to boot.
Captured images and other information can be viewed through the same app as the I2DS main system as shown in the screen capture below:
Removing the IR filter from the OV2640 increased low-light performance while not significantly degrading daylight performance (for indoor use only). The flash LED was replaced with a VCSEL IR illuminator.
Queueing a request to capture a frame to upload to Firebase can only be triggered by a warning packet from the internal sensor network. The only requests forwarded to the internal sensor network over IP are state change requests. Hence, it should difficult to falsely trigger this device to take a picture in such a way that privacy is compromised, without the attacker first having physical access and/or security key of the sensor network, appropriate sub-GHz transmitting device, and the Firebase access key.
This device will work with I2DS from commit #9158914.