Skip to content

Commit

Permalink
feats: trace visualizer, docs
Browse files Browse the repository at this point in the history
* docs: add readme for webcam

* docs: update header image

* feat: trace visualizer
  • Loading branch information
iChizer0 committed Apr 23, 2024
1 parent 8e443d9 commit 1cfbebb
Show file tree
Hide file tree
Showing 7 changed files with 842 additions and 685 deletions.
56 changes: 56 additions & 0 deletions examples/camera_web_server/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Grove Vision AI V2 Web Camera Example

![Tracking Example](img/example_0.gif)

Run AI models, streaming, tracking and visualizing the results using [Grove Vision AI V2](https://www.seeedstudio.com/Grove-Vision-AI-V2-Kit-p-5852.html) on a web page, an expanded rework of [ESP32 CameraWebServer](https://github.com/espressif/arduino-esp32/tree/master/libraries/ESP32/examples/Camera/CameraWebServer).

## Prerequisites

Software:

- [Arduino IDE](https://www.arduino.cc/en/software)
- [Seeed_Arduino_SSCMA](https://github.com/seeed-studio/Seeed_Arduino_SSCMA)
- [ArduinoJson](https://arduinojson.org/v7/how-to/install-arduinojson/)
- [ArduinoEigen](https://www.arduino.cc/reference/en/libraries/eigen/)

Hardware:

- [Grove Vision AI V2](https://www.seeedstudio.com/Grove-Vision-AI-V2-Kit-p-5852.html)
- Espressif Arduino capble MCU board that supports Wi-Fi, e.g. [XIAO (ESP32)](https://www.seeedstudio.com/XIAO-ESP32S3-p-5627.html)
- Router

## Getting Started

1. Install the required libraries in Arduino IDE.
2. Open the example in Arduino IDE: `File` -> `Examples` -> `Seeed_Arduino_SSCMA` -> `camera_web_server`.
3. Select the correct board and port.
4. Edit the `ssid` and `password` in the `camera_web_server.ino` sketch to match your Wi-Fi network.
5. Upload the sketch to the board, and open the Serial Monitor, when connected to the Wi-Fi network, the IP address of the board will be printed, connect to it using a web browser.

## Features

### HTTP API

- `/` - Main page with a video stream from the camera
- `/stream` - Stream from the camera (stream server runs on port 8080)
- `/stream/frame` - camera fream example
- `/stream/result` - invoke/sample raw results in JSON format
- `/command?base64=` - Send a base64 encoded AT command to the board

### URL only streaming

You can use the URL `http://<IP_ADDRESS>/stream/frame` to view the camera stream on browsers or other APPS that support MJPEG streaming without any additinal process.

![Streaming Example](img/example_1.gif)

### AI inference results visualization

The web camera page supports visualizing bounding boexes, poses, classes and keypoints, and for each trackable objects (bounding boxes), we put a unique ID on its top left corner.

![Pose Example](img/example_2.gif)

### Userfriendly error handling

The web camera page will notify the user if there is an issue during streaming and the potential cause of the issue, e.g. the camera cable is lossen, or the device lost connection to the Wi-Fi network.

![Error handling Example](img/example_3.gif)
Binary file added examples/camera_web_server/img/example_0.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/camera_web_server/img/example_1.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/camera_web_server/img/example_2.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/camera_web_server/img/example_3.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
60 changes: 59 additions & 1 deletion examples/camera_web_server/web/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -913,6 +913,14 @@
syncIou(iouSlider.value);
};

const traceLimit = 15;
let traceMap = {};
let traceStamp = 0;

const clearTraceMap = () => {
traceMap = {};
};

const stopStream = async () => {
disable(saveButton);
disable(scoreSlider);
Expand All @@ -925,6 +933,8 @@
}
isStreaming = false;

clearTraceMap();

await fetch((() => {
const cmd = 'BREAK';
const encodedCmd = btoa(cmd);
Expand All @@ -948,6 +958,50 @@
enable(differedSwitch);
};

const removeStaledTraces = () => {
for (const id in traceMap) {
if (traceMap[id]['stamp'] != traceStamp) {
delete traceMap[id];
}
}
traceStamp += 1;
};

const plotTrace = async(id, pt_x, pt_y, color) => {
if (id == -1) {
return;
}
try {
if (traceMap.hasOwnProperty(id)) {
const trace = traceMap[id];
trace['pts'].push([pt_x, pt_y]);
while (trace['pts'].length > traceLimit) {
trace['pts'].shift();
}
trace['stamp'] = traceStamp;
} else {
traceMap[id] = {
'stamp': traceStamp,
'pts': [[pt_x, pt_y]]
};
}

// draw trace line between every 2 points
streamPreviewCtx.strokeStyle = color;
streamPreviewCtx.lineWidth = 1.5;
streamPreviewCtx.beginPath();
for (let i = 1; i < traceMap[id]['pts'].length; i += 1) {
const pt = traceMap[id]['pts'][i];
const pt_prev = traceMap[id]['pts'][i - 1];
streamPreviewCtx.moveTo(pt_prev[0], pt_prev[1]);
streamPreviewCtx.lineTo(pt[0], pt[1]);
}
streamPreviewCtx.stroke();
} catch (e) {
console.error('Failed to plot trace:', e);
}
};

const visualizeBoundingBoxes = (boxes) => {
let classesMap = null;
if (modelInfo && modelInfo.hasOwnProperty('classes')) {
Expand Down Expand Up @@ -982,7 +1036,11 @@
streamPreviewCtx.font = 'bold 12px arial';
streamPreviewCtx.fillStyle = '#ffffff';
streamPreviewCtx.fillText(`${tag}: ${score}`, x - w / 2 + 5, y - h / 2 - 2);

plotTrace(id, x, y, color);
}

removeStaledTraces();
};

const visualizeClasses = (classes) => {
Expand Down Expand Up @@ -1488,7 +1546,7 @@
};

const streamPreviewReadableStream = async (url, { signal }) => {
const response = await fetch(url, { signal });
const response = await fetch(url, { cache: 'no-cache', signal:signal });
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
Expand Down
Loading

0 comments on commit 1cfbebb

Please sign in to comment.