Skip to content
Permalink
Browse files

Update README.

Signed-off-by: Bofu Chen (bafu) <bofu@dt42.io>
  • Loading branch information...
bafu committed Apr 5, 2019
1 parent a365da6 commit c5449234e32c53195d1cb9f32a2ea581f5c98601
Showing with 198 additions and 50 deletions.
  1. +42 −50 README.md
  2. +156 −0 README.md.old
@@ -44,96 +44,88 @@ $ ./configure

# Start and Stop BerryNet

BerryNet is managed by [systemd](https://freedesktop.org/wiki/Software/systemd/). You can manage BerryNet via `berrynet-manager`:
BerryNet performs an AIoT application by connecting independent components together. Component types include but not limited to AI engine, I/O processor, data processor (algorithm), or data collector.

We recommend to manage BerryNet componetns by by [supervisor](http://supervisord.org/), but you can also run BerryNet components manually. You can manage BerryNet via `berrynet-manager`:

```
$ berrynet-manager [start | stop | status | log]
```

The default application has three components:

# Configuration

All the configurations are in `config.js`.

* Choose AI Engine.

* Two types of AI engines currently: object classifier and object detector.

* Configure IP camera's snapshot access interface.
* Camera client to provide input images
* Object detection engine to find type and position of the detected objects in an image
* Dashboard to display the detection results

* Please refer to [IP camera setup](doc/ipcam.md) for more details.
You will learn how to configure or change the components in the Configuration section.

* MQTT topics.

# Dashboard: Freeboard

# Dashboard
## Open Freeboard on RPi3 (with touch screen)

## Open dashboard on RPi3 (with touch screen)
Freeboard is a web-based dashboard. Here are the steps to show the detection result iamge and text on Freeboard:

Open browser and enter the URL:
* 1: Enter `http://127.0.0.1:8080` in browser's URL bar, and press enter
* 2: [Download](https://raw.githubusercontent.com/DT42/BerryNet/master/config/dashboard-darknet.json) the Freeboard configuration for default application, `dashboard-darknet.json`
* 2: Click `LOAD FREEBOARD`, and select the newly downloaded `dashboard-darknet.json`
* 3: Wait for seconds, you should see the inference result image and text on Freeboard

`http://localhost:8080/index.html#source=dashboard.json`
## Open Freeboard on another computer

The default dashboard configuration file will be loaded.
Assuming that BerryNet default application runs on device A with IP `192.168.1.42`, and you want to see the detection result on device B with IP `192.168.1.43`:

## Open dashboard on browser from any computer
* 1: Enter `http://192.168.1.42:8080` in browser's URL bar, and press enter
* 2: [Download](https://raw.githubusercontent.com/DT42/BerryNet/master/config/dashboard-darknet.json) the Freeboard configuration for default application, `dashboard-darknet.json`
* 3: Replace all the `localhost` to `192.168.1.42` in `dashboard-darknet.json`
* 2: Click `LOAD FREEBOARD`, and select the newly downloaded `dashboard-darknet.json`
* 3: Wait for seconds, you should see the inference result image and text on Freeboard

Open browser and enter the URL:
For more details about dashboard configuration (e.g. how to add widgets), please refer to [Freeboard project](https://github.com/Freeboard/freeboard).

`http://<gateway-ip>:8080/index.html#source=dashboard.json`

Click the data sources, and change MQTT broker's IP address to the gateway's IP.

For more details about dashboard configuration (e.g. how to add widgets), please refer to [freeboard project](https://github.com/Freeboard/freeboard).

# Enable Data Collector

# Provide Image Input
You might want to store the snapshot and inference results for data analysis.

To capture an image via configured IP camera
To run BerryNet data collector manually, you can run the command below:

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m snapshot_ipcam
$ bn_data_collector --topic-config <topic-config-filepath> --data-dirpath <result-dirpath>
```

To capture an image via board-connected camera (RPi camera or USB webcam)
The topic config indicates what MQTT topic the data collector will listen, and what handler will be triggered. Here is a topic config exmaple:

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m snapshot_boardcam
{
"berrynet/engine/darknet/result": "self.update"
}
```

To provide a local image
The inference result image and text will be saved into the indicated result directory.

```
$ mosquitto_pub -h localhost -t berrynet/event/localImage -m <image_path>
```

To start and stop streaming from board-connected camera
# Configuration

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_boardcam_start
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_boardcam_stop
```
The default supervisor config is at `/etc/supervisor/conf.d/berrynet-darknet.conf`.

To start and stop streaming from Nest IP camera
## Camera Client

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_nest_ipcam_start
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_nest_ipcam_stop
```
BerryNet camera client can run in two modes: stream or file. In stream mode, local camera (e.g. USB camera and RPi camera) and IP camera can be supported, and input frame rate (FPS) can be changed on demand (default is 1). In file mode, user can indicate filepath as input source.

To run camera client in stream mode:

# Enable Data Collector

You might want to store the snapshot and inference results for data analysis.
```
$ bn_camera --fps 5
```

To enable data collector, you can set the storage directory path in config.js:
To run camera client in file mode:

```
config.storageDirPath = '<data-storage-dirpath>';
$ bn_camera --mode file --filepath <image-filepath>
```

and restart BerryNet.


# Use Your Data To Train

@@ -0,0 +1,156 @@
# BerryNet: Deep Learning Gateway on Raspberry Pi And Other Edge Devices

This project turns edge devices such as Raspberry Pi 3 into an intelligent gateway with deep learning running on it. No internet connection is required, everything is done locally on the edge device itself. Further, multiple edge devices can create a distributed AIoT network.

At DT42, we believe that bringing deep learning to edge devices is the trend towards the future. It not only saves costs of data transmission and storage but also makes devices able to respond according to the events shown in the images or videos without connecting to the cloud.

![Figure 1](https://user-images.githubusercontent.com/292790/45943626-a3d28b80-c019-11e8-829c-5eb6afd3faa4.png)

<p align="center">Figure 1: BerryNet architecture</p>

Figure 1 shows the software architecture of the project, we use Node.js/Python, MQTT and an AI engine to analyze images or video frames with deep learning. So far, there are two default types of AI engines, the classification engine (with Inception v3 [[1]](https://arxiv.org/pdf/1512.00567.pdf) model) and the object detection engine (with TinyYOLO [[2]](https://pjreddie.com/media/files/papers/YOLO9000.pdf) model or MobileNet SSD [[3]](https://arxiv.org/pdf/1704.04861.pdf) model). Figure 2 shows the differences between classification and object detection.

![Figure 2](https://cloud.githubusercontent.com/assets/292790/25520013/d9497738-2c2c-11e7-9693-3840647f2e1e.jpg)

<p align="center">Figure 2: Classification vs detection</p>

One of the application of this intelligent gateway is to use the camera to monitor the place you care about. For example, Figure 3 shows the analyzed results from the camera hosted in the DT42 office. The frames were captured by the IP camera and they were submitted into the AI engine. The output from the AI engine will be shown in the dashboard. We are working on the Email and IM notification so you can get a notification when there is a dog coming into the meeting area with the next release.

![Figure 3](https://cloud.githubusercontent.com/assets/292790/25498294/0ab79976-2bba-11e7-9114-46e328d15a18.gif)

<p align="center">Figure 3: Object detection result example</p>

To bring easy and flexible edge AI experience to user, we keep expending support of the AI engines and the reference HWs.

![Figure 4](https://user-images.githubusercontent.com/292790/45943641-b6e55b80-c019-11e8-9c90-28c96074c577.png)

<p align="center">Figure 4: Reference hardwares</p>


# Installation

You can install BerryNet by using pre-built image or from source. Please refer to the [Wiki page](https://github.com/DT42/BerryNet/wiki/Installation) for the details.

We are pushing BerryNet into Debian repository, so you will be able to install by only typing one command in the future.

Here is the quick steps to install from source:

```
$ git clone https://github.com/DT42/BerryNet.git
$ cd BerryNet
$ ./configure
```


# Start and Stop BerryNet

BerryNet is managed by [systemd](https://freedesktop.org/wiki/Software/systemd/). You can manage BerryNet via `berrynet-manager`:

```
$ berrynet-manager [start | stop | status | log]
```


# Configuration

All the configurations are in `config.js`.

* Choose AI Engine.

* Two types of AI engines currently: object classifier and object detector.

* Configure IP camera's snapshot access interface.

* Please refer to [IP camera setup](doc/ipcam.md) for more details.

* MQTT topics.


# Dashboard

## Open dashboard on RPi3 (with touch screen)

Open browser and enter the URL:

`http://localhost:8080/index.html#source=dashboard.json`

The default dashboard configuration file will be loaded.

## Open dashboard on browser from any computer

Open browser and enter the URL:

`http://<gateway-ip>:8080/index.html#source=dashboard.json`

Click the data sources, and change MQTT broker's IP address to the gateway's IP.

For more details about dashboard configuration (e.g. how to add widgets), please refer to [freeboard project](https://github.com/Freeboard/freeboard).


# Provide Image Input

To capture an image via configured IP camera

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m snapshot_ipcam
```

To capture an image via board-connected camera (RPi camera or USB webcam)

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m snapshot_boardcam
```

To provide a local image

```
$ mosquitto_pub -h localhost -t berrynet/event/localImage -m <image_path>
```

To start and stop streaming from board-connected camera

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_boardcam_start
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_boardcam_stop
```

To start and stop streaming from Nest IP camera

```
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_nest_ipcam_start
$ mosquitto_pub -h localhost -t berrynet/event/camera -m stream_nest_ipcam_stop
```


# Enable Data Collector

You might want to store the snapshot and inference results for data analysis.

To enable data collector, you can set the storage directory path in config.js:

```
config.storageDirPath = '<data-storage-dirpath>';
```

and restart BerryNet.


# Use Your Data To Train

The original instruction of retraining YOLOv2 model see [github repository of darknet](https://github.com/AlexeyAB/darknet#how-to-train-to-detect-your-custom-objects)

In the current of BerryNet, TinyYolo is used instead of YOLOv2.
The major differences are:

1. Create file yolo-obj.cfg with the same content as in `tiny-yolo.cfg`
2. Download pre-trained weights of darknet reference model, `darknet.weights.12`, for the convolutional layers (6.1MB)
https://drive.google.com/drive/folders/0B-oZJEwmkAObMzAtc2QzZDhyVGM?usp=sharing

The rest parts are the same as retraining YOLO.

If you use [LabelMe](http://labelme.csail.mit.edu/Release3.0/) to annotate data, `utils/xmlTotxt.py` can help convert the xml format to the text format that darknet uses.


# Discussion

Please refer to the [Telegram Group](https://t.me/berrynetdev) or [Google Group](https://groups.google.com/a/dt42.io/d/forum/berrynet) for questions, suggestions, or any idea discussion.

0 comments on commit c544923

Please sign in to comment.
You can’t perform that action at this time.