NV's [Computer Vision] Room Occupancy Sensor

What's That?

Open-source ASP.Net Core application and MQTT client that uses computer vision to identify someone's presence in the room. App uses background subtraction algorithms to identify if camera sees something which is not a part of furnishings. MQTT client supports configuration convention used by Home Assistant MQTT integration

How to Use It?

Setup the app to connect it with your camera, MQTT server and have fun! You can run it as a regular ASP.Net Core application on Windows host (x86_64) or Linux host (x86_64 or arm32). Application can use local cameras attached to host, IP cameras (at least the ones which send MJPEG streams) or even a video file.

Getting Started

Prerequisites

Hardware

Place your camera in the area you'd like to monitor.

Find an appropriate host machine that will run the app. Depending on configuration, application may consume significant amount of CPU and RAM, so it would be hard to provide a minimal and recommended configurations. It works on Raspberry Pi 4 with 4 Gb RAM, so you can try something similar or more powerful.

This application exposes HTTP API witout any authorization so you'll need to ensure it's hosted in a secure environment (inaccessible from Internet and so on).

This application does not provide anything that can control external devices or send notifications to end users out of the box. A home automation server with MQTT support would be required to setup various integrations which depends on someone's precense in the room. In most of the cases having a separate host for home automation server would be recommended option.

Software

Prepare your favorite API Explorer. Swagger-UI or Postman will work. Application produces OpenAPI definition that can be downloaded from path /swagger/v1/swagger.json. This format can be consumed by varios API testing tools.

You'll need .NET 6 installed on the host to build and run application.

Building the Application

Nothing special there - just run dotnet build and MSBuild will do the rest for you!

Quick Setup

1. Build application;
2. Update a few required settings in appsettings.json (or supply them as environment variables):
   1. Check that "Urls" property is correct;
   2. Ensure CV:Capture:Source is set to the proper source;
   3. Ensure MQTT settings are correct;
   4. Set Streaming to Enabled to enable mjpeg streams on debug page;
3. Deploy the application (either run dotnet publish or just move NVs.OccupancySensor.API folder to the preferred location); VS Code users can use "publish" task commited to the repository;
4. Start the app by running dotnet occ-sensor.dll in the target folder. By default app will start listening on port 5000;
5. Open /debug.html URL in your favorite browser. If application was successfully deployed you should see the debug page;
6. Start the sensor by making a POST HTTP request to /api/v1/Sensor/Start. You can download API definition from /swagger/v1/swagger.json and use API explorer like Swagger-UI or Postman to simplify requests submission;
7. Refresh the debug page;
8. Adjust the position of your camera using the translations on the debug page;
9. Start MQTT adapter by making a POST HTTP request to /api/v1/MQTTAdapter/Start;
10. Ensure that sensor started to publish MQTT topics to the server;
11. If everything is fine, set Streaming setting back to Disabled to improve your privacy;
12. Additionaly, update StartSensor and StartMQTT settings to True to enable automated start after reboots;
13. Restart application;
14. Read the docs below.

Docs

Configuration

On a high level, application is doing the following actions to find out if someone is present in the room:

1. Camera captures an image;
2. Application builds the foreground mask from the image image;
3. Captured image is getting processed by denoising block;
4. Foreground mask is getting adjusted using the correction mask to exclude false-positives (TV screen, edges of objects...);
5. Finally, detector computes foreground/background pixel ratio;
6. Computed value is getting compared with the threshold: if value is greater than threshold, application decides that someone is present in the room;
7. Detection results are getting published to MQTT broker by MQTT client.

Application has fair amount of configurable items:

• Most of the processing steps mentioned above have some settings to tweak, and most of these settings have defaults;
• Serilog is used to produce the logs;
• And since it's ASP.Net Core application you can configure it's settings, like "Urls", "AllowedHosts" etc.

App uses default .NET configuration providers, so you can change the settings by:

• updating appsettings.json with the values you need;
• overriding the properties using environment variables.
• Please refer to .NET Configuration documentation for additional details.

Camera

• CV:Capture:Source - the source for OpenCV capture. Can be either integer value that identifies your local camera, file path or link to the stream. Default is "0". This value is used to create EmguCV VideoCapture
• CV:Capture:FrameInterval - the timespan that defines how often application will request new frames from the camera. Default is 100 milliseconds.

Background Subtraction

• CV:Subtraction:Algorithm - background subtraction algorithm to use. Default is CNT.
  • CNT - CouNT subtraction algorithm created by sagi-z. This algorithm has a few settings to tweak (documentation):
    • CV:Subtraction:CNT:MinPixelStability - integer, optional. Default is 15
    • CV:Subtraction:CNT:UseHistory - boolean, optional. Default is True
    • CV:Subtraction:CNT:MaxPixelStability - integer, optional. Default is 900
    • CV:Subtraction:CNT:IsParallel - boolean, optional. Default is True

Denoising

• CV:Denoising:Algorithm - denoising algorithm to use. Default is None
  • None - no denoising performed. The image captured from the camera goes directly to detection logic
  • FastNlMeans - (FastNlMeansDenoising function used, documentation can be found here) There are several algorithm options that may be adjusted:
    • CV:Denoising:FastNlMeans:H - rational, optional. Default is 3`
    • CV:Denoising:FastNlMeans:TemplateWindowSize - odd integer, optional. Default is 7
    • CV:Denoising:FastNlMeans:SearchWindowSize - odd integer, optional. Default is 21
  • FastNlMeansColored - (FastNlMeansColoredDenoisingColored function used, documentation can be found here) There are several algorithm options that may be adjusted:
    • CV:Denoising:FastNlMeans:H - rational, optional. Default is 3`
    • CV:Denoising:FastNlMeans:HColor - rational, optional. Default is 3
    • CV:Denoising:FastNlMeans:TemplateWindowSize - odd integer, optional. Default is 7
    • CV:Denoising:FastNlMeans:SearchWindowSize - odd integer, optional. Default is 21
  • MedianBlur - MedianBlur function used. The only adjustable parameter is:
    • CV:Denoising:MedianBlur:K - odd integer greater then 1, optional. Default is 3

Correction

• CV:Correction:Algorithm - a post-processing correction options, optional. Default is None. Valid values are:
  • None - no correction will be performed
  • StaticMask - an additional static mask will be applied to the computed foreground mask. This mode uses additional parameter:
    • CV:Correction:StaticMask:PathToFile - a path to the static mask, optional. Default is "data/correction_mask.bmp". Should be a bi-colored (black and white) bitmap.

Detection

• CV:Detection:Threshold - a rational value between 0 and 1 that defines sensor sensitivity. Bigger values makes detector less sensitive. Default is 0.1

MQTT

Application uses MQTT.Net to build MQTT client. The following settings used to connect application to MQTT broker:

• MQTT:ClientId - the string value that defines client identifier for MQTT client. Required. Does not have a default value
• MQTT:Server - string that containts IP address or DNS name of MQTT server. Required. Does not have a default value
• MQTT:Port - integer, TCP port of MQTT server. Optional. Default is 1883
• MQTT:User - string, username used to authenticate client on the server. Required. Does not have a default value
• MQTT:Password - string, password used to authenticate client on the server. Required. Does not have a default value
• MQTT:Reconnect:AttemptsCount - integer, count of attemps to automatically reconnect to the server if connection was lost. Default is 0
• MQTT:Reconnect:IntervalBetweenAttempts - time span, base delay detween two attemps. Application uses progressive delays, multiplying this value to the current attempt number. Default is 00:00:00

Startup

• StartSensor - boolean toggle to start sensor on startup. Default False
• StartMQTT - boolean toggle to start MQTT client on startup. Default False

Streaming

Application can expose mjpeg streams over HTTP API to simplify setup and debugging. Streams are disabled by default.

• Streaming - setting that controls streaming API. Default is Disabled. Options are:
  • Disabled - no streams avaialable;
  • OnlyFinal - application will stream only resulting frames that passed subtraciton, denoising and correction. This steam is unavailable when sensor is stopped;
  • Enabled - application will stream frames from all the stages, starting from raw camera stream. By design, camera stream is available any time application is running, all other streams are available only when sensor is running.

Logging

This app uses Serilog to capture logs, with File and Console sinks available. Please refer to the documentation for Serilog.Settings.Configuration. Startup process gets logged to startup.ndjson file in the application working directory. Rolling interval is set to 1 day for this log. Application will keep last 10 startup.ndjson log files. This behaviour is hardcoded.

Swagger

When built in debug, application exposes Swagger UI on URL /swagger/index.html. This UI allows to explore and interact with HTTP API exposed by this app.