Basic PTZ object autotracking functionality

docs/docs/configuration/autotracking.md

@@ -0,0 +1,71 @@
+---
+id: autotracking
+title: Autotracking
+---
+
+An ONVIF-capable camera that supports relative movement within the field of view (FOV) can also be configured to automatically track moving objects and keep them in the center of the frame.
+
+## Autotracking behavior
+
+Once Frigate determines that an object is not a false positive and has entered one of the required zones, the autotracker will move the PTZ camera to keep the object centered in the frame until the object either moves out of the frame, the PTZ is not capable of any more movement, or Frigate loses track of it.
+
+Upon loss of tracking, Frigate will scan the region of the lost object for `timeout` seconds. If an object of the same type is found in that region, Frigate will track that new object.
+
+When tracking has ended, Frigate will return to the camera preset specified by the `return_preset` configuration entry.
+
+## Checking ONVIF camera support
+
+Frigate autotracking functions with PTZ cameras capable of relative movement within the field of view (as specified in the [ONVIF spec](https://www.onvif.org/specs/srv/ptz/ONVIF-PTZ-Service-Spec-v1712.pdf) as `RelativePanTiltTranslationSpace` having a `TranslationSpaceFov` entry).
+
+Many cheaper PTZs likely don't support this standard. Frigate will report an error message in the log and disable autotracking if your PTZ is unsupported.
+
+Alternatively, you can download and run [this simple Python script](https://gist.github.com/hawkeye217/152a1d4ba80760dac95d46e143d37112), replacing the details on line 4 with your camera's IP address, ONVIF port, username, and password to check your camera.
+
+## Configuration
+
+First, configure the ONVIF parameters for your camera, then specify the object types to track, a required zone the object must enter, and a camera preset name to return to when tracking has ended. Optionally, specify a delay in seconds before Frigate returns the camera to the preset.
+
+An [ONVIF connection](cameras.md) is required for autotracking to function.
+
+Note that `autotracking` is disabled by default but can be enabled in the configuration or by MQTT.
+
+```yaml
+cameras:
+  ptzcamera:
+    ...
+    onvif:
+      # Required: host of the camera being connected to.
+      host: 0.0.0.0
+      # Optional: ONVIF port for device (default: shown below).
+      port: 8000
+      # Optional: username for login.
+      # NOTE: Some devices require admin to access ONVIF.
+      user: admin
+      # Optional: password for login.
+      password: admin
+      # Optional: PTZ camera object autotracking. Keeps a moving object in
+      # the center of the frame by automatically moving the PTZ camera.
+      autotracking:
+        # Optional: enable/disable object autotracking. (default: shown below)
+        enabled: False
+        # Optional: list of objects to track from labelmap.txt (default: shown below)
+        track:
+          - person
+        # Required: Begin automatically tracking an object when it enters any of the listed zones.
+        required_zones:
+          - zone_name
+        # Required: Name of ONVIF camera preset to return to when tracking is over. (default: shown below)
+        return_preset: home
+        # Optional: Seconds to delay before returning to preset. (default: shown below)
+        timeout: 10
+```
+
+## Best practices and considerations
+
+Every PTZ camera is different, so autotracking may not perform ideally in every situation. This experimental feature was initially developed using an EmpireTech/Dahua SD1A404XB-GNR.
+
+The object tracker in Frigate estimates the motion of the PTZ so that tracked objects are preserved when the camera moves. In most cases (especially for faster moving objects), the default 5 fps is insufficient for the motion estimator to perform accurately. 10 fps is the current recommendation. Higher frame rates will likely not be more performant and will only slow down Frigate and the motion estimator. Adjust your camera to output at least 10 frames per second and change the `fps` parameter in the [detect configuration](index.md) of your configuration file.
+
+A fast [detector](object_detectors.md) is recommended. CPU detectors will not perform well or won't work at all. If Frigate already has trouble keeping track of your object, the autotracker will struggle as well.
+
+The autotracker will add PTZ motion requests to a queue while the motor is moving. Once the motor stops, the events in the queue will be executed together as one large move (rather than incremental moves). If your PTZ's motor is slow, you may not be able to reliably autotrack fast moving objects.

docs/docs/configuration/cameras.md

@@ -66,3 +66,5 @@ cameras:
 ```
 
 then PTZ controls will be available in the cameras WebUI.
+
+An ONVIF-capable camera that supports relative movement within the field of view (FOV) can also be configured to automatically track moving objects and keep them in the center of the frame. For autotracking setup, see the [autotracking](autotracking.md) docs.

docs/docs/configuration/index.md

@@ -555,6 +555,21 @@ cameras:
       user: admin
       # Optional: password for login.
       password: admin
+      # Optional: PTZ camera object autotracking. Keeps a moving object in
+      # the center of the frame by automatically moving the PTZ camera.
+      autotracking:
+        # Optional: enable/disable object autotracking. (default: shown below)
+        enabled: False
+        # Optional: list of objects to track from labelmap.txt (default: shown below)
+        track:
+          - person
+        # Required: Begin automatically tracking an object when it enters any of the listed zones.
+        required_zones:
+          - zone_name
+        # Required: Name of ONVIF camera preset to return to when tracking is over.
+        return_preset: preset_name
+        # Optional: Seconds to delay before returning to preset. (default: shown below)
+        timeout: 10
 
     # Optional: Configuration for how to sort the cameras in the Birdseye view.
     birdseye:

docs/docs/integrations/mqtt.md

@@ -188,3 +188,11 @@ Topic to send PTZ commands to camera.
 | `MOVE_<dir>`           | send command to continuously move in `<dir>`, possible values are [UP, DOWN, LEFT, RIGHT] |
 | `ZOOM_<dir>`           | send command to continuously zoom `<dir>`, possible values are [IN, OUT]                  |
 | `STOP`                 | send command to stop moving                                                               |
+
+### `frigate/<camera_name>/ptz_autotracker/set`
+
+Topic to turn the PTZ autotracker for a camera on and off. Expected values are `ON` and `OFF`.
+
+### `frigate/<camera_name>/ptz_autotracker/state`
+
+Topic with current state of the PTZ autotracker for a camera. Published values are `ON` and `OFF`.

frigate/app.py

@@ -41,7 +41,8 @@
 from frigate.object_processing import TrackedObjectProcessor
 from frigate.output import output_frames
 from frigate.plus import PlusApi
-from frigate.ptz import OnvifController
+from frigate.ptz.autotrack import PtzAutoTrackerThread
+from frigate.ptz.onvif import OnvifController
 from frigate.record.record import manage_recordings
 from frigate.stats import StatsEmitter, stats_init
 from frigate.storage import StorageMaintainer
@@ -134,6 +135,13 @@ def init_config(self) -> None:
                     "i",
                     self.config.cameras[camera_name].motion.improve_contrast,
                 ),
+                "ptz_autotracker_enabled": mp.Value(  # type: ignore[typeddict-item]
+                    # issue https://github.com/python/typeshed/issues/8799
+                    # from mypy 0.981 onwards
+                    "i",
+                    self.config.cameras[camera_name].onvif.autotracking.enabled,
+                ),
+                "ptz_stopped": mp.Event(),
                 "motion_threshold": mp.Value(  # type: ignore[typeddict-item]
                     # issue https://github.com/python/typeshed/issues/8799
                     # from mypy 0.981 onwards
@@ -162,6 +170,7 @@ def init_config(self) -> None:
                 "capture_process": None,
                 "process": None,
             }
+            self.camera_metrics[camera_name]["ptz_stopped"].set()
             self.feature_metrics[camera_name] = {
                 "audio_enabled": mp.Value(  # type: ignore[typeddict-item]
                     # issue https://github.com/python/typeshed/issues/8799
@@ -308,7 +317,7 @@ def init_web_server(self) -> None:
         )
 
     def init_onvif(self) -> None:
-        self.onvif_controller = OnvifController(self.config)
+        self.onvif_controller = OnvifController(self.config, self.camera_metrics)
 
     def init_dispatcher(self) -> None:
         comms: list[Communicator] = []
@@ -362,6 +371,15 @@ def start_detectors(self) -> None:
                 detector_config,
             )
 
+    def start_ptz_autotracker(self) -> None:
+        self.ptz_autotracker_thread = PtzAutoTrackerThread(
+            self.config,
+            self.onvif_controller,
+            self.camera_metrics,
+            self.stop_event,
+        )
+        self.ptz_autotracker_thread.start()
+
     def start_detected_frames_processor(self) -> None:
         self.detected_frames_processor = TrackedObjectProcessor(
             self.config,
@@ -371,6 +389,7 @@ def start_detected_frames_processor(self) -> None:
             self.event_processed_queue,
             self.video_output_queue,
             self.recordings_info_queue,
+            self.ptz_autotracker_thread,
             self.stop_event,
         )
         self.detected_frames_processor.start()
@@ -535,6 +554,7 @@ def start(self) -> None:
             sys.exit(1)
         self.start_detectors()
         self.start_video_output_processor()
+        self.start_ptz_autotracker()
         self.start_detected_frames_processor()
         self.start_camera_processors()
         self.start_camera_capture_processes()
@@ -579,6 +599,7 @@ def stop(self) -> None:
 
         self.dispatcher.stop()
         self.detected_frames_processor.join()
+        self.ptz_autotracker_thread.join()
         self.event_processor.join()
         self.event_cleanup.join()
         self.stats_emitter.join()

frigate/comms/dispatcher.py

@@ -5,7 +5,7 @@
 from typing import Any, Callable
 
 from frigate.config import FrigateConfig
-from frigate.ptz import OnvifCommandEnum, OnvifController
+from frigate.ptz.onvif import OnvifCommandEnum, OnvifController
 from frigate.types import CameraMetricsTypes, FeatureMetricsTypes
 from frigate.util.services import restart_frigate
 
@@ -55,6 +55,7 @@ def __init__(
             "audio": self._on_audio_command,
             "detect": self._on_detect_command,
             "improve_contrast": self._on_motion_improve_contrast_command,
+            "ptz_autotracker": self._on_ptz_autotracker_command,
             "motion": self._on_motion_command,
             "motion_contour_area": self._on_motion_contour_area_command,
             "motion_threshold": self._on_motion_threshold_command,
@@ -159,6 +160,25 @@ def _on_motion_improve_contrast_command(
 
         self.publish(f"{camera_name}/improve_contrast/state", payload, retain=True)
 
+    def _on_ptz_autotracker_command(self, camera_name: str, payload: str) -> None:
+        """Callback for ptz_autotracker topic."""
+        ptz_autotracker_settings = self.config.cameras[camera_name].onvif.autotracking
+
+        if payload == "ON":
+            if not self.camera_metrics[camera_name]["ptz_autotracker_enabled"].value:
+                logger.info(f"Turning on ptz autotracker for {camera_name}")
+                self.camera_metrics[camera_name]["ptz_autotracker_enabled"].value = True
+                ptz_autotracker_settings.enabled = True
+        elif payload == "OFF":
+            if self.camera_metrics[camera_name]["ptz_autotracker_enabled"].value:
+                logger.info(f"Turning off ptz autotracker for {camera_name}")
+                self.camera_metrics[camera_name][
+                    "ptz_autotracker_enabled"
+                ].value = False
+                ptz_autotracker_settings.enabled = False
+
+        self.publish(f"{camera_name}/ptz_autotracker/state", payload, retain=True)
+
     def _on_motion_contour_area_command(self, camera_name: str, payload: int) -> None:
         """Callback for motion contour topic."""
         try:

frigate/comms/mqtt.py

@@ -69,6 +69,11 @@ def _set_initial_topics(self) -> None:
                 "ON" if camera.motion.improve_contrast else "OFF",  # type: ignore[union-attr]
                 retain=True,
             )
+            self.publish(
+                f"{camera_name}/ptz_autotracker/state",
+                "ON" if camera.onvif.autotracking.enabled else "OFF",
+                retain=True,
+            )
             self.publish(
                 f"{camera_name}/motion_threshold/state",
                 camera.motion.threshold,  # type: ignore[union-attr]
@@ -152,6 +157,7 @@ def _start(self) -> None:
             "audio",
             "motion",
             "improve_contrast",
+            "ptz_autotracker",
             "motion_threshold",
             "motion_contour_area",
         ]

frigate/config.py

@@ -128,11 +128,31 @@ def validate_password(cls, v, values):
         return v
 
 
+class PtzAutotrackConfig(FrigateBaseModel):
+    enabled: bool = Field(default=False, title="Enable PTZ object autotracking.")
+    track: List[str] = Field(default=DEFAULT_TRACKED_OBJECTS, title="Objects to track.")
+    required_zones: List[str] = Field(
+        default_factory=list,
+        title="List of required zones to be entered in order to begin autotracking.",
+    )
+    return_preset: str = Field(
+        default="home",
+        title="Name of camera preset to return to when object tracking is over.",
+    )
+    timeout: int = Field(
+        default=10, title="Seconds to delay before returning to preset."
+    )
+
+
 class OnvifConfig(FrigateBaseModel):
     host: str = Field(default="", title="Onvif Host")
     port: int = Field(default=8000, title="Onvif Port")
     user: Optional[str] = Field(title="Onvif Username")
     password: Optional[str] = Field(title="Onvif Password")
+    autotracking: PtzAutotrackConfig = Field(
+        default_factory=PtzAutotrackConfig,
+        title="PTZ auto tracking config.",
+    )
 
 
 class RetainModeEnum(str, Enum):
@@ -892,6 +912,17 @@ def verify_zone_objects_are_tracked(camera_config: CameraConfig) -> None:
                 )
 
 
+def verify_autotrack_zones(camera_config: CameraConfig) -> ValueError | None:
+    """Verify that required_zones are specified when autotracking is enabled."""
+    if (
+        camera_config.onvif.autotracking.enabled
+        and not camera_config.onvif.autotracking.required_zones
+    ):
+        raise ValueError(
+            f"Camera {camera_config.name} has autotracking enabled, required_zones must be set to at least one of the camera's zones."
+        )
+
+
 class FrigateConfig(FrigateBaseModel):
     mqtt: MqttConfig = Field(title="MQTT Configuration.")
     database: DatabaseConfig = Field(
@@ -1067,6 +1098,7 @@ def runtime_config(self, plus_api: PlusApi = None) -> FrigateConfig:
             verify_recording_retention(camera_config)
             verify_recording_segments_setup_with_reasonable_time(camera_config)
             verify_zone_objects_are_tracked(camera_config)
+            verify_autotrack_zones(camera_config)
 
             if camera_config.rtmp.enabled:
                 logger.warning(

frigate/http.py

@@ -34,7 +34,7 @@
 from frigate.models import Event, Recordings, Timeline
 from frigate.object_processing import TrackedObject
 from frigate.plus import PlusApi
-from frigate.ptz import OnvifController
+from frigate.ptz.onvif import OnvifController
 from frigate.record.export import PlaybackFactorEnum, RecordingExporter
 from frigate.stats import stats_snapshot
 from frigate.storage import StorageMaintainer