Generative AI Custom Prompts

[wobl3r]

I have managed to get generative AI working but am trying to get more personalized replies.
For the detection of a car what I would like is, if the license plate is recognized a reply would have the details of the car.
eg. "Dads car is seen parking in the driveway or a particular zone of a particular camera"
I understand we can use variable replacements but what would they be for this. I would not wish to have a reply like "A car, license plate number, dads car" But just "Dads car" with a description of the scene.

With all that said a new discussion category would be nice where people could share the custom prompts they have created helping others fine tune there own configs.

[inkeep-ai-assistant[bot]]

Based on the documentation, Frigate's generative AI supports variable replacements in prompts including {label}, {sub_label}, and {camera} (1). The {sub_label} variable is particularly relevant for your use case, as it would contain the license plate information when license plate recognition is enabled and a plate is detected (1).

You can create custom prompts using these variables. Here's the configuration format from the documentation:

genai:
  enabled: True
  provider: ollama
  base_url: http://localhost:11434
  model: llava
  prompt: "Analyze the {label} in these images from the {camera} security camera. Focus on the actions, behavior, and potential intent of the {label}, rather than just describing its appearance."
  object_prompts:
    person: "Examine the main person in these images. What are they doing and what might their actions suggest about their intent (e.g., approaching a door, leaving an area, standing still)? Do not describe the surroundings or static details."
    car: "Observe the primary vehicle in these images. Focus on its movement, direction, or purpose (e.g., parking, approaching, circling). If it's a delivery vehicle, mention the company."

(1)

For your specific use case with license plates, you could customize the car prompt to use the {sub_label} variable. However, the documentation doesn't provide specific examples of how to integrate license plate information into custom prompts or how to map license plates to custom names like "Dad's car" (1)(2).

The prompt system works by using object-specific prompts when available, or falling back to the default prompt (3). If you have a specific prompt defined for an object type, that prompt is used without combining it with the default prompt (3).

For camera-level customization, you can also override prompts at the individual camera level (1):

cameras:
  front_door:
    genai:
      use_snapshot: True
      prompt: "Analyze the {label} in these images from the {camera} security camera at the front door. Focus on the actions and potential intent of the {label}."
      object_prompts:
        person: "Examine the person in these images. What are they doing, and how might their actions suggest their purpose (e.g., delivering something, approaching, leaving)? If they are carrying or interacting with a package, include details about its source or destination."

(1)

The documentation suggests experimenting with prompts using your AI provider's chat interface and downloading thumbnails from Frigate to test different prompt variations before updating your configuration (1).

📚 Sources:

1. Generative AI
2. Full Reference Config
3. Generative AI - Need clarification on options for prompt

---

Have another question? Just tag @inkeep.

[wobl3r]

Many Thanks!

I will give this a try and see how it goes.

car: "Describe this vehicle in a natural sentence using {sub_label} if available (name or license plate), otherwise {label}, and include its color, action, and the {camera} location. What {zone} is it in. Focus on its movement, direction, or purpose (e.g., parking, approaching, circling). If it's a delivery vehicle, mention the company."

[wobl3r]

No response was generated from the above prompt :(

[hawkeye217]

All frigate does is pass your request on to your GenAI provider. If no response is generated, you'll want to check the logs or usage limits of your provider.

[wobl3r]

Reading the logs i have KeyError: 'zone' this is obviously an unrecognized key, but... This completely crashes out the gen-ai process and the only way to restart the gen-ai is to restart frigate.

2025-08-20 12:10:24.158319394  Exception in thread embeddings_maintainer:
2025-08-20 12:10:24.158321709  Traceback (most recent call last):
2025-08-20 12:10:24.158322584    File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
2025-08-20 12:10:24.158323145      self.run()
2025-08-20 12:10:24.158323878    File "/opt/frigate/frigate/embeddings/maintainer.py", line 178, in run
2025-08-20 12:10:24.158324503      self._process_event_metadata()
2025-08-20 12:10:24.158333590    File "/opt/frigate/frigate/embeddings/maintainer.py", line 448, in _process_event_metadata
2025-08-20 12:10:24.158334507      self.handle_regenerate_description(
2025-08-20 12:10:24.158335339    File "/opt/frigate/frigate/embeddings/maintainer.py", line 678, in handle_regenerate_description
2025-08-20 12:10:24.158336275      self._genai_embed_description(event, embed_image)
2025-08-20 12:10:24.158337095    File "/opt/frigate/frigate/embeddings/maintainer.py", line 573, in _genai_embed_description
2025-08-20 12:10:24.158342791      description = self.genai_client.generate_description(
2025-08-20 12:10:24.158343497                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-20 12:10:24.158344496    File "/opt/frigate/frigate/genai/__init__.py", line 43, in generate_description
2025-08-20 12:10:24.158345172      prompt = camera_config.genai.object_prompts.get(
2025-08-20 12:10:24.158348417               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-20 12:10:24.158348974  KeyError: 'zone'

[hawkeye217]

Right, Frigate assumes your keys in the prompt are valid. A code change could be made to prevent the crash, but an invalid prompt would still be sent to your provider.

[hawkeye217]

I've put up a PR to catch invalid keys so that it doesn't crash the embeddings maintainer: #19657