Proactive Notifications Apps

[beastoin]

Overview

Issue: #1235 #1132

Features

• Developer could create new apps with the proactive notification capacity

Usages

• Developer create a new app with capacity proactive_notification, and external_integration to trigger the transcript_processed.

  {
    "id": "mentor-01",
    ...
    "capabilities": [
      "external_integration",
      "proactive_notification",
    ],
    "external_integration": {
      "triggers_on": "transcript_processed",
      "webhook_url": "https://based-hardware-development--plugins-api.modal.run/mentor",
      ...
    },
    "proactive_notification": {
        "scopes": ["user_name", "user_facts"]
    },
    ...
  }
  

• Webhook respond with the format:

  {
    "notification": {
        "prompt": "the prompt template, with `{{user_name}}` and `{{user_facts}}`, Omi will use this prompt to ask LLM then send a notification to user",
        "params": ["user_name", "user_facts"]
    }
  }
  

• (Optional) you could detect the codeword before responding with the mentor instruction, either by using the LLM or just a simple regex:

  ai_names = ['Omi', 'Omie', 'Homi', 'Homie']
  codewords = [f'hey {ai_name} what do you think' for ai_name in ai_names] ai_name in ai_names]
  transcript = TranscriptSegment.segments_as_string(segments)
  text_lower = normalize(transcript)
  pattern = r'\b(?:' + '|'.join(map(re.escape, [normalize(cw) for cw in codewords])) + r')\b'
  if bool(re.search(pattern, text_lower)):
      # respond 

Technical details

Examples

TODO

• A dead simple app which could trigger codeword and send the notification
• Put the LLM to it
• user_facts scope
• Put capacity proactive_notification
• Refine the document

Future idea

• memory_context scope
• rate limits per plugin per user per 5s - 1

🚀 Deploy Steps

• Merge https://github.com/BasedHardware/omi/pull/1240/files
• Create plugin.

  curl -X 'POST' \
  'https://based-hardware-development--backend-thinh-v2-api.modal.run/v3/plugins' \
  -H 'accept: application/json' \
  -H 'authorization: <KEY>' \
  -H 'Content-Type: multipart/form-data' \
  -F 'plugin_data={"name":"Mentor.01","author":"@thinh","description":"Mentor.01 - An AI-powered mentor, designed to elevate your meetings and help you achieve your goals. With its insightful guidance and real-time support, you'\''ll gain the confidence and skills to excel in every interaction.","image":"/plugins/logos/mentor_01.jpg","capabilities":["external_integration","proactive_notification"],"external_integration":{"triggers_on":"transcript_processed","webhook_url":"https://based-hardware-development--plugins-api.modal.run/mentor","setup_completed_url":"https://based-hardware-development--plugins-api.modal.run/setup/mentor","setup_instructions_file_path":"https://raw.githubusercontent.com/BasedHardware/Omi/main/plugins/instructions/mentor_01/README.md"},"proactive_notification":{"scopes":["user_name","user_facts"]},"deleted":false,"private":false}' \
  -F 'file=@mentor_01.jpg;type=image/jpeg'
  

• Deploy Pusher service
• Deploy Plugin service

Summary by CodeRabbit

Release Notes

• New Features

  • Introduced a new optional field for notifications to hold additional message content.
  • Added a ProactiveNotification class to enhance plugin capabilities with proactive notifications.
  • Implemented a mentoring plugin with endpoints for real-time interaction and setup.
  • Enhanced audio transcription handling with improved error management and task organization.
  • Added functionality for managing transcript segments with a new method for upserting segments.

• Bug Fixes

  • Improved error handling in plugin management and chat message processing.

• Documentation

  • Updated README with instructions for using the new mentoring feature.

• Refactor

  • Streamlined logic in various methods for better performance and readability.

[tiagoefreitas]

Does it work on IOS?
Can I do anything to help expedite this?
Will test soon.