Meowster Helper is a multi-modal, agentic pet mental and physical health companion system built for the AMD Developer Hackathon ACT II. By fusing real-time visual emotion recognition, acoustic vocalization analysis, and autonomous LLM agent execution, Meowster Helper transforms how pet owners understand, track, and manage their pets' well-being.

• Project Overview
• Problem Statement
• Key Features
• Releases
• System Architecture
• Technical Architecture
• GitHub Repositories
• Quickstart Guide
• Project Team
• License

Meowster Helper is designed to bridge the gap between pet perception (what pets feel and express) and autonomous intervention (the steps taken to keep them healthy). While Meowster Helper launches with optimized cat emotion models, the entire architecture is species-agnostic, designed to scale to dogs and other domestic pets. The system functions through a unified five-step lifecycle:

1. Perception: Starts in an interactive 2.5D pet room before launching an immersive camera experience to capture pet moments (images, 10s video clips, or vocalization audio).
2. Analysis: Evaluates the capture using custom Computer Vision (emotion detection).
3. Ingestion: Streams perception payloads into the Agentic Orchestrator over real-time Server-Sent Events (SSE).
4. Reasoning: Spawns specialized LLM subagents (e.g., vet-emergency for medical triage, supply-inventory for inventory management) utilizing a self-discovering Model Context Protocol (MCP) tool registry.
5. Action: Recommends clinics, orders supplies, triggers human-in-the-loop approvals, sends WhatsApp alerts to veterinarians, and plays voice-synthesized advice back to the owner.

Modern pet care suffers from three core bottlenecks:

1. Subtle Health Distress Signals: Pets (especially cats) naturally mask pain and illness. Early warning signs like changes in vocalization tones or slight facial expressions are difficult for owners to register.
2. Fragmented Care Workflows: Pet health tracking, medical advice lookup, emergency vet bookings, and nutritional supply-chain systems exist in siloed interfaces, leading to high response friction during critical medical windows.
3. Lack of Live, Integrated Multi-Modal Stream Orchestration: Most pet apps lack real-time pipelines connecting sensory data (audio/video) to immediate, agentic actions (alerting a clinic or looking up local inventories).

• 2.5D Isometric Pet Room (Home Page): An interactive, "Focus Friend"-style digital environment serving as the app's main dashboard. Features time-of-day dynamic lighting, parallax pinch-to-zoom, and anthropomorphic greetings (e.g., "Meow~ Welcome back!").
• Immersive Capture Screen: Seamless video (10s recording cap) and image upload with real-time pet health overlays, launched directly from the interactive pet stat card.
• AG-UI Streaming Event Client: Smooth chat experience that displays agent thoughts, active tools, real-time token execution, and current state snapshots.
• Modality Fusion: Combines visual emotion metrics (angry, sad, normal, surprised) with audio meow categorization for more accurate agent diagnostics.
• Self-Discovering MCP Config: Dynamically discover, load, and map tool specifications to individual subagents (e.g., Wikipedia lookup, Google Maps, Petstore).
• Human-in-the-Loop Triage: Interactive cards for care approval workflows, clinic booking schedules, and todo lists.
• Local Text-to-Speech (TTS): Lightning-fast voice synthesis powered by Kokoro-ONNX, running entirely offline on the CPU.
• WhatsApp Notification Integration: Automatic notification formatting and dispatch to local clinics via Green API.

• Production Release: Download the compiled Android APK on the Official Release page.
• Quick Preview: Test the mobile app directly in your web browser using Appetize.io Preview.

  ⚠️ Since this runs on the Appetize.io free tier, a queue may occur and you might need to wait briefly for an emulator instance to start up.

The following diagram illustrates how the frontend mobile app, machine learning inference microservices, LangChain agentic backend, and external tool integrations interact:

• Mobile Client: Flutter SDK. Integrates custom streaming HTTP clients, native microphone speech recognition, and AV audio/video players.
• Agent Orchestrator: Python 3.11 with FastAPI. Employs LangChain DeepAgents, Server-Sent Events (SSE), and a custom JSON stream parser compatible with AG-UI layout schemas.
• Cat Emotion Inference: FastAPI service in Python that loads a Hugging Face CNN model, preprocesses uploaded images and videos, and returns a unified emotion prediction payload with confidence breakdown.
• Model Context Protocol (MCP): Implements dynamic Node/Python-based MCP servers to resolve real-world state APIs (Google Maps, SQLite profiles, inventory stores).
• Deep Learning Inference: PyTorch-based networks. Mel-spectrogram processing via Librosa/FFmpeg. Model assets are run using ONNX Runtime for optimal CPU performance.
• Notification Services: Green API SDK integrations mapping database profiles to instant WhatsApp messages.

To run the full Meowster workspace locally, follow these steps to spin up the service stack in order:

cd amd-pet-agentic
cp .env.example .env  # Configure your LLM providers & MCP keys
uv sync
npm install           # Installs tool dependencies
uv run uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000

cd amd-pet-visual-cnn
uv sync
uv run uvicorn app.main:app --host 0.0.0.0 --port 8001 --reload

Ensure espeak-ng is installed on your OS:

• macOS: brew install espeak-ng
• Linux: sudo apt-get install -y espeak-ng

cd amd-pet-tts
uv sync
# Download model weights (ONNX v1.0)
curl -L -O https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/kokoro-v1.0.onnx
curl -L -O https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/voices-v1.0.bin
# Start service
uv run uvicorn main:app --reload --host 0.0.0.0 --port 8002

Set up .env files inside communication-service and search-service using their respective .env.example templates, then:

cd amd-pet-backend-services/communication-service
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload --port 8003

# On a separate terminal:
cd ../search-service
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload --port 8004

cd amd-pet-llm-audio
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -r requirements.txt
# Train the model first to generate the models/best_model.pt checkpoint
python training/train.py
# Start the API service
uvicorn app.main:app --host 0.0.0.0 --port 8005 --reload

Connect a physical device or simulator:

cd amd-pet-frontend
flutter pub get

# Start app linking all respective backend instances
flutter run \
  --dart-define=AGENT_BASE_URL=http://localhost:8000 \
  --dart-define=VISUAL_MODEL_BASE_URL=http://localhost:8001 \
  --dart-define=TTS_BASE_URL=http://localhost:8002

(Note: Replace localhost with your computer's LAN IP if running on a physical mobile device over Wi-Fi.)

Meowster is the result of a dedicated, multidisciplinary team, each member bringing unique expertise to deliver a seamless and innovative platform:

This project is for the (AMD Developer Hackathon: ACT II 2026) Hackathon Competition usage only.