SafeGuard Live is a real-time, low-latency AI "Co-Pilot" designed to protect users—especially the elderly—from human-led social engineering scams like Digital Arrests, Bank Impersonation, and OTP Theft.
Unlike passive blockers, SafeGuard Live acts as an active tactical assistant that listens, identifies scam patterns, and empowers the user to fight back in real-time.
When a scammer calls, they use "Amygdala Hijacking"—triggering fear and urgency so the victim can't think clearly. SafeGuard Live acts as the user's external prefrontal cortex. It monitors the call, stays calm, and provides the user with the logic and scripts needed to expose the fraudster.
- Pattern Matching: Identifies specific scam archetypes (e.g., "Illegal Parcel," "CBI Skype Interrogation") within the first 10 seconds.
- Acoustic Fingerprinting: Uses Gemini’s native audio intelligence to detect "Boiler Room" signatures (background chatter, VoIP jitter) common in scam centers.
- Safe-Mute: Upon high-confidence fraud detection, the agent can mute the user's microphone so the scammer cannot hear their panic.
- The Whisper: The AI "whispers" instructions to the user (via text or audio) that the scammer cannot hear.
- Reverse Interrogation: Proactively suggests "Counter-Questions" that scammers hate (e.g., "Ask for their Badge ID and official station landline").
- Visual Overlay: If the agent hears the scammer ask for a "code" or "OTP," it triggers a system-level red alert and blurs the user's notification tray to prevent them from reading the code aloud.
- On-the-Fly Queries: The user can ask the AI, "Is this a scam?" or "What should I do?" mid-call. The AI responds instantly with tactical advice.
- Silent Monitor (Caregiver Sync): Real-time transcript and "Risk Level" are mirrored to a secondary device for a family member to monitor and intervene if necessary.
| Layer | Technology |
|---|---|
| The "Brain" | Gemini 2.5 Flash Live (Native Multimodal Audio-to-Audio) |
| The "Body" | LiveKit Agents Framework (Python/Node.js) |
| Transmission | WebRTC / SIP (via LiveKit SIP Interconnect) |
| Real-time UI | React.js (HUD) + LiveKit Components |
| State/Alerts | Firebase Realtime Database (for Caregiver Sync) |
| Local Shield | MediaPipe (On-device Voice Activity & Pattern Detection) |
We use LiveKit as the orchestration layer. When a call starts, the user’s audio and the scammer’s audio are streamed as separate tracks into a LiveKit Room.
A Gemini Live Agent joins the room. Using the livekit-plugins-google, the agent consumes the raw audio stream. It is configured with a System Instruction to remain silent and monitor for specific "Deception Indicators."
When fraud is detected, Gemini triggers Tool Calls:
mute_user_mic(): Calls LiveKit’s Server API to silence the user’s uplink.send_hud_prompt(): Sends a Data Track message to the frontend with the "Reverse Interrogation" script.notify_caregiver(): Pushes an emergency alert to the Firebase backend.
- The Scammer: A teammate in another room calling the "Victim" via a LiveKit SIP number.
- The Victim: A teammate on stage with the SafeGuard HUD (Laptop/Tablet) open.
- The Visuals: A large screen showing the Live Waveform, the MO Classification, and the AI Whisper Prompts.
- The Hook: Scammer begins a "Digital Arrest" script: "This is the Police! Stay on the line or face arrest!"
- The Detection: SafeGuard HUD turns Yellow then Red as Gemini identifies the MO.
- The Mute: The AI Agent whispers: "I am muting you now. Don't worry. Ask him for his Case ID." (Judges see the user's mic icon turn red).
- The OTP Defense: Scammer asks for a verification code. The HUD flashes a massive "OTP THREAT DETECTED" warning.
- The Success: The user asks a counter-question, the scammer hangs up, and the AI provides a "Post-Call Summary" for reporting to the authorities.
Would you like me to generate the agent.py file that handles the LiveKit Muting logic and the Gemini system prompt?