Skip to content

smartonion/ScamProtect

BranchesTags

Repository files navigation

ScamProtect

Real-time financial scam detection for desktop. ScamProtect runs quietly in your system tray and automatically activates when you open a banking site, payment app, or remote-access tool. It monitors your screen and microphone, runs everything through a two-tier AI pipeline, and alerts you the moment it detects a scam in progress — popping up in your face with a full-screen flashing border and scam overlay.

The problem

Financial scams cost Americans over $10 billion annually. Scammers impersonate the IRS, bank fraud departments, and tech support agents to pressure victims into wiring money or sharing credentials — often while on a live phone call. By the time the victim realizes what happened, the money is gone.

ScamProtect acts as a real-time guardian that watches for the patterns susceptible humans miss in the moment.

How it works

ScamProtect is an Electron desktop app backed by a Python pipeline. When you open a sensitive application — like your bank's website, PayPal, or a remote-access tool — it automatically begins monitoring:

You open Chase.com
  |
  v
Watchdog detects sensitive app in focus
  |  Screen border turns BROWN
  |
  +---> Audio Capture (mic --> Google Cloud STT --> transcript)
  +---> Screen Capture (screenshot --> Google Cloud Vision --> OCR text)
  |
  v
Light Analyzer (every 30s)
  Reads last 30s of captures --> Gemini 2.5 Flash
  Extracts key quotes, assigns risk score 0-1
  |
  v
Risk Gate (threshold: 0.8)
  |  Screen border turns RED
  |
  +--- below 0.8 --> logged, no action
  +--- above 0.8 --> Session opens --> Heavy Analyzer
                                          |
                                          v
                                     Gemini 3.1 Pro
                                     Structured verdict:
                                       - refined_risk
                                       - action: none | notify | alert_scam
                                       - summary (user-facing)
                                          |
                                          v
                                     Screen border FLASHES RED
                                     App pops to front
                                     Desktop scam alert overlay
                                     Trusted contacts notified

What it detects

  • Government impersonation — "This is the IRS, you have a warrant..."
  • Tech support scams — fake virus popups, remote-access takeover
  • Bank fraud department impersonation — coached wire transfers
  • Gift card payment pressure — "Go buy $500 in Apple gift cards..."
  • Romance / recovery scams — emotional manipulation + money requests
  • Credential harvesting — reading out passwords, PINs, or OTP codes on a call
  • Remote-access exploitation — TeamViewer or AnyDesk running alongside a banking session

When it activates

ScamProtect only monitors when the foreground window matches sensitive patterns:

  • Banking & finance: Chase, Bank of America, Wells Fargo, PayPal, Venmo, Coinbase, Zelle, Schwab, Fidelity, Robinhood, Cash App, Wise, Western Union, and more
  • Remote-access tools: TeamViewer, AnyDesk, UltraViewer, Chrome Remote Desktop, ScreenConnect
  • Scam keyword triggers: Window titles containing "Tech Support", "Microsoft Support", "Apple Support", "Refund"

When you leave the sensitive app, monitoring continues for a 2-minute cooldown (in case you briefly alt-tab), then stops automatically.

Alert system

Level What happens
Capture active Brown border around your entire screen
Risk gate tripped Border turns solid red
notify verdict Toast notification slides in, app comes to front
alert_scam verdict Border flashes red, full scam overlay takes over, app forced to front, trusted contacts alerted

If the heavy model is unavailable (rate-limited, network error), a fallback verdict is emitted from the light model's risk score so you're never left unprotected.

System tray

Closing the app window minimizes it to the system tray — the backend keeps running. Right-click the tray icon to open the app or quit entirely. When a scam is detected, the app pops back up automatically.

Architecture

Layer Tech
Desktop Shell Electron 32, system tray, full-screen border overlay
Capture (audio + screen + watchdog) sounddevice, mss, Google Cloud STT/Vision, pywin32
Light Analyzer Gemini 2.5 Flash (every 30s)
Pipeline (gate, session, router, watcher) File-system watcher, circuit breaker, session accumulator
Heavy Analyzer Gemini 3.1 Pro (structured output, with fallback)
API Server FastAPI, SSE real-time verdict stream, verdict store
Notifications Toast popups, full-screen scam overlay, screen border overlay

Two-tier AI pipeline

Tier 1 — Light Analyzer (Gemini 2.5 Flash) Runs every 30 seconds. Reads the last 30 seconds of audio transcripts and screen OCR, extracts exact quotes that match scam indicators, and assigns a risk score from 0 to 1. Designed to be fast and cheap — the first line of defense.

Tier 2 — Heavy Analyzer (Gemini 3.1 Pro) Only activated when the light analyzer's risk score crosses 0.8. Receives the full session history (all light analysis records since the trigger), evaluates the trajectory of the conversation, and produces a structured verdict with a recommended action: none, notify (soft caution), or alert_scam (loud alert). If the heavy model fails (429/503/network), a fallback verdict is generated from the light model data.

Risk gate & sessions

The risk gate is a circuit breaker. Once it trips (risk > 0.8), it stays active until the heavy model explicitly resolves the session. This prevents the system from going silent if risk briefly dips below the threshold mid-scam. Gate state is reset on each fresh app launch to avoid stale state from previous sessions.

Quick start (Windows)

Prerequisites

Setup

# 1. Clone and install Python dependencies
git clone <repo-url>
cd ScamProtect
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

# 2. Install Electron dependencies
cd electron
npm install
cd ..

# 3. Create .env and add your Gemini API key
copy .env.example .env
# Edit .env: set GEMINI_API_KEY=AIzaSy...
# Get a key at https://aistudio.google.com/apikey

# 4. Google Cloud auth (one-time, for audio STT + screen OCR)
gcloud auth login
gcloud auth application-default login
gcloud services enable speech.googleapis.com vision.googleapis.com

# 5. Launch the app
cd electron
npm start

Electron spawns the Python backend automatically — you don't need to run python run_all.py separately.

Google Cloud SDK installation

  1. Download the installer from cloud.google.com/sdk/docs/install
  2. Run the installer (accept defaults)
  3. Open a new terminal after installation
  4. Run gcloud init and sign in with your Google account
  5. Run gcloud auth application-default login (opens browser — sign in again)
  6. Enable the required APIs: 
    gcloud services enable speech.googleapis.com vision.googleapis.com

Gemini API key setup

  1. Go to aistudio.google.com/apikey
  2. Click "Create API key"
  3. Copy the key and paste it into your .env file: 
    GEMINI_API_KEY=AIzaSy...your-key-here...
  4. Important: If you want the heavy model (Gemini 3.1 Pro) to work, you need to enable billing on your Google AI project. Without billing, only the free tier is available (20 requests/day for Flash, 0 for Pro). With billing enabled, limits jump to 2,000+ RPM. Enable billing at aistudio.google.com > Settings > Billing.

Usage

Electron app (recommended)

cd electron
npm start

This launches the full system: Electron shell + Python backend (capture + pipeline + API). Close the window to minimize to system tray. Right-click tray icon > Quit to fully exit.

Python backend only (no UI)

.venv\Scripts\activate
python run_all.py                  # Full system: capture + pipeline + API
python run_all.py --no-capture     # Pipeline + API only
python run_all.py --no-pipeline    # Capture + API only
python run_all.py --no-api         # Capture + pipeline only

API

ScamProtect exposes a local API on http://127.0.0.1:8000:

Method Path Description
GET /health Health check
GET /status Gate state + active session info
GET /verdicts/pending Unacknowledged verdicts
GET /verdicts/all All verdicts (acknowledged + unacknowledged)
POST /verdicts/{id}/ack Mark verdict as seen
GET /verdicts/stream SSE real-time verdict stream
GET /capture/status Capture watchdog state
POST /capture/toggle Enable/disable capture at runtime
POST /test/pipeline Inject fake analysis file (triggers full pipeline)
POST /test/notify Inject fake notify verdict
POST /test/alert_scam Inject fake scam alert verdict
POST /shutdown Clean shutdown
GET /docs Swagger UI

Data flow

Folder Written by Read by Contents
core/capture/_dumps/ Audio + screen capture Light analyzer Raw audio_*.json, screen_*.json
data/data_small/ Light analyzer Pipeline watcher Risk-scored summaries with key quotes
data/data_large/ Pipeline router Electron / API Final verdicts with actions

Configuration

File What it controls
.env API keys (GEMINI_API_KEY), server host/port
paths.txt I/O folder routing (editable without touching code)
config/settings.py Capture intervals, silence thresholds, trigger rules, model names, risk threshold

Key settings (config/settings.py)

Setting Default Description
ANALYSIS_INTERVAL 30s How often the light analyzer runs
ANALYSIS_WINDOW 30s How many seconds of captures to analyze
RISK_THRESHOLD 0.8 Risk score that trips the gate
WATCHDOG_COOLDOWN 120s Keep capturing after trigger app loses focus
AUDIO_SILENCE_RMS 0.0003 Below this RMS = silence (tune for your mic)
GEMINI_MODEL gemini-2.5-flash Light analyzer model
HEAVY_MODEL gemini-3.1-pro-preview Heavy analyzer model

Trigger rules are configurable under SENSITIVE_RULES. Add or remove banking sites, remote-access tools, or keyword patterns as needed.

Tech stack

  • Desktop: Electron 32 (system tray, full-screen border overlay, IPC)
  • Capture: sounddevice + Google Cloud Speech-to-Text, mss + Google Cloud Vision
  • LLM: Gemini 2.5 Flash (light), Gemini 3.1 Pro (heavy) via google-genai
  • Backend: FastAPI + Uvicorn, file-system watcher, pydantic
  • Platform: Windows (pywin32 for window detection)

Troubleshooting

Problem Fix
GEMINI_API_KEY is not set Edit .env and add your key
429 RESOURCE_EXHAUSTED (Flash) Free tier: 20 req/day. Enable billing or wait for reset
429 RESOURCE_EXHAUSTED (Pro) Free tier: 0 req. Must enable billing for Pro
google STT failed: 403 Run gcloud services enable speech.googleapis.com
google OCR failed: 403 Run gcloud services enable vision.googleapis.com
could not automatically determine credentials Run gcloud auth application-default login
sounddevice record failed or RMS is 0 Check mic permissions in Windows Settings > Privacy > Microphone
Audio always skipped as silent Lower AUDIO_SILENCE_RMS in config/settings.py
Watchdog never triggers Check window title matches a rule in SENSITIVE_RULES
Border stuck on screen after crash Kill remaining Electron processes in Task Manager
Electron can't find Python Ensure .venv exists at repo root, or python is on PATH

Team

Built at a hackathon by Martin, Owen, Lindsey and Shafay.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages