scraper

Part of the floodmask stack.

Phishing is hard to catch because it moves fast — a malicious site can appear, steal credentials, and disappear within hours. This service tries to catch those sites automatically by visiting them, looking at what they look like, and asking: is this pretending to be something it's not?

How it works

You give it a URL. It loads the page in a real browser, takes a screenshot, and looks for two things:

1. Brand logos — does this page display a logo belonging to a known, protected brand?
2. Credential forms — is this page asking for a password?

If both are true, and the domain doesn't belong to that brand, it's likely phishing. If only one is true, it's flagged at a lower confidence. The result comes back as a verdict — threat level, target brand, confidence score.

Brands are "protected" by scraping their real homepage first and saving what they look like. That becomes the reference.

Stack

• Express — REST API
• Puppeteer (with stealth plugins) — headless browser for scraping
• PostgreSQL — stores analyses, users, domains, logos
• Redis + bee-queue — async job queue so scraping doesn't block the API
• External ML APIs for logo detection and credential-page classification

API

All routes require an x-api-key header.

Method	Route	Description
POST	/api/internal/analyze	Analyze a URL for phishing
POST	/api/internal/protect	Add a domain to the protected list
GET	/api/internal/protected	List all protected domains
GET	/api/user/:id	Get user info
POST	/api/user	Create a user (admin only)
DELETE	/api/user/:id	Delete a user (admin only)

API docs are available at /docs (requires basic auth).

Environment variables

Variable	Description
APP_PORT	Port to run the server on
NODE_ENV	development enables test routes
SUPER_USER_KEY	Master API key with admin access
DB_NAME, DB_HOST, DB_USER, DB_PASS, DB_MAX_CONN	PostgreSQL connection
REDIS_HOST, REDIS_PORT, REDIS_SECRET	Redis connection
BROWSER_MIN, BROWSER_MAX, BROWSER_PARAMS	Browser pool config
ROOT_DIR	Directory where scraped files are saved
LOGO_MATCH_API	URL of the logo similarity API
CRP_API	URL of the credential-page classifier API
MAX_CONCURRENCY	Max parallel jobs processed at once
REQ_TIMEOUT	Request timeout in milliseconds
LOG_LEVEL	Winston log level (default: info)

Running with Docker

docker compose up

The app expects a .env file in the project root.