API Threat Protection Developer Guide

API threat protection — Developer Guide

5.1.0 · #287 · User guide: API threat protection · Design: Project documents in the API

🔑 The one-line version

Everything lives in includes/api_security.php: apiAuthenticate() calls its guards on every request, the project documents content route calls the download guard, and apiError() reports every 403/404 to the refusal watcher. One table, api_key_events, is both the security log and how alerts are de-duplicated.

📁 Files

File	What
includes/api_security.php	Settings, address and hours checks, the watchers, the download guard, the log and the alerts
api/v1/lib/auth.php	apiAuthenticate() - calls the guards in order; stores the key in $GLOBALS['FREEITSM_API_KEY'] for apiError()
api/v1/lib/response.php	apiError() - a 403/404 with a key set calls apiRefusalRecord()
api/v1/lib/permissions.php	documents.download
api/system/api_keys/threat_settings.php, threat_log.php	The Threat protection tab (administrators)
api/system/api_keys/create_key.php, update_key.php, list_keys.php	Per-key restrictions through apiKeyRestrictionValues()
system/api/index.php	The Keys / Threat protection tabs, the key window's Where and when
includes/services/notifications.php	The api.security bell type (entity api_key, linked to system/api/?tab=threats by entityLink())

🗄️ Data

-- api_keys: optional restrictions (NULL = any address / any time / the install's limits)
allowed_ips TEXT NULL,             -- one address or CIDR per line, normalised on save
allowed_hours TEXT NULL,           -- JSON {"days":[1..7],"from":"07:00","to":"19:00","tz":"Europe/London","mode":"refuse|alert"}
download_alert_files INT NULL, download_limit_files INT NULL, download_limit_mb INT NULL,
suspended_reason VARCHAR(255) NULL -- why the guard switched it off; cleared when switched back on

-- document_access_log: api_key_id INT NULL (NULL = the app), bytes BIGINT NULL; KEY (api_key_id, created_datetime)

CREATE TABLE api_key_events (      -- the security log
    id INT AUTO_INCREMENT PRIMARY KEY, api_key_id INT NULL,   -- NULL = an address with no valid key
    event VARCHAR(30) NOT NULL, ip_address VARCHAR(45) NULL, detail VARCHAR(500) NULL,
    created_datetime DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    KEY (api_key_id, event, created_datetime),
    FOREIGN KEY (api_key_id) REFERENCES api_keys (id) ON DELETE SET NULL   -- the history outlives the key
);

Install settings (API_SECURITY_DEFAULTS): api_download_alert_files 50, api_download_alert_mb 500, api_download_limit_files 200, api_download_limit_mb 2048, api_download_limit_action refuse|suspend, api_new_ip_alert bell_email|bell|off, api_failed_auth_limit 50, api_failed_auth_minutes 15, api_refusal_alert 20, api_refusal_limit 100.

Events: download_alert, download_limit, key_suspended, refused, ip_refused, outside_hours, new_ip, seen_ip (a key's first address - its baseline), auth_failed, address_blocked, alert_not_emailed, and <event>_alerted (the de-duplication marker). The log endpoint hides seen_ip and *_alerted unless asked by name.

🔁 The order on every request

function apiAuthenticate(PDO $conn): array {
    apiAddressBlockGuard($conn);          // 1. an address sending bad keys - 429 address_blocked
    $rawKey = apiExtractKey();
    if (!$rawKey) { apiAuthFailed($conn); apiError(401, ...); }   // counted against the address
    // ... look the key up; invalid -> apiAuthFailed() + 401
    // ... disabled (with suspended_reason), expired, inactive analyst -> 403
    apiKeyRequestGuard($conn, $key);      // 2. allowed addresses, allowed hours, new-address alert
    apiRateLimit(...);                    // 3. requests per minute (unchanged)
    // ...
    $GLOBALS['FREEITSM_API_KEY'] = $key;  // 4. from here a 403/404 counts against this key
}

🔑 Address and hours come before anything else runs, so a refused request learns nothing - not even whether the route exists.

🔴 The address is REMOTE_ADDR, the connection's own. Never X-Forwarded-For: the caller writes that header, so an allow-list read from it is one header away from useless. Behind a reverse proxy, the list must name the proxy (the help says so).

⬇️ The download guard

// api/v1/resources/project_documents.php - apiProjectDocumentContent()
apiRequirePermission($apiKey, 'documents', 'download');   // its own permission
$size = (int)filesize($path);
apiDownloadGuard($conn, $apiKey, $size, $did);            // BEFORE a byte is read
apiDownloadRecord($conn, $apiKey, $did, 'download', $size);
readfile($path);

apiDownloadGuard() counts the key's last 60 minutes in document_access_log (action = 'download'), adds this file, and:

• over the limit → a refused row in the access log, then either api_keys.active = 0 + suspended_reason + 403 key_disabled (suspend), or 429 download_limit (refuse); both logged and alerted;
• at or over the alert level → apiSecurityAlertOnce().

The limits are the key's own columns where set, otherwise the install's (apiDownloadLimits()).

🕵️ The watchers

Refusals - apiError() calls apiRefusalRecord() for every 403/404 once a key is authenticated. It logs a refused row (status, code, method, path), counts the key's last 10 minutes, alerts at api_refusal_alert and switches the key off at api_refusal_limit. A static $busy flag stops recursion, and the refusals a key's own restrictions raise (ip_not_allowed, outside_allowed_hours, key_disabled, key_expired) are not double-counted.

💡 Because "forbidden" and "does not exist" are answered alike (404) on purpose, this one counter also catches enumeration of ids.

Bad keys - apiAuthFailed() logs auth_failed against the address; apiAddressBlockGuard() refuses the address while it has api_failed_auth_limit of them in the window. The alert fires once, at exactly the limit (address_blocked).

New address - in apiKeyRequestGuard(): when the request's address differs from last_used_ip and has no seen_ip/new_ip row for that key in 30 days, a new_ip row and an alert. The first address a key is ever used from is recorded as seen_ip - a baseline, not an alert.

📣 Alerts

apiSecurityAlertOnce($conn, $keyId, 'download_alert', $ip, $message);
// once per key per event per hour: looks for '<event>_alerted' in the last 60 minutes, then
apiSecurityNotifyAdmins($conn, $keyId, $title, $message, $email = true);
// a bell (NotificationsService, type api.security) for every active administrator, and an email
// from the first mailbox that mailboxCanSend(); no mailbox -> an 'alert_not_emailed' row.

🧱 Saving a key's restrictions

apiKeyRestrictionValues($conn, $input) validates and returns column => value for the fields present: addresses through apiAllowedIpsParse() (bad entries are refused by name), hours through apiAllowedHoursClean() (at least one day, valid HH:MM, from ≠ to; the time zone defaults to the server's), overrides as positive integers or NULL. Switching a key back on clears suspended_reason.

🛂 Keys that act as an administrator

api_admin_keys = allow | require_ips | forbid, checked in apiKeyRequestGuard() after the hours check, with apiKeyIsAdmin() reading analysts.is_admin now (an analyst promoted after the key was made is caught). Refusals: 403 admin_key_unrestricted / admin_key_not_allowed, event admin_key_refused, alerted once an hour. No hard ban: api/v1/resources/analysts.php (provisioning) requires an administrator's key, and jobs.php accepts one.

📜 The activity log

CREATE TABLE api_request_log (
    id BIGINT AUTO_INCREMENT PRIMARY KEY, api_key_id INT NULL,   -- NULL = no valid key
    channel VARCHAR(8) NOT NULL DEFAULT 'rest',                  -- rest | mcp
    method VARCHAR(8), path VARCHAR(255), status SMALLINT, error_code VARCHAR(50) NULL,
    duration_ms INT NULL, bytes_out BIGINT NULL, ip_address VARCHAR(45) NULL,
    created_datetime DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    KEY (created_datetime), KEY (api_key_id, created_datetime), KEY (status, created_datetime)
);

// api/v1/index.php and api/mcp/index.php, first thing:
apiActivityStart('rest');   // registers a shutdown function: ONE row however the call ends

The shutdown function reads http_response_code(), $GLOBALS['FREEITSM_API_KEY'] (set by apiAuthenticate()), $GLOBALS['FREEITSM_API_ERROR_CODE'] and FREEITSM_API_BYTES (set by apiError() / apiRespond()), or a Content-Length header (a download). A success is skipped when api_log_successes is 0. It never throws.

The tidy-up - cron/api_log_tidy.php, job api_log_tidy in includes/job_registry.php (daily; needed once a row exists) → apiLogTidy(): every day older than api_log_keep_days is deleted, or first passed to apiLogArchiveDay(), which adds api-log-YYYY-MM-DD.csv to its period's zip (api_log_archive_period day / week / month → apiLogArchivePeriodKey()). The zip is found by the period of the first datestamp in its name (API_LOG_ARCHIVE_NAME = api-log-YYYYMMDD-YYYYMMDD.zip) and renamed when a later day joins. Built on a .part copy, opened and checked, then renamed - the day's rows are deleted only after that. uploads/api-log-archive/ is made by uploadPrepareDir() (deny-all guards), shipped in git (.gitignore un-ignores the guards), denied by nginx's /uploads/ and checked by D009.

Endpoints: api/system/api_keys/activity_log.php (list, filters, today's totals) and activity_archive.php (list, ?download=, POST {delete} - only names matching API_LOG_ARCHIVE_NAME). Settings go through threat_settings.php.

⚠️ Traps

• TRAP: never trust a header for the address - REMOTE_ADDR only.
• TRAP: documents.download must never be in a preset or ticked by Select all - the key form skips [data-sensitive].
• TRAP: a new route that serves file bytes must call apiDownloadGuard() before reading the file and apiDownloadRecord() after - or it bypasses the limits.
• TRAP: don't call apiError() from inside apiRefusalRecord() - it is called by apiError().
• TRAP: dates in apiLogTidy() are UTC. gmdate('Y-m-d', strtotime($day . ' +1 day')) - a LOCAL +1 day printed in UTC - is the same day during summer time, so every day's window was empty and nothing was ever tidied; the test caught it. Use strtotime($day . ' 00:00:00 UTC +1 day').

🧪 Test

tests/project-documents-api.php (45) - the guard's alert, limit and suspend; allowed addresses and ranges; hours (refuse, alert-only, overnight); new-address; probing switch-off; address block; administrator keys (all three settings); the activity log (rows, error codes, no-key calls, failures only); the tidy-up (archive by day and by month with the first-last name and the rename, delete, recent rows kept, the job registered). See Developer tests: System.

---

See also: Project documents in the API — Developer Guide · REST API — how it works