Repository navigation
2.13.0
Pairs with yeti-feeds-frontend 2.13.0. prod/docker-compose.yaml drives both from one YETI_IMAGE_TAG, so deploy them together.
Upgrade notes
abuse.ch feeds need an Auth-Key (#1455). abuse.ch retired the keyless exports for MalwareBazaar, ThreatFox and URLhaus, which now answer 410 Gone. URLhaus had been failing silently. Get a free key from https://auth.abuse.ch/ and set YETI_ABUSECH_AUTH_KEY (or [abusech] in yeti.conf) for the task worker. Without a key these three feeds fail with a message saying so. Feodo Tracker, SSL Blacklist and YARAify still need no key.
API keys: revocation and expiry now hold (#1456, #1457).
- A deleted or disabled API key sent directly as a bearer token used to keep working forever. It is now refused.
- Expiry was inverted: a key with an expiry date was refused until that date, then accepted forever after it. After upgrading, keys whose expiry date is still ahead start working, and expired keys stop.
Observables are validated against the type they are given (#1468). POST /observables/ and POST /observables/bulk used to store any value under an explicit type, e.g. bogus as an ipv4. Such values are now refused: per item in a bulk add, and with a 400 for a single add. Feeds and scripts that sent mistyped values will log them as skipped instead of storing them.
/graph/match lists an entity under every looked-up value it is linked to (#1464). It used to list each entity once per request, under the first value processed. entities now has one [relationship, entity] pair per link. Clients that de-duplicate by entity id should group by the relationship's source/target instead. yeti-feeds-frontend 2.13.0 does.
Event task patterns match on Python 3.11+ (#1399). Event strings rendered as EventType.new:observable:url, so acts_on patterns starting with the event type never matched. That included all three shipped event tasks. They now fire. As noted in 2.12.0, DatadogMetrics will start reporting a yeti.object series with type:acl.
Tag names keep letters in every script (#1453). Normalizing used to strip everything outside [a-zA-Z0-9_.:-]. A name written in another script became empty, and the whole tag call failed: malpedia's Gelsemium had no tags at all. Names mixing scripts were silently truncated. Such names are now kept, so the affected objects gain their tags on the next feed run.
webserver-prod runs YETI_WEB_WORKERS workers, default 4 (#1440). It used to start one worker per host core, whatever the container's CPU quota, at about 340 MiB each. Busy deployments can raise the variable.
URL imports time out (#1459, #1461). /observables/import/url now times out: 5 s to connect, 30 s between bytes. A failed fetch is a 400 instead of a 500. Set allow_internal_url_imports = false (YETI_SYSTEM_ALLOW_INTERNAL_URL_IMPORTS) to fetch public addresses only. The default still allows internal ones.
New setting: audit_log_bodies (#1476). Set it to false (YETI_SYSTEM_AUDIT_LOG_BODIES) to leave request bodies out of the console and the audit log file. The default keeps today's behaviour. The auditlog collection still records the bodies.
Security
LINUX_PATH_REGEXno longer backtracks catastrophically: GHSA-ggvx-5xq4-mh3h (#1447).- Revoked API keys are refused as bearer tokens, and expiry is enforced (#1456, #1457).
- OIDC:
- Uploaded files are logged by name and size, no longer in full (#1475).
- The bloomcheck exception stays out of the API's error detail (#1419).
authlib1.9.1 (#1469).
New feeds and analytics
- Feeds:
- On-demand analytics:
Performance
- ChromaDB re-embeds only the documents whose content changed. On a 73k-document instance, a pass took longer than its 10-minute schedule (#1390).
- The ChromaDB prune read is split to fit the backend's batch limit (#1387).
- List responses serialize faster thanks to type discriminators on their item unions (#1443).
- The total of an unfiltered list comes from the collection count (#1446).
- Audit timeline lookups are indexed (#1444).
Fixes
- API:
- Single-object responses validate against their own type only. A regex indicator with an all-digits pattern made every read of it a 500 (#1467).
- Each distinct value is matched once in
/graph/match(#1420). - Graph descriptors without an object id are rejected (#1425).
- The API answers 503 when the agent service is unreachable (#1415).
- The worker status is empty when no Celery worker answers (#1416).
- Async job results that were only polled are now fetched (#1441).
- Users:
- Graph and tags:
- Feeds and tasks:
- Task registration is atomic across processes (#1393).
- IOCExtractor analyzes only URLs whose
extract_iocstag is still fresh (#1394). - The agent's response is validated before it is written to the graph (#1351).
- YARAify's multi-rule files are imported one rule at a time (#1452).
- Upstream data quirks in two feeds are no longer logged as errors (#1454).