Skip to content

2.13.0

Choose a tag to compare

@tomchop tomchop released this 19 Nov 20:54
2e3cbbb

Pairs with yeti-feeds-frontend 2.13.0. prod/docker-compose.yaml drives both from one YETI_IMAGE_TAG, so deploy them together.

Upgrade notes

abuse.ch feeds need an Auth-Key (#1455). abuse.ch retired the keyless exports for MalwareBazaar, ThreatFox and URLhaus, which now answer 410 Gone. URLhaus had been failing silently. Get a free key from https://auth.abuse.ch/ and set YETI_ABUSECH_AUTH_KEY (or [abusech] in yeti.conf) for the task worker. Without a key these three feeds fail with a message saying so. Feodo Tracker, SSL Blacklist and YARAify still need no key.

API keys: revocation and expiry now hold (#1456, #1457).

  • A deleted or disabled API key sent directly as a bearer token used to keep working forever. It is now refused.
  • Expiry was inverted: a key with an expiry date was refused until that date, then accepted forever after it. After upgrading, keys whose expiry date is still ahead start working, and expired keys stop.

Observables are validated against the type they are given (#1468). POST /observables/ and POST /observables/bulk used to store any value under an explicit type, e.g. bogus as an ipv4. Such values are now refused: per item in a bulk add, and with a 400 for a single add. Feeds and scripts that sent mistyped values will log them as skipped instead of storing them.

/graph/match lists an entity under every looked-up value it is linked to (#1464). It used to list each entity once per request, under the first value processed. entities now has one [relationship, entity] pair per link. Clients that de-duplicate by entity id should group by the relationship's source/target instead. yeti-feeds-frontend 2.13.0 does.

Event task patterns match on Python 3.11+ (#1399). Event strings rendered as EventType.new:observable:url, so acts_on patterns starting with the event type never matched. That included all three shipped event tasks. They now fire. As noted in 2.12.0, DatadogMetrics will start reporting a yeti.object series with type:acl.

Tag names keep letters in every script (#1453). Normalizing used to strip everything outside [a-zA-Z0-9_.:-]. A name written in another script became empty, and the whole tag call failed: malpedia's Gelsemium had no tags at all. Names mixing scripts were silently truncated. Such names are now kept, so the affected objects gain their tags on the next feed run.

webserver-prod runs YETI_WEB_WORKERS workers, default 4 (#1440). It used to start one worker per host core, whatever the container's CPU quota, at about 340 MiB each. Busy deployments can raise the variable.

URL imports time out (#1459, #1461). /observables/import/url now times out: 5 s to connect, 30 s between bytes. A failed fetch is a 400 instead of a 500. Set allow_internal_url_imports = false (YETI_SYSTEM_ALLOW_INTERNAL_URL_IMPORTS) to fetch public addresses only. The default still allows internal ones.

New setting: audit_log_bodies (#1476). Set it to false (YETI_SYSTEM_AUDIT_LOG_BODIES) to leave request bodies out of the console and the audit log file. The default keeps today's behaviour. The auditlog collection still records the bodies.

Security

  • LINUX_PATH_REGEX no longer backtracks catastrophically: GHSA-ggvx-5xq4-mh3h (#1447).
  • Revoked API keys are refused as bearer tokens, and expiry is enforced (#1456, #1457).
  • OIDC:
    • logins whose provider reports email_verified=false are refused (#1460);
    • every login of a disabled user is refused, not only the first (#1483);
    • a missing email claim or an invalid ID token gets a 401, not a 500. The reason is logged, not shown to the browser (#1481).
  • Uploaded files are logged by name and size, no longer in full (#1475).
  • The bloomcheck exception stays out of the API's error detail (#1419).
  • authlib 1.9.1 (#1469).

New feeds and analytics

  • Feeds:
    • CIRCL OSINT, in the MISP feed format: no MISP server or key needed (#1472);
    • OffSeq Mirage attacker IPs and payload URLs (#1470);
    • NetAPI Top 1M domains (#1451);
    • phishunt.io phishing (#1401).
  • On-demand analytics:

Performance

  • ChromaDB re-embeds only the documents whose content changed. On a 73k-document instance, a pass took longer than its 10-minute schedule (#1390).
  • The ChromaDB prune read is split to fit the backend's batch limit (#1387).
  • List responses serialize faster thanks to type discriminators on their item unions (#1443).
  • The total of an unfiltered list comes from the collection count (#1446).
  • Audit timeline lookups are indexed (#1444).

Fixes

  • API:
    • Single-object responses validate against their own type only. A regex indicator with an all-digits pattern made every read of it a 500 (#1467).
    • Each distinct value is matched once in /graph/match (#1420).
    • Graph descriptors without an object id are rejected (#1425).
    • The API answers 503 when the agent service is unreachable (#1415).
    • The worker status is empty when no Celery worker answers (#1416).
    • Async job results that were only polled are now fetched (#1441).
  • Users:
    • Users created by a first OIDC login get the default groups (#1482), as CLI-created users now do (#1426).
    • A trailing slash in system.webroot no longer breaks the OIDC callback URL (#1484).
  • Graph and tags:
    • Concurrent link_to() and link_to_acl() calls no longer duplicate edges (#1411).
    • A replacing tag is applied for every name it replaces (#1442).
    • Incoming context is compared in the form it is stored (#1391), and per-run timestamps no longer count as a change to a feed's context (#1392).
  • Feeds and tasks:
    • Task registration is atomic across processes (#1393).
    • IOCExtractor analyzes only URLs whose extract_iocs tag is still fresh (#1394).
    • The agent's response is validated before it is written to the graph (#1351).
    • YARAify's multi-rule files are imported one rule at a time (#1452).
    • Upstream data quirks in two feeds are no longer logged as errors (#1454).

Development

  • :dev images build only the newest commit (#1406).
  • Feed tests can run on demand and from a label (#1410), with the plugins dependency group installed (#1408).
  • Test fixes: #1261, #1450.