Skip to content

v2.1.0 — Headwater

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 Sep 05:56
· 68 commits to main since this release
v2.1.0
0c2cc15
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

One self-hosted API for Google Maps, News, Trends and Autocomplete, plus YouTube
transcripts. This release renames the project, fixes several bugs that failed
silently, and moves the image to Python 3.14 on Debian 13.

Breaking

The project is now headwater. The old Docker Hub repository
rainmanjam/social-flood has been deleted, and Docker Hub does not redirect
a renamed or deleted repository. Any docker pull rainmanjam/social-flood in a
compose file, script or CI job stops working immediately.

- image: rainmanjam/social-flood:latest
+ image: rainmanjam/headwater:2.1.0

The GitHub repository moved to rainmanjam/headwater; GitHub does redirect, so
existing git remotes keep working. No API path, parameter or response field
changed — only the name.

Maps now rejects impossible requests up front. A max_results/timeout
combination that cannot finish returns 400 with the count you can afford,
instead of timing out minutes later. max_results is capped at 45.

Fixed — bugs that returned success while doing nothing

These are the reason for the release. Each one looked healthy from the outside.

  • Record storage never reached Redis. RecordStore called
    manager.is_available() on what is a @property, so every call raised
    'bool' object is not callable, was swallowed, and silently fell back to
    in-memory storage. Maps jobs, monitors and webhooks were lost on restart and
    invisible to sibling workers. /health/detailed now reports
    record_storage_durable, and startup asserts it, so this cannot recur quietly.
  • The Redis health check tested nothing. It called a _get_redis_client()
    method that does not exist. It now pings the real client.
  • A broken import crashed every News search, hidden behind the cache.
  • Google News ignored the proxy. GNews takes a {"http": ..., "https": ...}
    mapping, not a bare URL string.

Security

  • Proxy credentials are no longer logged. A proxy URL carries user:pass@
    inline; two code paths wrote it to the transcript and application logs.
    Credentials are masked at every call site now.
  • NLTK removed, which removes PYSEC-2026-2026 rather than suppressing it.
    Article extraction still returns title, authors, date and full text; summary
    and keywords are now null, with nlp_available: false to say so plainly.
  • Debian security updates are applied at image build.

Proxying is now per host

ENABLE_PROXY was global, which forced one decision for every upstream. The
upstreams disagree: Reddit needs a proxy, YouTube is refused by some providers at
the tunnel, and Google Maps loads through a plain GET but a full browser
navigation through a datacentre proxy never settles.

ENABLE_PROXY=true
PROXY_URLS=http://user:pass@proxy.example.com:8080
NO_PROXY_HOSTS=youtube.com,youtu.be,ytimg.com,google.com

NO_PROXY_HOSTS matches on a dot boundary, so youtube.com does not also match
notyoutube.com.example.

Performance

  • Google News search: ~80s → ~3s. GNews was launching a whole Chromium
    instance per article to resolve redirect URLs. It now decodes them directly.
  • Maps searches are cached for an hour. The blocking path had no cache at
    all, and Maps costs roughly 12 seconds per result.
  • Trends reference data is cached for a day. /geo (3,681 locations) and
    /categories (1,133) change on the order of months.

API

  • /geo is served from the native Trends endpoint rather than a scrape.
  • POST /batch-get-transcripts accepts a JSON body, up to 50 video ids.
  • Trends endpoints stop passing None into trendspy, so its own defaults apply.
  • Exhausted Google Trends quota returns 502, not an empty 200 — a quiet
    week and a broken scraper should never look the same.

Image and supply chain

  • Python 3.14 on Debian 13 (trixie), pinned by digest.
  • Multi-arch: linux/amd64 and linux/arm64.
  • SBOM and provenance attestations attached, plus a cosign signature.
  • Full OCI metadata — source, revision, version, licence, base image.

Docs

The README is rewritten against what the API actually exposes, with real
responses rather than invented ones, and all 29 markdown files were audited
against the code.

Images

docker pull rainmanjam/headwater:2.1.0
docker pull ghcr.io/rainmanjam/headwater:2.1.0

Digest: sha256:ab46c57ed6ce94a5f3d24c0e2c16f067749c5c743356a07e46c879d85b127c7c

Verify the signature

cosign verify \
  --certificate-identity-regexp 'https://github.com/rainmanjam/headwater/.github/workflows/release.yml@.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/rainmanjam/headwater@sha256:ab46c57ed6ce94a5f3d24c0e2c16f067749c5c743356a07e46c879d85b127c7c

Full commit log: v2.0.0...v2.1.0

Assets 3
Loading