Skip to content

Reproducible updaters for bundled lists + Disconnect category-drift flag - #635

Merged
kasnder merged 1 commit into
masterfrom
claude/remote-asset-updates-06c17e
Jul 11, 2026
Merged

Reproducible updaters for bundled lists + Disconnect category-drift flag#635
kasnder merged 1 commit into
masterfrom
claude/remote-asset-updates-06c17e

Conversation

@kasnder

@kasnder kasnder commented Jul 11, 2026

Copy link
Copy Markdown
Member

What & why

Several remote datasets are bundled in app/src/main/assets/ and read straight from the read-only APK, so they can only be refreshed by shipping a new build. Only the on-device hosts blocklist had an update path; the identification lists (Disconnect/shavar, DuckDuckGo TDS, GeoLite2) were refreshed by hand — easy to forget, and their provenance was undocumented.

Rather than add on-device fetches (a privacy cost for data that changes slowly, and GeoLite2 is license-gated anyway), this refreshes them at build time and captures the source + transform for each.

Updaters (stdlib-only, --check/--input, validate-before-write, write-only-on-change)

Script Source Transform
scripts/update_disconnect_list.py Disconnect services.json (the list Mozilla repackages as Firefox "shavar") byte-reverse (AV workaround, #30)
scripts/update_ddg_tds.py DuckDuckGo Android TDS field-strip to {owner:{name,displayName}, default} — the only fields the loader reads (~1.6 MB → ~100 KB)
scripts/update_geolite2.py MaxMind GeoLite2 (needs MAXMIND_LICENSE_KEY) download tar.gz → extract .mmdb

These follow the same convention as scripts/update_exodus_trackers.py from #622, which stays the single Exodus updater — the workflow calls it if present and no-ops otherwise, so there's no duplication.

.github/workflows/update-lists.yml runs the set monthly (and on demand) and opens a PR. The PR's existing Test CI is the gate that vets a bad upstream.

Disconnect category-drift flag

Disconnect occasionally renames/adds categories. The app already handles unknown categories safely — they fold to Uncategorised via canonicalise (#571) — but silently, so a rename would quietly drop trackers out of their bucket unnoticed.

  • The Disconnect→bucket mapping is refactored out of the inline switch in loadDisconnectTrackers into TrackerCategory (DISCONNECT_ALIASES + mapDisconnectCategory / isRecognisedDisconnectCategory) — one testable source of truth. Behavior-preserving.
  • DisconnectCategoryCoverageTest reads the bundled list and fails the build if any category can no longer be mapped to a UI bucket — turning silent drift into a loud CI failure, regardless of how the asset was updated.

Notes

Verification

  • All three scripts: live --check, real write + idempotency, clean revert; GeoLite2 no-key path errors cleanly.
  • testFdroidDebugUnitTest green: DisconnectCategoryCoverageTest + TrackerListModeTest + BlockingBaselineSubsetTest pass; the refactor is behavior-preserving.

Related: #622

🤖 Generated with Claude Code

@kasnder
kasnder force-pushed the claude/remote-asset-updates-06c17e branch from 7a855a2 to a126f51 Compare July 11, 2026 12:52
…ft flag

The app bundles several remote datasets in app/src/main/assets/ that are read
straight from the read-only APK, so they can only be refreshed by shipping a new
build. Only the on-device hosts blocklist had an update path; the identification
lists (Disconnect/shavar, DuckDuckGo TDS, GeoLite2) were refreshed by hand, which
is easy to forget and left their provenance undocumented.

Rather than add on-device fetches (a privacy cost for data that changes slowly,
and GeoLite2 is license-gated anyway), refresh them at build time:

- scripts/update_disconnect_list.py — Disconnect services.json -> byte-reverse
- scripts/update_ddg_tds.py — DuckDuckGo Android TDS -> field-strip to the
  fields the loader reads ({owner:{name,displayName}, default})
- scripts/update_geolite2.py — MaxMind GeoLite2 (needs MAXMIND_LICENSE_KEY)

Each is stdlib-only, validates before writing, writes only on change, and has
--check/--input, matching the convention of scripts/update_exodus_trackers.py
(#622), which stays the single Exodus updater.

.github/workflows/update-lists.yml runs these monthly and opens a PR; the PR's
existing Test CI is the gate that vets a bad upstream.

Disconnect category drift:
- TrackerControl already handles unknown Disconnect categories safely (they fold
  to Uncategorised via canonicalise, #571) — but silently. The Disconnect->bucket
  mapping is refactored out of an inline switch in loadDisconnectTrackers into
  TrackerCategory (DISCONNECT_ALIASES + mapDisconnectCategory /
  isRecognisedDisconnectCategory) so it is one testable source of truth.
- DisconnectCategoryCoverageTest reads the bundled list and fails the build if a
  category can no longer be mapped to a UI bucket, turning silent drift into a
  loud CI failure regardless of how the asset was updated.

No asset data is changed in this PR; the updaters were run and reverted so this
ships mechanism only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kasnder
kasnder force-pushed the claude/remote-asset-updates-06c17e branch from a126f51 to 8c5c647 Compare July 11, 2026 12:58
@kasnder
kasnder merged commit 6c9aab4 into master Jul 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant