Reproducible updaters for bundled lists + Disconnect category-drift flag - #635
Merged
Conversation
kasnder
force-pushed
the
claude/remote-asset-updates-06c17e
branch
from
July 11, 2026 12:52
7a855a2 to
a126f51
Compare
…ft flag
The app bundles several remote datasets in app/src/main/assets/ that are read
straight from the read-only APK, so they can only be refreshed by shipping a new
build. Only the on-device hosts blocklist had an update path; the identification
lists (Disconnect/shavar, DuckDuckGo TDS, GeoLite2) were refreshed by hand, which
is easy to forget and left their provenance undocumented.
Rather than add on-device fetches (a privacy cost for data that changes slowly,
and GeoLite2 is license-gated anyway), refresh them at build time:
- scripts/update_disconnect_list.py — Disconnect services.json -> byte-reverse
- scripts/update_ddg_tds.py — DuckDuckGo Android TDS -> field-strip to the
fields the loader reads ({owner:{name,displayName}, default})
- scripts/update_geolite2.py — MaxMind GeoLite2 (needs MAXMIND_LICENSE_KEY)
Each is stdlib-only, validates before writing, writes only on change, and has
--check/--input, matching the convention of scripts/update_exodus_trackers.py
(#622), which stays the single Exodus updater.
.github/workflows/update-lists.yml runs these monthly and opens a PR; the PR's
existing Test CI is the gate that vets a bad upstream.
Disconnect category drift:
- TrackerControl already handles unknown Disconnect categories safely (they fold
to Uncategorised via canonicalise, #571) — but silently. The Disconnect->bucket
mapping is refactored out of an inline switch in loadDisconnectTrackers into
TrackerCategory (DISCONNECT_ALIASES + mapDisconnectCategory /
isRecognisedDisconnectCategory) so it is one testable source of truth.
- DisconnectCategoryCoverageTest reads the bundled list and fails the build if a
category can no longer be mapped to a UI bucket, turning silent drift into a
loud CI failure regardless of how the asset was updated.
No asset data is changed in this PR; the updaters were run and reverted so this
ships mechanism only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kasnder
force-pushed
the
claude/remote-asset-updates-06c17e
branch
from
July 11, 2026 12:58
a126f51 to
8c5c647
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Several remote datasets are bundled in
app/src/main/assets/and read straight from the read-only APK, so they can only be refreshed by shipping a new build. Only the on-device hosts blocklist had an update path; the identification lists (Disconnect/shavar, DuckDuckGo TDS, GeoLite2) were refreshed by hand — easy to forget, and their provenance was undocumented.Rather than add on-device fetches (a privacy cost for data that changes slowly, and GeoLite2 is license-gated anyway), this refreshes them at build time and captures the source + transform for each.
Updaters (stdlib-only,
--check/--input, validate-before-write, write-only-on-change)scripts/update_disconnect_list.pyservices.json(the list Mozilla repackages as Firefox "shavar")scripts/update_ddg_tds.py{owner:{name,displayName}, default}— the only fields the loader reads (~1.6 MB → ~100 KB)scripts/update_geolite2.pyMAXMIND_LICENSE_KEY).mmdbThese follow the same convention as
scripts/update_exodus_trackers.pyfrom #622, which stays the single Exodus updater — the workflow calls it if present and no-ops otherwise, so there's no duplication..github/workflows/update-lists.ymlruns the set monthly (and on demand) and opens a PR. The PR's existing Test CI is the gate that vets a bad upstream.Disconnect category-drift flag
Disconnect occasionally renames/adds categories. The app already handles unknown categories safely — they fold to Uncategorised via
canonicalise(#571) — but silently, so a rename would quietly drop trackers out of their bucket unnoticed.switchinloadDisconnectTrackersintoTrackerCategory(DISCONNECT_ALIASES+mapDisconnectCategory/isRecognisedDisconnectCategory) — one testable source of truth. Behavior-preserving.DisconnectCategoryCoverageTestreads the bundled list and fails the build if any category can no longer be mapped to a UI bucket — turning silent drift into a loud CI failure, regardless of how the asset was updated.Notes
MAXMIND_LICENSE_KEYrepo secret to be included; otherwise it's skipped.Verification
--check, real write + idempotency, clean revert; GeoLite2 no-key path errors cleanly.testFdroidDebugUnitTestgreen:DisconnectCategoryCoverageTest+TrackerListModeTest+BlockingBaselineSubsetTestpass; the refactor is behavior-preserving.Related: #622
🤖 Generated with Claude Code