Add reproducible Exodus tracker-signature updater script (#318) - #622
Merged
Conversation
TrackerControl detects tracker libraries and maps traffic using signatures from Exodus Privacy. The app already fetches the latest signatures from the official Exodus JSON API at runtime (TrackerSignatureManager); the copy at app/src/main/assets/trackers.json is the offline fallback. Issue #318 asked for a way to refresh/expand these signatures and proposed a fragile HTML scraper of reports.exodus-privacy.eu.org. This replaces that with scripts/update_exodus_trackers.py, which pulls from the same documented JSON API the app uses, validates the payload against the schema the app's parser expects (top-level "trackers" object, required id/name fields, sane tracker count, non-empty network signatures, regex compile check), and rewrites the bundled asset only when content actually changed. --check validates without writing; --input reads a local file. No data change is bundled: the live API is currently byte-identical to the committed asset (432 trackers, newest 2023-12-05), so refreshing today is a verified no-op. The script gives maintainers a one-command, idempotent refresh for whenever Exodus updates upstream. Exodus signature data remains under Exodus Privacy's own terms; the script only mirrors their published API output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kasnder
marked this pull request as ready for review
July 11, 2026 12:50
kasnder
added a commit
that referenced
this pull request
Jul 11, 2026
…ft flag
The app bundles several remote datasets in app/src/main/assets/ that are read
straight from the read-only APK, so they can only be refreshed by shipping a new
build. Only the on-device hosts blocklist had an update path; the identification
lists (Disconnect/shavar, DuckDuckGo TDS, GeoLite2) were refreshed by hand, which
is easy to forget and left their provenance undocumented.
Rather than add on-device fetches (a privacy cost for data that changes slowly,
and GeoLite2 is license-gated anyway), refresh them at build time:
- scripts/update_disconnect_list.py — Disconnect services.json -> byte-reverse
- scripts/update_ddg_tds.py — DuckDuckGo Android TDS -> field-strip to the
fields the loader reads ({owner:{name,displayName}, default})
- scripts/update_geolite2.py — MaxMind GeoLite2 (needs MAXMIND_LICENSE_KEY)
Each is stdlib-only, validates before writing, writes only on change, and has
--check/--input, matching the convention of scripts/update_exodus_trackers.py
(#622), which stays the single Exodus updater.
.github/workflows/update-lists.yml runs these monthly and opens a PR; the PR's
existing Test CI is the gate that vets a bad upstream.
Disconnect category drift:
- TrackerControl already handles unknown Disconnect categories safely (they fold
to Uncategorised via canonicalise, #571) — but silently. The Disconnect->bucket
mapping is refactored out of an inline switch in loadDisconnectTrackers into
TrackerCategory (DISCONNECT_ALIASES + mapDisconnectCategory /
isRecognisedDisconnectCategory) so it is one testable source of truth.
- DisconnectCategoryCoverageTest reads the bundled list and fails the build if a
category can no longer be mapped to a UI bucket, turning silent drift into a
loud CI failure regardless of how the asset was updated.
No asset data is changed in this PR; the updaters were run and reverted so this
ships mechanism only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kasnder
added a commit
that referenced
this pull request
Jul 11, 2026
…ft flag
The app bundles several remote datasets in app/src/main/assets/ that are read
straight from the read-only APK, so they can only be refreshed by shipping a new
build. Only the on-device hosts blocklist had an update path; the identification
lists (Disconnect/shavar, DuckDuckGo TDS, GeoLite2) were refreshed by hand, which
is easy to forget and left their provenance undocumented.
Rather than add on-device fetches (a privacy cost for data that changes slowly,
and GeoLite2 is license-gated anyway), refresh them at build time:
- scripts/update_disconnect_list.py — Disconnect services.json -> byte-reverse
- scripts/update_ddg_tds.py — DuckDuckGo Android TDS -> field-strip to the
fields the loader reads ({owner:{name,displayName}, default})
- scripts/update_geolite2.py — MaxMind GeoLite2 (needs MAXMIND_LICENSE_KEY)
Each is stdlib-only, validates before writing, writes only on change, and has
--check/--input, matching the convention of scripts/update_exodus_trackers.py
(#622), which stays the single Exodus updater.
.github/workflows/update-lists.yml runs these monthly and opens a PR; the PR's
existing Test CI is the gate that vets a bad upstream.
Disconnect category drift:
- TrackerControl already handles unknown Disconnect categories safely (they fold
to Uncategorised via canonicalise, #571) — but silently. The Disconnect->bucket
mapping is refactored out of an inline switch in loadDisconnectTrackers into
TrackerCategory (DISCONNECT_ALIASES + mapDisconnectCategory /
isRecognisedDisconnectCategory) so it is one testable source of truth.
- DisconnectCategoryCoverageTest reads the bundled list and fails the build if a
category can no longer be mapped to a UI bucket, turning silent drift into a
loud CI failure regardless of how the asset was updated.
No asset data is changed in this PR; the updaters were run and reverted so this
ships mechanism only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #318.
What changed
scripts/update_exodus_trackers.py— a robust, idempotent updater that refreshes the bundled Exodus tracker signatures from the official Exodus JSON API, replacing the fragile HTML scraper proposed in the issue.README.md— one acknowledgement line documenting the Exodus data source and the refresh script.No tracker data is bundled in this PR (see verification below).
Data flow I mapped
TrackerSignatureManager(app/src/main/java/.../analysis/) already downloads the latest signatures at runtime fromhttps://reports.exodus-privacy.eu.org/api/trackersand caches them infilesDir/trackers.json.app/src/main/assets/trackers.jsonis the offline fallback loaded on first launch / when the live fetch fails. Its schema is the API response verbatim:{"trackers": {"<id>": {id, name, code_signature, network_signature, website, categories, ...}}}.parseTrackers) readsid,name,code_signature,network_signature,website. Current asset: 432 trackers, 260 with network signatures, 428 with code signatures (newestcreation_date2023-12-05).The issue's HTML scraper of
reports.exodus-privacy.eu.orgis unnecessary and non-reproducible — Exodus exposes a documented JSON API, which is exactly the source the app already consumes.How a maintainer refreshes the bundled fallback
Stdlib-only (no dependencies). The script validates before writing (top-level
trackersobject, requiredid/name, sane count, non-empty network signatures, regex compile sanity check), writes the raw API bytes verbatim so the committed asset stays byte-identical to what the app downloads, and only rewrites on an actual change.Verification status
Asset already up to date— the live API is byte-identical to the committed asset, so a data refresh right now is a verified no-op. This is why no data blob is committed here.trackerspayload both correctly fail with exit 1 and no write.Licensing
The tracker signatures are produced by Exodus Privacy and distributed under their own terms. This PR commits no Exodus data — the script only mirrors their published API output on demand, and the licensing note is in the script header and README.
🤖 Generated with Claude Code