Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Measurements were not uploaded properly in OONI Probe android 2.0.0 #281

hellais opened this Issue Feb 22, 2019 · 0 comments


None yet
1 participant
Copy link

hellais commented Feb 22, 2019

Impact: measurements which should have been uploader were not and measurements which shouldn't have were in OONI Probe android 2.0.0.

Detection: user reported a bug

Timeline in UTC:
9 Nov 2018
A critical bug is introduced into OONI Probe Android in commit f2d0aa400459d8a8095d89dd2619a4a14a5f6c28

18 Nov 2018
ooniprobe-android 2.0.0-beta.1+39 is shipped including the bug

14 Jan 2018
The bug goes unnoticed and is included in the first public release of OONI Probe Android 2.0.0+41

22:00 30th Jan 2019 UTC
Android bug is filed: ooni/probe-android#188

09:00 31st Jan 2019 UTC
A fix is implemented for the bug

07:00 1st Feb 2019 UTC
The fix is included in a release and shipped to end users

14:35 1st Feb 2019 UTC
Cleanup of affected measurements started: removing affected autoclaved archives and report files coming from 2.0.0* from public S3

09:00 4th Feb 2019 UTC, Monday
Cleanup of affected measurements from publicly accessible data sources finished.

17:30 8th Feb 2019 UTC
We send a tweet asking users to upgrade to the latest version

~05:00 8th Feb 2019 UTC
Users still on OONI Probe Android 2.0.0 were sent a push notification saying:
You may be running an out of date version of OONI Probe which includes a critical bug. Please update to the latest version.

01:00 15th Feb 2019 UTC
All the metadata & data stored in the backups and other publicly inaccessible online data sources is cleaned up.

19th Feb 2019 UTC
Started reaching out to organisations and groups who may have a mirror or a copy of the affected OONI data asking them to resync and delete data from buckets:

~21:00 19th Feb 2019 UTC
A second push notification is sent to OONI Probe Android users still on 2.0.0 saying:
You may be running an out of date version of OONI Probe which includes a critical bug. Please update to the latest version.

What we will do to reduce the chance of this happening in the future:

  • Improve our QA process for the mobile apps by
  • Having unit tests for the app
  • Having end to end integration tests
  • Making a checklist that we should go through before release for items to manually tests and validate before doing a release
  • Expose inside of the app which measurements have been uploaded and which ones have not and support re-uploading measurements
  • Monitor more carefully for anomalies the number of incoming measurements per app version and platform
  • Document our process for resolving critical bugs of this sort and how people can report security issues to us (similar to:

@hellais hellais added the incident label Feb 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.