2.4 Improve upon OONI Explorer

[hellais]

To enable the internet freedom community to rapidly respond to emergent censorship events, we aim to make a series of improvements to OONI Explorer that will simplify the process of exploring OONI data to uncover censorship events. To this end, we will continue to improve upon the UX of OONI Explorer based on community feedback that we will collect through usability studies and workshops. We will also implement new features to better meet community needs. Some improvements that we would like to prioritize on include:

• User feedback reporting mechanism. To improve our heuristics and the quality of OONI measurements, we aim to add support to enable users to share feedback via OONI Explorer to report false positives, or share other information that is useful for evaluating measurements (such as information about blocked websites).
• OONI Explorer user accounts. To support more OONI Explorer use cases (such as database-heavy loads and content management) we will add support for creating OONI Explorer user accounts. This system will provide privileges when using OONI Explorer and we will grant accounts to our partners and collaborators. To prevent abuse of our systems and protect OONI Explorer users, we will ensure that only the minimum amount of personal information required will be collected and that users are aware of any potential risks that may arise from account creation. Accounts will be voluntary and they will only require a username and password. An email address can be provided optionally to provide account recovery.
• Export of OONI data in CSV files. Currently, OONI data can be downloaded from the OONI API in JSON format. However, analyzing JSON data requires a technical background and many community members interested in OONI data (such as political scientists, lawyers, and journalists) have requested that OONI Explorer supports the export of CSV files. By supporting the export of CSV files, we will enable community members to use OONI data as part of their research, policy, and advocacy efforts.
• Creation of network pages and domain pages in OONI Explorer. OONI Explorer currently presents findings on a country level granularity, but internet censorship often varies significantly from ISP to ISP. We will therefore create pages that present internet censorship findings on a more granular, network level. We will also create domain-specific pages to show how tested web resources are interfered with on a global scale.
• Integration of relevant network measurements into OONI Explorer. We will seek collaboration with other censorship measurement projects on the integration of their datasets into OONI Explorer. This will help provide stronger signals of censorship, particularly when it’s possible to correlate and corroborate censorship events through multiple data sources. We have already begun discussions with IODA regarding the possible integration of their data and we will encourage other censorship measurement projects as well (such as Censored Planet, ICLab, and NetBlocks).

Over the last few months, the latency of the analysis and publication of OONI measurements on OONI Explorer has been reduced from 24 hours to seconds. Having written a new “fast-path” pipeline, OONI measurements collected from around the world are now analyzed and published in near real-time. This means that as soon as an OONI Probe user runs a test anywhere around the world, their test results are automatically analyzed and openly published on OONI Explorer within a minute. The internet freedom community can therefore benefit from near real-time OONI measurements for rapid response to emergent censorship events around the world.

To enable the internet freedom community to examine website measurements in more depth, we will develop website-centric aggregate views of the data that will be exposed on OONI Explorer. We will write data processing tools and database queries to compute the necessary metrics and populate the required database tables. We will make it possible to integrate website-related metrics into other tooling with some machine-to-machine data export format.

As the load on the OONI database increases by supporting more advanced workflows, it is necessary for us to start introducing user accounts (similarly to IODA’s Dashboard which also provides user accounts for accessing a more advanced analysis dashboard) that allow us to enforce some quotas on database use. This is required in order to not penalize the general public with an unresponsive OONI Explorer.

Anyone from the public will be able to request a user account to access functionality available only to logged-in users. We will add support for user accounts (either in OONI Explorer or on another web platform) so that people can register and login to access functions which are more database-heavy. Account users will be able to access certain OONI Explorer pages with more detailed views that are particularly database-intensive (which we cannot afford to expose publicly due to resource constraints and the risk of getting DoSed).

The main criterion that we will use to determine whether to accept an account request or not is whether they are likely to abuse our services or if they will be mindful of our resources and not abuse them. We generally aim to support as many researchers from the internet freedom community as possible with accounts. When someone asks for an account, we will probably just request an email address and the reason for the account. We will likely make account creation automatic, but may lock it down if accounts start getting abused.

We will implement a frontend that presents results in a website-centric way to logged-in users. If possible, we would like to make some subset of these views also available to the general public, but we will have to evaluate this based on how much database load it creates.

As censorship events emerge, URL priorities may need to change on an ongoing basis. We will therefore create a system that allows logged-in users to request changes in URL priorities, and we will also enable members of the OONI community to request changes via an interface. This will help ensure that when censorship events emerge, relevant URLs are measured.

[hellais]

• User feedback reporting mechanism
• OONI Explorer user accounts.
• Export of OONI data in CSV files.
• Creation of network pages and domain pages in OONI Explorer, see: Add third party data to country and network page explorer#831 & Domain centric pages explorer#830
• Integration of relevant network measurements into OONI Explorer, see: Add third party data to country and network page explorer#831

[agrabeli]

Everything here has been done!