New proposal for instant experiment result data model #43
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #43 +/- ##
==========================================
+ Coverage 76.12% 80.92% +4.80%
==========================================
Files 69 69
Lines 5989 6067 +78
==========================================
+ Hits 4559 4910 +351
+ Misses 1430 1157 -273 ☔ View full report in Codecov by Sentry. |
|
Once #44 is landed in here. I should be sure to merge this without squashing everything compacting all the above commits into one. For the rationale behind this decision see this comment: #44 (comment) |
Implements: #42 Changes: * Some refactoring to the web_analysis to align with new experiment result model * Refactor feature based analysis into dask * Rename create table query to WebAnalysis * Implement workaround for ooni/probe#2628 * Implement probe_cc based load spreading * Fix start_analysis signature * Kill some dead code * Setup system for running tests on web_analysis * Add checks for ensuring analysis and observations agree on measurement counts * Refactor and add tests for make_cc_batches * Add tests for ground truth generation * A lot of bugfixing and testing improvements
= Overview of scope of this PR Inside of this PR I implement an Experiment Results analysis approach based on the analysis tables. The basic idea is to take the analysis keys that are generated by comparing an individual observation with ground truth data. Through a very large set of rules we are able to assign individual blocking, down and ok rules based on how confident we are in that particular signal being a sign for censorship. We then take all the scores pertaining to a particular observation group relevant to a measurement and generate a `MeasurementExperimentResult` which should be backward compatible with out existing PR. Based on this we add support for generating the experiment results based on the analysis inside of the `mkanalysis` command and a simple web interface for inspecting them. In terms of performance some cursory benchmarks were run the dataset from 2023-09-01 - 2023-11-01 and it was processing data at a rate of ~7k observations per second scaling on 34 cores. = Summary of changes * Implement Experiment Result generation based on the analysis tables * Implement minimal UI for MeasurementExperimentResult * Add support for generating MeasurementExperimentResult as part of mkanalysis cli command * Add more tests for all of the above
* Make utcnow() calls timezone aware * Implement workardound for clickhouse bug mymarilyn/clickhouse-driver#388 * Implement more tests for range deletions * Refactoring of get_prev_range functions * Fix problem in experiment result generation
hellais
force-pushed
the
experiment-results
branch
from
c9ffc63
to
6f628df
Compare
|
The git history is now clean. This branch is ready for review. |
|
I am going to land this into main to make progress on shipping this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements: #42