webconnectivity: long term evolution (aka v0.5.x aka LTE) #2237

bassosimone · 2022-08-26T09:31:22Z

The idea here is to rewrite webconnectivity to use step-by-step and to incorporate some websteps features without deviating from the original data format. Here's what we can reasonably implement:

secondary DNS over UDP resolver
secondary DNS over HTTPS resolver (maybe not for each request b/c if it's blocked, it's blocked?)
TCP connect and TLS handshake for all addresses discovered by any resolver
TCP connect and TLS handshake for extra addresses returned by the TH
TLS handshake regardless of whether the URL is HTTP
Support in the TH for knowing about the result of TLS handshakes
stay true to webconnectivity and just use a single connection for determining redirects
collect extra information also for redirects
probeservices: A/B testing using check-in API feature flags #2396

The general idea is to write this prototype using boilerplate (see ooni/probe-cli#853), and make it available to miniooni users optionally under a disabled-by-default feature flag to collect feedback from users (see #2009). After an initial miniooni only deployment and refinement, we aim to selectively enable this new implementation just for some desktop, cli, android, and ios users via the check-in API, so to collect more data and further improve the implementation.

The overall aim is to start collecting websteps like data without breaking webconnectivity. This strategy should give us better data as well as better information regarding a full rollout of websteps.

Users will be able to run this experiment like this miniooni web_connectivity@v0.5, which will select the new version as opposed to selecting the v0.4.1 version that is being run by default.

The text was updated successfully, but these errors were encountered:

Required by ooni/probe#2237

In a pure step-by-step model, we don't need to trace HTTP round trips like we did before. We _may_ want in the future to also have some form of HTTP tracing (see #868 for a prototype) but doing that is currently not in scope for moving forward the step-by-step design. For this reason, I only added a public convenience function for formatting an OONI spec compatible request. I also added new fields, which should be documented inside the ooni/spec repository (see ooni/probe#2238). Required by ooni/probe#2237

See ooni/probe#2237

bassosimone · 2022-08-26T14:43:38Z

Initial release (v0.5.0)

We merged ooni/probe-cli#882, which contains an initial implementation. I will spend a couple of days running some extra tests and then I'll bless an alpha release, so power users can help us moving forward.

Part of ooni/probe#2237

bassosimone · 2022-08-26T16:28:31Z

Support for late DNS replies (v0.5.1)

See ooni/probe-cli#883, which supports recording late DNS replies.

Part of ooni/probe#2237

This pull request documents more precise stdlib resolver naming that will benefit web_connectivity@v0.5. See ooni/probe#2238, ooni/probe#2237, ooni/probe-cli#885. While there, fix the template for new PRs.

See what we documented at ooni/spec#257 Reference issue: ooni/probe#2238 See also the related ooni/spec PR: ooni/spec#257 See also ooni/probe#2237 While there, bump webconnectivity@v0.5 version because this change has an impact onto the generated data format. The drop in coverage is unavoidable because we've written some tests for `measurex` to ensure we deal with DNS resolvers and transport names correctly depending on the splitting policy we use. (However, `measurex` is only used for the `tor` experiment and, per the step-by-step design document, new experiments should use `measurexlite` instead, so this is hopefully fine(TM).) While there, fix a broken integration test that does not run in `-short` mode.

bassosimone · 2022-08-27T13:48:24Z

Use better resolver engine names (v0.5.2)

See ooni/probe-cli#885, which implements ooni/spec#257.

While working on ooni/probe#2237, I noticed there's no enforced timeout for measurement tasks. So, this diff introduces the following timeouts: 1. use a 4 seconds timeout for the DNS lookup; 2. use a 10 seconds timeout for TCP; 3. use a 15 seconds timeout for HTTP. They are a bit stricter than what we have on the probe because the TH should supposedly have better bandwidth and connectivity.

This diff introduces the following `oohelperd` enhancements: 1. measure both IP addresses resolved by the TH and IP addresses resolved by the probe; 2. when the URL scheme is http and there's no explicit port, measure both 80 and 443 (which will pay off big once we introduce support for optionally performing TLS handshakes); 3. include information about the probe and TH IP addresses into the results: who resolved each IP address, whether an address is a bogon, the ASN associated to an address. This diff is part of ooni/probe#2237

This diff improves oohelperd to measure :443 endpoints with TLS. Part of ooni/probe#2237.

bassosimone · 2022-08-28T12:37:47Z

The above PRs extended oohelperd to (1) measure IP addresses resolved by the probe and by the TH; (2) perform TLS measurements; and (3) return additional information about IP addresses. These enhancements are backwards compatible and should not cause any harm to existing clients. Though, new clients could take advantage of new data to provide better analysis in output to users. We will test these changes in the next days.

bassosimone · 2022-09-05T09:36:35Z

Use TLS info from the TH (v0.5.3)

See ooni/probe-cli#933

bassosimone · 2022-09-08T09:26:57Z

Sync data format with v0.4 and urlgetter (v0.5.5)

We're bumping the version number to reflect recent improvements in the data format implemented in these pull requests: ooni/probe-cli#942, ooni/probe-cli#943, ooni/probe-cli#944

bassosimone · 2022-09-10T14:38:47Z

DoH failure should not set failure flags (v0.5.6)

See ooni/probe-cli#948

It's confusing to see ``` measuring additional addrs from TH: [] ``` when actually nothing is going to be measured. So, instead, let us log about the additional addrs discovered by the TH instead, which is less confusing. Part of ooni/probe#2237

We're bumping the experiment's version number because we changed the name of the field used to contain late/duplicate DNS responses. We have also changed the algorithm to determine #dnsDiff, however, the change should only impact how we log this information. Overall, here the idea is to provide users with a reasonably clear explanation of how the probe maps observations to blocking and accessible. Part of ooni/probe#2237

We're bumping the experiment's version number because we changed the name of the field used to contain late/duplicate DNS responses. We have also changed the algorithm to determine `#dnsDiff`. However, the change should only impact how we log this information. Overall, here the idea is to provide users with a reasonably clear explanation of how the probe maps observations to blocking and accessible using expected/unexpected as the conceptual framework. Part of ooni/probe#2237

Part of ooni/probe#2237

) We're bumping the experiment's version number because we changed the name of the field used to contain late/duplicate DNS responses. We have also changed the algorithm to determine `#dnsDiff`. However, the change should only impact how we log this information. Overall, here the idea is to provide users with a reasonably clear explanation of how the probe maps observations to blocking and accessible using expected/unexpected as the conceptual framework. Part of ooni/probe#2237

Part of ooni/probe#2237

bassosimone · 2023-02-01T08:58:26Z

With #2396 (which is part of the 3.17.x release train), we added support for A/B testing of the webconnectivity version to use depending on feature flags provided by the check-in API. With this functionality, we can start testing this new version of webconnectivity more widely.

bassosimone · 2024-01-22T21:07:04Z

It's about time to call this done! Keeping issues open would just increase our cognitive load! Geronimo!

bassosimone added the triage label Aug 26, 2022

bassosimone self-assigned this Aug 26, 2022

bassosimone changed the title ~~webconnectivity: long term evolution and convergence with websteps~~ webconnectivity: long term evolution Aug 26, 2022

bassosimone added a commit to ooni/probe-cli that referenced this issue Aug 26, 2022

feat(measurexlite): generate HTTP traces

3bcfb5a

Required by ooni/probe#2237

bassosimone mentioned this issue Aug 26, 2022

feat(measurexlite): generate HTTP traces ooni/probe-cli#881

Merged

bassosimone added a commit to ooni/probe-cli that referenced this issue Aug 26, 2022

feat: webconnectivity long term evolution prototype

8b9723d

See ooni/probe#2237

bassosimone mentioned this issue Aug 26, 2022

feat(webconnectivity): long-term-evolution prototype ooni/probe-cli#882

Merged

bassosimone added a commit to ooni/probe-cli that referenced this issue Aug 26, 2022

feat(webconnectivity): long-term-evolution prototype (#882)

1a1d312

See ooni/probe#2237

bassosimone added a commit to ooni/probe-cli that referenced this issue Aug 26, 2022

feat(webconnectivity@v0.5): record late DNS replies

eaa9019

Part of ooni/probe#2237

bassosimone mentioned this issue Aug 26, 2022

feat(webconnectivity@v0.5): record late DNS replies ooni/probe-cli#883

Merged

bassosimone added a commit to ooni/probe-cli that referenced this issue Aug 26, 2022

feat(webconnectivity@v0.5): record late DNS replies (#883)

c3964e4

Part of ooni/probe#2237

This was referenced Aug 27, 2022

doc(dns): more precise stdlib resolver naming ooni/spec#257

Merged

feat: clearly indicate which resolver we're using ooni/probe-cli#885

Merged

bassosimone mentioned this issue Aug 28, 2022

feat(oohelperd): follow (and record) TH and probe endpoints ooni/probe-cli#890

Merged

bassosimone mentioned this issue Aug 28, 2022

feat(oohelperd): measure TLS for :443 endpoints ooni/probe-cli#886

Merged

3 tasks

bassosimone added a commit to ooni/probe-cli that referenced this issue Aug 28, 2022

feat(oohelperd): measure TLS for :443 endpoints (#886)

1e7384d

This diff improves oohelperd to measure :443 endpoints with TLS. Part of ooni/probe#2237.

bassosimone changed the title ~~webconnectivity: long term evolution~~ webconnectivity: long term evolution (aka v0.5.x) Sep 5, 2022

bassosimone mentioned this issue Sep 8, 2022

webconnectivity 0.6: flag server side blocking #2269

Open

bassosimone mentioned this issue Sep 12, 2022

fix(webconnectivity@v0.5): avoid confusing log message ooni/probe-cli#954

Merged

bassosimone mentioned this issue Sep 15, 2022

refactor(webconnectivity@v0.5): improve logging clarity ooni/probe-cli#964

Merged

bassosimone added a commit to ooni/probe-cli that referenced this issue Sep 15, 2022

fix(web_connectivity@v0.5): limit number or redirects

afe63b1

Part of ooni/probe#2237

bassosimone mentioned this issue Sep 15, 2022

fix(web_connectivity@v0.5): limit number of redirects ooni/probe-cli#965

Merged

bassosimone added a commit to ooni/probe-cli that referenced this issue Sep 15, 2022

fix(web_connectivity@v0.5): limit number of redirects (#965)

ad01856

Part of ooni/probe#2237

bassosimone changed the title ~~webconnectivity: long term evolution (aka v0.5.x)~~ webconnectivity: long term evolution (aka v0.5.x aka LTE) Nov 18, 2022

bassosimone mentioned this issue Nov 18, 2022

webconnectivity LTE collects additional metrics to facilitate throttling measurements ooni/ooni.org#1297

Closed

bassosimone added a commit to ooni/probe-cli that referenced this issue Nov 21, 2022

[backport] fix(web_connectivity@v0.5): limit number of redirects (#965)

9bf6e38

Part of ooni/probe#2237

bassosimone mentioned this issue Jan 12, 2023

probeservices: A/B testing using check-in API feature flags #2396

Closed

bassosimone added the fixed-by-webconnectivity-lte The next version of webconnectivity fixes this issue label Feb 1, 2023

bassosimone closed this as completed Jan 22, 2024

bassosimone added the 2024-01-data-quality-cleanup Data quality issues addressed on 2024-01 label Jan 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

webconnectivity: long term evolution (aka v0.5.x aka LTE) #2237

webconnectivity: long term evolution (aka v0.5.x aka LTE) #2237

bassosimone commented Aug 26, 2022 •

edited

Loading

bassosimone commented Aug 26, 2022 •

edited

Loading

bassosimone commented Aug 26, 2022

bassosimone commented Aug 27, 2022 •

edited

Loading

bassosimone commented Aug 28, 2022

bassosimone commented Sep 5, 2022

bassosimone commented Sep 8, 2022

bassosimone commented Sep 10, 2022 •

edited

Loading

bassosimone commented Feb 1, 2023

bassosimone commented Jan 22, 2024 •

edited

Loading

webconnectivity: long term evolution (aka v0.5.x aka LTE) #2237

webconnectivity: long term evolution (aka v0.5.x aka LTE) #2237

Comments

bassosimone commented Aug 26, 2022 • edited Loading

bassosimone commented Aug 26, 2022 • edited Loading

Initial release (v0.5.0)

bassosimone commented Aug 26, 2022

Support for late DNS replies (v0.5.1)

bassosimone commented Aug 27, 2022 • edited Loading

Use better resolver engine names (v0.5.2)

bassosimone commented Aug 28, 2022

bassosimone commented Sep 5, 2022

Use TLS info from the TH (v0.5.3)

bassosimone commented Sep 8, 2022

Sync data format with v0.4 and urlgetter (v0.5.5)

bassosimone commented Sep 10, 2022 • edited Loading

DoH failure should not set failure flags (v0.5.6)

bassosimone commented Feb 1, 2023

bassosimone commented Jan 22, 2024 • edited Loading

bassosimone commented Aug 26, 2022 •

edited

Loading

bassosimone commented Aug 26, 2022 •

edited

Loading

bassosimone commented Aug 27, 2022 •

edited

Loading

bassosimone commented Sep 10, 2022 •

edited

Loading

bassosimone commented Jan 22, 2024 •

edited

Loading