You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
GSoC 2021: Measuring QUIC and TLS censorship with OONI Probe
Introduction
The main objectives of my Summer of Code project were twofold: Furthering the integration HTTP/3 measurements into OONI Probe, and enabling OONI Probe to use custom TLS fingerprints in order to circumvent potential TLS fingerprint blocking. These objectives have been realized in the form of a new experiment prototype, tentatively called Websteps, which supports HTTP/3 measurements and uses uTLS to emulate popular TLS fingerprints.
In the following I will describe the deliverables of my GSoC experience, and link to the corresponding pull requests in the ooni/probe-cli repository.
Along with the main deliverables, I also fixed bugs that I discovered in the existing implementation, which are for example documented here: ooni/probe-cli#432 (comment).
The design choices are a result of the collaboration with my GSoC mentor, @bassosimone, who I am very thankful for.
Improved handling of system errors by introducing system errno constants to detect and identify system errors. This solved the problem of classifying localized Windows error strings: #1526.
Both improvements provide more robust alternatives for string comparisons.
As part of the refactoring I also divided the classifier code by layers, so that each of the following protocol layers has it‘s own classifier with layer-specific errors: DNS, TLS over TCP, and QUIC.
A general classifier remains to handle errors that can occur on multiple layers.
This change helps with the maintainability of the classification code and improves the interface of the errorx package.
This commit introduced UTLSConn, a new type of TLS-like connection, into the Probe engine. This enables the OONI Probe networking library to use utls, which provides ClientHello fingerprinting resistance and low-level access to TLS handshake specifics.
Limitations: net/http does not support using alternative TLS implementations apart from crypto/tls, due to an implementation that casts to (tls.Conn). This is why the http.Transport breaks when using utls (see refraction-networking/utls#16).
Unmerged pull request: ooni/probe-cli#414, with more utls integration work, which is described in the pull request description. After considering and discussing the integration design with my mentor, we decided to replace it with a different approach which is described below.
Deliverable 3: Websteps - A new OONI Probe network experiment and Testhelper
Specification ooni/spec#219 of the new testhelper design (original design idea by Simone Basso, with my contribution to the further elaboration).
The OONI testhelper runs the control measurements for the Web Connectivity experiment. Control measurements replicate the probe client's measurements from an uncensored network and indicate what to expect. That way, the OONI Probe client software can detect anomalies in the client's network.
Web Connectivity is a mainstream, large-scale network censorship experiment of OONI Probe, which is deployed in the enduser application. It takes input in the form of URLs and measures whether the hosts are accessible to the probe client.
ooni/probe-cli#432 contains the new implementation of the testhelper along with comprehensive unit and integration tests, as well as the prototype of a corresponding new mainstream censorship measurement which is similar to Web Connectivity.
The new testhelper introduces HTTP/3 control measurements, and can use utls for TLS parroting. It also supports testing multiple IP endpoints per domain which the legacy design does not.
As precisely described in ooni/spec#219, the testhelper algorithm consists of three main steps:
InitialChecks verifies that the input URL can be parsed, has an expected scheme, and contains a valid domain name.
Explore enumerates all the URLs that it discovers by redirection from the original URL, or by detecting h3 support at the target host.
Generate performs a step-by-step measurement of each discovered URL.
The idea of the new testhelper is that it tells the probe client what to measure, and what to expect. It produces a set of URLs that originated from the initial input URL. Follow-up URLs may be caused by HTTP redirections or when the server announces support for HTTP/3.
Unmerged pull request including the first design of a new testhelper, which we discarded due to its high complexity in regards to manual HTTP redirection and cookies handling.
Websteps (preliminary name) is a new OONI Probe network measurement, which is at the prototype stage still. Websteps is the client probe software which can work in cooperation with the above described testhelper to know what to measure and what to expect. Websteps has the same high-level measurement objective as Web Connectivity and further supports HTTP/3 measurements, TLS parroting and testing all IP endpoints of a domain.
The prototype does not yet have:
unit and integration tests,
a policy for choosing which endpoints to measure, out of the endpoints proposed by the testhelper,
an analysis tool to compare the control and the probe measurement.
Websteps and the testhelper use the same code for performing network operations such as the TCP three-way-handshake, or the HTTP roundtrip.
Unmerged pull request: ooni/probe-cli#431, including the first design of a newWeb Connectivity-like experiment, which we discarded due to its high complexity in regards to manual HTTP redirection and cookies handling.
This pull request enables Websteps and the testhelper to use uTLS for TLS parroting.
In order to solve the previously described problems with using a fork of the crypto/tls library, we have forked the net/http standard library (ooni/oohttp) and modified it so that it supports all tls-like connections, like utls.Conn. As an API, I built a wrapper for the Transport (oohttp.StdlibTransport) which looks like a "net/http".Transport but uses the "ooni/oohttp".Transport internally.
The testhelper and Websteps code uses this transport wrapper for making HTTP requests with uTLS connections. As a result, we gain access to the TLS fingerprint. Tentatively, we apply the utls.HelloChrome_Auto fingerprint, i.e. the simulated TLS fingerprint of the Google Chrome browser.
How to build and test Websteps
Build and deploy the testhelper locally
The default listening endpoint is ":8080".
go build ./cmd/oohelperd
./oohelperd [-endpoint]
Try out the testserver with curl
Minimal example, with the default testhelper endpoint ":8080".
curl -X POST -d '{"url": "http://www.google.com"}' http://localhost:8080/api/unstable/websteps
Build and run Websteps
go build ./cmd/miniooni
./miniooni -i "https://www.google.com" --reportfile="measurement.json" websteps
The text was updated successfully, but these errors were encountered:
Thanks for writing this comprehensive report @kelmenhorst! It has been a pleasure to host you for this GSoC, and I think you definitely did great work to help us at @ooni improve our measurement tools!
I put this issue in my Sprint planning and will keep it open until I have finished the final evaluation on the GSoC website.
GSoC 2021: Measuring QUIC and TLS censorship with OONI Probe
Introduction
The main objectives of my Summer of Code project were twofold: Furthering the integration HTTP/3 measurements into OONI Probe, and enabling OONI Probe to use custom TLS fingerprints in order to circumvent potential TLS fingerprint blocking. These objectives have been realized in the form of a new experiment prototype, tentatively called
Websteps
, which supports HTTP/3 measurements and usesuTLS
to emulate popular TLS fingerprints.In the following I will describe the deliverables of my GSoC experience, and link to the corresponding pull requests in the
ooni/probe-cli
repository.Along with the main deliverables, I also fixed bugs that I discovered in the existing implementation, which are for example documented here: ooni/probe-cli#432 (comment).
The design choices are a result of the collaboration with my GSoC mentor, @bassosimone, who I am very thankful for.
Deliverable 1: Refactored OONI error classification
Merged pull request: ooni/probe-cli#386
This contribution is a refactoring of the classifier code which is responsible for translating network errors to OONI error types.
The outcome in particular:
Improved handling of QUIC errors by
a) importing QUIC error codes from quic-go,
b) importing TLS alert codes from RFC5246,
Improved handling of system errors by introducing system errno constants to detect and identify system errors. This solved the problem of classifying localized Windows error strings: #1526.
Both improvements provide more robust alternatives for string comparisons.
As part of the refactoring I also divided the classifier code by layers, so that each of the following protocol layers has it‘s own classifier with layer-specific errors: DNS, TLS over TCP, and QUIC.
A general classifier remains to handle errors that can occur on multiple layers.
This change helps with the maintainability of the classification code and improves the interface of the errorx package.
Unmerged pull request: ooni/probe-cli#377, with the first design.
Deliverable 2:
utls
for TLS parrotingMerged commit: ooni/probe-cli@285290a
This commit introduced
UTLSConn
, a new type of TLS-like connection, into the Probe engine. This enables the OONI Probe networking library to useutls
, which provides ClientHello fingerprinting resistance and low-level access to TLS handshake specifics.Limitations:
net/http
does not support using alternative TLS implementations apart fromcrypto/tls
, due to an implementation that casts to(tls.Conn)
. This is why the http.Transport breaks when usingutls
(see refraction-networking/utls#16).Unmerged pull request: ooni/probe-cli#414, with more
utls
integration work, which is described in the pull request description. After considering and discussing the integration design with my mentor, we decided to replace it with a different approach which is described below.Deliverable 3: Websteps - A new OONI Probe network experiment and Testhelper
Specification ooni/spec#219 of the new testhelper design (original design idea by Simone Basso, with my contribution to the further elaboration).
Merged pull request: ooni/probe-cli#432
The OONI testhelper runs the control measurements for the
Web Connectivity
experiment. Control measurements replicate the probe client's measurements from an uncensored network and indicate what to expect. That way, the OONI Probe client software can detect anomalies in the client's network.Web Connectivity
is a mainstream, large-scale network censorship experiment of OONI Probe, which is deployed in the enduser application. It takes input in the form of URLs and measures whether the hosts are accessible to the probe client.ooni/probe-cli#432 contains the new implementation of the testhelper along with comprehensive unit and integration tests, as well as the prototype of a corresponding new mainstream censorship measurement which is similar to
Web Connectivity
.The new testhelper introduces HTTP/3 control measurements, and can use
utls
for TLS parroting. It also supports testing multiple IP endpoints per domain which the legacy design does not.As precisely described in ooni/spec#219, the testhelper algorithm consists of three main steps:
InitialChecks
verifies that the input URL can be parsed, has an expected scheme, and contains a valid domain name.Explore
enumerates all the URLs that it discovers by redirection from the original URL, or by detecting h3 support at the target host.Generate
performs a step-by-step measurement of each discovered URL.The idea of the new testhelper is that it tells the probe client what to measure, and what to expect. It produces a set of URLs that originated from the initial input URL. Follow-up URLs may be caused by HTTP redirections or when the server announces support for HTTP/3.
Unmerged pull request including the first design of a new testhelper, which we discarded due to its high complexity in regards to manual HTTP redirection and cookies handling.
Websteps
(preliminary name) is a new OONI Probe network measurement, which is at the prototype stage still.Websteps
is the client probe software which can work in cooperation with the above described testhelper to know what to measure and what to expect.Websteps
has the same high-level measurement objective asWeb Connectivity
and further supports HTTP/3 measurements, TLS parroting and testing all IP endpoints of a domain.The prototype does not yet have:
Websteps
and the testhelper use the same code for performing network operations such as the TCP three-way-handshake, or the HTTP roundtrip.Unmerged pull request: ooni/probe-cli#431, including the first design of a new
Web Connectivity
-like experiment, which we discarded due to its high complexity in regards to manual HTTP redirection and cookies handling.Deliverable 4: Enable TLS parroting for
Websteps
Merged pull request: ooni/probe-cli#442
This pull request enables
Websteps
and the testhelper to useuTLS
for TLS parroting.In order to solve the previously described problems with using a fork of the
crypto/tls
library, we have forked the net/http standard library (ooni/oohttp
) and modified it so that it supports all tls-like connections, likeutls.Conn
. As an API, I built a wrapper for the Transport (oohttp.StdlibTransport
) which looks like a"net/http".Transport
but uses the"ooni/oohttp".Transport
internally.The testhelper and
Websteps
code uses this transport wrapper for making HTTP requests withuTLS
connections. As a result, we gain access to the TLS fingerprint. Tentatively, we apply theutls.HelloChrome_Auto
fingerprint, i.e. the simulated TLS fingerprint of the Google Chrome browser.How to build and test
Websteps
Build and deploy the testhelper locally
The default listening endpoint is
":8080"
.Try out the testserver with curl
Minimal example, with the default testhelper endpoint
":8080"
.Build and run
Websteps
The text was updated successfully, but these errors were encountered: