fix(urlgetter): allow body signaled by EOF

[bassosimone]

% ./miniooni -i http://www.un.org/ urlgetter
[      0.000454] <info> miniooni home directory: $HOME/.miniooni
[      0.002826] <info> Looking up OONI backends; please be patient...
[      0.613004] <info> session: using probe services: {Address:https://ps2.ooni.io Type:https Front:}
[      0.613052] <info> Looking up your location; please be patient...
[      1.062837] <info> - country: IT
[      1.062869] <info> - network: Vodafone Italia S.p.A. (AS30722)
[      1.062879] <info> - resolver's IP: 91.80.36.85
[      1.062887] <info> - resolver's network: Vodafone Italia S.p.A. (AS30722)
[      1.062930] <info> [1/1] running with input: http://www.un.org/
[      1.406371] <warn> measurement failed: map[error:eof_error]
[      1.406422] <info> submitting measurement to OONI collector; please be patient...
[      1.504477] <info> New reportID: 20210119T133854Z_urlgetter_IT_30722_n1_KBFwIlfxPe0lfoUr
[      1.603306] <info> saving measurement to disk
[      1.604541] <info> experiment: recv 135  byte, sent 276  byte
[      1.605301] <info> sessionresolver: failure rate: primary: 0/3; fallback: 0/0
[      1.605573] <info> whole session: recv  10 kbyte, sent   7 kbyte

The bug is in httptransport.SaverBodyHTTPTransport. How we save just a slice of the body is broken when EOF signals the response body end. We need to find an alternative implementation. While working on this issue, we should also consider whether we could address https://github.com/ooni/probe-engine/issues/1061 as well.

[hellais]

More cases found here:
https://explorer.ooni.org/measurement/20210127T202108Z_webconnectivity_AZ_39232_n1_dDtZNOnO4zK3JIQj?input=http%3A%2F%2Fadultfriendfinder.com%2F
https://explorer.ooni.org/measurement/20210205T084354Z_webconnectivity_AZ_13099_n1_XL1w2iidu6Dvvb2Q?input=http%3A%2F%2Fcartercenter.org%2F

[bassosimone]

With this diff applied

diff --git a/internal/engine/netx/netx.go b/internal/engine/netx/netx.go
index 59b38af..45594db 100644
--- a/internal/engine/netx/netx.go
+++ b/internal/engine/netx/netx.go
@@ -249,8 +249,10 @@ func NewHTTPTransport(config Config) HTTPRoundTripper {
        if config.HTTPSaver != nil {
                txp = httptransport.SaverMetadataHTTPTransport{
                        RoundTripper: txp, Saver: config.HTTPSaver, Transport: transport}
-               txp = httptransport.SaverBodyHTTPTransport{
-                       RoundTripper: txp, Saver: config.HTTPSaver}
+               /*
+                       txp = httptransport.SaverBodyHTTPTransport{
+                               RoundTripper: txp, Saver: config.HTTPSaver}
+               */
                txp = httptransport.SaverPerformanceHTTPTransport{
                        RoundTripper: txp, Saver: config.HTTPSaver}
                txp = httptransport.SaverTransactionHTTPTransport{

The problem disappears: https://explorer.ooni.org/measurement/20210211T120627Z_urlgetter_IT_30722_n1_rl94sUjPKE6A4z4Z?input=http%3A%2F%2Fwww.un.org%2F

[bassosimone]

With ooni/probe-cli#226 applied, all works as intended.

See: https://explorer.ooni.org/measurement/20210211T124748Z_urlgetter_IT_30722_n1_BaokShWakUEuDYnB?input=http%3A%2F%2Fwww.un.org%2F

[bassosimone]

We're done!