Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(datafmt): sync measurexlite and v0.5 with previous code #942

Merged
merged 5 commits into from
Sep 8, 2022
Merged

fix(datafmt): sync measurexlite and v0.5 with previous code #942

merged 5 commits into from
Sep 8, 2022

Conversation

bassosimone
Copy link
Contributor

This diff is the first step towards ooni/probe#2238. Let us make sure that new code uses the same data format of previously existing code. Once we're sure about this fact, we can update ooni/spec.

@bassosimone
fix(model/archival.go): more optional keys
39dd48f 
Basically, `t0` and `transaction_id` should be optional. Version 0.4.x
of web_connectivity should not include them, version 0.5.x should.

There is a technical reason why v0.4.x should not include them. The code
it is based on, tracex, does not record these two fields.

Whereas, v0.5.x, uses measurexlite, which records these two fields.

Part of ooni/probe#2238
@bassosimone
fix(webconnectivity@v0.5): add more fields
5fb22cf 
This diff adds the following fields to webconnectivity@v0.5:

1. agent, always set to "redirect" (legacy field);

2. client_resolver, properly initialized w/ the resolver's IPv4 address;

3. retries, legacy field always set to null;

4. socksproxy, legacy field always set to null.

Part of ooni/probe#2238
@bassosimone
fix(webconnectivity@v0.5): register extensions
84a523f 
The general idea behind this field is that we would be able
in the future to tweak the data model for some fields, by declaring
we're using a later version, so it seems useful to add it.

See ooni/probe#2238
@bassosimone
fix(measurexlite): use tcp or quic for tls handshake network
be099c9 
This diff fixes a bug where measurexlite was using "tls" as the
protocol for the TLS handshake when using TCP.

While this choice _could_ make sense, the rest of the code we have
written so far uses "tcp" instead.

Using "tcp" makes more sense because it allows you to search for
the same endpoint across different events by checking for the same
network and for the same endpoint rather than special casing TLS
handshakes for using "tls" when the endpoint is "tcp".

See ooni/probe#2238
@bassosimone
chore: run alltests.yml for "alltestsbuild" branches
32de37d 
Part of ooni/probe#2238
@bassosimone bassosimone requested a review from hellais as a code owner September 8, 2022 07:47
@bassosimone
Copy link
Contributor Author

Let's break down how alltests is failing:

?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/autorun	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/app	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/autorun	[no test files]
ok  	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/geoip	0.098s
ok  	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/info	0.110s
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/list	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/onboard	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/reset	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/rm	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/root	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/run	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/show	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/upload	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/cli/version	[no test files]
ok  	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/config	0.090s
ok  	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/database	0.931s
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/enginex	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/log/handlers/batch	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/log/handlers/cli	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/log/handlers/cli/progress	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/log/handlers/syslog	[no test files]
ok  	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/nettests	6.739s
ok  	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/ooni	0.126s
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/oonitest	[no test files]
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/output	[no test files]
ok  	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/utils	0.024s
?   	github.com/ooni/probe-cli/v3/cmd/ooniprobe/internal/utils/homedir	[no test files]
ok  	github.com/ooni/probe-cli/v3/internal/atomicx	0.032s
ok  	github.com/ooni/probe-cli/v3/internal/bytecounter	0.092s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/apitool	2.561s
?   	github.com/ooni/probe-cli/v3/internal/cmd/e2epostprocess	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/cmd/getresources	[no test files]
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar	2.052s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar/badproxy	3.215s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar/flagx	0.037s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar/httpproxy	1.274s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar/iptables	4.653s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar/resolver	3.257s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar/tlsproxy	0.700s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/jafar/uncensored	0.212s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/miniooni	2.300s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/oohelper	0.835s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/oohelper/internal	1.107s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/oohelperd	0.919s
ok  	github.com/ooni/probe-cli/v3/internal/cmd/oonireport	2.789s
?   	github.com/ooni/probe-cli/v3/internal/cmd/printversion	[no test files]

Everything is good up to this point.

2022/09/08 07:49:13  info Platform: linux          
2022/09/08 07:49:13  info ProbeASN: 0              
2022/09/08 07:49:13  info ProbeASNString: AS0      
2022/09/08 07:49:13  info ProbeCC: US              
2022/09/08 07:49:13  info ProbeIP: 20.225.83.60    
2022/09/08 07:49:13  info ProbeNetworkName:        
2022/09/08 07:49:13  info ResolverASN: 0           
2022/09/08 07:49:13  info ResolverASNString: AS0   
2022/09/08 07:49:13  info ResolverIP: 20.49.89.107 
2022/09/08 07:49:13  info ResolverNetworkName:        
[snip]
--- FAIL: TestSessionLocationLookup (0.09s)
    session_integration_test.go:279: unexpected ProbeASNString
[snip]
FAIL
FAIL	github.com/ooni/probe-cli/v3/internal/engine	108.816s

This test is failing because the address used by GitHub isn't included into the ASN database we're using. 😭

ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/dash	0.106s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/dnscheck	1.723s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/dnsping	0.344s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/example	0.104s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/fbmessenger	0.890s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/hhfm	0.169s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/hirl	5.123s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/httphostheader	0.390s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/ndt7	27.527s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/psiphon	2.102s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/quicping	28.377s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/riseupvpn	0.561s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/run	0.096s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/signal	1.232s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/simplequicping	0.388s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/sniblocking	0.491s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/stunreachability	11.024s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/tcpping	0.095s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/telegram	2.656s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/tlsping	0.255s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/tlstool	0.527s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/tlstool/internal	1.199s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/tor	0.295s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/torsf	3.166s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/urlgetter	9.727s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/vanillator	3.103s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/webconnectivity	4.835s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/webconnectivity/internal	0.026s
ok  	github.com/ooni/probe-cli/v3/internal/engine/experiment/whatsapp	1.467s
ok  	github.com/ooni/probe-cli/v3/internal/engine/geolocate	1.245s
?   	github.com/ooni/probe-cli/v3/internal/engine/mockable	[no test files]
ok  	github.com/ooni/probe-cli/v3/internal/engine/netx	6.083s
ok  	github.com/ooni/probe-cli/v3/internal/engine/probeservices	10.319s
?   	github.com/ooni/probe-cli/v3/internal/engine/probeservices/testorchestra	[no test files]
ok  	github.com/ooni/probe-cli/v3/internal/engine/sessionresolver	0.186s
?   	github.com/ooni/probe-cli/v3/internal/experiment/webconnectivity	[no test files]
ok  	github.com/ooni/probe-cli/v3/internal/fsx	0.083s
ok  	github.com/ooni/probe-cli/v3/internal/geoipx	0.031s
ok  	github.com/ooni/probe-cli/v3/internal/httpx	0.118s
ok  	github.com/ooni/probe-cli/v3/internal/humanize	0.022s
ok  	github.com/ooni/probe-cli/v3/internal/kvstore	0.031s
ok  	github.com/ooni/probe-cli/v3/internal/legacy/assetsdir	0.020s
ok  	github.com/ooni/probe-cli/v3/internal/logx	0.033s
ok  	github.com/ooni/probe-cli/v3/internal/measurex	0.104s
ok  	github.com/ooni/probe-cli/v3/internal/measurexlite	4.134s
ok  	github.com/ooni/probe-cli/v3/internal/mlablocate	0.516s
ok  	github.com/ooni/probe-cli/v3/internal/mlablocatev2	0.518s
ok  	github.com/ooni/probe-cli/v3/internal/model	0.043s
ok  	github.com/ooni/probe-cli/v3/internal/model/mocks	0.073s
ok  	github.com/ooni/probe-cli/v3/internal/multierror	0.031s
ok  	github.com/ooni/probe-cli/v3/internal/netxlite	47.322s
ok  	github.com/ooni/probe-cli/v3/internal/netxlite/filtering	12.200s
?   	github.com/ooni/probe-cli/v3/internal/netxlite/internal/gencertifi	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/netxlite/internal/generrno	[no test files]
ok  	github.com/ooni/probe-cli/v3/internal/netxlite/mocks	0.034s
ok  	github.com/ooni/probe-cli/v3/internal/netxlite/quictesting	0.026s
ok  	github.com/ooni/probe-cli/v3/internal/oonirun	0.124s
ok  	github.com/ooni/probe-cli/v3/internal/platform	0.021s
ok  	github.com/ooni/probe-cli/v3/internal/ptx	4.211s
ok  	github.com/ooni/probe-cli/v3/internal/randx	0.040s
ok  	github.com/ooni/probe-cli/v3/internal/registry	0.115s
ok  	github.com/ooni/probe-cli/v3/internal/runtimex	0.031s
ok  	github.com/ooni/probe-cli/v3/internal/scrubber	0.104s
ok  	github.com/ooni/probe-cli/v3/internal/shellx	0.035s
ok  	github.com/ooni/probe-cli/v3/internal/stuninput	0.021s
ok  	github.com/ooni/probe-cli/v3/internal/testingx	0.022s
ok  	github.com/ooni/probe-cli/v3/internal/torlogs	0.043s
ok  	github.com/ooni/probe-cli/v3/internal/tracex	0.128s
ok  	github.com/ooni/probe-cli/v3/internal/tunnel	8.005s
ok  	github.com/ooni/probe-cli/v3/internal/tunnel/mocks	0.045s
?   	github.com/ooni/probe-cli/v3/internal/tutorial/experiment/torsf/chapter01	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/experiment/torsf/chapter02	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/experiment/torsf/chapter03	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/experiment/torsf/chapter04	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/generator	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter01	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter02	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter03	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter04	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter05	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter06	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter07	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter08	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter09	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter10	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter11	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter12	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter13	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/measurex/chapter14	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter01	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter02	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter03	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter04	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter05	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter06	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter07	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/tutorial/netxlite/chapter08	[no test files]
?   	github.com/ooni/probe-cli/v3/internal/version	[no test files]

Everything above is WAI.

2022/09/08 07:52:29 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
--- FAIL: TestPrivacyAndScrubbing (2.32s)
    task_test.go:146: unexpected result
2022/09/08 07:52:37  info sessionresolver: [{"URL":"https://dns.quad9.net/dns-query","Score":0.5928741295229714},{"URL":"https://mozilla.cloudflare-dns.com/dns-query","Score":0.535208448310007},{"URL":"http3://dns.google/dns-query","Score":0.44736898782354756},{"URL":"https://cloudflare-dns.com/dns-query","Score":0.35396532605638786},{"URL":"http3://cloudflare-dns.com/dns-query","Score":0.27021100371134904},{"URL":"http3://mozilla.cloudflare-dns.com/dns-query","Score":0.15224974440998434},{"URL":"https://dns.google/dns-query","Score":0.11693296951817514},{"URL":"system:///","Score":0}]
--- FAIL: TestGeolocateGood (0.02s)
    session_integration_test.go:101: location.Org is empty
FAIL
FAIL	github.com/ooni/probe-cli/v3/pkg/oonimkall	24.312s
FAIL
Error: Process completed with exit code 1.

Again, we see failures because the ASN geolocation fails.

Based on these results I think the following:

  1. it might be that the ASN database just does not include such an IP address (while unfortunate, the lack of a cloud IP address is not very worrying but we should probably run more tests on our end for this release);

  2. none of the failed tests is related to the changes in this PR, so merging is still okay.

@bassosimone bassosimone merged commit 39cb595 into master Sep 8, 2022
@bassosimone bassosimone deleted the alltestsbuild branch September 8, 2022 08:02
bassosimone added a commit that referenced this pull request Sep 8, 2022
@bassosimone
[backport] fix(datafmt): sync measurexlite and v0.5 with previous code (
668378e 
#942)

* fix(model/archival.go): more optional keys

Basically, `t0` and `transaction_id` should be optional. Version 0.4.x
of web_connectivity should not include them, version 0.5.x should.

There is a technical reason why v0.4.x should not include them. The code
it is based on, tracex, does not record these two fields.

Whereas, v0.5.x, uses measurexlite, which records these two fields.

Part of ooni/probe#2238

* fix(webconnectivity@v0.5): add more fields

This diff adds the following fields to webconnectivity@v0.5:

1. agent, always set to "redirect" (legacy field);

2. client_resolver, properly initialized w/ the resolver's IPv4 address;

3. retries, legacy field always set to null;

4. socksproxy, legacy field always set to null.

Part of ooni/probe#2238

* fix(webconnectivity@v0.5): register extensions

The general idea behind this field is that we would be able
in the future to tweak the data model for some fields, by declaring
we're using a later version, so it seems useful to add it.

See ooni/probe#2238

* fix(measurexlite): use tcp or quic for tls handshake network

This diff fixes a bug where measurexlite was using "tls" as the
protocol for the TLS handshake when using TCP.

While this choice _could_ make sense, the rest of the code we have
written so far uses "tcp" instead.

Using "tcp" makes more sense because it allows you to search for
the same endpoint across different events by checking for the same
network and for the same endpoint rather than special casing TLS
handshakes for using "tls" when the endpoint is "tcp".

See ooni/probe#2238

* chore: run alltests.yml for "alltestsbuild" branches

Part of ooni/probe#2238
bassosimone added a commit that referenced this pull request Sep 8, 2022
@bassosimone
chore: web_connectivity v0.5.5
d5a2691 
We're bumping the version number to reflect recent improvements in the
data format implemented in these pull requests:

- #942

- #943

- #944

Reference issue: ooni/probe#2238
@bassosimone bassosimone mentioned this pull request Sep 8, 2022
Merged
bassosimone added a commit that referenced this pull request Sep 8, 2022
@bassosimone
chore: bump web_connectivity@v0.5 version to 0.5.5 (#945)
800217d 
chore: web_connectivity v0.5.5

We're bumping the version number to reflect recent improvements in the
data format implemented in these pull requests:

- #942

- #943

- #944

Reference issue: ooni/probe#2238
bassosimone added a commit that referenced this pull request Sep 8, 2022
@bassosimone
[backport] chore: bump web_connectivity@v0.5 version to 0.5.5 (#945)
4f4e5a0 
chore: web_connectivity v0.5.5

We're bumping the version number to reflect recent improvements in the
data format implemented in these pull requests:

- #942

- #943

- #944

Reference issue: ooni/probe#2238
This was referenced Sep 8, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hellais hellais Awaiting requested review from hellais hellais is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@bassosimone