WDC migration and re-numbering

[darkk]

GH / WDC migrates to Miami.
Hardware will be physically moved, IP addresses will be re-numbered.
So, everything that is there, should be killed, frozen or moved.
This migration will happen in the week 26 Nov ... 02 Dec.

• munin.ooni.io
• shwdc.ooni.io
• shark.ooni.nu
• a.echo.th.ooni.io
• c.web-connectivity.th.ooni.io
• b.collector.ooni.io
• analytics.ooni.io

[darkk]

• munin — gone, disk image will be removed in a while, Tor TH will be re-deployed preserving keys
• shwdc — gone, replaced with another bastion host, disk image is deleted as it had no files in home directories

[darkk]

• shark.ooni.nu — VM deleted, but disk image is kept for a while. I believe, this VM was used for one-off testing and it has not seen any shell logins since Oct 16, so I'm unsure if allocated resources are still needed. @hellais, please tell, if you still need the data stored there.

[darkk]

a.echo.th.ooni.io

It was unclear to me if it gets any load (as it's not mentioned at bouncer config).

It does. E.g. 2018-09-01 bucket mentions it once, 2018-11-20 also once, both cases mention it by FQDN, not by IP.

tcp-echo test helper 37.218.247.110 from bouncer config get a bit more hits -- 360 in 2018-11-20 bucket, 800 in 2018-09-01

Given uptime of 300 days and following stats from DOCKER iptables chain, I belive that dport:80 is the only service that actualy gets some load for that FQDN. Also, the IP address had no other DNS names pointing to it in ooni zones but a.echo.th.ooni.io.

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    7   339 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           tcp dpt:57005
    0     0 ACCEPT     udp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           udp dpt:57004
    6   299 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           tcp dpt:57003
   45  2535 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           tcp dpt:57001
1156K   63M ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           tcp dpt:80

I believe, the way to handle that host is to switch a.echo.th.ooni.io/A DNS record to another test-helper doing echo at tcp/80 and just kill that VM at this moment (keeping disk for some time).
Also, the VM has 3 GiB of RAM, but uses only ~300 MiB.

So, I'm pointing a.echo.th.ooni.io to 37.218.247.110 (aka c.echo.th.ooni.io) from bouncer config.

[darkk]

analytics.ooni.io

I believe, it can go down for a week and come back online after migration. Moreover, it uses only 1.2 GB of RAM (470 MB without disk cache) out of 6 GB allocated. The IP will be changed from 199.119.112.42 to 37.218.241.42.

[darkk]

c.web-connectivity.th.ooni.io & b.collector.ooni.io

I decided to avoid migration of these services to another DC as I'm not 100% sure, what can go wrong. So I just disabled corresponding collector and test-helper at the bouncer and will re-enable when VMs are renumbered.

[darkk]

b.collector.ooni.io is hardcoded, so bouncer reply is probably not respected (anyway, traffic did not go away after bouncer configuration change) -- https://github.com/measurement-kit/measurement-kit/blob/v0.9.0-beta/src/libmeasurement_kit/ooni/collector_client.hpp#L18-L19
Seems, endpoint should be migrated somewhere.

Somewhere is 103.104.244.110 (aka a.collector.ooni.io), nginx frontend is a convenience.

[darkk]

UPD from @bassosimone: the collector is indeed hardcoded, but the hardcoded value is used as a fallback when bouncer does not reply. It explains both the existence of traffic and very low amount of measurements hitting b.collector.ooni.io endpoint.

[darkk]

Unfortunately c.web-connectivity.th.ooni.io still gets some traffic to y3zq5fwelrzkkv3s.onion. The source is unclear.

[hellais]

shark.ooni.nu — VM deleted, but disk image is kept for a while. I believe, this VM was used for one-off testing and it has not seen any shell logins since Oct 16, so I'm unsure if allocated resources are still needed. @hellais, please tell, if you still need the data stored there

This is the machine I am using for building probe-cli on linux. The disk image has several build artifacts that it would be useful to preserve (so I don't have to rebuild them) and it would be good to re-instate it at the new location.

[darkk]

it would be good to re-instate it at the new location

Do you need computing power or just the build artifacts?

[hellais]

Do you need computing power or just the build artifacts?

Both. I am ok keeping it in stopped as I don't use it on a daily basis, but I do need access to a linux based host to build probe-cli.

[darkk]

The VMs went down half an hour ago (at ~14:00 UTC).
I've moved y3zq5fwelrzkkv3s.onion frontend to b.web-connectivity.th.ooni.io, seems, it's working.
I hope, that's enough for the clients that had that onion address hardcoded to survive.

[darkk]

FTR, number of test-helpers from 2018-11-20 bucket:

$ ( for f in *.tar.lz4; do tar -I lz4 -xf - --to-stdout < $f; done; for f in *.json.lz4; do lz4cat <$f; done ) | jq -c .test_helpers | sort | uniq -c
  55748 {}
    360 {"backend":"37.218.247.110"}
    387 {"backend":"http://37.218.247.95:80"}
      1 {"backend":"http://a.echo.th.ooni.io"}
     20 {"backend":{"type":"https","address":"https://b.web-connectivity.th.ooni.io"}}
   9268 {"backend":{"type":"https","address":"https://b.web-connectivity.th.ooni.io:443"}}
   9767 {"backend":{"type":"https","address":"https://c.web-connectivity.th.ooni.io:443"}}
 235919 {"backend":{"type":"onion","address":"httpo://y3zq5fwelrzkkv3s.onion"}}
      1 null