Bouncer giving out down test-helpers for ~16 hours

[hellais]

As part of #356 the echo and http test helpers were redeployed to new hosts (mia-echoth.ooni.nu & mia-httpth.ooni.nu) however the bouncer was not updated to reflect these changes.

The old hosts which had a different IP (37.218.247.110 for echo & 37.218.247.95 for http) were shutdown at around 19:00 UTC+2.

Detection: alert from user

Timeline:
~17:00 UTC 24-10-2019 the old test helper hosts are shutdown
03:00 UTC 25-10-2019 a user informs us via twitter of this issue
09:00 UTC 25-10-2019 the bouncer is updated with the IPs of the new services

What went well:

• Users detected this early

What went wrong:

• We did not have alerting takes into account the values returned by the bouncer

What we should do to prevent it happening the future:

• Configure monitoring to use the values returned by the bouncer to check uptime of services

[darkk]

Another option is to take RUM route: count volume or percentage of erratic measurements coming to the collector.

[hellais]

So it turned out that actually the httpth was also wrongly configured on the bouncer and we were giving out an address of a down service for that until ~21:00 2019-10-26 UTC.

[hellais]

For people accessing this incident issue from the internet, the incident has been fully resolved since Saturday evening.

The incident affects measurements from the following time period: 2019-10-24 16:00 UTC - 2019-10-26 23:00 UTC.

We keep the issue ticket open until we have taken all the steps we consider necessary to prevent it re-occurring in the future.

[hellais]

The way to check if the measurement you are looking at is a false positive is:

1. Look at the test_keys->failure key and seeing if it says connection_refused (it should not say this)
2. Checking if the test_helpers->backend says something other than http://37.218.247.94:80 or 37.218.247.93

[hellais]

ooni/backend#343