Collaborative Launch 1706191200 (2024-01-25 14:00 GMT, Kartograf v0.4.2)

[fjahr]

Next collaborative launch is planned for 1706191200. Please make sure to upgrade to Kartograf v0.4.2 which was released today (https://github.com/fjahr/kartograf/releases/tag/0.4.2).

./run map -w=1706191200 -irr -rv

#4 has been a success but a lot of issues were fixed along the way making it hard to parse and grasp the actual process. I hope this one can the first clean run that concisely shows what the process could look like when being run routinely.

Thanks for your participation in advance! Upon completion please post your hash in the comments. When we have majority for one hash, we will open a PR, similar to #6.

[Sjors]

Engines ready! T - 4 hours...

[brunoerg]

I will join! Ready here!

[0xB10C]

Sadly, my run failed. Same error as @virtu reported in #8.

[Sjors]

The SHA-256 hash of the result file is: 30048a44cf31116a841dffee4a6760b624fcde5b20ba6679f1ef6ce6c31d9d3b
Total runtime: 2:39:37.270724

[fjahr]

Not a match with @Sjors unfortunately:

The SHA-256 hash of the result file is: 3910796c4a706de88043646bed3de62e5a87482ee7c58b8e312ae29fc09e5638
Total runtime: 3:11:21.409145

[brunoerg]

The SHA-256 hash of the result file is: 6c850c482428d19e6c5b78f8882699b824ab1ad14432816f71e1f1a9f3d50490
Total runtime: 2:30:40.648287

Not a match with @Sjors or @fjahr

[Emzy]

The SHA-256 hash of the result file is: 30048a44cf31116a841dffee4a6760b624fcde5b20ba6679f1ef6ce6c31d9d3b
Total runtime: 15:38:24.217769

Got a match with @Sjors

[0xB10C]

I'm assuming, but I don't know, that the different hashes stem from different input data. E.g. the data was updated on the server between participants downloaded it. A few thoughts came to mind:

1. The data might be updated with a cronjob (or similar). Humans like round numbers, so these cron jobs might run hourly (or every 30 min, or every 10 min, or every 5 min - I'm not sure what update frequency makes sense for them). Starting a collaborative launch at exactly the full hour gives us a high chance of some participants downloading data before and some after the cronjob has finished.
2. The downloads run in sequence. Participants with a slower connection might ask for the last to-be-downloaded file later than a faster connection. I think, it might make sense to experiment with starting the downloads in fetch_irr() and in download_rir_tals() in parallel. This assumes the download would still continue even if the file is changed on the server. I don't know if that's the case.
3. To figure out if the issue stems from different input data, it might be good to implement Data hashes in logs asmap/kartograf#2.

[0xB10C]

These are the timestamps that the FTP servers provide for the IRR files (code):

2024-01-27   00:05                      afrinic.db.gz
2024-01-26   15:01:07	                apnic.db.route6.gz
2024-01-26   15:01:15			apnic.db.route.gz
2024-01-26   14:00:07			arin.db.gz
2024-01-26   23:08:37			ripe.db.route6.gz
2024-01-26   23:08:37			ripe.db.route.gz
2024-01-27   07:35                      lacnic.db.gz

It doesn't seem like they are updated too often. There might have been updates around the time of this collaborative launch (not sure what timezone these timestamps are).

Edit: lacnic.db.gz seems to be updated every 5 minutes and seems to be in GMT-3 (which makes sense as their in Uruguay). When they update, I get an EOFError.

[fjahr]

@0xB10C Yes, the different hashes should come from different inputs unless we have found another bug :) The data that is fluctuating the most by far is RPKI, because there we download the data directly from many different decentralized repositories that can change their data every second. This is the reason why I was not expecting collaborative launch to work as well as it did the last couple of times and why a result like today should be expected sometimes.

The RPKI updates are usually manually triggered in very small increments so I don't think the start time matters there. I think it's possible that the IRR DB FTP download is updated with a cron job regularly but there is no public information on this and from my experience with the space they probably don't have a very strict or transparent policy on this. I will see if I know someone who has an idea. But if they are triggered every full hour for example, that should work perfectly well with the current implementation because RPKI is downloaded first, taking a few minutes on a fast connection. Then the IRR file is downloaded which should be updated at that point.

Running the downloads in sequence is by design currently because the RPKI data is the most volatile by far from my experience. So far every time I have analyzed the results of failed collaborative launch matches, the source was RPKI (aside from bugs). So I prioritize that everyone is getting the RPKI data as fast as possible to minimize the risk there. I believe starting the IRR downloads in parallel would lead to slightly worse results due to higher likelihood of differing RPKI data.

From what I have seen on these IRR download failures I would really put my money on some quirky issue between the ftp server and the libs that are used in the nix env because:

1. I have downloaded these IRR files over ftp probably about 300 times in the last couple of months. I have never seen this issue and no other tester has reported such an issue until this week. Now two nix users see it happening very frequently while the non-nix users don't see it. The only thing irritating about that is that you also haven't seen such an error in earlier runs, right?
2. I am not an expert on this but I did some research and usually the download shouldn't break even if the file is updated because during the download the download file is cached in memory on the server side. Some FTP implementations may not do this but I would be surprised if the RIRs would not use the most robust solution possible.

But yes, I will implement printing the hashes now, it just saved us a lot of guess-work/manual hashing when we fail :)

[fjahr]

Hm, ok, your lacnic finding that i didn't see before writing seem to indicate that it might be interesting to look into parallel download. I am still very surprised then that I didn't see lacnic fail every. I will try to reproduce this. Maybe the nix env uses a lib where the download breaks while others don't have the same behavior.

[Emzy]

2. I am not an expert on this but I did some research and usually the download shouldn't break even if the file is updated because during the download the download file is cached in memory on the server side. Some FTP implementations may not do this but I would be surprised if the RIRs would not use the most robust solution possible.

At an Unix like (Linux) OS, if the file is replaced the old file will stay in place for the process that opened it and gets for real erased after the file handle is closed. So the ftp server should not be interrupted by it.