Collaborative Launch: 1730210400 (Tuesday, October 29th, 14:00 GMT)

[jurraca]

Next collaborative launch is planned for 1730210400.

./run map -w=1730210400 -irr -rv

The latest kartograf release is here. The requirements are Python 3.11 and rpki-client 9.3. We recommend you use Nix and run nix develop to enter an environment with the most recent dependency versions, including rpki-client 9.3.

This release will also generate a debug.log file at out/1730210400/debug.log to log rpki-client run info. This will be useful to understand discrepancies or RPKI failures if they occur.

[jurraca]

Happening two hours from now!

You'll see a warning that the RPKI version hasn't been tested passed 9.1 -- you can safely ignore and run the 9.3 version. Will get that fixed shortly.

[dunxen]

My result:

The SHA-256 hash of the result file is: 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce

Logs

./run map -w=1730210400 -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.8
Using rpki-client version 9.3 (recommended).
Coordinated launch mode: Waiting until 1730210400 (2024-10-29 16:00:00 SAST) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1730210400 (2024-10-29 14:00:00 UTC, local: 2024-10-29 16:00:00 SAST)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35
Downloaded TAL for APNIC to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94
Downloaded TAL for ARIN to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48
Downloaded TAL for LACNIC to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a
Downloaded TAL for RIPE to /home/dunxen/repos/fjahr/kartograf/data/1730210400/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 6cd35224cff561814f67b66a336b1a1c0d96d5a84882565f8da1a39cf72a36e8
...finished in 0:02:47.040959

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloaded afrinic.db.gz, file hash: 06b38332b7a943102f28183a9f9d228eecc979f610eb0d584d42328f2c8b3506
Downloading apnic.db.route.gz
Downloaded apnic.db.route.gz, file hash: c89fc97d30042e09bf52076702c1ce168246f47419859126955b2b3aec600f19
Downloading apnic.db.route6.gz
Downloaded apnic.db.route6.gz, file hash: e29b02a507a94c6fa44dbee7e6d56ebe04799c18f4972e7deecd28ea62ba124f
Downloading arin.db.gz
Downloaded arin.db.gz, file hash: 08bb92d3b3cb6ca927674e149f96299d2af42a9068aebf6e8f879dee5fc4fa4d
Downloading lacnic.db.gz
Downloaded lacnic.db.gz, file hash: fc4146b26dddc5f4a2658fad86e75f663efb2909a191f58f91384a4ec9b4c90c
Downloading ripe.db.route.gz
Downloaded ripe.db.route.gz, file hash: 7e51fb885feba90181387eb71e3f80e377a54dd63ac07d8d1378b0342383d258
Downloading ripe.db.route6.gz
Downloaded ripe.db.route6.gz, file hash: 13e2665d306358db6f53d3ea0f483a83f371d027803a01a3af7c7c99351f7c48
...finished in 0:01:00.416695

--- Fetching Routeviews pfx2as ---

Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/10/routeviews-rv2-20241027-1200.pfx2as.gz
Downloaded /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: de48c5f0e488be2f32dbf3af8db23f168790b3cf3cdb90dd8ca6ac3e72ba4b7a
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/10/routeviews-rv6-20241028-1200.pfx2as.gz
Downloaded /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: c1d03e9d306d66d6a1c3fae374f7077b6f94e0fa09483452a6c0abe6fa031189
...finished in 0:00:12.936680

--- Validating RPKI ---

Validating RPKI ROAs
326288 raw RKPI ROA files found.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 326288/326288 [08:33<00:00, 635.36it/s]
326288 RKPI ROAs validated and saved to /home/dunxen/repos/fjahr/kartograf/out/1730210400/rpki/rpki_raw.json, file hash: 18472ec6170db5e05dbaa407ae889398a20d629c1190b2f77eef0ce68bdafbf9
...finished in 0:09:08.937212

--- Parsing RPKI ---

Parsing 326288 ROAs
Result entries written: 557068
Duplicates found: 70434
Invalids found: 57827
Incompletes: 0
Non-ROA files: 0
...finished in 0:01:10.319102

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/afrinic.db
Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/apnic.db.route
Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/apnic.db.route6
Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/arin.db
Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/lacnic.db
Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/ripe.db.route
Parsing /home/dunxen/repos/fjahr/kartograf/out/1730210400/irr/ripe.db.route6
Found valid, unique entries: 1813212
...finished in 0:04:35.926408

--- Merging RPKI and IRR data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113326 /   113326 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113325 /   113325 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113325 /   113325 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113325 /   113325 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   113325 /   113325 |

Finished filtering! Originally 1813212 entries filtered down to 377073
Merging base file with filtered extra file
...finished in 2:01:31.230769

--- Parsing Routeviews pfx2as ---

Unzipping /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/dunxen/repos/fjahr/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/dunxen/repos/fjahr/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/dunxen/repos/fjahr/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/dunxen/repos/fjahr/kartograf/out/1730210400/collectors/pfx2asn.txt
...finished in 0:02:09.012038

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            

 100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |                                                                                                                                                                                                                                                                                                                                                              100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |                                                                                                                                                                                                                                                                                                                                                              100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |                                                                                                                                                                                                                                                                                                                                                              100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |                                                                                                                                                                                                                                                                                                                                                              100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |                                                                                                                                                                                                                                                                                                                                                              100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77568 /    77568 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77567 /    77567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77567 /    77567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77567 /    77567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77567 /    77567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77567 /    77567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77567 /    77567 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |    77567 /    77567 |

Finished filtering! Originally 1241081 entries filtered down to 384523
Merging base file with filtered extra file
...finished in 2:32:12.673296

--- Sorting results ---

...finished in 0:00:12.763655

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce
Total runtime: 4:55:02.716502

[fjahr]

I have 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce.

(You can ignore the time tracking in my run, I did the download part in one location and then finished the processing in another.

Logs

$ ./run map -w=1730210400 -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.8
Using rpki-client version 9.3 (recommended).
Coordinated launch mode: Waiting until 1730210400 (2024-10-29 15:00:00 CET) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1730210400 (2024-10-29 14:00:00 UTC, local: 2024-10-29 15:00:00 CET)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35
Downloaded TAL for APNIC to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94
Downloaded TAL for ARIN to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48
Downloaded TAL for LACNIC to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a
Downloaded TAL for RIPE to /Users/FJ/projects/python/kartograf/data/1730210400/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb
Downloading RPKI Data, this may take a while.
Downloaded RPKI Data, hash sum: 457eb54bc20072ec302f5fa5b7f61bba11c6842f7f3ec6d8c28b8eabeab22f8e
...finished in 0:07:30.840128

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloaded afrinic.db.gz, file hash: 06b38332b7a943102f28183a9f9d228eecc979f610eb0d584d42328f2c8b3506
Downloading apnic.db.route.gz
Downloaded apnic.db.route.gz, file hash: c89fc97d30042e09bf52076702c1ce168246f47419859126955b2b3aec600f19
Downloading apnic.db.route6.gz
Downloaded apnic.db.route6.gz, file hash: e29b02a507a94c6fa44dbee7e6d56ebe04799c18f4972e7deecd28ea62ba124f
Downloading arin.db.gz
Downloaded arin.db.gz, file hash: 08bb92d3b3cb6ca927674e149f96299d2af42a9068aebf6e8f879dee5fc4fa4d
Downloading lacnic.db.gz
Downloaded lacnic.db.gz, file hash: fc4146b26dddc5f4a2658fad86e75f663efb2909a191f58f91384a4ec9b4c90c
Downloading ripe.db.route.gz
Downloaded ripe.db.route.gz, file hash: 7e51fb885feba90181387eb71e3f80e377a54dd63ac07d8d1378b0342383d258
Downloading ripe.db.route6.gz
Downloaded ripe.db.route6.gz, file hash: 13e2665d306358db6f53d3ea0f483a83f371d027803a01a3af7c7c99351f7c48
...finished in 0:01:08.242644

--- Fetching Routeviews pfx2as ---

Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/10/routeviews-rv2-20241027-1200.pfx2as.gz
Downloaded /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: de48c5f0e488be2f32dbf3af8db23f168790b3cf3cdb90dd8ca6ac3e72ba4b7a
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/10/routeviews-rv6-20241028-1200.pfx2as.gz
Downloaded /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: c1d03e9d306d66d6a1c3fae374f7077b6f94e0fa09483452a6c0abe6fa031189
...finished in 0:00:07.301552

--- Validating RPKI ---

Validating RPKI ROAs
326295 raw RKPI ROA files found.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 326295/326295 [3:23:04<00:00, 26.78it/s]
326295 RKPI ROAs validated and saved to /Users/FJ/projects/python/kartograf/out/1730210400/rpki/rpki_raw.json, file hash: 190fa2cc1b5bdf3960c0ca450a76cdf39bc7d4a1def9932d0875cb73f617a0e7
...finished in 0:39:54.935860

--- Parsing RPKI ---

Parsing 326295 ROAs
Result entries written: 557068
Duplicates found: 70434
Invalids found: 57834
Incompletes: 0
Non-ROA files: 0
...finished in 0:01:05.939468

--- Parsing IRR ---

Extracting afrinic.db.gz
Extracting apnic.db.route.gz
Extracting apnic.db.route6.gz
Extracting arin.db.gz
Extracting lacnic.db.gz
Extracting ripe.db.route.gz
Extracting ripe.db.route6.gz
Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/lacnic.db
Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/ripe.db.route
Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/apnic.db.route6
Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/apnic.db.route
Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/arin.db
Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/afrinic.db
Parsing /Users/FJ/projects/python/kartograf/out/1730210400/irr/ripe.db.route6
Found valid, unique entries: 1813212
...finished in 0:03:55.230642

--- Merging RPKI and IRR data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   181322 /   181322 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181322 /   181322 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   181321 /   181321 |

Finished filtering! Originally 1813212 entries filtered down to 377073
Merging base file with filtered extra file
...finished in 1:29:00.817305

--- Parsing Routeviews pfx2as ---

Unzipping /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /Users/FJ/projects/python/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /Users/FJ/projects/python/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /Users/FJ/projects/python/kartograf/out/1730210400/collectors/pfx2asn.txt
...finished in 0:02:01.592040

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   124109 /   124109 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   124108 /   124108 |

Finished filtering! Originally 1241081 entries filtered down to 384523
Merging base file with filtered extra file
...finished in 1:40:25.020590

--- Sorting results ---

...finished in 0:00:16.142230

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce
Total runtime: 6:49:01.317201

[jurraca]

I got 37c70f403b65f9d7df887f73f0417fe401e8d0489b9c807f18351d190259d219

I seem to have more IRR and RouteViews data to parse than both of you.

Logs

--- Start Kartograf ---                                                                                                                                                                                             
                                                                                                          
Kartograf version: 0.4.8 
Using rpki-client version 9.3 (recommended).                                                                                                                                                                                     
Coordinated launch mode: Waiting until 1730210400 (2024-10-29 14:00:00 WET) to launch mapping process.                                                                                                              
Countdown: 0 second(s)                                                                                                                                                                                              
Starting...                                                                                               
The epoch for this run is: 1730210400 (2024-10-29 14:00:00 UTC, local: 2024-10-29 14:00:00 WET)                                                                                                                     
                                                                                                                                                                                                                    
--- Fetching RPKI ---                                                                                                                                                                                               
                                                                                                                                                                                                                    
Downloaded TAL for AFRINIC to /home/base/code/kartograf/data/1730210400/rpki/tals/afrinic.tal, file hash: 2838ef30ea27ce5705abf5f5adb131d8c35b1f50858338a2f3c84bb207c2fa35                                          
Downloaded TAL for APNIC to /home/base/code/kartograf/data/1730210400/rpki/tals/apnic.tal, file hash: 472e551f7c551c2e999e582b7c9437d3bee4900fe53afff62aeb28d4940ade94                                              
Downloaded TAL for ARIN to /home/base/code/kartograf/data/1730210400/rpki/tals/arin.tal, file hash: 4f6c1e456fe5ab468beac1495e57d99a1eeeaa4d9f9e34519eaf58857c21af48
Downloaded TAL for LACNIC to /home/base/code/kartograf/data/1730210400/rpki/tals/lacnic.tal, file hash: d44bb9394ab009c8b53e5efebf2a1c9450bab61a27efe00de5a3e4587a3a2f6a
Downloaded TAL for RIPE to /home/base/code/kartograf/data/1730210400/rpki/tals/ripe.tal, file hash: 59ca27ef93f23682749fcefe7c6d70fbc723343549ff9e4d3996acaff79817fb
Downloading RPKI Data, this may take a while.                                                             
Downloaded RPKI Data, hash sum: aa4e25b4207e7e9dc63c3bdc52ee21e8ddedf609adb03bf418ef8fffc2819b83                                                                                                                    
...finished in 0:06:15.300586                                                                             
                                                                                                                                                                                                                    
--- Fetching IRR ---                                                                                      
                                                                                                                                                                                                                    Downloading afrinic.db.gz                                                                                                                                                                                           Downloaded afrinic.db.gz, file hash: 06b38332b7a943102f28183a9f9d228eecc979f610eb0d584d42328f2c8b3506                                                                                                               
Downloading apnic.db.route.gz                                                                                                                                                                                       
Downloaded apnic.db.route.gz, file hash: c89fc97d30042e09bf52076702c1ce168246f47419859126955b2b3aec600f19                                                                                                           
Downloading apnic.db.route6.gz                                                                                                                                                                                      
Downloaded apnic.db.route6.gz, file hash: e29b02a507a94c6fa44dbee7e6d56ebe04799c18f4972e7deecd28ea62ba124f                                                                                                          Downloading arin.db.gz                                                                                                                                                                                              Downloaded arin.db.gz, file hash: 08bb92d3b3cb6ca927674e149f96299d2af42a9068aebf6e8f879dee5fc4fa4d                                                                                                                  
Downloading lacnic.db.gz                                                                                                                                                                                            
Downloaded lacnic.db.gz, file hash: fc4146b26dddc5f4a2658fad86e75f663efb2909a191f58f91384a4ec9b4c90c                                                                                                                
Downloading ripe.db.route.gz                                                                              
Downloaded ripe.db.route.gz, file hash: 7e51fb885feba90181387eb71e3f80e377a54dd63ac07d8d1378b0342383d258                                                                                                            
Downloading ripe.db.route6.gz                                                                             
Downloaded ripe.db.route6.gz, file hash: 13e2665d306358db6f53d3ea0f483a83f371d027803a01a3af7c7c99351f7c48                                                                                                           
...finished in 0:04:22.929134                                                                             
                                                                                                          
--- Fetching Routeviews pfx2as ---                                                                        
                                                                                                                                                                                                                    
Downloading from https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2024/10/routeviews-rv2-20241027-1200.pfx2as.gz
Downloaded /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz, file hash: de48c5f0e488be2f32dbf3af8db23f168790b3cf3cdb90dd8ca6ac3e72ba4b7a                                          
Downloading from https://publicdata.caida.org/datasets/routing/routeviews6-prefix2as/2024/10/routeviews-rv6-20241028-1200.pfx2as.gz                                                                                 
Downloaded /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz, file hash: c1d03e9d306d66d6a1c3fae374f7077b6f94e0fa09483452a6c0abe6fa031189                                          
...finished in 0:00:07.072789     
--- Validating RPKI ---                                                                                                                                                                                             
                                                                                                                                                                                                                    
Validating RPKI ROAs                                                                                                                                                                                                
326282 raw RKPI ROA files found.                                                                          
100%|██████████████████████████████████████████████████████████████| 326282/326282 [23:13<00:00, 234.22it/s]                                                                                                        
326282 RKPI ROAs validated and saved to /home/base/code/kartograf/out/1730210400/rpki/rpki_raw.json, file hash: 8a5cdcf52aca0d9ba07c56d482e1805ceec9cb32a4efa54e7d3ff46f2ef04e70
...finished in 0:23:32.685831                                                                             

--- Parsing RPKI ---                                                                                      

Parsing 326282 ROAs                                                                                                                                                                                                 
Result entries written: 412435                                                                                                                                                                                      
Duplicates found: 41473                                                                                                                                                                                             
Invalids found: 210561                                                                                                                                                                                              
Incompletes: 0                                                                                                                                                                                                      
Non-ROA files: 0                                                                                          
...finished in 0:01:19.068106                                                                             
                                                                                                          
--- Parsing IRR ---                                  
                                                                                                          
Extracting afrinic.db.gz                                                                                  
Extracting apnic.db.route.gz                                                                                                                                                                                        
Extracting apnic.db.route6.gz                                                                             
Extracting arin.db.gz                                                                                                                                                                                               Extracting lacnic.db.gz                                                                                                                                                                                             Extracting ripe.db.route.gz                                                                               
Extracting ripe.db.route6.gz                                                                                                                                                                                        
Parsing /home/base/code/kartograf/out/1730210400/irr/afrinic.db                                                                                                                                                     
Parsing /home/base/code/kartograf/out/1730210400/irr/arin.db                                                                                                                                                        
Parsing /home/base/code/kartograf/out/1730210400/irr/ripe.db.route6                                                                                                                                                 Parsing /home/base/code/kartograf/out/1730210400/irr/apnic.db.route6                                                                                                                                                Parsing /home/base/code/kartograf/out/1730210400/irr/lacnic.db                                                                                                                                                      
Parsing /home/base/code/kartograf/out/1730210400/irr/ripe.db.route                                                                                                                                                  
Parsing /home/base/code/kartograf/out/1730210400/irr/apnic.db.route                                                                                                                                                 
Found valid, unique entries: 1813212                                                                      
...finished in 0:06:55.376568                                                                             

--- Merging RPKI and IRR data ---                                                                         

Parse base file to numpy arrays                                                                           
Parse extra file to Pandas DataFrame                                                                      
Filtering extra prefixes that were already included in the base file:                                                                                                                                               

 100.00% :::::::::::::::::::::::::::::::::::::::: |   226652 /   226652 |                                                                                                                                           
 100.00% :::::::::::::::::::::::::::::::::::::::: |   226652 /   226652 |                                                                                                                                           
 100.00% :::::::::::::::::::::::::::::::::::::::: |   226652 /   226652 |                                                                                                                                           
 100.00% :::::::::::::::::::::::::::::::::::::::: |   226652 /   226652 |                    
 100.00% :::::::::::::::::::::::::::::::::::::::: |   226651 /   226651 |                                  
 100.00% :::::::::::::::::::::::::::::::::::::::: |   226651 /   226651 |                                  
 100.00% :::::::::::::::::::::::::::::::::::::::: |   226651 /   226651 |                                  
 100.00% :::::::::::::::::::::::::::::::::::::::: |   226651 /   226651 |                                  

Finished filtering! Originally 1813212 entries filtered down to 412660
Merging base file with filtered extra file
...finished in 3:08:55.204572

--- Parsing Routeviews pfx2as ---

Unzipping /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip4.txt.gz
Formatting /home/base/code/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip4.txt
Unzipping /home/base/code/kartograf/data/1730210400/collectors/routeviews_pfx2asn_ip6.txt.gz
Formatting /home/base/code/kartograf/out/1730210400/collectors/routeviews_pfx2asn_ip6.txt
Cleaning /home/base/code/kartograf/out/1730210400/collectors/pfx2asn.txt
...finished in 0:03:23.099779

--- Merging Routeviews and base data ---

Parse base file to numpy arrays
Parse extra file to Pandas DataFrame
Filtering extra prefixes that were already included in the base file:

 100.00% :::::::::::::::::::::::::::::::::::::::: |   155135 /   155135 |                                  
 100.00% :::::::::::::::::::::::::::::::::::::::: |   155135 /   155135 |                                  
 100.00% :::::::::::::::::::::::::::::::::::::::: |   155135 /   155135 |                                  
 100.00% :::::::::::::::::::::::::::::::::::::::: |   155135 /   155135 |                                  
 100.00% :::::::::::::::::::::::::::::::::::::::: |   155135 /   155135 |
 100.00% :::::::::::::::::::::::::::::::::::::::: |   155135 /   155135 | 
 100.00% :::::::::::::::::::::::::::::::::::::::: |   155135 /   155135 |                                  

Finished filtering! Originally 1241081 entries filtered down to 472611
Merging base file with filtered extra file
...finished in 4:36:25.700276

--- Sorting results ---

...finished in 0:00:19.232583

--- Finishing Kartograf ---

The SHA-256 hash of the result file is: 37c70f403b65f9d7df887f73f0417fe401e8d0489b9c807f18351d190259d219
Total runtime: 8:31:37.862229

[jurraca]

OK, a bit low on participants, so thank you @dunxen for showing up! Could I ask you to upload your zipped final_result.txt? Would like to figure out where I differed.

Working on performance improvements which should make this script significantly faster.

[dunxen]

Could I ask you to upload your zipped final_result.txt? Would like to figure out where I differed.

Sure thing!

Here's the compressed result:
1730210400-final_result.txt.gz

[0xB10C]

I got 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce.

[jurraca]

Nice, looks like 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce is the "winner".

Looking at this a bit more closely, the difference for me seems to be RPKI validation results. We all got about the same IRR data, the same RPKI files and almost the same number of entries, but I have a lot more invalids, which seems to account for the difference in the result files. I see a ton of CRL not yet valid from several RIRs in debug.log. Looking into it.

[Emzy]

I got:

8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce

[jurraca]

Thanks everyone for participating!

Using the output file with hash 8986e7baf0aea86b6b63766968d77edb384d1c332b23725596e18693235419ce, I've opened issue #19 with the compressed file which you can verify. Would be good to get at least two ACKs before we make it the latest_asmap we point others to.

[fjahr]

Closing per #19 , thanks everyone for participating again ❤️