Update to Route stats & stacked area plot vs im/export

[SichangHe]

Following #72 after change in special cases.

2.4GiB. Took 3200133ms (53min) on feddev and 22550119ms (376min) on timbersaw.

In [6]: df
Out[6]:
          import_ok  export_ok  import_skip  export_skip  import_unrec  export_unrec  import_meh  export_meh  import_err  export_err  skip_regex_tilde  skip_regex_with_set  skip_community  unrec_import_empty  unrec_export_empty  unrec_filter_set  unrec_as_routes  unrec_route_set  unrec_as_set  unrec_as_set_route  unrec_some_as_set_route  unrec_aut_num  unrec_peering_set  spec_uphill  spec_uphill_tier1  spec_tier1_pair  spec_import_peer_oifps  spec_import_customer_oifps  spec_export_customers  spec_import_from_neighbor  spec_as_is_origin_but_no_route  spec_as_set_contains_origin_but_no_route  err_filter  err_filter_as_num  err_filter_as_set  err_filter_prefixes  err_filter_route_set  err_remote_as_num  err_remote_as_set  err_except_peering_right  err_peering  err_regex  rpsl_as_name  rpsl_filter  rpsl_regex  rpsl_unknown_filter  recursion
0                 1          1            0            0             0             0           0           0           0           0                 0                    0               0                   0                   0                 0                0                0             0                   0                        0              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0
1                 0          0            0            0             1             1           0           0           0           0                 0                    0               0                   0                   1                 0                0                0             0                   0                        1              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0
2                 0          0            0            0             1             1           0           0           0           0                 0                    0               0                   0                   1                 0                0                0             0                   0                        1              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0
3                 0          0            0            0             0             1           1           0           0           0                 0                    0               0                   0                   1                 0                0                0             0                   0                        0              0                  0            0                  0                0                       1                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0
4                 0          0            0            0             1             1           0           0           0           0                 0                    0               0                   1                   1                 0                0                0             0                   0                        0              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0
...             ...        ...          ...          ...           ...           ...         ...         ...         ...         ...               ...                  ...             ...                 ...                 ...               ...              ...              ...           ...                 ...                      ...            ...                ...          ...                ...              ...                     ...                         ...                    ...                        ...                             ...                                       ...         ...                ...                ...                  ...                   ...                ...                ...                       ...          ...        ...           ...          ...         ...                  ...        ...
26447595          0          0            0            0             1             2           0           0           1           0                 0                    0               0                   1                   2                 0                0                0             0                   0                        0              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                120                  0                         0          120          0             0            0           0                    0          0
26447596          1          0            0            0             2             4           0           0           1           0                 0                    0               0                   2                   3                 0                0                0             0                   0                        1              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  4                  0                         0            4          0             0            0           0                    0          0
26447597          1          0            0            0             2             3           0           0           0           0                 0                    0               0                   2                   3                 0                0                0             0                   0                        0              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0
26447598          0          0            0            0             4             4           0           0           0           0                 0                    0               0                   2                   3                 0                0                0             0                   3                        0              2                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0
26447599          1          0            0            0             2             3           0           0           0           0                 0                    0               0                   2                   3                 0                0                0             0                   0                        0              0                  0            0                  0                0                       0                           0                      0                          0                               0                                         0           0                  0                  0                    0                     0                  0                  0                         0            0          0             0            0           0                    0          0

[26447600 rows x 47 columns]

In [7]: desc = df.describe()

In [8]: desc
Out[8]:
          import_ok     export_ok   import_skip   export_skip  import_unrec  export_unrec    import_meh    export_meh    import_err    export_err  skip_regex_tilde  skip_regex_with_set  skip_community  unrec_import_empty  unrec_export_empty  unrec_filter_set  unrec_as_routes  unrec_route_set  unrec_as_set  unrec_as_set_route  unrec_some_as_set_route  unrec_aut_num  unrec_peering_set   spec_uphill  spec_uphill_tier1  spec_tier1_pair  spec_import_peer_oifps  spec_import_customer_oifps  spec_export_customers  spec_import_from_neighbor  spec_as_is_origin_but_no_route  spec_as_set_contains_origin_but_no_route    err_filter  err_filter_as_num  err_filter_as_set  err_filter_prefixes  err_filter_route_set  err_remote_as_num  err_remote_as_set  err_except_peering_right   err_peering     err_regex  rpsl_as_name  rpsl_filter  rpsl_regex  rpsl_unknown_filter   recursion
count  2.644760e+07  2.644760e+07  2.644760e+07  2.644760e+07  2.644760e+07  2.644760e+07  2.644760e+07  2.644760e+07  2.644760e+07  2.644760e+07      2.644760e+07         2.644760e+07    2.644760e+07        2.644760e+07        2.644760e+07      2.644760e+07     2.644760e+07     2.644760e+07  2.644760e+07        2.644760e+07             2.644760e+07   2.644760e+07       2.644760e+07  2.644760e+07       2.644760e+07     2.644760e+07            2.644760e+07                2.644760e+07           2.644760e+07               2.644760e+07                    2.644760e+07                              2.644760e+07  2.644760e+07       2.644760e+07       2.644760e+07         2.644760e+07          2.644760e+07       2.644760e+07       2.644760e+07                26447600.0  2.644760e+07  2.644760e+07  2.644760e+07   26447600.0  26447600.0         2.644760e+07  26447600.0
mean   9.460705e-01  5.450561e-01  4.499463e-05  5.969918e-04  1.099601e+00  1.524752e+00  5.446093e-01  5.804130e-01  3.146360e-01  2.544874e-01      3.694097e-05         6.919342e-06    1.148497e-03        5.879887e-01        7.067451e-01      1.928341e-06     1.919758e-03     1.606416e-01  2.993335e+00        8.265607e+02             4.962050e+02   6.232728e-01       5.183457e-04  4.756757e-01       3.257567e-01     1.092092e-01            1.500892e-02                1.237541e-04           1.716322e-02               7.100750e-02                    1.095841e-01                              5.669660e-02  1.517673e-02       2.662018e-03       1.188161e-02         1.308247e-05          5.082125e-04       1.247648e+03       1.712681e-02                       0.0  1.247423e+03  9.157731e-05  1.026558e-04          0.0         0.0         1.058697e-06         0.0
std    8.687449e-01  8.451503e-01  6.707653e-03  2.538112e-02  1.002192e+00  1.128988e+00  6.697928e-01  7.306020e-01  4.774113e-01  4.447039e-01      6.077796e-03         2.630455e-03    9.218714e-02        7.472118e-01        8.549986e-01      1.750045e-03     4.858963e-02     7.936015e+00  3.825467e+01        2.372436e+03             1.383372e+03   1.124857e+00       3.619264e-02  8.403122e-01       5.378626e-01     3.168373e-01            1.215965e-01                1.112380e-02           1.668009e-01               4.407332e-01                    4.826153e-01                              4.277722e-01  1.816781e-01       1.219198e-01       1.410812e-01         3.994485e-03          3.493396e-02       2.697730e+03       2.425601e-01                       0.0  2.697789e+03  9.600724e-03  1.517045e-02          0.0         0.0         1.028930e-03         0.0
min    0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00      0.000000e+00         0.000000e+00    0.000000e+00        0.000000e+00        0.000000e+00      0.000000e+00     0.000000e+00     0.000000e+00  0.000000e+00       -3.259000e+04             0.000000e+00   0.000000e+00       0.000000e+00  0.000000e+00       0.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00  0.000000e+00       0.000000e+00       0.000000e+00         0.000000e+00          0.000000e+00       0.000000e+00       0.000000e+00                       0.0  0.000000e+00  0.000000e+00  0.000000e+00          0.0         0.0         0.000000e+00         0.0
25%    0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  1.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00      0.000000e+00         0.000000e+00    0.000000e+00        0.000000e+00        0.000000e+00      0.000000e+00     0.000000e+00     0.000000e+00  0.000000e+00        0.000000e+00             0.000000e+00   0.000000e+00       0.000000e+00  0.000000e+00       0.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00  0.000000e+00       0.000000e+00       0.000000e+00         0.000000e+00          0.000000e+00       0.000000e+00       0.000000e+00                       0.0  0.000000e+00  0.000000e+00  0.000000e+00          0.0         0.0         0.000000e+00         0.0
50%    1.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  1.000000e+00  1.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00      0.000000e+00         0.000000e+00    0.000000e+00        0.000000e+00        0.000000e+00      0.000000e+00     0.000000e+00     0.000000e+00  0.000000e+00        0.000000e+00             0.000000e+00   0.000000e+00       0.000000e+00  0.000000e+00       0.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00  0.000000e+00       0.000000e+00       0.000000e+00         0.000000e+00          0.000000e+00       0.000000e+00       0.000000e+00                       0.0  0.000000e+00  0.000000e+00  0.000000e+00          0.0         0.0         0.000000e+00         0.0
75%    1.000000e+00  1.000000e+00  0.000000e+00  0.000000e+00  2.000000e+00  2.000000e+00  1.000000e+00  1.000000e+00  1.000000e+00  1.000000e+00      0.000000e+00         0.000000e+00    0.000000e+00        1.000000e+00        1.000000e+00      0.000000e+00     0.000000e+00     0.000000e+00  0.000000e+00        0.000000e+00             1.000000e+00   1.000000e+00       0.000000e+00  1.000000e+00       1.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00  0.000000e+00       0.000000e+00       0.000000e+00         0.000000e+00          0.000000e+00       1.710000e+02       0.000000e+00                       0.0  1.610000e+02  0.000000e+00  0.000000e+00          0.0         0.0         0.000000e+00         0.0
max    8.000000e+00  7.000000e+00  1.000000e+00  3.000000e+00  1.100000e+01  1.100000e+01  6.000000e+00  6.000000e+00  5.000000e+00  5.000000e+00      1.000000e+00         1.000000e+00    3.200000e+01        8.000000e+00        8.000000e+00      3.000000e+00     3.000000e+00     8.430000e+02  7.314000e+03        2.308000e+04             2.787000e+04   1.800000e+01       6.000000e+00  9.000000e+00       2.000000e+00     2.000000e+00            2.000000e+00                1.000000e+00           1.700000e+01               2.400000e+01                    3.700000e+01                              7.200000e+01  1.600000e+01       1.600000e+01       7.000000e+00         2.000000e+00          3.000000e+00       1.277900e+04       2.800000e+01                       0.0  1.277900e+04  2.000000e+00  4.000000e+00          0.0         0.0         1.000000e+00         0.0

In [9]: desc.applymap("{0:.3f}".format)
Out[9]:
          import_ok     export_ok   import_skip   export_skip  import_unrec  export_unrec    import_meh    export_meh    import_err    export_err skip_regex_tilde skip_regex_with_set skip_community unrec_import_empty unrec_export_empty unrec_filter_set unrec_as_routes unrec_route_set  unrec_as_set unrec_as_set_route unrec_some_as_set_route unrec_aut_num unrec_peering_set   spec_uphill spec_uphill_tier1 spec_tier1_pair spec_import_peer_oifps spec_import_customer_oifps spec_export_customers spec_import_from_neighbor spec_as_is_origin_but_no_route spec_as_set_contains_origin_but_no_route    err_filter err_filter_as_num err_filter_as_set err_filter_prefixes err_filter_route_set err_remote_as_num err_remote_as_set err_except_peering_right   err_peering     err_regex  rpsl_as_name   rpsl_filter    rpsl_regex rpsl_unknown_filter     recursion
count  26447600.000  26447600.000  26447600.000  26447600.000  26447600.000  26447600.000  26447600.000  26447600.000  26447600.000  26447600.000     26447600.000        26447600.000   26447600.000       26447600.000       26447600.000     26447600.000    26447600.000    26447600.000  26447600.000       26447600.000            26447600.000  26447600.000      26447600.000  26447600.000      26447600.000    26447600.000           26447600.000               26447600.000          26447600.000              26447600.000                   26447600.000                             26447600.000  26447600.000      26447600.000      26447600.000        26447600.000         26447600.000      26447600.000      26447600.000             26447600.000  26447600.000  26447600.000  26447600.000  26447600.000  26447600.000        26447600.000  26447600.000
mean          0.946         0.545         0.000         0.001         1.100         1.525         0.545         0.580         0.315         0.254            0.000               0.000          0.001              0.588              0.707            0.000           0.002           0.161         2.993            826.561                 496.205         0.623             0.001         0.476             0.326           0.109                  0.015                      0.000                 0.017                     0.071                          0.110                                    0.057         0.015             0.003             0.012               0.000                0.001          1247.648             0.017                    0.000      1247.423         0.000         0.000         0.000         0.000               0.000         0.000
std           0.869         0.845         0.007         0.025         1.002         1.129         0.670         0.731         0.477         0.445            0.006               0.003          0.092              0.747              0.855            0.002           0.049           7.936        38.255           2372.436                1383.372         1.125             0.036         0.840             0.538           0.317                  0.122                      0.011                 0.167                     0.441                          0.483                                    0.428         0.182             0.122             0.141               0.004                0.035          2697.730             0.243                    0.000      2697.789         0.010         0.015         0.000         0.000               0.001         0.000
min           0.000         0.000         0.000         0.000         0.000         0.000         0.000         0.000         0.000         0.000            0.000               0.000          0.000              0.000              0.000            0.000           0.000           0.000         0.000         -32590.000                   0.000         0.000             0.000         0.000             0.000           0.000                  0.000                      0.000                 0.000                     0.000                          0.000                                    0.000         0.000             0.000             0.000               0.000                0.000             0.000             0.000                    0.000         0.000         0.000         0.000         0.000         0.000               0.000         0.000
25%           0.000         0.000         0.000         0.000         0.000         1.000         0.000         0.000         0.000         0.000            0.000               0.000          0.000              0.000              0.000            0.000           0.000           0.000         0.000              0.000                   0.000         0.000             0.000         0.000             0.000           0.000                  0.000                      0.000                 0.000                     0.000                          0.000                                    0.000         0.000             0.000             0.000               0.000                0.000             0.000             0.000                    0.000         0.000         0.000         0.000         0.000         0.000               0.000         0.000
50%           1.000         0.000         0.000         0.000         1.000         1.000         0.000         0.000         0.000         0.000            0.000               0.000          0.000              0.000              0.000            0.000           0.000           0.000         0.000              0.000                   0.000         0.000             0.000         0.000             0.000           0.000                  0.000                      0.000                 0.000                     0.000                          0.000                                    0.000         0.000             0.000             0.000               0.000                0.000             0.000             0.000                    0.000         0.000         0.000         0.000         0.000         0.000               0.000         0.000
75%           1.000         1.000         0.000         0.000         2.000         2.000         1.000         1.000         1.000         1.000            0.000               0.000          0.000              1.000              1.000            0.000           0.000           0.000         0.000              0.000                   1.000         1.000             0.000         1.000             1.000           0.000                  0.000                      0.000                 0.000                     0.000                          0.000                                    0.000         0.000             0.000             0.000               0.000                0.000           171.000             0.000                    0.000       161.000         0.000         0.000         0.000         0.000               0.000         0.000
max           8.000         7.000         1.000         3.000        11.000        11.000         6.000         6.000         5.000         5.000            1.000               1.000         32.000              8.000              8.000            3.000           3.000         843.000      7314.000          23080.000               27870.000        18.000             6.000         9.000             2.000           2.000                  2.000                      1.000                17.000                    24.000                         37.000                                   72.000        16.000            16.000             7.000               2.000                3.000         12779.000            28.000                    0.000     12779.000         2.000         4.000         0.000         0.000               1.000         0.000

[SichangHe]

Data: https://github.com/SichangHe/internet_route_verification/releases/download/data-88/route_stats1.csv.gz

[SichangHe]

Stack area plot: percentages of exchange vs routes.

Stack area plot: percentages of export vs routes.

Stack area plot: percentages of import vs routes.

PDFs (rasterized).

route-exchange-percentages-stacked-area-squared.pdf
route-exchange-percentages-stacked-area.pdf
route-export-percentages-stacked-area-squared.pdf
route-export-percentages-stacked-area.pdf
route-import-percentages-stacked-area-squared.pdf
route-import-percentages-stacked-area.pdf

Generated using SichangHe/internet_route_verification_meta@8f8e9c4. Edit: added "exchange" in SichangHe/internet_route_verification_meta@a6df558.

[SichangHe]

Skips are invisible. The plots are still blocky.

[cunha]

Makes sense to me. It's nice to have mostly OK and unrec.

…

On Wed, Nov 8, 2023, 09:15 Steven Hé (Sīchàng) ***@***.***> wrote: Skips are invisible. The plots are still blocky. — Reply to this email directly, view it on GitHub <#88 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACPO52VCRNRB74ATQGGQG3YDNZWDAVCNFSM6AAAAAA7CW7PC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBRG43TGMRZGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[SichangHe]

From #72.

@cunha, why blocky?

[SichangHe]

Is it blocky because of the sampling? I think it's still OK: although we lose some resolution, we can still see the cases that happen together and how often each case happens.

—@cunha in #72 (comment)

No. Plots here are rasterized. Vector plots with sampling look highly similar except that some lines are slightly thicker.

[SichangHe]

Latest plots.

route-exchange-percentages-stacked-area-squared.pdf
route-exchange-percentages-stacked-area.pdf
route-export-percentages-stacked-area-squared.pdf
route-export-percentages-stacked-area.pdf
route-import-percentages-stacked-area-squared.pdf
route-import-percentages-stacked-area.pdf

[SichangHe]

Aggregated stats for all RIBs (commit SichangHe/internet_route_verification_meta@abf8a2e).

route_all_export_stats.csv
route_all_import_stats.csv
route_all_total_stats.csv

[SichangHe]

For all RIBs:

Very little changes.

Commit: SichangHe/internet_route_verification_meta@2071083.

[SichangHe]

Update1: Aggregated stats for all RIBs (commit SichangHe/internet_route_verification_meta@796066b).
route_all_total_stats1.csv
route_all_export_stats1.csv
route_all_import_stats1.csv

Causes little difference in the plot generated.