Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated Neighbors VS Rules Stats considering ASes in ASRelDB but not IRR #67

Closed
SichangHe opened this issue Oct 15, 2023 · 11 comments
Closed
Labels
Stats Statistics record wontnote Won't note this in the writing

Comments

@SichangHe
Copy link
Owner

SichangHe commented Oct 15, 2023

-1 for neighbor if no data in AS Relationship DB. -1 for import/export if no data in IRR.

Polars output.
shape: (95_911, 4)
┌─────────┬──────────┬────────┬────────┐
│ aut_numneighborimportexport │
│ ------------    │
│ u64i32i32i32    │
╞═════════╪══════════╪════════╪════════╡
│ 375912-1-1     │
│ 204753133      │
│ 9689211      │
│ 626854-1-1     │
│ …       ┆ …        ┆ …      ┆ …      │
│ 262378711      │
│ 139330100      │
│ 2656511-1-1     │
│ 170771088      │
└─────────┴──────────┴────────┴────────┘
shape: (9, 5)
┌────────────┬───────────────┬────────────┬───────────┬───────────┐
│ describe   ┆ aut_num       ┆ neighbor   ┆ import    ┆ export    │
│ ---        ┆ ------------       │
│ strf64f64f64f64       │
╞════════════╪═══════════════╪════════════╪═══════════╪═══════════╡
│ count95911.095911.095911.095911.0   │
│ null_count0.00.00.00.0       │
│ mean125671.69340310.156024.1222494.008946  │
│ std112437.127515102.24357252.11039348.272178 │
│ min1.0-1.0-1.0-1.0      │
│ 25%34821.01.00.00.0       │
│ 50%62122.01.01.01.0       │
│ 75%205279.53.02.02.0       │
│ max        ┆ 6.131644e69628.05724.05344.0    │
└────────────┴───────────────┴────────────┴───────────┴───────────┘

as_neighbors_vs_rules.csv

@SichangHe SichangHe added the lateststats Latest and up-to-date statistics label Oct 15, 2023
@SichangHe
Copy link
Owner Author

as_neighbor_vs_counts_zoom

Code used.
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv("as_neighbors_vs_rules.csv")
plt.figure(figsize=(15, 15))
plt.scatter(df['neighbor'], df['import'], color='blue', alpha=0.3, s=2, label='Number of Import')
plt.scatter(df['neighbor'], df['export'], color='red', alpha=0.3, s=2, label='Number of Export')
plt.xlabel('Neighbor')
plt.ylabel('Counts')
plt.title('Neighbor Counts vs Import/Export Counts for Autonomous Systems')
plt.legend()
plt.grid(True)
plt.show()

@SichangHe
Copy link
Owner Author

16960 ASes appear in ASRelDB but not IRR.
20468 ASes appear in IRR but not ASRelDB.
Total: 95911 ASes.

IPython.
In [14]: df[(df['import'] == -1) | (df['export'] == -1)]
Out[14]:
       aut_num  neighbor  import  export
0        37591         2      -1      -1
3        62685         4      -1      -1
6       399566         1      -1      -1
16       11471         1      -1      -1
19       14444         2      -1      -1
...        ...       ...     ...     ...
95883    31957         2      -1      -1
95884    52860         1      -1      -1
95885   264164         2      -1      -1
95895   393635         2      -1      -1
95909   265651         1      -1      -1

[16960 rows x 4 columns]

In [15]: df[(df['neighbor'] == -1)]
Out[15]:
       aut_num  neighbor  import  export
11       55704        -1       0       0
33      202322        -1       3       3
44      199337        -1       2       2
45       44502        -1       1       1
48       24788        -1       0       0
...        ...       ...     ...     ...
95896   142031        -1       0       0
95899    17902        -1       3       3
95900    28529        -1       0       0
95901   208690        -1       0       0
95905   211088        -1       9       9

[20468 rows x 4 columns]

@cunha
Copy link
Collaborator

cunha commented Oct 16, 2023

16960 ASes appear in ASRelDB but not IRR.

Wild. Do you mean ~17K ASes have no import/export rules, or that there are no aut-num objects at all?

@SichangHe
Copy link
Owner Author

Wild. Do you mean ~17K ASes have no import/export rules, or that there are no aut-num objects at all?

No aut-num in IRR databases we use.

@SichangHe
Copy link
Owner Author

SichangHe commented Oct 17, 2023

whois some of the AS not in IRR.
$ whois -h whois.afrinic.net AS37591
% This is the AfriNIC Whois server.
% The AFRINIC whois database is subject to  the following terms of Use. See https://afrinic.net/whois/terms

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

%ERROR:101: no entries found
%
% No entries found in source AFRINIC.

$ whois -h whois.arin.net AS62685
ASNumber:       62685
ASName:         ORIONVM
ASHandle:       AS62685
RegDate:        2013-08-07
Updated:        2013-08-07
Ref:            https://rdap.arin.net/registry/autnum/62685


OrgName:        OrionVM Inc
OrgId:          ORION-34
Address:        320 Judah Street
Address:        Suite 5
City:           San Francisco
StateProv:      CA
PostalCode:     94122
Country:        US
RegDate:        2013-01-22
Updated:        2020-05-26
Comment:        Please send spam and abuse reports to abuse@orionvm.com.
Comment:        Please include detailed information and times in UTC.
Ref:            https://rdap.arin.net/registry/entity/ORION-34


OrgTechHandle: NOC12797-ARIN
OrgTechName:   NOC
OrgTechPhone:  +1-844-674-6686
OrgTechEmail:  noc@orionvm.com
OrgTechRef:    https://rdap.arin.net/registry/entity/NOC12797-ARIN

OrgAbuseHandle: NOC12797-ARIN
OrgAbuseName:   NOC
OrgAbusePhone:  +1-844-674-6686
OrgAbuseEmail:  noc@orionvm.com
OrgAbuseRef:    https://rdap.arin.net/registry/entity/NOC12797-ARIN

OrgNOCHandle: NOC12797-ARIN
OrgNOCName:   NOC
OrgNOCPhone:  +1-844-674-6686
OrgNOCEmail:  noc@orionvm.com
OrgNOCRef:    https://rdap.arin.net/registry/entity/NOC12797-ARIN

$ whois -h whois.arin.net AS31957
ASNumber:       31957
ASName:         AGILE
ASHandle:       AS31957
RegDate:        2015-07-10
Updated:        2015-07-10
Ref:            https://rdap.arin.net/registry/autnum/31957


OrgName:        Agile Networks
OrgId:          ANBL-2
Address:        213 Market Ave N Suite 310
City:           Canton
StateProv:      OH
PostalCode:     44702
Country:        US
RegDate:        2012-10-03
Updated:        2023-08-11
Comment:        http://www.agilenetworks.com
Ref:            https://rdap.arin.net/registry/entity/ANBL-2


OrgTechHandle: SPILL46-ARIN
OrgTechName:   Spillman, Phillip
OrgTechPhone:  +1-419-631-8495
OrgTechEmail:  phil.spillman@agilenetworks.com
OrgTechRef:    https://rdap.arin.net/registry/entity/SPILL46-ARIN

OrgTechHandle: ROBER1880-ARIN
OrgTechName:   Robertson, Zane
OrgTechPhone:  +1-330-806-6246
OrgTechEmail:  zane.robertson@agilenetworks.com
OrgTechRef:    https://rdap.arin.net/registry/entity/ROBER1880-ARIN

OrgTechHandle: KREIG3-ARIN
OrgTechName:   Kreigh, Russ
OrgTechPhone:  +1-260-446-4094
OrgTechEmail:  russ.kreigh@agilenetworks.com
OrgTechRef:    https://rdap.arin.net/registry/entity/KREIG3-ARIN

OrgAbuseHandle: SPILL46-ARIN
OrgAbuseName:   Spillman, Phillip
OrgAbusePhone:  +1-419-631-8495
OrgAbuseEmail:  phil.spillman@agilenetworks.com
OrgAbuseRef:    https://rdap.arin.net/registry/entity/SPILL46-ARIN

OrgRoutingHandle: SPILL46-ARIN
OrgRoutingName:   Spillman, Phillip
OrgRoutingPhone:  +1-419-631-8495
OrgRoutingEmail:  phil.spillman@agilenetworks.com
OrgRoutingRef:    https://rdap.arin.net/registry/entity/SPILL46-ARIN

OrgDNSHandle: SPILL46-ARIN
OrgDNSName:   Spillman, Phillip
OrgDNSPhone:  +1-419-631-8495
OrgDNSEmail:  phil.spillman@agilenetworks.com
OrgDNSRef:    https://rdap.arin.net/registry/entity/SPILL46-ARIN

OrgNOCHandle: SPILL46-ARIN
OrgNOCName:   Spillman, Phillip
OrgNOCPhone:  +1-419-631-8495
OrgNOCEmail:  phil.spillman@agilenetworks.com
OrgNOCRef:    https://rdap.arin.net/registry/entity/SPILL46-ARIN

OrgTechHandle: HOTTL1-ARIN
OrgTechName:   Hottle, Jesse
OrgTechPhone:  +1-888-494-0549
OrgTechEmail:  jesse.hottle@agilenetworks.com
OrgTechRef:    https://rdap.arin.net/registry/entity/HOTTL1-ARIN

@SichangHe
Copy link
Owner Author

I realized a potential problem, @cunha. We are using the AS Relationship DB from 2013, which is probably outdated.

Maybe we should update to one of CAIDA's other DBs such as as-relationships?

@SichangHe SichangHe added Stats Statistics record and removed lateststats Latest and up-to-date statistics labels Nov 3, 2023
@SichangHe
Copy link
Owner Author

No longer latest stats after fixing default.

@SichangHe SichangHe added the wontnote Won't note this in the writing label Nov 14, 2023
@SichangHe
Copy link
Owner Author

Covered by #95.

@cunha
Copy link
Collaborator

cunha commented Feb 3, 2024

I realized a potential problem, @cunha. We are using the AS Relationship DB from 2013, which is probably outdated.

Maybe we should update to one of CAIDA's other DBs such as as-relationships?

We should definitely be using a newer database. Something to match the IRR datasets we're using. Did you do this at one point?

@cunha cunha reopened this Feb 3, 2024
@SichangHe
Copy link
Owner Author

No, we are not using databases from 2013. It is just under that folder. It has been undated regularly since and we have been using the July 2023 one.

@cunha
Copy link
Collaborator

cunha commented Feb 4, 2024

Sounds good. Just checking.

@cunha cunha closed this as completed Feb 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Stats Statistics record wontnote Won't note this in the writing
Projects
None yet
Development

No branches or pull requests

2 participants