Collectors

lib_bgp_data

Collectors perform ETL. They download data from websites, clean it, and insert it into the database.

Below are the collectors we have thus far, and a short description of each (with hyperlink)

• AS Metadata: Parses country, continent, holding organization of an ASN
• AS Rank: Parses AS Rank data
• Base: Base class for all parsers/collectors
• BGPStream Website: Downloads BGPMon public hijackings, Leaks, and Outages
• Blacklists: (needs docs and merged branch) Downloads ASN and prefix blacklists from multiple sources
• CDN Whitelist: Downloads ASNs that belong to CDNs
• Historical ROAs: (needs docs and review) Parses historical ROA information
• Hurricane: (needs docs and merged branch) Parses Hurricane data
• MRT: Downloads RIPE, RouteViews, Isolario MRT Rib dumbs
• Relationships: Downloads internet topology from Caida
• ROAs: Downloads public ROAs
• RPKI Validator: Contains RPKI Validator Parser, which inputs MRT announcements and gets out ROA data. Also has a wrapper for the RPKI Validator.