Skip to content
Permalink
Browse files

ENH: cymru full bogons: add ipv6 feed

plus docs and tests
  • Loading branch information
wagner-certat committed Mar 25, 2020
1 parent 1a00443 commit 66110aa48411ea6a8f73bedc38a2c5baa3b6d730
@@ -44,6 +44,9 @@ CHANGELOG
- `intelmq.bots.parsers.taichung.parser`:
- Migrate to `ParserBot`.
- Also parse geolocation information if available.
- `intelmq.bots.parsers.cymru.parser_full_bogons`:
- Migrate to `ParserBot`.
- Add last updated information in raw.

#### Experts
- `intelmq.bots.experts.csv_converter`: Added as converter to CSV.
@@ -61,6 +64,8 @@ CHANGELOG
### Documentation
- Document usage of the `INTELMQ_ROOT_DIR` environment variable.
- Added document on MISP integration possibilities.
- Feeds:
- Added "Full Bogons IPv6" feed.

### Packaging
- `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway.
@@ -79,6 +84,7 @@ CHANGELOG
- Added tests for the new bot `intelmq.bots.experts.misp.expert` (#1473).
- Added tests for `intelmq.lib.exceptions`.
- Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`.
- Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`.

### Tools
- `intelmqctl`:
@@ -1827,7 +1827,7 @@ server {
* **Configuration Parameters:**


## Full Bogons
## Full Bogons IPv4

* **Public:** yes
* **Revision:** 2018-01-20
@@ -1839,7 +1839,29 @@ server {
* **Module:** intelmq.bots.collectors.http.collector_http
* **Configuration Parameters:**
* * `http_url`: `https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt`
* * `name`: `Full Bogons`
* * `name`: `Full Bogons IPv4`
* * `provider`: `Team Cymru`
* * `rate_limit`: `129600`

### Parser

* **Module:** intelmq.bots.parsers.cymru.parser_full_bogons
* **Configuration Parameters:**


## Full Bogons IPv6

* **Public:** yes
* **Revision:** 2018-01-20
* **Documentation:** https://www.team-cymru.com/bogon-reference-http.html
* **Description:** Fullbogons are a larger set which also includes IP space that has been allocated to an RIR, but not assigned by that RIR to an actual ISP or other end-user. IANA maintains a convenient IPv4 summary page listing allocated and reserved netblocks, and each RIR maintains a list of all prefixes that they have assigned to end-users. Our bogon reference pages include additional links and resources to assist those who wish to properly filter bogon prefixes within their networks.

### Collector

* **Module:** intelmq.bots.collectors.http.collector_http
* **Configuration Parameters:**
* * `http_url`: `https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt`
* * `name`: `Full Bogons IPv6`
* * `provider`: `Team Cymru`
* * `rate_limit`: `129600`

@@ -2,40 +2,39 @@
import dateutil

from intelmq.lib import utils
from intelmq.lib.bot import Bot
from intelmq.lib.bot import ParserBot


class CymruFullBogonsParserBot(Bot):

def process(self):
report = self.receive_message()
class CymruFullBogonsParserBot(ParserBot):

def parse(self, report):
raw_report = utils.base64_decode(report.get("raw")).strip()

if not len(raw_report): # We depend on first line = date
self.acknowledge_message()
return

row = raw_report.splitlines()[0]
time_str = row[row.find('(') + 1:row.find(')')]
time = dateutil.parser.parse(time_str).isoformat()
first_row = raw_report[:raw_report.find('\n')]
time_str = first_row[first_row.find('(') + 1:first_row.find(')')]
self.last_updated = dateutil.parser.parse(time_str).isoformat()
self.tempdata.append(first_row)

for row in raw_report.splitlines():
val = row.strip()
if not len(val) or val.startswith('#') or val.startswith('//'):
continue
yield row.strip()

def parse_line(self, val, report):
if not len(val) or val.startswith('#') or val.startswith('//'):
return

event = self.new_event(report)
event = self.new_event(report)

if not event.add('source.ip', val, raise_failure=False):
event.add('source.network', val)
if not event.add('source.ip', val, raise_failure=False):
event.add('source.network', val)

event.add('time.source', time)
event.add('classification.type', 'blacklist')
event.add('raw', row)
event.add('time.source', self.last_updated)
event.add('classification.type', 'blacklist')
event.add('raw', self.recover_line(val))

self.send_message(event)
self.acknowledge_message()
yield event


BOT = CymruFullBogonsParserBot
@@ -516,7 +516,7 @@ providers:
revision: 2018-01-20
documentation: https://www.team-cymru.com/CSIRT-AP.html https://www.cymru.com/$certname/report_info.txt
public: no
Full Bogons:
Full Bogons IPv4:
description: Fullbogons are a larger set which also includes IP space that has
been allocated to an RIR, but not assigned by that RIR to an actual ISP or
other end-user. IANA maintains a convenient IPv4 summary page listing allocated
@@ -539,6 +539,29 @@ providers:
revision: 2018-01-20
documentation: https://www.team-cymru.com/bogon-reference-http.html
public: yes
Full Bogons IPv6:
description: Fullbogons are a larger set which also includes IP space that has
been allocated to an RIR, but not assigned by that RIR to an actual ISP or
other end-user. IANA maintains a convenient IPv4 summary page listing allocated
and reserved netblocks, and each RIR maintains a list of all prefixes that
they have assigned to end-users. Our bogon reference pages include additional
links and resources to assist those who wish to properly filter bogon prefixes
within their networks.
additional_information:
bots:
collector:
module: intelmq.bots.collectors.http.collector_http
parameters:
http_url: https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
rate_limit: 129600
name: __FEED__
provider: __PROVIDER__
parser:
module: intelmq.bots.parsers.cymru.parser_full_bogons
parameters:
revision: 2018-01-20
documentation: https://www.team-cymru.com/bogon-reference-http.html
public: yes
Taichung:
Netflow Recent:
description: "Abnormal flows detected: Attacking (DoS, Brute-Force, Scanners) and malicious hosts (C&C servers, hosting malware)"
@@ -17,15 +17,28 @@
'source.network': '0.0.0.0/8',
'classification.type': 'blacklist',
'time.observation': '2015-11-01T00:01:45+00:05',
'raw': 'MC4wLjAuMC84',
'raw': 'IyBsYXN0IHVwZGF0ZWQgMTQ1MDE5MzcwMiAoVHVlIERlYyAxNSAxNTozNTowMiAyMDE1IEdNVCkKMC4wLjAuMC84',
}
EVENT2 = {'__type': 'Event',
'feed.url': 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt',
'time.source': '2015-12-15T15:35:02+00:00',
'source.network': '2.56.0.0/14',
'classification.type': 'blacklist',
'time.observation': '2015-11-01T00:01:45+00:05',
'raw': 'Mi41Ni4wLjAvMTQ='
'raw': 'IyBsYXN0IHVwZGF0ZWQgMTQ1MDE5MzcwMiAoVHVlIERlYyAxNSAxNTozNTowMiAyMDE1IEdNVCkKMi41Ni4wLjAvMTQ='
}
V6REPO = {'__type': 'Report',
'feed.url': 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt',
'raw': 'IyBsYXN0IHVwZGF0ZWQgMTU4NTE0MDYwMSAoV2VkIE1hciAyNSAxMjo1MDowMSAyMDIwIEdNVCkKOjovOAoxMDA6Oi84Cg==',
'time.observation': '2020-03-25T16:42:45+00:00',
}
V6EVEN = {'__type': 'Event',
'feed.url': 'https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt',
'time.source': '2020-03-25T12:50:01+00:00',
'source.network': '::/8',
'classification.type': 'blacklist',
'time.observation': '2020-03-25T16:42:45+00:00',
'raw': 'IyBsYXN0IHVwZGF0ZWQgMTU4NTE0MDYwMSAoV2VkIE1hciAyNSAxMjo1MDowMSAyMDIwIEdNVCkKOjovOA==',
}


@@ -37,14 +50,19 @@ class TestCymruFullBogonsParserBot(test.BotTestCase, unittest.TestCase):
@classmethod
def set_bot(cls):
cls.bot_reference = CymruFullBogonsParserBot
cls.default_input_message = {'__type': 'Report', 'raw': 'Cg=='}

def test_events(self):
""" Test if correct Events have been produced. """
def test_ipv4_events(self):
""" Test if correct IPv4 Events have been produced. """
self.input_message = REPORT
self.run_bot()
self.assertMessageEqual(0, EVENT1)
self.assertMessageEqual(1, EVENT2)

def test_ipv6_events(self):
""" Test if correct IPv6 Events have been produced. """
self.input_message = V6REPO
self.run_bot()
self.assertMessageEqual(0, V6EVEN)

if __name__ == '__main__': # pragma: no cover
unittest.main()

0 comments on commit 66110aa

Please sign in to comment.
You can’t perform that action at this time.