Skip to content
Permalink
Browse files

Merge branch 'maintenance' into develop

  • Loading branch information...
wagner-certat committed Aug 9, 2019
2 parents 9085d30 + a3d617e commit 2fbd378512fbbbde5f4651bc7da3a245a7c59f2f
@@ -86,8 +86,9 @@ CHANGELOG
- `intelmq.lib.bot`:
- fix parameters of ParserBot and CollectorBot constructors, allowing `intelmqctl run` with these bots again (#1414).
- Also run `rate_limit` after retry counter reset (#1431).
- `__version_info__` is now available in the top level module.
- `__version__`: uses now integer values if possible.
- `__version_info__`:
- is now available in the top level module.
- uses now integer values instead of strings for numerical version parts
- Also provide (empty) `ROOT_DIR` for non-pip installations.
- `intelmq.lib.upgrades`: New library file `upgrades` with upgrade functions.
- `intelmq.lib.utils`:
@@ -99,7 +100,11 @@ CHANGELOG
- `log` takes a new argument `logging_level_stream` for the logging level of the console handler.
- New constant `LOG_FORMAT_SIMPLE`, used by intelmqctl.
- New function `write_configuration` to write dicts to files in the correct json formatting.
- `intelmq.lib.pipeline`: AMQP: Actually use `source/destination_pipeline_amqp_virtual_host` parameter.
- `intelmq.lib.pipeline`:
- AMQP: Actually use `source/destination_pipeline_amqp_virtual_host` parameter.
- pipeline base class: add missing dummy methods.
- Add missing return types.
- Redis: Evaluate return parameter of queue/key deletion.
- Variable `STATE_FILE_PATH` added.

### Development
@@ -123,18 +128,22 @@ CHANGELOG
- `intelmq.bots.experts.modify.expert`:
- Compile regular expressions (all string rules) at initializations, improves the speed.
- Warn about old configuration style deprecation.
- `intelmq.bots.experts.do_portal.expert`: Use `http_timeout_max_tries` parameter for retries on connection timeouts (#1432).
- `intelmq.bots.experts.ripe.expert`: Use `http_timeout_max_tries` parameter for retries on connection timeouts.

#### Outputs
- `intelmq.bots.outputs.postgresql`: Recommend psycopg2-binary package.
- `intelmq.bots.outputs.amqptopic.output`: Shutdown: Close connection only if connection exists.
- `intelmq.bots.outputs.amqptopic`:
- Shutdown: Close connection only if connection exists.
- Add support for pika > 1, the way the (Non-)Acknowledgments are provided has been changed.
- Gracefully handle unroutable messages and give advice.
- Support for no used authentication.
- Replace deprecated parameter `type` with `exchange_type` for `exchange_declare`, supporting pika >= 0.11 (#1425).
- New parameters `message_hierarchical_output`, `message_with_type`, `message_jsondict_as_string`.
- `intelmq.bots.outputs.mongodb.output`: Support for pymongo >= 3.0.0 (#1063, PR#1421).
- `intelmq.bots.outputs.file`: `time.*` field serialization: support for microseconds.
- `intelmq.bots.outputs.mongodb.output`: Support for authentication in pymongo >= 3.5 (#1062).
- `intelmq.bots.outputs.restapi.output`: Use `http_timeout_max_tries` parameter for retries on connection timeouts.

### Documentation
- Add certbund-contact to the ecosystem document.
@@ -144,6 +153,7 @@ CHANGELOG
- Clarify on Uninstallation

### Packaging
- Do not execute the tcp collector tests during debian and ubuntu builds as they fail there.

### Tests
- `intelmq.lib.test`: Disable statistics for test runs of bots.
@@ -24,8 +24,9 @@ build:
patch -p1 setup.py debian/patches/fix-dnspython-name.patch;\
fi
dh build --with python3 --without python2 --buildsystem=pybuild --with quilt --with systemd
# This tests frequently failes on ubuntu and debian systems
rm -r intelmq/tests/bots/collectors/tcp/
# This tests frequently failes on ubuntu and debian systems.
# On some systems this command will be executed twice, so -f
rm -rf intelmq/tests/bots/collectors/tcp/

override_dh_auto_build:

@@ -1,10 +1,10 @@
**Table of Contents:**
- [Requirements](#requirements)
- [Install Dependencies](#install-dependencies)
- [Ubuntu 14.04 / Debian 8](#ubuntu-1404-debian-8)
- [Ubuntu 16.04 / Ubuntu 18.04 / Debian 9](#ubuntu-1604-ubuntu-1804-debian-9)
- [CentOS 7 / RHEL 7](#centos-7-rhel-7)
- [openSUSE Leap 42.3 / 15.0](#opensuse-leap-423-150)
- [Debian 8](#debian-8)
- [Ubuntu 16.04 / Ubuntu 18.04 / Debian 9](#ubuntu-1604--ubuntu-1804--debian-9)
- [CentOS 7 / RHEL 7](#centos-7--rhel-7)
- [openSUSE Leap 15.0 / 15.1](#opensuse-leap-150--151)
- [Installation](#installation)
- [Native Packages](#native-packages)
- [PyPi](#pypi)
@@ -24,17 +24,17 @@ The following instructions assume the following requirements:

Supported and recommended operating systems are:
* CentOS 7
* Debian 8 and 9
* OpenSUSE Leap 42.3 and 15.0
* Ubuntu: 14.04, 16.04 and 18.04
* Debian 8, 9 and 10
* OpenSUSE Leap 15.0 and 15.1
* Ubuntu: 16.04, 18.04 and 19.04

Other distributions which are (most probably) supported include RHEL, Fedora and openSUSE Tumbleweed.

# Install Dependencies

If you are using native packages, you can simply skip this section as all dependencies are installed automatically.

## Ubuntu 14.04 / Debian 8
## Debian 8

```bash
apt-get install python3 python3-pip
@@ -79,7 +79,7 @@ curl "https://bootstrap.pypa.io/get-pip.py" -o "/tmp/get-pip.py"
python3.4 /tmp/get-pip.py
```

## openSUSE Leap 42.3 / 15.0
## openSUSE 15.0 / 15.1

```bash
zypper install python3-dateutil python3-dnspython python3-psutil python3-pytz python3-redis python3-requests python3-python-termstyle
@@ -106,17 +106,18 @@ Installation methods available:
Supported Operating Systems:

* **CentOS 7** (requires `epel-release`)
* **RHEL 7** (requires `epel-release`)
* **Debian 8** (requires `python3-typing`)
* **Debian 9**
* **Fedora 27**
* **Fedora 28**
* **Debian 10**
* **Fedora 29**
* **openSUSE Leap 42.3**
* **Fedora 30**
* **RHEL 7** (requires `epel-release`)
* **openSUSE Leap 15.0**
* **openSUSE Leap 15.1**
* **openSUSE Tumbleweed**
* **Ubuntu 16.04** (enable the universe repositories by appending ` universe` in `/etc/apt/sources.list` to `deb http://[...].archive.ubuntu.com/ubuntu/ xenial main`)
* **Ubuntu 18.04** (enable the universe repositories by appending ` universe` in `/etc/apt/sources.list` to `deb http://[...].archive.ubuntu.com/ubuntu/ bionic main`)
* **Ubuntu 19.04** (enable the universe repositories by appending ` universe` in `/etc/apt/sources.list` to `deb http://[...].archive.ubuntu.com/ubuntu/ disco main`)

Get the installation instructions for your operating system here: [Installation Native Packages](https://software.opensuse.org/download.html?project=home%3Asebix%3Aintelmq&package=intelmq).

@@ -24,26 +24,42 @@ def init(self):

def process(self):
event = self.receive_message()
if "source.ip" in event:
req = requests.get(self.url % event['source.ip'],
headers=self.http_header,
auth=self.auth,
proxies=self.proxy,
verify=self.http_verify_cert,
cert=self.ssl_client_cert,
timeout=self.http_timeout_sec)
if req.status_code == 404 and req.json()['message'].startswith("('no such cidr'"):
result = []
else:
req.raise_for_status()
result = req.json()['abusecs']

if self.mode == 'append':
existing = event.get("source.abuse_contact", '').split(',')
combined = ','.join(existing + result).strip(',')
event.add("source.abuse_contact", combined, overwrite=True)
else:
event.add("source.abuse_contact", ','.join(result), overwrite=True)
if "source.ip" not in event:
self.send_message(event)
self.acknowledge_message()
return

timeoutretries = 0
req = None

while timeoutretries < self.http_timeout_max_tries and req is None:
try:
req = requests.get(self.url % event['source.ip'],
headers=self.http_header,
auth=self.auth,
proxies=self.proxy,
verify=self.http_verify_cert,
cert=self.ssl_client_cert,
timeout=self.http_timeout_sec)
except requests.exceptions.Timeout:
timeoutretries += 1

if req is None and timeoutretries >= self.http_timeout_max_tries:
raise ValueError("Request timed out %i times in a row."
"" % timeoutretries)

if req.status_code == 404 and req.json()['message'].startswith("('no such cidr'"):
result = []
else:
req.raise_for_status()
result = req.json()['abusecs']

if self.mode == 'append':
existing = event.get("source.abuse_contact", '').split(',')
combined = ','.join(existing + result).strip(',')
event.add("source.abuse_contact", combined, overwrite=True)
else:
event.add("source.abuse_contact", ','.join(result), overwrite=True)

self.send_message(event)
self.acknowledge_message()
@@ -137,7 +137,20 @@ def __perform_cached_query(self, type, resource):
else:
return json.loads(cached_value)
else:
response = self.http_session.get(self.QUERY[type].format(resource), data="", timeout=self.http_timeout_sec)
timeoutretries = 0
response = None

while timeoutretries < self.http_timeout_max_tries and response is None:
try:
response = self.http_session.get(self.QUERY[type].format(resource),
data="", timeout=self.http_timeout_sec)
except requests.exceptions.Timeout:
timeoutretries += 1

if response is None and timeoutretries >= self.http_timeout_max_tries:
raise ValueError("Request timed out %i times in a row."
"" % timeoutretries)

if response.status_code != 200:
if type == 'db_asn' and response.status_code == 404:
""" If no abuse contact could be found, a 404 is given. """
@@ -38,12 +38,23 @@ def process(self):
else:
kwargs = {'data': event.to_dict(hierarchical=self.parameters.hierarchical_output)}

r = self.session.post(self.parameters.host,
timeout=self.http_timeout_sec,
**kwargs)
if not r.ok:
timeoutretries = 0
req = None
while timeoutretries < self.http_timeout_max_tries and req is None:
try:
req = self.session.post(self.parameters.host,
timeout=self.http_timeout_sec,
**kwargs)
except requests.exceptions.Timeout:
timeoutretries += 1

if req is None and timeoutretries >= self.http_timeout_max_tries:
raise ValueError("Request timed out %i times in a row."
"" % timeoutretries)

if not req.ok:
self.logger.debug("Error during message sending with response body: %r.", r.text)
r.raise_for_status()
req.raise_for_status()
self.logger.debug('Sent message.')
self.acknowledge_message()

@@ -110,10 +110,19 @@ def set_queues(self, queues, queues_type):
else:
raise exceptions.InvalidArgument('queues_type', got=queues_type, expected=['source', 'destination'])

def nonempty_queues(self) -> set:
def send(self, message, path="_default", path_permissive=False):
raise NotImplementedError

def send(self, message, path="_default", path_permissive=False):
def receive(self) -> str:
raise NotImplementedError

def acknowledge(self):
raise NotImplementedError

def clear_queue(self, queue):
raise NotImplementedError

def nonempty_queues(self) -> set:
raise NotImplementedError


@@ -196,7 +205,7 @@ def send(self, message, path="_default", path_permissive=False):
'Look at redis\'s logs.')
raise exceptions.PipelineError(exc)

def receive(self):
def receive(self) -> str:
if self.source_queue is None:
raise exceptions.ConfigurationError('pipeline', 'No source queue given.')
try:
@@ -220,7 +229,7 @@ def acknowledge(self):
except Exception as e:
raise exceptions.PipelineError(e)

def count_queued_messages(self, *queues):
def count_queued_messages(self, *queues) -> dict:
queue_dict = {}
for queue in queues:
try:
@@ -233,7 +242,10 @@ def clear_queue(self, queue):
"""Clears a queue by removing (deleting) the key,
which is the same as an empty list in Redis"""
try:
return self.pipe.delete(queue)
retval = self.pipe.delete(queue)
if retval not in (0, 1):
raise ValueError("Error on redis queue deletion: Return value was not 0 "
"or 1 but %s." % retval)
except Exception as exc:
raise exceptions.PipelineError(exc)

@@ -290,7 +302,7 @@ def send(self, message, path="_default", path_permissive=False):
else:
self.state[destination_queue] = [utils.encode(message)]

def receive(self):
def receive(self) -> str:
"""
Receives the last not yet acknowledged message.
@@ -310,9 +322,9 @@ def receive(self):

def acknowledge(self):
"""Removes a message from the internal queue and returns it"""
return self.state.get(self.internal_queue, [None]).pop(0)
self.state.get(self.internal_queue, [None]).pop(0)

def count_queued_messages(self, *queues):
def count_queued_messages(self, *queues) -> dict:
"""Returns the amount of queued messages
over all given queue names.
"""
@@ -1,2 +1,2 @@
__version_info__ = (2, 1, 0, 'alpha', 1)
__version__ = '.'.join(map(str,__version_info__))
__version__ = '.'.join(map(str, __version_info__))

0 comments on commit 2fbd378

Please sign in to comment.
You can’t perform that action at this time.