Permalink
Browse files

Merge branch 'maintenance' into develop

  • Loading branch information...
wagner-certat committed Oct 10, 2018
2 parents 2915e9f + 6f36c7a commit 08c2f9017a7c66140e988b86eb988059e0669c8c
Showing with 21 additions and 5 deletions.
  1. +4 −0 CHANGELOG.md
  2. +1 −1 docs/Feeds.md
  3. +15 −3 intelmq/bots/collectors/mail/collector_mail_url.py
  4. +1 −1 intelmq/etc/feeds.yaml
@@ -69,6 +69,9 @@ CHANGELOG
### Bots
#### Collectors
- `intelmq.bots.http.collector_http`: Fix parameter `extract_files` in BOTS (#1331).
- `intelmq.bots.collectos.mail.collector_mail_url`:
- Handle HTTP status codes != 2xx the same as HTTP timeouts: No exception, but graceful handling.
- Handle HTTP errors (bad status code and timeouts) with `error_procedure` == 'pass' but marking the mail as read and logging the error.
#### Parsers
- `intelmq.bots.parsers.misp`: Fix Object attribute (#1318).
@@ -91,6 +94,7 @@ CHANGELOG
### Documentation
- FAQ: Explanation and solution on orphaned queues.
- Add or fix the tables of contents for all documentation files.
- Feeds: Fix Autoshun Feed URL (#1325).
### Packaging
- Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).
@@ -224,7 +224,7 @@ To add feeds to this file add them to `intelmq/etc/feeds.yaml` and then run
* **Module:** intelmq.bots.collectors.http.collector_http
* **Configuration Parameters:**
* * `http_url`: `https://www.autoshun.org/files/shunlist.html`
* * `http_url`: `https://www.autoshun.org/download/?api_key=__APIKEY__&format=html`
* * `rate_limit`: `3600`
### Parser
@@ -84,8 +84,11 @@ def process(self):
continue
if resp.status_code // 100 != 2:
raise ValueError('HTTP response status code was {}.'
''.format(resp.status_code))
self.logger.error('HTTP response status code was {}.'
''.format(resp.status_code))
erroneous = True
continue
if not resp.content:
self.logger.warning('Got empty reponse from server.')
else:
@@ -111,7 +114,16 @@ def process(self):
if not erroneous:
self.logger.info("Email report read.")
else:
self.logger.error("Email report read with errors, the report was not processed.")
if self.parameters.error_procedure == 'pass':
try:
mailbox.mark_seen(uid)
except imaplib.abort:
mailbox = self.connect_mailbox()
mailbox.mark_seen(uid)
self.logger.error("Download of report failed with above error, marked Email as read "
"(according to `error_procedure` parameter).")
else:
self.logger.error("Email report read with above errors, the report was not processed.")
else:
self.logger.debug("No unread mails to check.")
mailbox.logout()
@@ -688,7 +688,7 @@ providers:
collector:
module: intelmq.bots.collectors.http.collector_http
parameters:
http_url: https://www.autoshun.org/files/shunlist.html
http_url: https://www.autoshun.org/download/?api_key=__APIKEY__&format=html
rate_limit: 3600
parser:
module: intelmq.bots.parsers.autoshun.parser

0 comments on commit 08c2f90

Please sign in to comment.