certtools / intelmq

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/UPGRADING.md

Documentation

• Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht).

Harmonization

• See NEWS.md for information on a fixed bug in the taxonomy expert.

Bots

Collectors

• intelmq.bots.rt.collector_rt: Log the size of the downloaded file in bytes on debug logging level.

Parsers

• intelmq.bots.parsers.cymru.parser_cap_program:
  • Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt).
  • Add support for field additional_asns in optional information column.
• intelmq.bots.parsers.microsoft.parser_ctip:
  • Fix mapping of DestinationIpInfo.DestinationIpConnectionType field (contained a typo).
  • Explicitly ignore field DestinationIpInfo.DestinationIpv4Int as the data is already in another field.
• intelmq.bots.parsers.generic.parser_csv:
  • Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh).
  • Data fields containing - are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer).

Experts

• intelmq.bots.experts.taxonomy.expert: Map type scanner to information-gathering instead of information gathering. See NEWS file for more information.

Tests

• Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython.

Known issues

• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
• Corrupt dump files when interrupted during writing (#870).

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md

Core

• intelmq.lib.upgrades:
  • Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist".

Bots

Parsers

• intelmq.bots.parsers.shadowserver:
  • Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
  • Added support for the feeds Accessible Radmin and CAIDA IP Spoofer (PR#1600 by sinus-x).
• intelmq.bots.parsers.anubisnetworks.parser: Fix parsing error where dst.ip was not equal to comm.http.host.
• intelmq/bots/parsers/danger_rulez/parser: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus).
• `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23).
• intelmq.bots.parsers.microsoft.parser_ctip:
  • Add support for DestinationIpInfo.* and Signatures.Sha256 fields, used by the ctip-c2 feed (PR#1623 by Mikk Margus Möll).
  • Use extra.payload.text for the feed's field Payload if the content cannot be decoded (PR#1610 by Giedrius Ramas).

Experts

• intelmq.bots.experts.cymru_whois:
  • Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606).
  • The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606).

Documentation

• README:
  • Add Core Infrastructure Initiative Best Practices Badge.
• Bots:
  • Generic CSV Parser: Add note on escaping backslashes (#1579).
  • Remove section of non-existing "Copy Extra" Bot.
  • Explain taxonomy expert.
  • Add documentation on n6 parser.
  • Gethostbyname expert: Add documentation how errors are treated.
• Feeds:
  • Fixed bot modules of Calidog CertStream feed.
  • Add information on Microsoft CTIP C2 feed.

Packaging

• In Debian packages, intelmqctl check and intelmqctl upgrade-config are executed in the postinst step (#1551, PR#1624 by Birger Schacht).

Tests

• intelmq.tests.lib.test_pipeline: Skip TestAmqp.test_acknowledge on Travis with Python 3.8.
• intelmq.tests.bots.outputs.elasticsearch.test_output: Refresh index intelmq manually to fix random test failures (#1593, PR#1595 by Zach Stone).

Tools

• intelmqctl check:
  • For disabled bots which do not have any pipeline connections, do not raise an error, but only warning.
  • Fix check on source/destination queues for bots as well the orphaned queues.

Contrib

• Bash completion scripts: Check both /opt/intelmq/ as well as LSB-paths (/etc/intelmq/ and /var/log/intelmq/) for loading bot information (#1561, PR#1628 by Birger Schacht).

Known issues

• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
• Corrupt dump files when interrupted during writing (#870).

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md

Core

• intelmq.lib.upgrades:
  • Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
  • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559).
  • intelmq.lib.harmonization:
    • For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).

Development

• Ignore line length (E501) in code-style checks altogether.

Bots

Collectors

• intelmq.bots.collectors.misp: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)
• intelmq.bots.collectors.stomp: Remove empty client.pem file.

Parsers

• intelmq.bots.parsers.shadowserver.config:
  • Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
  • Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321).
• intelmq.bots.parser.anubisnetworks.parser: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all.
• intelmq.bots.parsers.generic.parser_csv: Allow values of type dictionary for parameter type_translation.
• intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559).
• intelmq.bots.parsers.cymru.parser_cap_program: Add support for comment "username" for "scanner" category.
• intelmq.bots.parsers.malwareurl.parser: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).

Experts

• intelmq.bots.experts.maxmind_geoip: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.

Outputs

• intelmq.bot.outputs.udp: Fix error handling on sending, had a bug itself.

Documentation

• Feeds:
  • Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
• Bots:
  • Overhaul of all bots' description fields (#1570).
• User-Guide:
  • Overhaul pipeline configuration section and explain named queues better (#1577).

Tests

• intelmq.tests.bots.experts.cymru: Adapt test_empty_result, remove test_unicode_as_name and test_country_question_mark (#1576).

Tools

• intelmq.bin.intelmq_gen_docs: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form.
• intelmq.bin.intelmqctl:
  • debug: In JSON mode, use dictionaries instead of lists.
  • debug: Add PATH to the paths shown.
  • check: Show $PATH environment variable if executable cannot be found.

Contrib

• malware_name_mapping: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.

Known issues

• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
• Corrupt dump files when interrupted during writing (#870).
• Bash completion scripts search in wrong directory in packages (#1561).
• Cymru Expert: Wrong Cache-Key Calculation (#1592).

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md

Dropped support for Python 3.4.

Core

• __init__: Changes to the path-handling, see User Guide, section /opt and LSB paths for more information
  • The environment variable INTELMQ_ROOT_DIR can be used to set custom root directories instead of /opt/intelmq/ (#805) in case of non LSB-path installations.
  • The environment variable ROOT_DIR can be used to set custom root directories instead of / (#805) in case of LSB-path installations.
• intelmq.lib.exceptions: Added MissingDependencyError for show error messages about a missing library and how to install it (#1471).
  • Added optional parameter installed to show the installed version.
  • Added optional parameter additional_text to show arbitrary text.
• Adding more type annotations for core libraries.
• intelmq.lib.pipeline.Pythonlist.sleep: Drop deprecated method.
• intelmq.lib.utils: write_configuration: Append a newline at end of configuration/file to allow proper comparisons & diffs.
• intelmq.lib.test: BotTestCase drops privileges upon initialization (#1489).
• intelmq.lib.bot:
  • New class OutputBot:
    • Method export_event to format/export events according to the parameters given by the user.
  • ParserBot: New methods parse_json_stream and recover_line_json_stream.
  • ParserBot.recover_line_json: Fix format by adding a list around the line data.
  • Bot.send_message: In debugging log level, the path to which the message is sent is now logged too.

Bots

• Bots with dependencies: Use of intelmq.lib.exceptions.MissingDependencyError.

Collectors

• intelmq.bots.collectors.misp.collector: Deprecate parameter misp_verify in favor of generic parameter http_verify_cert.
• intelmq.bots.collectors.tcp.collector: Drop compatibility with Python 3.4.
• intelmq.bots.collectors.stomp.collector:
  • Check the stomp.py version and show an error message if it does not match.
  • For stomp.py versions >= 5.0.0 redirect the stomp.PrintingListener output to debug logging.
• intelmq.bots.collectors.microsoft.collector_azure: Support current Python library azure-storage-blob>= 12.0.0, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
• intelmq.bots.collectors.amqp.collector_amqp: Require pika minimum version 1.0.
• intelmq.bots.collectors.github_api.collector_github_contents_api: Added (PR#1481).

Parsers

• intelmq.bots.parsers.autoshun.parser: Drop compatibility with Python 3.4.
• intelmq.bots.parsers.html_table.parser: Drop compatibility with Python 3.4.
• intelmq.bots.parsers.shadowserver.parser: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
• intelmq.bots.parsers.taichung.parser:
  • Migrate to ParserBot.
  • Also parse geolocation information if available.
• intelmq.bots.parsers.cymru.parser_full_bogons:
  • Migrate to ParserBot.
  • Add last updated information in raw.
• intelmq.bots.parsers.anubisnetworks.parser: Add new parameter use_malware_familiy_as_classification_identifier.
• intelmq.bots.parsers.microsoft.parser_ctip: Compatibility for new CTIP data format used provided by the Azure interface.
• intelmq.bots.parsers.cymru.parser_cap_program: Support for openresolver type.
• intelmq.bots.parsers.github_feed.parser: Added (PR#1481).
• intelmq.bots.parsers.urlvir.parser: Removed, as the feed is discontinued (#1537).

Experts

• intelmq.bots.experts.csv_converter: Added as converter to CSV.
• intelmq.bots.experts.misp: Added (PR#1475).
• intelmq.bots.experts.modify: New parameter maximum_matches.

Outputs

• intelmq.bots.outputs.amqptopic:
  • Use OutputBot and export_event.
  • Allow formatting the routing key with event data by the new parameter format_routing_key (boolean).
• intelmq.bots.outputs.file: Use OutputBot and export_event.
• intelmq.bots.outputs.files: Use OutputBot and export_event.
• intelmq.bots.outputs.misp.output_feed: Added, creates a MISP Feed (PR#1473).
• intelmq.bots.outputs.misp.output_api: Added, pushes to MISP via the API (PR#1506, PR#1536).
• intelmq.bots.outputs.elasticsearch.output: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513).

Documentation

• Document usage of the INTELMQ_ROOT_DIR environment variable.
• Added document on MISP integration possibilities.
• Feeds:
  • Added "Full Bogons IPv6" feed.
  • Remove discontinued URLVir Feeds (#1537).

Packaging

• setup.py do not try to install any data to /opt/intelmq/ as the behavior is inconsistent on various systems and with intelmqsetup we have a tool to create the structure and files anyway.
• debian/rules:
  • Provide a blank state file in the package.
• Patches:
  • Updated fix-intelmq-paths.patch.

Tests

• Travis: Use intelmqsetup here too.
  • Install required build dependencies for the Debian package build test.
  • This version is no longer automatically tested on Python < 3.5.
  • Also run the tests on Python 3.8.
  • Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
• Added tests for the new bot intelmq.bots.outputs.misp.output_feed (#1473).
• Added tests for the new bot intelmq.bots.experts.misp.expert (#1473).
• Added tests for intelmq.lib.exceptions.
• Added tests for intelmq.lib.bot.OutputBot and intelmq.lib.bot.OutputBot.export_event.
• Added IPv6 tests for intelmq.bots.parsers.cymru.parser_full_bogons.
• Added tests for intelmq.lib.bot.ParserBot's new methods parse_json_stream and recover_line_json_stream.
• intelmq.tests.test_conf: Set encoding to UTF-8 for reading the feeds.yaml file.

Tools

• intelmqctl:
  • upgrade-config:
    • Allow setting the state file location with the --state-file parameter.
    • Do not require a second run anymore, if the state file is newly created (#1491).
    • New parameter no_backup/--no-backup to skip creation of .bak files for state and configuration files.
  • Only require psutil for the IntelMQProcessManager, not for process manager independent calls like upgrade-config or check.
  • Add new command debug to output some information for debugging. Currently implemented:
    • paths
    • environment variables
  • IntelMQController: New argument --no-file-logging to disable logging to file.
  • If dropping privileges does not work, intelmqctl will now abort (#1489).
• intelmqsetup:
  • Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
  • Call intelmqctl upgrade-config and add argument for the state file path (#1491).
• intelmq_generate_misp_objects_templates.py: Tool to create a MISP object template (#1470).
• intelmqdump: New parameter -t or --truncate to optionally give the maximum length of raw data to show, 0 for no truncating.

Contrib

• Added development-tools.
• ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513).
• Malware Name Mapping Downloader:
  • New parameter --mwnmp-ignore-adware.
  • The parameter --add-default supports an optional parameter to define the default value.

Known issues

• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
• Corrupt dump files when interrupted during writing (#870).

Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md

Requirements

• The python library requests is (again) listed as dependency of the core (#1519).

Core

• intelmq.lib.upgrades:
  • Harmonization upgrade: Also check and update regular expressions.
  • Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
  • Add function to migrate changed Taichung URL feed.
  • Check for discontinued Abuse.CH Zeus Tracker feed.
• intelmq.lib.bot:
  • ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
  • start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
  • start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
  • _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
• intelmq.lib.test:
  • BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
  • Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
  • Fix test for (allowed) errors and warnings.
• intelmq.lib.exceptions:
  • InvalidKey: Add KeyError as parent class.
  • DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
• intelmq.lib.pipeline:
  • Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
• intelmq.lib.utils:
  • decode: Raise new DecodingError if decoding fails.

Harmonization

• protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).

Bots

Collectors

• intelmq.bots.collectors.mail.collector_mail_attach:
  • Fix handling of deprecated parameter name attach_unzip.
  • Fix handling of attachments without filenames (#1538).
• intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
• intelmq.bots.collectors.microsoft:
  • Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
  • intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.

Parsers

• intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
• intelmq.bots.parsers.taichung.parser: Support more types/classifications:
  • Application Compromise: Apache vulnerability & SQL injections
  • Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
  • C2 Sever: Attack controller
  • DDoS
  • DoS: DNS, DoS, Excess connection
  • IDS Alert / known vulnerability exploitation: backdoor
  • Malware: Malware Proxy
  • Warn on new unknown types.
• intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
• intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
• intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
• intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
• intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
• intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
• intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
• intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
• intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
• intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.

Experts

• intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.

Outputs

• intelmq.bots.outputs.mongodb:
  • Set default port 27017.
  • Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).

Documentation

• Feeds:
  • Remove unavailable feed Abuse.CH Zeus Tracker.
  • Remove the field status, offline feeds should be removed.
  • Add a new field public to differentiate between private and public feeds.
  • Adding documentation URLs to nearly all feeds.
  • Remove unavailable Bitcash.cz feed.
  • Remove unavailable Fraunhofer DDos Attack feeds.
  • Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
  • Update information on Bambenek Feeds, many require a license now (#1525).
  • Remove discontinued Nothink Honeypot Feeds (#1537).
• Developers Guide: Fix the instructions for /opt/intelmq file permissions.

Packaging

• Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
• Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
• Add runtime dependency debianutils for the program which, which is required for intelmqctl.

Tests

• Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
• intelmq.tests.bots.experts.cymru_whois:
  • Drop missing ASN test, does not work anymore.
  • IPv6 to IPv4 test: Test for two possible results.
• intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
• intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
• Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
• Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
• Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
• Travis: Explicitly enable RabbitMQ management plugin.
• intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).

Tools

• intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
• intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
• intelmqctl:
  • IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
  • IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
• intelmqdump:
  • Handle base64-type messages for show, editor and recovery actions.

Contrib

• intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, hadiasghari/pyasn#62).

Known issues

• HTTP stream collector: retry on regular connection problems? (#1435).
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
• Reverse DNS: Only first record is used (#877).
• Corrupt dump files when interrupted during writing (#870).

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md

Core

• __init__: Resolve absolute path for STATE_FILE_PATH variable (resolves ..).
• intelmq.lib.utils:
  • log: Do not raise an exception if logging to neither file nor syslog is requested.
  • logging StreamHandler: Colorize all warning and error messages red.
  • logging FileHandler: Strip all shell colorizations from the messages (#1436).
• intelmq.lib.message:
  • Message.to_json: Set sort_keys=True to get reproducible results.
  • drop_privileges: Handle situations where the user or group intelmq does not exist.
• intelmq.lib.pipeline:
  • Amqp._send and Amqp._acknowledge: Log traceback in debug mode in case of errors and necessary re-connections.
  • Amqp._acknowledge: Reset delivery tag if acknowledge was successful.

Bots

Collectors

• intelmq.bots.collectors.misp.collector:
  • Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).

Parsers

• intelmq.bots.parsers.shadowserver.config: Add some missing fields for the feed accessible-rdp (#1463).
• intelmq.bots.parsers.shadowserver.parser:
  • Feed-detection based on file names: The prefixed date is optional now.
  • Feed-detection based on file names: Re-detect feed for every report received (#1493).

Experts

• intelmq.bots.experts.national_cert_contact_certat: Handle empty responses by server (#1467).
• intelmq.bots.experts.maxmind_geoip: The script update-geoip-data now requires a license key as second parameter because of upstream changes (#1484)).

Outputs

• intelmq.bots.outputs.restapi.output: Fix logging of response body if response status code was not ok.

Documentation

• Remove some hardcoded /opt/intelmq/ paths from code comments and program outputs.

Packaging

• debian/rules: Only replace /opt/intelmq/ with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements.
• debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.

Tests

• lib/test_utils: Skip some tests on Python 3.4 because contextlib.redirect_stdout and contextlib.redirect_sterr are not supported on this version.
• Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
• tests.bots.parsers.html_table: Make tests independent of current year.

Tools

• intelmqctl upgrade-config: Fix missing substitution in error message "State file %r is not writable.".

Known issues

• bots trapped in endless loop if decoding of raw message fails (#1494)
• intelmqctl status of processes: need to check bot id too (#1492)
• MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
• ctl: shell colorizations are logged (#1436)
• http stream collector: retry on regular connection problems? (#1435)
• tests: capture logging with context manager (#1342)
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
• n6 parser: mapping is modified within each run (#905)
• reverse DNS: Only first record is used (#877)
• Corrupt dump files when interrupted during writing (#870)

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md

Configuration

• Default configuration:
  • Remove discontinued feed "Feodo Tracker Domains" from default configuration.
  • Add "Feodo Tracker Browse" feed to default configuration.

Core

• intelmq.lib.pipeline: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (intelmqctl_rabbitmq_monitoring_url).
• intelmq.lib.upgrades: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.
• intelmq.lib.utils:
  • TimeoutHTTPAdapter: A subclass of requests.adapters.HTTPAdapter with the possibility to set the timeout per adapter.
  • create_request_session_from_bot: Use the TimeoutHTTPAdapter with the user-defined timeout. Previously the timeout was not functional.

Bots

Parsers

• intelmq.bots.parsers.shadowserver.parser: Fix logging message if the parameter feedname is not present.
• intelmq.bots.parsers.shodan.parser: Also add field classification.identifier ('network-scan') in minimal mode.
• intelmq.bots.parsers.spamhaus.parser_cert: Add support for category 'misc'.
• intelmq.bots.parsers.cymru.parser_cap_program:
  • Add support for phishing events without URL.
  • Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
• intelmq.bots.parsers.microsoft.parser_bingmurls:
  • Save the Tags data as source.geolocation.cc.

Experts

• intelmq.bots.experts.modify.expert: Fix bug with setting non-string values (#1460).

Outputs

• intelmq.bots.outputs.smtp:
  • Allow non-existent field in text formatting by using a default value None instead of throwing errors.
  • Fix Authentication (#1464).
  • Fix sending to multiple recipients (#1464).

Documentation

• Feeds:
  • Fix configuration of Feodo Tracker Browse feed.
• Bots:
  • Sieve expert: Document behavior of != with lists.

Tests

• Adaption and extension of the test cases to the changes.

Tools

• intelmq.bin.intelmqctl:
  • check: Check if running the upgrade function for harmonization is necessary.
  • upgrade-config: Run the upgrade function for harmonization.
  • intelmqctl restart did throw an error as the message for restarting was not defined (#1465).

Known issues

• MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
• ctl: shell colorizations are logged (#1436)
• http stream collector: retry on regular connection problems? (#1435)
• tests: capture logging with context manager (#1342)
• Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
• n6 parser: mapping is modified within each run (#905)
• reverse DNS: Only first record is used (#877)
• Corrupt dump files when interrupted during writing (#870)