Skip to content

@wagner-certat wagner-certat released this Mar 4, 2021 · 51 commits to develop since this release

Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html

IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.

Configuration

Core

  • intelmq.lib.bot:
    • ParserBot.recover_line_json_stream: Make line parameter optional, as it is not needed for this method (by Sebastian Wagner).
    • Bot.argparser: Added class method _create_argparser (returns argparse.ArgumentParser) for easy command line arguments parsing (PR#1586 by Filip Pokorný).
    • Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorný).
    • Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
    • Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner).
  • intelmq.lib.upgrades:
    • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559, by Sebastian Wagner).
  • intelmq.lib.exceptions:
    • PipelineError: Remove unused code to format exceptions (by Sebastian Wagner).
  • intelmq.lib.utils:
    • create_request_session_from_bot:
      • Changed bot argument to optional, uses defaults.conf as fallback, renamed to create_request_session. Name create_request_session_from_bot will be removed in version 3.0.0 (PR#1524 by Filip Pokorný).
      • Fixed setting of http_verify_cert from defaults configuration (PR#1758 by Birger Schacht).
    • log: Use RotatingFileHandler for allow log file rotation without external tools (PR#1637 by Vasek Bruzek).
  • intelmq.lib.harmonization:
    • The IPAddress type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner).
    • DateTime.parse_utc_isoformat: Add parameter return_datetime to return datetime object instead of string ISO format (by Sebastian Wagner).
    • DateTime.convert: Fix utc_isoformat format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner).
    • DateTime.from_timestamp: Ensure that time zone information (+00:00) is always present (by Sebastian Wagner).
    • DateTime.__parse now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner).
  • intelmq.lib.upgrades:
    • Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner).
    • Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht).

Development

  • intelmq.bin.intelmq_gen_docs:
    • Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht).
    • Merged into docs/autogen.py (PR#1622 by Birger Schacht).

Bots

Collectors

  • intelmq.bots.collectors.eset.collector: Added (PR#1554 by Mikk Margus Möll).
  • intelmq.bots.collectors.http.collector_http:
    • Added PGP signature check functionality (PR#1602 by sinus-x).
    • If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner).
  • intelmq.bots.collectors.kafka.collector: Added (PR#1654 by Birger Schacht, closes #1634).
  • intelmq.bots.collectors.xmpp.collector: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
  • intelmq.bots.collectors.shadowserver.collector_api:
    • Added (#1683, PR#1700 by Birger Schacht).
    • Change file names in the report to .json instead of the original and wrong .csv (PR#1769 by Sebastian Wagner).
  • intelmq.bots.collectors.mail: Add content of the email's Date header as extra.email_date to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner).
  • intelmq.bots.collectors.http.collector_http_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
  • intelmq.bots.collectors.shodan.collector_stream: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).
  • intelmq.bots.collectors.twitter.collector_twitter:
    • Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner).
    • Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR#1754 by Sebastian Wagner).

Parsers

  • intelmq.bots.parsers.eset.parser: Added (PR#1554 by Mikk Margus Möll).
    • Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll).
  • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559, by Sebastian Wagner).
  • intelmq.bots.parsers.cznic.parser_haas: Added (PR#1560 by Filip Pokorný and Edvard Rejthar).
  • intelmq.bots.parsers.cznic.parser_proki: Added (PR#1599 by sinus-x).
  • intelmq.bots.parsers.key_value.parser: Added (PR#1607 by Karl-Johan Karlsson).
  • intelmq.bots.parsers.generic.parser_csv: Added new parameter compose_fields (by Sebastian Wagner).
  • intelmq.bots.parsers.shadowserver.parser_json: Added (PR#1700 by Birger Schacht).
  • intelmq.bots.parsers.shadowserver.config:
    • Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer).
    • Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer).
    • Ignore value 0 for source.asn and destination.asn in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner).
  • intelmq.bots.parsers.abusech.parser_ip: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus).
  • intelmq.bots.parsers.malwaredomainlist: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).

Experts

  • intelmq.bots.experts.rfc1918.expert:
    • Add support for ASNs (PR#1557 by Mladen Markovic).
    • Speed improvements.
    • More output in debug logging mode (by Sebastian Wagner).
    • Checks parameter length on initialization and in check method (by Sebastian Wagner).
  • intelmq.bots.experts.gethostbyname.expert:
    • Added parameter fallback_to_url and set to True (PR#1586 by Edvard Rejthar).
    • Added parameter gaierrors_to_ignore to optionally ignore other gethostbyname errors (#1553).
    • Added parameter overwrite to optionally overwrite existing IP addresses (by Sebastian Wagner).
  • intelmq.bots.experts.asn_lookup.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • The script update-asn-data is now deprecated and will be removed in version 3.0.
  • intelmq.bots.experts.maxmind_geoip.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • Added license_key parameter (PR#1524 by Filip Pokorný).
    • The script update-geoip-data is now deprecated and will be removed in version 3.0.
  • intelmq.bots.experts.tor_nodes.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • The script update-tor-nodes is now deprecated and will be removed in version 3.0.
  • intelmq.bots.experts.recordedfuture_iprisk.expert:
    • Added --update-database option (PR#1524 by Filip Pokorný).
    • Added api_token parameter (PR#1524 by Filip Pokorný).
    • The script update-rfiprisk-data is now deprecated and will be removed in version 3.0.
  • Added intelmq.bots.experts.threshold (PR#1608 by Karl-Johan Karlsson).
  • Added intelmq.bots.experts.splunk_saved_search.expert (PR#1666 by Karl-Johan Karlsson).
  • intelmq.bots.experts.sieve.expert:
    • Added possibility to give multiple queue names for the path directive (#1462, by Sebastian Wagner).
    • Added possibility to run actions without filtering expression (#1706, PR#1708 by Sebastian Waldbauer).
    • Added datetime math operations (#1680, PR#1696 by Sebastian Waldbauer).
  • intelmq.bots.experts.maxmind_geoip.expert:
    • Fixed handing over of overwrite parameter to event.add (PR#1743 by Birger Schacht).

Outputs

  • intelmq.bots.outputs.rt: Added Request Tracker output bot (PR#1589 by Marius Urkis).
  • intelmq.bots.outputs.xmpp.output: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).
  • intelmq.bots.outputs.smtp.output: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner).

Documentation

  • Feeds:
    • Add ESET URL and Domain feeds (by Sebastian Wagner).
    • Remove unavailable HPHosts Hosts file feed (#1559 by Sebastian Wagner).
    • Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar).
    • Added CZ.NIC Proki feed (PR#1599 by sinus-x).
    • Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný).
    • Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner).
    • Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner).
    • Fixed parsing of the public field in the generated feeds documentation (PR#1641 by Birger Schacht).
    • Change the rate_limit parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds).
    • Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorný and Sebastian Wagner).
    • Added Shadowserver Reports API (by Sebastian Wagner).
    • Change the rate_limit parameter for many feeds from 2 days to the default one day (by Sebastian Wagner).
    • Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
  • Bots:
    • Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner).
    • Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar).
    • Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new --update-database option (PR#1524 by Filip Pokorný).
    • Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner).
  • Add n6 integration documentation (by Sebastian Wagner).
  • Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner).
  • Generate documentation using Sphinx (PR#1622 by Birger Schacht).
  • Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht).

Packaging

  • Fix paths in the packaged logcheck rules (by Sebastian Wagner).
  • Build the sphinx documentation on package build (PR#1701 by Birger Schacht).
  • Ignore non-zero exit-codes for the intelmqctl check call in postinst (#1748, by Sebastian Wagner).

Tests

  • Added tests for intelmq.lib.exceptions.PipelineError (by Sebastian Wagner).
  • intelmq.tests.bots.collectors.http_collector.test_collector: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
  • intelmq.tests.bots.outputs.restapi.test_output:
    • Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
    • Add a test for checking the response status code (by Sebastian Wagner).
  • intelmq.tests.bots.collectors.mail.test_collector_url: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
  • intelmq.tests.bots.experts.ripe.test_expert: Use requests_mock to mock all requests and do not require a local webserver (by Sebastian Wagner).
  • The test flag (environment variable) INTELMQ_TEST_LOCAL_WEB is no longer used (by Sebastian Wagner).
  • Added tests for intelmq.harmonization.DateTime.parse_utc_isoformat and convert_fuzzy (by Sebastian Wagner).
  • Move from Travis to GitHub Actions (PR#1707 by Birger Schacht).
  • intelmq.lib.test:
    • test_static_bot_check_method checks the bot's static check(parameters) method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner).
    • setUpClass: Skip tests if cache was requests with use_cache member, but Redis is deactivated with the environment variable INTELMQ_SKIP_REDIS (by Sebastian Wagner).
  • intelmq.tests.bots.experts.cymru_whois.test_expert:
    • Switch from example.com to ns2.univie.ac.at for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer).
    • Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner).
  • intelmq.tests.bots.parsers.abusech: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus).
  • Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner).

Tools

  • intelmqdump:
    • Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic).
  • intelmqctl:
    • intelmq list queues: --sum, --count, -s flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic).
    • intelmq check: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner).
    • Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).

Contrib

  • EventDB:
    • Add SQL script for keeping track of the oldest inserted/update "time.source" information (by Sebastian Wagner).
  • Cron Jobs: The script intelmq-update-data has been renamed to intelmq-update-database (by Filip Pokorný).
  • Dropped utterly outdated contrib modules (by Sebastian Wagner):
    • ansible
    • vagrant
    • vagrant-ansible
  • logrotate:
    • Do not use the deprecated "copytruncate" option as intelmq re-opens the log anyways (by Sebastian Wagner).
    • Set file permissions to 0644 (by Sebastian Wagner).

Known issues

  • Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
  • Corrupt dump files when interrupted during writing (#870).
  • CSV line recovery forces Windows line endings (#1597).
  • intelmqdump: Honor logging_path variable (#1605).
  • Timeout error in mail URL fetcher (#1621).
  • AMQP pipeline: get_queues needs to check vhost of response (#1746).
Assets 2
Pre-release

@wagner-certat wagner-certat released this Feb 19, 2021 · 87 commits to develop since this release

2.3.0.rc1

2.3.0 Release candidate 1
Assets 2

@wagner-certat wagner-certat released this Dec 23, 2020 · 598 commits to develop since this release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/UPGRADING.md

Documentation

  • Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht).

Harmonization

  • See NEWS.md for information on a fixed bug in the taxonomy expert.

Bots

Collectors

  • intelmq.bots.rt.collector_rt: Log the size of the downloaded file in bytes on debug logging level.

Parsers

  • intelmq.bots.parsers.cymru.parser_cap_program:
    • Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt).
    • Add support for field additional_asns in optional information column.
  • intelmq.bots.parsers.microsoft.parser_ctip:
    • Fix mapping of DestinationIpInfo.DestinationIpConnectionType field (contained a typo).
    • Explicitly ignore field DestinationIpInfo.DestinationIpv4Int as the data is already in another field.
  • intelmq.bots.parsers.generic.parser_csv:
    • Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh).
    • Data fields containing - are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer).

Experts

  • intelmq.bots.experts.taxonomy.expert: Map type scanner to information-gathering instead of information gathering. See NEWS file for more information.

Tests

  • Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython.

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2

@wagner-certat wagner-certat released this Oct 28, 2020 · 618 commits to develop since this release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md

Core

  • intelmq.lib.upgrades:
    • Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist".

Bots

Parsers

  • intelmq.bots.parsers.shadowserver:
    • Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
    • Added support for the feeds Accessible Radmin and CAIDA IP Spoofer (PR#1600 by sinus-x).
  • intelmq.bots.parsers.anubisnetworks.parser: Fix parsing error where dst.ip was not equal to comm.http.host.
  • intelmq/bots/parsers/danger_rulez/parser: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus).
  • `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23).
  • intelmq.bots.parsers.microsoft.parser_ctip:
    • Add support for DestinationIpInfo.* and Signatures.Sha256 fields, used by the ctip-c2 feed (PR#1623 by Mikk Margus Möll).
    • Use extra.payload.text for the feed's field Payload if the content cannot be decoded (PR#1610 by Giedrius Ramas).

Experts

  • intelmq.bots.experts.cymru_whois:
    • Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606).
    • The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606).

Documentation

  • README:
    • Add Core Infrastructure Initiative Best Practices Badge.
  • Bots:
    • Generic CSV Parser: Add note on escaping backslashes (#1579).
    • Remove section of non-existing "Copy Extra" Bot.
    • Explain taxonomy expert.
    • Add documentation on n6 parser.
    • Gethostbyname expert: Add documentation how errors are treated.
  • Feeds:
    • Fixed bot modules of Calidog CertStream feed.
    • Add information on Microsoft CTIP C2 feed.

Packaging

  • In Debian packages, intelmqctl check and intelmqctl upgrade-config are executed in the postinst step (#1551, PR#1624 by Birger Schacht).

Tests

  • intelmq.tests.lib.test_pipeline: Skip TestAmqp.test_acknowledge on Travis with Python 3.8.
  • intelmq.tests.bots.outputs.elasticsearch.test_output: Refresh index intelmq manually to fix random test failures (#1593, PR#1595 by Zach Stone).

Tools

  • intelmqctl check:
    • For disabled bots which do not have any pipeline connections, do not raise an error, but only warning.
    • Fix check on source/destination queues for bots as well the orphaned queues.

Contrib

  • Bash completion scripts: Check both /opt/intelmq/ as well as LSB-paths (/etc/intelmq/ and /var/log/intelmq/) for loading bot information (#1561, PR#1628 by Birger Schacht).

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2

@wagner-certat wagner-certat released this Jul 30, 2020 · 673 commits to develop since this release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md

Core

  • intelmq.lib.upgrades:
    • Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
    • Add upgrade function for removal of HPHosts Hosts file feed and intelmq.bots.parsers.hphosts parser (#1559).
    • intelmq.lib.harmonization:
      • For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).

Development

  • Ignore line length (E501) in code-style checks altogether.

Bots

Collectors

  • intelmq.bots.collectors.misp: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)
  • intelmq.bots.collectors.stomp: Remove empty client.pem file.

Parsers

  • intelmq.bots.parsers.shadowserver.config:
    • Add support for Accessible-CoAP feed (PR #1555 by Thomas Hungenberg).
    • Add support for Accessible-ARD feed (PR #1584 by Tomas Bellus @tomas321).
  • intelmq.bots.parser.anubisnetworks.parser: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all.
  • intelmq.bots.parsers.generic.parser_csv: Allow values of type dictionary for parameter type_translation.
  • intelmq.bots.parsers.hphosts: Removed, feed is unavailable (#1559).
  • intelmq.bots.parsers.cymru.parser_cap_program: Add support for comment "username" for "scanner" category.
  • intelmq.bots.parsers.malwareurl.parser: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).

Experts

  • intelmq.bots.experts.maxmind_geoip: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.

Outputs

  • intelmq.bot.outputs.udp: Fix error handling on sending, had a bug itself.

Documentation

  • Feeds:
    • Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
  • Bots:
    • Overhaul of all bots' description fields (#1570).
  • User-Guide:
    • Overhaul pipeline configuration section and explain named queues better (#1577).

Tests

  • intelmq.tests.bots.experts.cymru: Adapt test_empty_result, remove test_unicode_as_name and test_country_question_mark (#1576).

Tools

  • intelmq.bin.intelmq_gen_docs: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form.
  • intelmq.bin.intelmqctl:
    • debug: In JSON mode, use dictionaries instead of lists.
    • debug: Add PATH to the paths shown.
    • check: Show $PATH environment variable if executable cannot be found.

Contrib

  • malware_name_mapping: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
  • Bash completion scripts search in wrong directory in packages (#1561).
  • Cymru Expert: Wrong Cache-Key Calculation (#1592).
Assets 2

@wagner-certat wagner-certat released this Jun 18, 2020 · 721 commits to develop since this release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md

Dropped support for Python 3.4.

Core

  • __init__: Changes to the path-handling, see User Guide, section /opt and LSB paths for more information
    • The environment variable INTELMQ_ROOT_DIR can be used to set custom root directories instead of /opt/intelmq/ (#805) in case of non LSB-path installations.
    • The environment variable ROOT_DIR can be used to set custom root directories instead of / (#805) in case of LSB-path installations.
  • intelmq.lib.exceptions: Added MissingDependencyError for show error messages about a missing library and how to install it (#1471).
    • Added optional parameter installed to show the installed version.
    • Added optional parameter additional_text to show arbitrary text.
  • Adding more type annotations for core libraries.
  • intelmq.lib.pipeline.Pythonlist.sleep: Drop deprecated method.
  • intelmq.lib.utils: write_configuration: Append a newline at end of configuration/file to allow proper comparisons & diffs.
  • intelmq.lib.test: BotTestCase drops privileges upon initialization (#1489).
  • intelmq.lib.bot:
    • New class OutputBot:
      • Method export_event to format/export events according to the parameters given by the user.
    • ParserBot: New methods parse_json_stream and recover_line_json_stream.
    • ParserBot.recover_line_json: Fix format by adding a list around the line data.
    • Bot.send_message: In debugging log level, the path to which the message is sent is now logged too.

Bots

  • Bots with dependencies: Use of intelmq.lib.exceptions.MissingDependencyError.

Collectors

  • intelmq.bots.collectors.misp.collector: Deprecate parameter misp_verify in favor of generic parameter http_verify_cert.
  • intelmq.bots.collectors.tcp.collector: Drop compatibility with Python 3.4.
  • intelmq.bots.collectors.stomp.collector:
    • Check the stomp.py version and show an error message if it does not match.
    • For stomp.py versions >= 5.0.0 redirect the stomp.PrintingListener output to debug logging.
  • intelmq.bots.collectors.microsoft.collector_azure: Support current Python library azure-storage-blob>= 12.0.0, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
  • intelmq.bots.collectors.amqp.collector_amqp: Require pika minimum version 1.0.
  • intelmq.bots.collectors.github_api.collector_github_contents_api: Added (PR#1481).

Parsers

  • intelmq.bots.parsers.autoshun.parser: Drop compatibility with Python 3.4.
  • intelmq.bots.parsers.html_table.parser: Drop compatibility with Python 3.4.
  • intelmq.bots.parsers.shadowserver.parser: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
  • intelmq.bots.parsers.taichung.parser:
    • Migrate to ParserBot.
    • Also parse geolocation information if available.
  • intelmq.bots.parsers.cymru.parser_full_bogons:
    • Migrate to ParserBot.
    • Add last updated information in raw.
  • intelmq.bots.parsers.anubisnetworks.parser: Add new parameter use_malware_familiy_as_classification_identifier.
  • intelmq.bots.parsers.microsoft.parser_ctip: Compatibility for new CTIP data format used provided by the Azure interface.
  • intelmq.bots.parsers.cymru.parser_cap_program: Support for openresolver type.
  • intelmq.bots.parsers.github_feed.parser: Added (PR#1481).
  • intelmq.bots.parsers.urlvir.parser: Removed, as the feed is discontinued (#1537).

Experts

  • intelmq.bots.experts.csv_converter: Added as converter to CSV.
  • intelmq.bots.experts.misp: Added (PR#1475).
  • intelmq.bots.experts.modify: New parameter maximum_matches.

Outputs

  • intelmq.bots.outputs.amqptopic:
    • Use OutputBot and export_event.
    • Allow formatting the routing key with event data by the new parameter format_routing_key (boolean).
  • intelmq.bots.outputs.file: Use OutputBot and export_event.
  • intelmq.bots.outputs.files: Use OutputBot and export_event.
  • intelmq.bots.outputs.misp.output_feed: Added, creates a MISP Feed (PR#1473).
  • intelmq.bots.outputs.misp.output_api: Added, pushes to MISP via the API (PR#1506, PR#1536).
  • intelmq.bots.outputs.elasticsearch.output: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513).

Documentation

  • Document usage of the INTELMQ_ROOT_DIR environment variable.
  • Added document on MISP integration possibilities.
  • Feeds:
    • Added "Full Bogons IPv6" feed.
    • Remove discontinued URLVir Feeds (#1537).

Packaging

  • setup.py do not try to install any data to /opt/intelmq/ as the behavior is inconsistent on various systems and with intelmqsetup we have a tool to create the structure and files anyway.
  • debian/rules:
    • Provide a blank state file in the package.
  • Patches:
    • Updated fix-intelmq-paths.patch.

Tests

  • Travis: Use intelmqsetup here too.
    • Install required build dependencies for the Debian package build test.
    • This version is no longer automatically tested on Python < 3.5.
    • Also run the tests on Python 3.8.
    • Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
  • Added tests for the new bot intelmq.bots.outputs.misp.output_feed (#1473).
  • Added tests for the new bot intelmq.bots.experts.misp.expert (#1473).
  • Added tests for intelmq.lib.exceptions.
  • Added tests for intelmq.lib.bot.OutputBot and intelmq.lib.bot.OutputBot.export_event.
  • Added IPv6 tests for intelmq.bots.parsers.cymru.parser_full_bogons.
  • Added tests for intelmq.lib.bot.ParserBot's new methods parse_json_stream and recover_line_json_stream.
  • intelmq.tests.test_conf: Set encoding to UTF-8 for reading the feeds.yaml file.

Tools

  • intelmqctl:
    • upgrade-config:
      • Allow setting the state file location with the --state-file parameter.
      • Do not require a second run anymore, if the state file is newly created (#1491).
      • New parameter no_backup/--no-backup to skip creation of .bak files for state and configuration files.
    • Only require psutil for the IntelMQProcessManager, not for process manager independent calls like upgrade-config or check.
    • Add new command debug to output some information for debugging. Currently implemented:
      • paths
      • environment variables
    • IntelMQController: New argument --no-file-logging to disable logging to file.
    • If dropping privileges does not work, intelmqctl will now abort (#1489).
  • intelmqsetup:
    • Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
    • Call intelmqctl upgrade-config and add argument for the state file path (#1491).
  • intelmq_generate_misp_objects_templates.py: Tool to create a MISP object template (#1470).
  • intelmqdump: New parameter -t or --truncate to optionally give the maximum length of raw data to show, 0 for no truncating.

Contrib

  • Added development-tools.
  • ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513).
  • Malware Name Mapping Downloader:
    • New parameter --mwnmp-ignore-adware.
    • The parameter --add-default supports an optional parameter to define the default value.

Known issues

  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2
May 30, 2020
2.2.0 Release candidate 1

@wagner-certat wagner-certat released this May 26, 2020 · 1033 commits to develop since this release

Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md

Requirements

  • The python library requests is (again) listed as dependency of the core (#1519).

Core

  • intelmq.lib.upgrades:
    • Harmonization upgrade: Also check and update regular expressions.
    • Add function to migrate the deprecated parameter attach_unzip to extract_files for the mail attachment collector.
    • Add function to migrate changed Taichung URL feed.
    • Check for discontinued Abuse.CH Zeus Tracker feed.
  • intelmq.lib.bot:
    • ParserBot.recover_line: Parameter line needs to be optional, fix usage of fallback value self.current_line.
    • start: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).
    • start: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors.
    • _dump_message: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
  • intelmq.lib.test:
    • BotTestCase.run_bot: Add parameters allowed_error_count and allowed_warning_count to allow set the number per run, not per test class.
    • Set source_pipeline_broker and destination_pipeline_broker to pythonlist instead of the old broker, fixes intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising.
    • Fix test for (allowed) errors and warnings.
  • intelmq.lib.exceptions:
    • InvalidKey: Add KeyError as parent class.
    • DecodingError: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
  • intelmq.lib.pipeline:
    • Decode messages in Pipeline.receive not in the implementation's _receive so that the internal counter is correct in case of decoding errors (#1494).
  • intelmq.lib.utils:
    • decode: Raise new DecodingError if decoding fails.

Harmonization

  • protocol.transport: Adapt regular expression to allow the value nvp-ii (protocol 11).

Bots

Collectors

  • intelmq.bots.collectors.mail.collector_mail_attach:
    • Fix handling of deprecated parameter name attach_unzip.
    • Fix handling of attachments without filenames (#1538).
  • intelmq.bots.collectors.stomp.collector: Fix compatibility with stomp.py versions > 4.1.20 and catch errors on shutdown.
  • intelmq.bots.collectors.microsoft:
    • Update REQUIREMENTS.txt temporarily fixing deprecated Azure library (#1530, PR#1532).
    • intelmq.bots.collectors.microsoft.collector_interflow: Add method for printing the file list.

Parsers

  • intelmq.bots.parsers.cymru.parser_cap_program: Support for protocol 11 (nvp-ii) and conficker type.
  • intelmq.bots.parsers.taichung.parser: Support more types/classifications:
    • Application Compromise: Apache vulnerability & SQL injections
    • Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
    • C2 Sever: Attack controller
    • DDoS
    • DoS: DNS, DoS, Excess connection
    • IDS Alert / known vulnerability exploitation: backdoor
    • Malware: Malware Proxy
    • Warn on new unknown types.
  • intelmq.bots.parsers.bitcash.parser: Removed as feed is discontinued.
  • intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc and intelmq.bots.parsers.fraunhofer.parser_ddosattack_target: Removed as feed is discontinued.
  • intelmq.bots.parsers.malwaredomains.parser: Correctly classify C&C and phishing events.
  • intelmq.bots.parsers.shadowserver.parser: More verbose error message for missing report specification (#1507).
  • intelmq.bots.parsers.n6.parser_n6stomp: Always add n6 field name as malware.name independent of category.
  • intelmq.bots.parsers.anubisnetworks: Update parser with new data format.
  • intelmq.bots.parsers.bambenek: Add new feed URLs with Host faf.bambenekconsulting.com (#1525, PR#1526).
  • intelmq.bots.parsers.abusech.parser_ransomware: Removed, as the feed is discontinued (#1537).
  • intelmq.bots.parsers.nothink.parser: Removed, as the feed is discontinued (#1537).
  • intelmq.bots.parsers.n6.parser: Remove not allowed characters in the name field for malware.name and write original value to event_description.text instead.

Experts

  • intelmq.bots.experts.cymru_whois.lib: Fix parsing of AS names with Unicode characters.

Outputs

  • intelmq.bots.outputs.mongodb:
    • Set default port 27017.
    • Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).

Documentation

  • Feeds:
    • Remove unavailable feed Abuse.CH Zeus Tracker.
    • Remove the field status, offline feeds should be removed.
    • Add a new field public to differentiate between private and public feeds.
    • Adding documentation URLs to nearly all feeds.
    • Remove unavailable Bitcash.cz feed.
    • Remove unavailable Fraunhofer DDos Attack feeds.
    • Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
    • Update information on Bambenek Feeds, many require a license now (#1525).
    • Remove discontinued Nothink Honeypot Feeds (#1537).
  • Developers Guide: Fix the instructions for /opt/intelmq file permissions.

Packaging

  • Patches: fix-logrotate-path.patch: also include path to rotated file in patch.
  • Fix paths from /opt to LSB for setup.py and contrib/logrotate/intelmq in build process (#1500).
  • Add runtime dependency debianutils for the program which, which is required for intelmqctl.

Tests

  • Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
  • intelmq.tests.bots.experts.cymru_whois:
    • Drop missing ASN test, does not work anymore.
    • IPv6 to IPv4 test: Test for two possible results.
  • intelmq.lib.test: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).
  • intelmq.bots.collectors.tcp.test_collector: Removing custom mocking and bot starting, not necessary anymore.
  • Added tests for intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline.
  • Fix and split tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json.
  • Added tests for invalid encodings in input messages in intelmq.tests.lib.test_bot and intelmq.tests.lib.test_pipeline (#1494).
  • Travis: Explicitly enable RabbitMQ management plugin.
  • intelmq.tests.lib.test_message: Fix usage of the parameter blacklist for Message hash tests (#1539).

Tools

  • intelmqsetup: Copy missing BOTS file to IntelMQ's root directory (#1498).
  • intelmq_gen_docs: Feed documentation generation: Handle missing/empty parameters.
  • intelmqctl:
    • IntelMQProcessManager: For the status of running bots also check the bot ID of the commandline and ignore the path of the executable (#1492).
    • IntelMQController: Fix exit codes of check command for JSON output (now 0 on success and 1 on error, was swapped, #1520).
  • intelmqdump:
    • Handle base64-type messages for show, editor and recovery actions.

Contrib

  • intelmq/bots/experts/asn_lookup/update-asn-data: Use pyasn_util_download.py to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, hadiasghari/pyasn#62).

Known issues

  • HTTP stream collector: retry on regular connection problems? (#1435).
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952).
  • Reverse DNS: Only first record is used (#877).
  • Corrupt dump files when interrupted during writing (#870).
Assets 2

@wagner-certat wagner-certat released this Jan 28, 2020 · 1127 commits to develop since this release

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md

Core

  • __init__: Resolve absolute path for STATE_FILE_PATH variable (resolves ..).
  • intelmq.lib.utils:
    • log: Do not raise an exception if logging to neither file nor syslog is requested.
    • logging StreamHandler: Colorize all warning and error messages red.
    • logging FileHandler: Strip all shell colorizations from the messages (#1436).
  • intelmq.lib.message:
    • Message.to_json: Set sort_keys=True to get reproducible results.
    • drop_privileges: Handle situations where the user or group intelmq does not exist.
  • intelmq.lib.pipeline:
    • Amqp._send and Amqp._acknowledge: Log traceback in debug mode in case of errors and necessary re-connections.
    • Amqp._acknowledge: Reset delivery tag if acknowledge was successful.

Bots

Collectors

  • intelmq.bots.collectors.misp.collector:
    • Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).

Parsers

  • intelmq.bots.parsers.shadowserver.config: Add some missing fields for the feed accessible-rdp (#1463).
  • intelmq.bots.parsers.shadowserver.parser:
    • Feed-detection based on file names: The prefixed date is optional now.
    • Feed-detection based on file names: Re-detect feed for every report received (#1493).

Experts

  • intelmq.bots.experts.national_cert_contact_certat: Handle empty responses by server (#1467).
  • intelmq.bots.experts.maxmind_geoip: The script update-geoip-data now requires a license key as second parameter because of upstream changes (#1484)).

Outputs

  • intelmq.bots.outputs.restapi.output: Fix logging of response body if response status code was not ok.

Documentation

  • Remove some hardcoded /opt/intelmq/ paths from code comments and program outputs.

Packaging

  • debian/rules: Only replace /opt/intelmq/ with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements.
  • debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.

Tests

  • lib/test_utils: Skip some tests on Python 3.4 because contextlib.redirect_stdout and contextlib.redirect_sterr are not supported on this version.
  • Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
  • tests.bots.parsers.html_table: Make tests independent of current year.

Tools

  • intelmqctl upgrade-config: Fix missing substitution in error message "State file %r is not writable.".

Known issues

  • bots trapped in endless loop if decoding of raw message fails (#1494)
  • intelmqctl status of processes: need to check bot id too (#1492)
  • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
  • ctl: shell colorizations are logged (#1436)
  • http stream collector: retry on regular connection problems? (#1435)
  • tests: capture logging with context manager (#1342)
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
  • n6 parser: mapping is modified within each run (#905)
  • reverse DNS: Only first record is used (#877)
  • Corrupt dump files when interrupted during writing (#870)
Assets 2

@wagner-certat wagner-certat released this Nov 11, 2019 · 1183 commits to develop since this release

Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md

Configuration

  • Default configuration:
    • Remove discontinued feed "Feodo Tracker Domains" from default configuration.
    • Add "Feodo Tracker Browse" feed to default configuration.

Core

  • intelmq.lib.pipeline: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (intelmqctl_rabbitmq_monitoring_url).
  • intelmq.lib.upgrades: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.
  • intelmq.lib.utils:
    • TimeoutHTTPAdapter: A subclass of requests.adapters.HTTPAdapter with the possibility to set the timeout per adapter.
    • create_request_session_from_bot: Use the TimeoutHTTPAdapter with the user-defined timeout. Previously the timeout was not functional.

Bots

Parsers

  • intelmq.bots.parsers.shadowserver.parser: Fix logging message if the parameter feedname is not present.
  • intelmq.bots.parsers.shodan.parser: Also add field classification.identifier ('network-scan') in minimal mode.
  • intelmq.bots.parsers.spamhaus.parser_cert: Add support for category 'misc'.
  • intelmq.bots.parsers.cymru.parser_cap_program:
    • Add support for phishing events without URL.
    • Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
  • intelmq.bots.parsers.microsoft.parser_bingmurls:
    • Save the Tags data as source.geolocation.cc.

Experts

  • intelmq.bots.experts.modify.expert: Fix bug with setting non-string values (#1460).

Outputs

  • intelmq.bots.outputs.smtp:
    • Allow non-existent field in text formatting by using a default value None instead of throwing errors.
    • Fix Authentication (#1464).
    • Fix sending to multiple recipients (#1464).

Documentation

  • Feeds:
    • Fix configuration of Feodo Tracker Browse feed.
  • Bots:
    • Sieve expert: Document behavior of != with lists.

Tests

  • Adaption and extension of the test cases to the changes.

Tools

  • intelmq.bin.intelmqctl:
    • check: Check if running the upgrade function for harmonization is necessary.
    • upgrade-config: Run the upgrade function for harmonization.
    • intelmqctl restart did throw an error as the message for restarting was not defined (#1465).

Known issues

  • MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
  • ctl: shell colorizations are logged (#1436)
  • http stream collector: retry on regular connection problems? (#1435)
  • tests: capture logging with context manager (#1342)
  • Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
  • n6 parser: mapping is modified within each run (#905)
  • reverse DNS: Only first record is used (#877)
  • Corrupt dump files when interrupted during writing (#870)
Assets 2