wagner-certat
released this
Installation documentation:
https://intelmq.readthedocs.io/en/maintenance/user/installation.html
Upgrade documentation:
https://intelmq.readthedocs.io/en/maintenance/user/upgrade.html
IntelMQ no longer supports Python 3.5 (and thus Debian 9 and Ubuntu 16.04), the minimum supported Python version is 3.6.
Configuration
Core
intelmq.lib.bot
:ParserBot.recover_line_json_stream
: Makeline
parameter optional, as it is not needed for this method (by Sebastian Wagner).Bot.argparser
: Added class method_create_argparser
(returnsargparse.ArgumentParser
) for easy command line arguments parsing (PR#1586 by Filip Pokorný).- Runtime configuration does not necessarily need a parameter entry for each block. Previously at least an empty block was required (PR#1604 by Filip Pokorný).
- Allow setting the pipeline host and the Redis cache host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
- Better logging message for SIGHUP handling if the handling of the signal is not delayed (by Sebastian Wagner).
intelmq.lib.upgrades
:- Add upgrade function for removal of HPHosts Hosts file feed and
intelmq.bots.parsers.hphosts
parser (#1559, by Sebastian Wagner).
- Add upgrade function for removal of HPHosts Hosts file feed and
intelmq.lib.exceptions
:PipelineError
: Remove unused code to format exceptions (by Sebastian Wagner).
intelmq.lib.utils
:create_request_session_from_bot
:- Changed bot argument to optional, uses defaults.conf as fallback, renamed to
create_request_session
. Namecreate_request_session_from_bot
will be removed in version 3.0.0 (PR#1524 by Filip Pokorný). - Fixed setting of
http_verify_cert
from defaults configuration (PR#1758 by Birger Schacht).
- Changed bot argument to optional, uses defaults.conf as fallback, renamed to
log
: UseRotatingFileHandler
for allow log file rotation without external tools (PR#1637 by Vasek Bruzek).
intelmq.lib.harmonization
:- The
IPAddress
type sanitation now accepts integer IP addresses and converts them to the string representation (by Sebastian Wagner). DateTime.parse_utc_isoformat
: Add parameterreturn_datetime
to returndatetime
object instead of string ISO format (by Sebastian Wagner).DateTime.convert
: Fixutc_isoformat
format, it pointed to a string and not a function, causing an exception when used (by Sebastian Wagner).DateTime.from_timestamp
: Ensure that time zone information (+00:00
) is always present (by Sebastian Wagner).DateTime.__parse
now handles OverflowError exceptions from the dateutil library, happens for large numbers, e.g. telehpone numbers (by Sebastian Wagner).
- The
intelmq.lib.upgrades
:- Added upgrade function for CSV parser parameter misspelling (by Sebastian Wagner).
- Check for existence of collector and parser for the obsolete Malware Domain List feed and raise warning if found (#1762, PR#1771 by Birger Schacht).
Development
intelmq.bin.intelmq_gen_docs
:- Add bot name to the resulting feed documentation (PR#1617 by Birger Schacht).
- Merged into
docs/autogen.py
(PR#1622 by Birger Schacht).
Bots
Collectors
intelmq.bots.collectors.eset.collector
: Added (PR#1554 by Mikk Margus Möll).intelmq.bots.collectors.http.collector_http
:- Added PGP signature check functionality (PR#1602 by sinus-x).
- If status code is not 2xx, the request's and response's headers and body are logged in debug logging level (#1615, by Sebastian Wagner).
intelmq.bots.collectors.kafka.collector
: Added (PR#1654 by Birger Schacht, closes #1634).intelmq.bots.collectors.xmpp.collector
: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).intelmq.bots.collectors.shadowserver.collector_api
:- Added (#1683, PR#1700 by Birger Schacht).
- Change file names in the report to
.json
instead of the original and wrong.csv
(PR#1769 by Sebastian Wagner).
intelmq.bots.collectors.mail
: Add content of the email'sDate
header asextra.email_date
to the report in all email collectors (PR#1749 by aleksejsv and Sebastian Wagner).intelmq.bots.collectors.http.collector_http_stream
: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).intelmq.bots.collectors.shodan.collector_stream
: Retry on common connection issues without raising exceptions (#1435, PR#1747 by Sebastian Waldbauer and Sebastian Wagner).intelmq.bots.collectors.twitter.collector_twitter
:- Proper input validation in URLs using urllib. CWE-20, found by GitHub's CodeQL (PR#1754 by Sebastian Wagner).
- Limit replacement ("pastebin.com", "pastebin.com/raw") to a maximum of one (PR#1754 by Sebastian Wagner).
Parsers
intelmq.bots.parsers.eset.parser
: Added (PR#1554 by Mikk Margus Möll).- Ignore invalid "NXDOMAIN" IP addresses (PR#1573 by Mikk Margus Möll).
intelmq.bots.parsers.hphosts
: Removed, feed is unavailable (#1559, by Sebastian Wagner).intelmq.bots.parsers.cznic.parser_haas
: Added (PR#1560 by Filip Pokorný and Edvard Rejthar).intelmq.bots.parsers.cznic.parser_proki
: Added (PR#1599 by sinus-x).intelmq.bots.parsers.key_value.parser
: Added (PR#1607 by Karl-Johan Karlsson).intelmq.bots.parsers.generic.parser_csv
: Added new parametercompose_fields
(by Sebastian Wagner).intelmq.bots.parsers.shadowserver.parser_json
: Added (PR#1700 by Birger Schacht).intelmq.bots.parsers.shadowserver.config
:- Fixed mapping for Block list feed to accept network ranges in CIDR notation (#1720, PR#1728 by Sebastian Waldbauer).
- Added mapping for new feed MSRDPUDP, Vulnerable-HTTP, Sinkhole DNS (#1716, #1726, #1733, PR#1732, PR#1735, PR#1736 by Sebastian Waldbauer).
- Ignore value
0
forsource.asn
anddestination.asn
in all mappings to avoid parsing errors (PR#1769 by Sebastian Wagner).
intelmq.bots.parsers.abusech.parser_ip
: Adapt to changes in the Feodo Tracker Botnet C2 IP Blocklist feed (PR#1741 by Thomas Bellus).intelmq.bots.parsers.malwaredomainlist
: Removed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
Experts
intelmq.bots.experts.rfc1918.expert
:- Add support for ASNs (PR#1557 by Mladen Markovic).
- Speed improvements.
- More output in debug logging mode (by Sebastian Wagner).
- Checks parameter length on initialization and in check method (by Sebastian Wagner).
intelmq.bots.experts.gethostbyname.expert
:- Added parameter
fallback_to_url
and set to True (PR#1586 by Edvard Rejthar). - Added parameter
gaierrors_to_ignore
to optionally ignore othergethostbyname
errors (#1553). - Added parameter
overwrite
to optionally overwrite existing IP addresses (by Sebastian Wagner).
- Added parameter
intelmq.bots.experts.asn_lookup.expert
:- Added
--update-database
option (PR#1524 by Filip Pokorný). - The script
update-asn-data
is now deprecated and will be removed in version 3.0.
- Added
intelmq.bots.experts.maxmind_geoip.expert
:- Added
--update-database
option (PR#1524 by Filip Pokorný). - Added
license_key
parameter (PR#1524 by Filip Pokorný). - The script
update-geoip-data
is now deprecated and will be removed in version 3.0.
- Added
intelmq.bots.experts.tor_nodes.expert
:- Added
--update-database
option (PR#1524 by Filip Pokorný). - The script
update-tor-nodes
is now deprecated and will be removed in version 3.0.
- Added
intelmq.bots.experts.recordedfuture_iprisk.expert
:- Added
--update-database
option (PR#1524 by Filip Pokorný). - Added
api_token
parameter (PR#1524 by Filip Pokorný). - The script
update-rfiprisk-data
is now deprecated and will be removed in version 3.0.
- Added
- Added
intelmq.bots.experts.threshold
(PR#1608 by Karl-Johan Karlsson). - Added
intelmq.bots.experts.splunk_saved_search.expert
(PR#1666 by Karl-Johan Karlsson). intelmq.bots.experts.sieve.expert
:intelmq.bots.experts.maxmind_geoip.expert
:- Fixed handing over of
overwrite
parameter toevent.add
(PR#1743 by Birger Schacht).
- Fixed handing over of
Outputs
intelmq.bots.outputs.rt
: Added Request Tracker output bot (PR#1589 by Marius Urkis).intelmq.bots.outputs.xmpp.output
: Marked as deprecated, see https://lists.cert.at/pipermail/intelmq-users/2020-October/000177.html (#1614, PR#1685 by Birger Schacht).intelmq.bots.outputs.smtp.output
: Fix sending to multiple recipients when recipients are defined by event-data (#1759, PR#1760 by Sebastian Waldbauer and Sebastian Wagner).
Documentation
- Feeds:
- Add ESET URL and Domain feeds (by Sebastian Wagner).
- Remove unavailable HPHosts Hosts file feed (#1559 by Sebastian Wagner).
- Added CZ.NIC HaaS feed (PR#1560 by Filip Pokorný and Edvard Rejthar).
- Added CZ.NIC Proki feed (PR#1599 by sinus-x).
- Updated Abuse.ch URLhaus feed (PT#1572 by Filip Pokorný).
- Added CERT-BUND CB-Report Malware infections feed (PR#1598 by sinus-x and Sebastian Wagner).
- Updated Turris Greylist feed with PGP verification information (by Sebastian Wagner).
- Fixed parsing of the
public
field in the generated feeds documentation (PR#1641 by Birger Schacht). - Change the
rate_limit
parameter of some feeds from 2 days (129600 seconds) to one day (86400 seconds). - Update the cAPTure Ponmocup Domains feed documentation (PR#1574 by Filip Pokorný and Sebastian Wagner).
- Added Shadowserver Reports API (by Sebastian Wagner).
- Change the
rate_limit
parameter for many feeds from 2 days to the default one day (by Sebastian Wagner). - Removed Malware Domain List feed, as the feed is obsolete (#1762, PR#1771 by Birger Schacht).
- Bots:
- Enhanced documentation of RFC1918 Expert (PR#1557 by Mladen Markovic and Sebastian Wagner).
- Enhanced documentation of SQL Output (PR#1620 by Edvard Rejthar).
- Updated documentation for MaxMind GeoIP, ASN Lookup, TOR Nodes and Recorded Future experts to reflect new
--update-database
option (PR#1524 by Filip Pokorný). - Added documentation for Shadowserver API collector and parser (PR#1700 by Birger Schacht and Sebastian Wagner).
- Add n6 integration documentation (by Sebastian Wagner).
- Moved 'Orphaned Queues' section from the FAQ to the intelmqctl documentation (by Sebastian Wagner).
- Generate documentation using Sphinx (PR#1622 by Birger Schacht).
- The documentation is now available at https://intelmq.readthedocs.io/en/latest/
- Refactor documentation and fix broken syntax (#1639, PRs #1638 #1640 #1642 by Birger Schacht).
- Integrate intelmq-manager and intelmq-api user documentation to provide unified documentation place (PR#1714 & PR#1714 by Birger Schacht).
Packaging
- Fix paths in the packaged logcheck rules (by Sebastian Wagner).
- Build the sphinx documentation on package build (PR#1701 by Birger Schacht).
- Ignore non-zero exit-codes for the
intelmqctl check
call in postinst (#1748, by Sebastian Wagner).
Tests
- Added tests for
intelmq.lib.exceptions.PipelineError
(by Sebastian Wagner). intelmq.tests.bots.collectors.http_collector.test_collector
: Userequests_mock
to mock all requests and do not require a local webserver (by Sebastian Wagner).intelmq.tests.bots.outputs.restapi.test_output
:- Use
requests_mock
to mock all requests and do not require a local webserver (by Sebastian Wagner). - Add a test for checking the response status code (by Sebastian Wagner).
- Use
intelmq.tests.bots.collectors.mail.test_collector_url
: Userequests_mock
to mock all requests and do not require a local webserver (by Sebastian Wagner).intelmq.tests.bots.experts.ripe.test_expert
: Userequests_mock
to mock all requests and do not require a local webserver (by Sebastian Wagner).- The test flag (environment variable)
INTELMQ_TEST_LOCAL_WEB
is no longer used (by Sebastian Wagner). - Added tests for
intelmq.harmonization.DateTime.parse_utc_isoformat
andconvert_fuzzy
(by Sebastian Wagner). - Move from Travis to GitHub Actions (PR#1707 by Birger Schacht).
intelmq.lib.test
:test_static_bot_check_method
checks the bot's staticcheck(parameters)
method for any exceptions, and a valid formatted return value (#1505, by Sebastian Wagner).setUpClass
: Skip tests if cache was requests withuse_cache
member, but Redis is deactivated with the environment variableINTELMQ_SKIP_REDIS
(by Sebastian Wagner).
intelmq.tests.bots.experts.cymru_whois.test_expert
:- Switch from
example.com
tons2.univie.ac.at
for hopefully more stable responses (#1730, PR#1731 by Sebastian Waldbauer). - Do not test for exact expected values in the 6to4 network test, as the values are changing regularly (by Sebastian Wagner).
- Switch from
intelmq.tests.bots.parsers.abusech
: Remove tests cases of discontinued feeds (PR#1741 by Thomas Bellus).- Activate GitHub's CodeQL Code Analyzing tool as GitHub Action (by Sebastian Wagner).
Tools
intelmqdump
:- Check if given queue is configured upon recovery (#1433, PR#1587 by Mladen Markovic).
intelmqctl
:intelmq list queues
:--sum
,--count
,-s
flag for showing total count of messages (#1408, PR#1581 by Mladen Markovic).intelmq check
: Added a possibility to ignore queues from the orphaned queues check (by Sebastian Wagner).- Allow setting the pipeline host by environment variables for docker usage (PR#1669 by Sebastian Waldbauer).
Contrib
- EventDB:
- Add SQL script for keeping track of the oldest inserted/update "time.source" information (by Sebastian Wagner).
- Cron Jobs: The script
intelmq-update-data
has been renamed tointelmq-update-database
(by Filip Pokorný). - Dropped utterly outdated contrib modules (by Sebastian Wagner):
- ansible
- vagrant
- vagrant-ansible
- logrotate:
- Do not use the deprecated "copytruncate" option as intelmq re-opens the log anyways (by Sebastian Wagner).
- Set file permissions to
0644
(by Sebastian Wagner).
Known issues
- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
- Corrupt dump files when interrupted during writing (#870).
- CSV line recovery forces Windows line endings (#1597).
- intelmqdump: Honor logging_path variable (#1605).
- Timeout error in mail URL fetcher (#1621).
- AMQP pipeline: get_queues needs to check vhost of response (#1746).
Assets
2
wagner-certat
released this
2.3.0.rc1 2.3.0 Release candidate 1
Assets
2
wagner-certat
released this
Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/UPGRADING.md
Documentation
- Bots/Sieve expert: Add information about parenthesis in if-expressions (#1681, PR#1687 by Birger Schacht).
Harmonization
- See NEWS.md for information on a fixed bug in the taxonomy expert.
Bots
Collectors
intelmq.bots.rt.collector_rt
: Log the size of the downloaded file in bytes on debug logging level.
Parsers
intelmq.bots.parsers.cymru.parser_cap_program
:- Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt).
- Add support for field
additional_asns
in optional information column.
intelmq.bots.parsers.microsoft.parser_ctip
:- Fix mapping of
DestinationIpInfo.DestinationIpConnectionType
field (contained a typo). - Explicitly ignore field
DestinationIpInfo.DestinationIpv4Int
as the data is already in another field.
- Fix mapping of
intelmq.bots.parsers.generic.parser_csv
:- Ignore line having spaces or tabs only or comment having leading tabs or spaces (PR#1669 by Brajneesh).
- Data fields containing
-
are now ignored and do not raise an exception anymore (#1651, PR#74 by Sebastian Waldbauer).
Experts
intelmq.bots.experts.taxonomy.expert
: Map typescanner
toinformation-gathering
instead ofinformation gathering
. See NEWS file for more information.
Tests
- Travis: Deactivate tests with optional requirements on Python 3.5, as the build fails because of abusix/querycontacts version conflicts on dnspython.
Known issues
Assets
2
wagner-certat
released this
Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.2/docs/UPGRADING.md
Core
intelmq.lib.upgrades
:- Add upgrade function for renamed Shadowserver feed name "Blacklisted-IP"/"Blocklist".
Bots
Parsers
intelmq.bots.parsers.shadowserver
:- Rename "Blacklisted-IP" feed to "Blocklist", old name is still valid until IntelMQ version 3.0 (PR#1588 by Thomas Hungenberg).
- Added support for the feeds
Accessible Radmin
andCAIDA IP Spoofer
(PR#1600 by sinus-x).
intelmq.bots.parsers.anubisnetworks.parser
: Fix parsing error wheredst.ip
was not equal tocomm.http.host
.intelmq/bots/parsers/danger_rulez/parser
: correctly skip malformed rows by defining variables before referencing (PR#1601 by Tomas Bellus).- `intelmq.bots.parsers.misp.parser: Fix MISP Event URL (#1619, PR#1618 by Nedfire23).
intelmq.bots.parsers.microsoft.parser_ctip
:- Add support for
DestinationIpInfo.*
andSignatures.Sha256
fields, used by thectip-c2
feed (PR#1623 by Mikk Margus Möll). - Use
extra.payload.text
for the feed's fieldPayload
if the content cannot be decoded (PR#1610 by Giedrius Ramas).
- Add support for
Experts
intelmq.bots.experts.cymru_whois
:- Fix cache key calculation which previously led to duplicate keys and therefore wrong results in rare cases. The cache key calculation is intentionally not backwards-compatible (#1592, PR#1606).
- The bot now caches and logs (as level INFO) empty responses from Cymru (PR#1606).
Documentation
- README:
- Add Core Infrastructure Initiative Best Practices Badge.
- Bots:
- Generic CSV Parser: Add note on escaping backslashes (#1579).
- Remove section of non-existing "Copy Extra" Bot.
- Explain taxonomy expert.
- Add documentation on n6 parser.
- Gethostbyname expert: Add documentation how errors are treated.
- Feeds:
- Fixed bot modules of Calidog CertStream feed.
- Add information on Microsoft CTIP C2 feed.
Packaging
- In Debian packages,
intelmqctl check
andintelmqctl upgrade-config
are executed in the postinst step (#1551, PR#1624 by Birger Schacht).
Tests
intelmq.tests.lib.test_pipeline
: SkipTestAmqp.test_acknowledge
on Travis with Python 3.8.intelmq.tests.bots.outputs.elasticsearch.test_output
: Refresh indexintelmq
manually to fix random test failures (#1593, PR#1595 by Zach Stone).
Tools
intelmqctl check
:- For disabled bots which do not have any pipeline connections, do not raise an error, but only warning.
- Fix check on source/destination queues for bots as well the orphaned queues.
Contrib
- Bash completion scripts: Check both
/opt/intelmq/
as well as LSB-paths (/etc/intelmq/
and/var/log/intelmq/
) for loading bot information (#1561, PR#1628 by Birger Schacht).
Known issues
Assets
2
wagner-certat
released this
Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.1/docs/UPGRADING.md
Core
intelmq.lib.upgrades
:- Add upgrade function for changed configuration of the feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
- Add upgrade function for removal of HPHosts Hosts file feed and
intelmq.bots.parsers.hphosts
parser (#1559). intelmq.lib.harmonization
:- For IP Addresses, explicitly reject IPv6 addresses with scope ID (due to changed behavior in Python 3.9, #1550).
Development
- Ignore line length (E501) in code-style checks altogether.
Bots
Collectors
intelmq.bots.collectors.misp
: Fix access to actual MISP object (PR#1548 by Tomas Bellus @tomas321)intelmq.bots.collectors.stomp
: Remove emptyclient.pem
file.
Parsers
intelmq.bots.parsers.shadowserver.config
:intelmq.bots.parser.anubisnetworks.parser
: Ignore "TestSinkholingLoss" events, these are not intended to be sent out at all.intelmq.bots.parsers.generic.parser_csv
: Allow values of type dictionary for parametertype_translation
.intelmq.bots.parsers.hphosts
: Removed, feed is unavailable (#1559).intelmq.bots.parsers.cymru.parser_cap_program
: Add support for comment "username" for "scanner" category.intelmq.bots.parsers.malwareurl.parser
: Check for valid FQDN and IP address in URL and IP address columns (PR#1585 by Marius Urkis).
Experts
intelmq.bots.experts.maxmind_geoip
: On Python < 3.6, require maxminddb < 2, as that version does no longer support Python 3.5.
Outputs
intelmq.bot.outputs.udp
: Fix error handling on sending, had a bug itself.
Documentation
- Feeds:
- Update documentation of feed "Abuse.ch URLHaus" (#1571, PR#1572 by Filip Pokorný).
- Bots:
- Overhaul of all bots' description fields (#1570).
- User-Guide:
- Overhaul pipeline configuration section and explain named queues better (#1577).
Tests
intelmq.tests.bots.experts.cymru
: Adapttest_empty_result
, removetest_unicode_as_name
andtest_country_question_mark
(#1576).
Tools
intelmq.bin.intelmq_gen_docs
: Format parameters of types lists with double quotes around values to produce conform JSON, ready to copy and paste the value into the IntelMQ Manager's bot parameter form.intelmq.bin.intelmqctl
:debug
: In JSON mode, use dictionaries instead of lists.debug
: AddPATH
to the paths shown.check
: Show$PATH
environment variable if executable cannot be found.
Contrib
malware_name_mapping
: Change MISP Threat Actors URL to new URL (branch master -> main) in download script.
Known issues
Assets
2
wagner-certat
released this
Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.0/docs/UPGRADING.md
Dropped support for Python 3.4.
Core
__init__
: Changes to the path-handling, see User Guide, section /opt and LSB paths for more informationintelmq.lib.exceptions
: AddedMissingDependencyError
for show error messages about a missing library and how to install it (#1471).- Added optional parameter
installed
to show the installed version. - Added optional parameter
additional_text
to show arbitrary text.
- Added optional parameter
- Adding more type annotations for core libraries.
intelmq.lib.pipeline.Pythonlist.sleep
: Drop deprecated method.intelmq.lib.utils
:write_configuration
: Append a newline at end of configuration/file to allow proper comparisons & diffs.intelmq.lib.test
:BotTestCase
drops privileges upon initialization (#1489).intelmq.lib.bot
:- New class
OutputBot
:- Method
export_event
to format/export events according to the parameters given by the user.
- Method
ParserBot
: New methodsparse_json_stream
andrecover_line_json_stream
.ParserBot.recover_line_json
: Fix format by adding a list around the line data.Bot.send_message
: In debugging log level, the path to which the message is sent is now logged too.
- New class
Bots
- Bots with dependencies: Use of
intelmq.lib.exceptions.MissingDependencyError
.
Collectors
intelmq.bots.collectors.misp.collector
: Deprecate parametermisp_verify
in favor of generic parameterhttp_verify_cert
.intelmq.bots.collectors.tcp.collector
: Drop compatibility with Python 3.4.intelmq.bots.collectors.stomp.collector
:- Check the stomp.py version and show an error message if it does not match.
- For stomp.py versions
>= 5.0.0
redirect thestomp.PrintingListener
output to debug logging.
intelmq.bots.collectors.microsoft.collector_azure
: Support current Python libraryazure-storage-blob>= 12.0.0
, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.intelmq.bots.collectors.amqp.collector_amqp
: Requirepika
minimum version 1.0.intelmq.bots.collectors.github_api.collector_github_contents_api
: Added (PR#1481).
Parsers
intelmq.bots.parsers.autoshun.parser
: Drop compatibility with Python 3.4.intelmq.bots.parsers.html_table.parser
: Drop compatibility with Python 3.4.intelmq.bots.parsers.shadowserver.parser
: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).intelmq.bots.parsers.taichung.parser
:- Migrate to
ParserBot
. - Also parse geolocation information if available.
- Migrate to
intelmq.bots.parsers.cymru.parser_full_bogons
:- Migrate to
ParserBot
. - Add last updated information in raw.
- Migrate to
intelmq.bots.parsers.anubisnetworks.parser
: Add new parameteruse_malware_familiy_as_classification_identifier
.intelmq.bots.parsers.microsoft.parser_ctip
: Compatibility for new CTIP data format used provided by the Azure interface.intelmq.bots.parsers.cymru.parser_cap_program
: Support foropenresolver
type.intelmq.bots.parsers.github_feed.parser
: Added (PR#1481).intelmq.bots.parsers.urlvir.parser
: Removed, as the feed is discontinued (#1537).
Experts
intelmq.bots.experts.csv_converter
: Added as converter to CSV.intelmq.bots.experts.misp
: Added (PR#1475).intelmq.bots.experts.modify
: New parametermaximum_matches
.
Outputs
intelmq.bots.outputs.amqptopic
:- Use
OutputBot
andexport_event
. - Allow formatting the routing key with event data by the new parameter
format_routing_key
(boolean).
- Use
intelmq.bots.outputs.file
: UseOutputBot
andexport_event
.intelmq.bots.outputs.files
: UseOutputBot
andexport_event
.intelmq.bots.outputs.misp.output_feed
: Added, creates a MISP Feed (PR#1473).intelmq.bots.outputs.misp.output_api
: Added, pushes to MISP via the API (PR#1506, PR#1536).intelmq.bots.outputs.elasticsearch.output
: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (#1513).
Documentation
- Document usage of the
INTELMQ_ROOT_DIR
environment variable. - Added document on MISP integration possibilities.
- Feeds:
- Added "Full Bogons IPv6" feed.
- Remove discontinued URLVir Feeds (#1537).
Packaging
setup.py
do not try to install any data to/opt/intelmq/
as the behavior is inconsistent on various systems and withintelmqsetup
we have a tool to create the structure and files anyway.debian/rules
:- Provide a blank state file in the package.
- Patches:
- Updated
fix-intelmq-paths.patch
.
- Updated
Tests
- Travis: Use
intelmqsetup
here too.- Install required build dependencies for the Debian package build test.
- This version is no longer automatically tested on Python
<
3.5. - Also run the tests on Python 3.8.
- Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
- Added tests for the new bot
intelmq.bots.outputs.misp.output_feed
(#1473). - Added tests for the new bot
intelmq.bots.experts.misp.expert
(#1473). - Added tests for
intelmq.lib.exceptions
. - Added tests for
intelmq.lib.bot.OutputBot
andintelmq.lib.bot.OutputBot.export_event
. - Added IPv6 tests for
intelmq.bots.parsers.cymru.parser_full_bogons
. - Added tests for
intelmq.lib.bot.ParserBot
's new methodsparse_json_stream
andrecover_line_json_stream
. intelmq.tests.test_conf
: Set encoding to UTF-8 for reading thefeeds.yaml
file.
Tools
intelmqctl
:upgrade-config
:- Allow setting the state file location with the
--state-file
parameter. - Do not require a second run anymore, if the state file is newly created (#1491).
- New parameter
no_backup
/--no-backup
to skip creation of.bak
files for state and configuration files.
- Allow setting the state file location with the
- Only require
psutil
for theIntelMQProcessManager
, not for process manager independent calls likeupgrade-config
orcheck
. - Add new command
debug
to output some information for debugging. Currently implemented:- paths
- environment variables
IntelMQController
: New argument--no-file-logging
to disable logging to file.- If dropping privileges does not work,
intelmqctl
will now abort (#1489).
intelmqsetup
:- Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
- Call
intelmqctl upgrade-config
and add argument for the state file path (#1491).
intelmq_generate_misp_objects_templates.py
: Tool to create a MISP object template (#1470).intelmqdump
: New parameter-t
or--truncate
to optionally give the maximum length ofraw
data to show, 0 for no truncating.
Contrib
- Added
development-tools
. - ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (#1513).
- Malware Name Mapping Downloader:
- New parameter
--mwnmp-ignore-adware
. - The parameter
--add-default
supports an optional parameter to define the default value.
- New parameter
Known issues
Assets
2
wagner-certat
released this
Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md
Requirements
- The python library
requests
is (again) listed as dependency of the core (#1519).
Core
intelmq.lib.upgrades
:- Harmonization upgrade: Also check and update regular expressions.
- Add function to migrate the deprecated parameter
attach_unzip
toextract_files
for the mail attachment collector. - Add function to migrate changed Taichung URL feed.
- Check for discontinued Abuse.CH Zeus Tracker feed.
intelmq.lib.bot
:ParserBot.recover_line
: Parameterline
needs to be optional, fix usage of fallback valueself.current_line
.start
: Handle decoding errors in the pipeline different so that the bot is not stuck in an endless loop (#1494).start
: Only acknowledge a message in case of errors, if we actually had a message to dump, which is not the case for collectors._dump_message
: Dump messages with encoding errors base64 encoded, not in JSON format as it's not possible to decode them (#1494).
intelmq.lib.test
:BotTestCase.run_bot
: Add parametersallowed_error_count
andallowed_warning_count
to allow set the number per run, not per test class.- Set
source_pipeline_broker
anddestination_pipeline_broker
topythonlist
instead of the oldbroker
, fixesintelmq.tests.lib.test_bot.TestBot.test_pipeline_raising
. - Fix test for (allowed) errors and warnings.
intelmq.lib.exceptions
:InvalidKey
: AddKeyError
as parent class.DecodingError
: Added, string representation has all relevant information on the decoding error, including encoding, reason and the affected string (#1494).
intelmq.lib.pipeline
:- Decode messages in
Pipeline.receive
not in the implementation's_receive
so that the internal counter is correct in case of decoding errors (#1494).
- Decode messages in
intelmq.lib.utils
:decode
: Raise newDecodingError
if decoding fails.
Harmonization
protocol.transport
: Adapt regular expression to allow the valuenvp-ii
(protocol 11).
Bots
Collectors
intelmq.bots.collectors.mail.collector_mail_attach
:- Fix handling of deprecated parameter name
attach_unzip
. - Fix handling of attachments without filenames (#1538).
- Fix handling of deprecated parameter name
intelmq.bots.collectors.stomp.collector
: Fix compatibility with stomp.py versions> 4.1.20
and catch errors on shutdown.intelmq.bots.collectors.microsoft
:- Update
REQUIREMENTS.txt
temporarily fixing deprecated Azure library (#1530, PR#1532). intelmq.bots.collectors.microsoft.collector_interflow
: Add method for printing the file list.
- Update
Parsers
intelmq.bots.parsers.cymru.parser_cap_program
: Support for protocol 11 (nvp-ii
) andconficker
type.intelmq.bots.parsers.taichung.parser
: Support more types/classifications:- Application Compromise: Apache vulnerability & SQL injections
- Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH & SIP attacks
- C2 Sever: Attack controller
- DDoS
- DoS: DNS, DoS, Excess connection
- IDS Alert / known vulnerability exploitation: backdoor
- Malware: Malware Proxy
- Warn on new unknown types.
intelmq.bots.parsers.bitcash.parser
: Removed as feed is discontinued.intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc
andintelmq.bots.parsers.fraunhofer.parser_ddosattack_target
: Removed as feed is discontinued.intelmq.bots.parsers.malwaredomains.parser
: Correctly classifyC&C
andphishing
events.intelmq.bots.parsers.shadowserver.parser
: More verbose error message for missing report specification (#1507).intelmq.bots.parsers.n6.parser_n6stomp
: Always add n6 fieldname
asmalware.name
independent ofcategory
.intelmq.bots.parsers.anubisnetworks
: Update parser with new data format.intelmq.bots.parsers.bambenek
: Add new feed URLs with Hostfaf.bambenekconsulting.com
(#1525, PR#1526).intelmq.bots.parsers.abusech.parser_ransomware
: Removed, as the feed is discontinued (#1537).intelmq.bots.parsers.nothink.parser
: Removed, as the feed is discontinued (#1537).intelmq.bots.parsers.n6.parser
: Remove not allowed characters in the name field formalware.name
and write original value toevent_description.text
instead.
Experts
intelmq.bots.experts.cymru_whois.lib
: Fix parsing of AS names with Unicode characters.
Outputs
intelmq.bots.outputs.mongodb
:- Set default port 27017.
- Use different authentication mechanisms per MongoDB server version to fix compatibility with server version >= 3.4 (#1439).
Documentation
- Feeds:
- Remove unavailable feed Abuse.CH Zeus Tracker.
- Remove the field
status
, offline feeds should be removed. - Add a new field
public
to differentiate between private and public feeds. - Adding documentation URLs to nearly all feeds.
- Remove unavailable Bitcash.cz feed.
- Remove unavailable Fraunhofer DDos Attack feeds.
- Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
- Update information on Bambenek Feeds, many require a license now (#1525).
- Remove discontinued Nothink Honeypot Feeds (#1537).
- Developers Guide: Fix the instructions for
/opt/intelmq
file permissions.
Packaging
- Patches:
fix-logrotate-path.patch
: also include path to rotated file in patch. - Fix paths from
/opt
to LSB forsetup.py
andcontrib/logrotate/intelmq
in build process (#1500). - Add runtime dependency
debianutils
for the programwhich
, which is required forintelmqctl
.
Tests
- Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
intelmq.tests.bots.experts.cymru_whois
:- Drop missing ASN test, does not work anymore.
- IPv6 to IPv4 test: Test for two possible results.
intelmq.lib.test
: Fix compatibility of logging capture with Python >= 3.7 by reworking the whole process (#1342).intelmq.bots.collectors.tcp.test_collector
: Removing custom mocking and bot starting, not necessary anymore.- Added tests for
intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline
. - Fix and split
tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json
. - Added tests for invalid encodings in input messages in
intelmq.tests.lib.test_bot
andintelmq.tests.lib.test_pipeline
(#1494). - Travis: Explicitly enable RabbitMQ management plugin.
intelmq.tests.lib.test_message
: Fix usage of the parameterblacklist
for Message hash tests (#1539).
Tools
intelmqsetup
: Copy missing BOTS file to IntelMQ's root directory (#1498).intelmq_gen_docs
: Feed documentation generation: Handle missing/empty parameters.intelmqctl
:intelmqdump
:- Handle base64-type messages for show, editor and recovery actions.
Contrib
intelmq/bots/experts/asn_lookup/update-asn-data
: Usepyasn_util_download.py
to download the data instead from RIPE, which cannot be parsed currently (#1517, PR#1518, hadiasghari/pyasn#62).
Known issues
Assets
2
wagner-certat
released this
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.2/docs/UPGRADING.md
Core
__init__
: Resolve absolute path forSTATE_FILE_PATH
variable (resolves..
).intelmq.lib.utils
:- log: Do not raise an exception if logging to neither file nor syslog is requested.
- logging StreamHandler: Colorize all warning and error messages red.
- logging FileHandler: Strip all shell colorizations from the messages (#1436).
intelmq.lib.message
:Message.to_json
: Setsort_keys=True
to get reproducible results.drop_privileges
: Handle situations where the user or groupintelmq
does not exist.
intelmq.lib.pipeline
:Amqp._send
andAmqp._acknowledge
: Log traceback in debug mode in case of errors and necessary re-connections.Amqp._acknowledge
: Reset delivery tag if acknowledge was successful.
Bots
Collectors
intelmq.bots.collectors.misp.collector
:- Add compatibility with current pymisp versions and versions released after January 2020 (PR #1468).
Parsers
intelmq.bots.parsers.shadowserver.config
: Add some missing fields for the feedaccessible-rdp
(#1463).intelmq.bots.parsers.shadowserver.parser
:- Feed-detection based on file names: The prefixed date is optional now.
- Feed-detection based on file names: Re-detect feed for every report received (#1493).
Experts
intelmq.bots.experts.national_cert_contact_certat
: Handle empty responses by server (#1467).intelmq.bots.experts.maxmind_geoip
: The scriptupdate-geoip-data
now requires a license key as second parameter because of upstream changes (#1484)).
Outputs
intelmq.bots.outputs.restapi.output
: Fix logging of response body if response status code was not ok.
Documentation
- Remove some hardcoded
/opt/intelmq/
paths from code comments and program outputs.
Packaging
- debian/rules: Only replace
/opt/intelmq/
with LSB-paths in some certain files, not the whole tree, avoiding wrong replacements. - debian/rules and debian/intelmq.install: Do install the examples configuration directly instead of working around the abandoned examples directory.
Tests
lib/test_utils
: Skip some tests on Python 3.4 becausecontextlib.redirect_stdout
andcontextlib.redirect_sterr
are not supported on this version.- Travis: Stop running tests with all optional dependencies on Python 3.4, as more and more libraries are dropping support for it. Tests on the core and code without non-optional requirements are not affected.
tests.bots.parsers.html_table
: Make tests independent of current year.
Tools
intelmqctl upgrade-config
: Fix missing substitution in error message "State file %r is not writable.".
Known issues
- bots trapped in endless loop if decoding of raw message fails (#1494)
- intelmqctl status of processes: need to check bot id too (#1492)
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)
Assets
2
wagner-certat
released this
Install documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.1/docs/UPGRADING.md
Configuration
- Default configuration:
- Remove discontinued feed "Feodo Tracker Domains" from default configuration.
- Add "Feodo Tracker Browse" feed to default configuration.
Core
intelmq.lib.pipeline
: AMQP: using port 15672 as default (like RabbitMQ's defaults) for the monitoring interface for getting statistical data (intelmqctl_rabbitmq_monitoring_url
).intelmq.lib.upgrades
: Added a generic upgrade function for harmonization, checking of all message types, it's fields and their types.intelmq.lib.utils
:TimeoutHTTPAdapter
: A subclass ofrequests.adapters.HTTPAdapter
with the possibility to set the timeout per adapter.create_request_session_from_bot
: Use theTimeoutHTTPAdapter
with the user-defined timeout. Previously the timeout was not functional.
Bots
Parsers
intelmq.bots.parsers.shadowserver.parser
: Fix logging message if the parameterfeedname
is not present.intelmq.bots.parsers.shodan.parser
: Also add fieldclassification.identifier
('network-scan'
) in minimal mode.intelmq.bots.parsers.spamhaus.parser_cert
: Add support for category'misc'
.intelmq.bots.parsers.cymru.parser_cap_program
:- Add support for phishing events without URL.
- Add support for protocols >= 143 (unassigned, experiments, testing, reserved), saving the number to extra, as the data would be bogus.
intelmq.bots.parsers.microsoft.parser_bingmurls
:- Save the
Tags
data assource.geolocation.cc
.
- Save the
Experts
intelmq.bots.experts.modify.expert
: Fix bug with setting non-string values (#1460).
Outputs
intelmq.bots.outputs.smtp
:
Documentation
- Feeds:
- Fix configuration of
Feodo Tracker Browse
feed.
- Fix configuration of
- Bots:
- Sieve expert: Document behavior of
!=
with lists.
- Sieve expert: Document behavior of
Tests
- Adaption and extension of the test cases to the changes.
Tools
intelmq.bin.intelmqctl
:- check: Check if running the upgrade function for harmonization is necessary.
- upgrade-config: Run the upgrade function for harmonization.
intelmqctl restart
did throw an error as the message for restarting was not defined (#1465).
Known issues
- MongoDB authentication: compatibility on different MongoDB and pymongo versions (#1439)
- ctl: shell colorizations are logged (#1436)
- http stream collector: retry on regular connection problems? (#1435)
- tests: capture logging with context manager (#1342)
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (#952)
- n6 parser: mapping is modified within each run (#905)
- reverse DNS: Only first record is used (#877)
- Corrupt dump files when interrupted during writing (#870)