Permalink
Browse files

Merge branch 'develop' into rsync-collector

  • Loading branch information...
wagner-certat committed Aug 17, 2018
2 parents 0c78597 + 84c9938 commit 8093c0600ce23d39a0b737894d7016d199e627de
Showing with 1,064 additions and 523 deletions.
  1. +29 −3 CHANGELOG.md
  2. +0 −108 README.md
  3. +117 −0 README.rst
  4. +1 −1 contrib/bash-completion/intelmqctl
  5. +10 −0 contrib/malware_name_mapping/README.md
  6. +121 −0 contrib/malware_name_mapping/apply_mapping_eventdb.py
  7. +6 −0 debian/changelog
  8. +1 −4 debian/intelmq.install
  9. +3 −1 debian/patches/fix-intelmq-paths.patch
  10. +6 −0 debian/rules
  11. +14 −0 docs/Bots.md
  12. +18 −0 docs/Developers-Guide.md
  13. +4 −2 docs/INSTALL.md
  14. +5 −6 docs/README.md
  15. +1 −0 intelmq/__init__.py
  16. +63 −25 intelmq/bin/intelmqctl.py
  17. +11 −1 intelmq/bots/BOTS
  18. +1 −1 intelmq/bots/collectors/calidog/collector_certstream.py
  19. +2 −2 intelmq/bots/experts/asn_lookup/update-asn-data
  20. +1 −1 intelmq/bots/experts/maxmind_geoip/update-geoip-data
  21. +8 −2 intelmq/bots/experts/ripencc_abuse_contact/expert.py
  22. +1 −2 intelmq/bots/experts/tor_nodes/update-tor-nodes
  23. +3 −0 intelmq/bots/outputs/amqptopic/output.py
  24. 0 intelmq/bots/outputs/blackhole/__init__.py
  25. +12 −0 intelmq/bots/outputs/blackhole/output.py
  26. 0 intelmq/bots/parsers/cert_eu/__init__.py
  27. +79 −0 intelmq/bots/parsers/cert_eu/parser_csv.py
  28. +46 −12 intelmq/bots/parsers/shadowserver/config.py
  29. +5 −25 intelmq/bots/parsers/shadowserver/parser.py
  30. +11 −5 intelmq/lib/bot.py
  31. +3 −3 intelmq/lib/test.py
  32. 0 intelmq/tests/bots/outputs/blackhole/__init__.py
  33. +21 −0 intelmq/tests/bots/outputs/blackhole/test_output.py
  34. 0 intelmq/tests/bots/parsers/cert_eu/__init__.py
  35. +3 −0 intelmq/tests/bots/parsers/cert_eu/example.csv
  36. +90 −0 intelmq/tests/bots/parsers/cert_eu/test_parser_csv.py
  37. +0 −3 intelmq/tests/bots/parsers/shadowserver/Accessible-SMB_reconstructed.csv
  38. +0 −3 intelmq/tests/bots/parsers/shadowserver/Sinkhole-HTTP-Drone-RECONSTRUCTED.csv
  39. +3 −0 intelmq/tests/bots/parsers/shadowserver/accessible-adb.csv
  40. +3 −0 intelmq/tests/bots/parsers/shadowserver/accessible-adb_reconstructed.csv
  41. +0 −3 intelmq/tests/bots/parsers/shadowserver/accessible-cisco-smart-install-reconstructed.csv
  42. +0 −3 intelmq/tests/bots/parsers/shadowserver/accessible-hadoop-reconstructed.csv
  43. +0 −3 intelmq/tests/bots/parsers/shadowserver/accessible_vnc_RECONSTRUCTED.csv
  44. +0 −7 intelmq/tests/bots/parsers/shadowserver/chargen_RECONSTRUCTED.csv
  45. +0 −3 intelmq/tests/bots/parsers/shadowserver/compromised_website_RECONSTRUCTED.csv
  46. +0 −9 intelmq/tests/bots/parsers/shadowserver/drone-hadoop_RECONSTRUCTED.csv
  47. +0 −2 intelmq/tests/bots/parsers/shadowserver/drone_brute_force_RECONSTRUCTED.csv
  48. +0 −3 intelmq/tests/bots/parsers/shadowserver/elasticsearch_RECONSTRUCTED.csv
  49. +0 −15 intelmq/tests/bots/parsers/shadowserver/microsoft-sinkhole_RECONSTRUCTED.csv
  50. +0 −5 intelmq/tests/bots/parsers/shadowserver/qotd_RECONSTRUCTED.csv
  51. +0 −4 intelmq/tests/bots/parsers/shadowserver/sinkhole6_http_RECONSTRUCTED.csv
  52. +0 −11 intelmq/tests/bots/parsers/shadowserver/snmp_RECONSTRUCTED.csv
  53. +2 −0 intelmq/tests/bots/parsers/shadowserver/ssl_poodle.csv
  54. +93 −0 intelmq/tests/bots/parsers/shadowserver/test_accessible_adb.py
  55. +4 −9 intelmq/tests/bots/parsers/shadowserver/test_accessible_cisco_smart_install.py
  56. +4 −9 intelmq/tests/bots/parsers/shadowserver/test_accessible_hadoop.py
  57. +4 −9 intelmq/tests/bots/parsers/shadowserver/test_accessible_smb.py
  58. +4 −9 intelmq/tests/bots/parsers/shadowserver/test_accessible_vnc.py
  59. +11 −29 intelmq/tests/bots/parsers/shadowserver/test_chargen.py
  60. +4 −9 intelmq/tests/bots/parsers/shadowserver/test_compromised_website.py
  61. +2 −7 intelmq/tests/bots/parsers/shadowserver/test_drone_brute_force.py
  62. +16 −21 intelmq/tests/bots/parsers/shadowserver/test_drone_hadoop.py
  63. +32 −0 intelmq/tests/bots/parsers/shadowserver/test_helpers.py
  64. +28 −33 intelmq/tests/bots/parsers/shadowserver/test_microsoft_sinkhole.py
  65. +4 −9 intelmq/tests/bots/parsers/shadowserver/test_open_elasticsearch.py
  66. +8 −13 intelmq/tests/bots/parsers/shadowserver/test_qotd.py
  67. +6 −11 intelmq/tests/bots/parsers/shadowserver/test_sinkhole6_http.py
  68. +4 −9 intelmq/tests/bots/parsers/shadowserver/test_sinkhole_http_drone.py
  69. +20 −25 intelmq/tests/bots/parsers/shadowserver/test_snmp.py
  70. +86 −0 intelmq/tests/bots/parsers/shadowserver/test_ssl_poodle.py
  71. +18 −23 intelmq/tests/bots/parsers/shadowserver/test_xdmcp.py
  72. +0 −10 intelmq/tests/bots/parsers/shadowserver/xdmcp_RECONSTRUCTED.csv
  73. +1 −1 intelmq/tests/lib/test_expert_bot.py
  74. +4 −1 intelmq/tests/lib/test_parser_bot.py
  75. +6 −6 intelmq/tests/test_conf.py
  76. +1 −2 setup.py
View
@@ -19,13 +19,15 @@ CHANGELOG
- added `intelmq.bots.parsers.mcafee.parser_atd` (#1265).
- `intelmq.bots.parsers.generic.parser_csv`:
- New parameter `columns_required` to optionally ignore parse errors for columns.
- added `intelmq.bots.parsers.cert_eu.parser_csv` (#1287).
#### Experts
- added `intelmq.bots.experts.recordedfuture_iprisk` (#1267).
- added `intelmq.bots.experts.mcafee.expert_mar` (1265).
#### Outputs
- added `intelmq.bots.experts.mcafee.output_esm` (1265).
- added `intelmq.bots.outputs.blackhole` (#1279).
### Documentation
@@ -37,6 +39,7 @@ CHANGELOG
### Tools
### Contrib
* `malware_name_mapping`: Added the script `apply_mapping_eventdb.py` to apply the mapping to an eventdb.
### Known issues
@@ -63,12 +66,14 @@ CHANGELOG
- `intelmqctl restart` bug fix; returned some half-nonsense, now returns return state of start and stop operation in a list (#1226).
- `intelmqctl check`: New parameter `--no-connections` to prevent the command from making connections e.g. to the redis pipeline.s
- `intelmqctl list queues`: don't display named paths amongst standard queues.
- The process status test failed if the PATH did not include the bot executables and the `which` command failed. Then the proccess's command line could not be compared correctly. The fix warns of this and adds a new status 'unknown' (#1297).
### Contrib
- tool `feeds-config-generator` to automatically generate the collector and parser runtime and pipeline configurations.
- `malware_name_mapping`: Download and convert tool for malware family name mapping has been added.
- Added a systemd script which creates systemd units for bots (#953).
- `contrib/cron-jobs/update-asn-data`, `contrib/cron-jobs/update-geoip-data`, `contrib/cron-jobs/update-tor-nodes`: Errors produce proper output.
### Core
- lib/bot
@@ -79,6 +84,11 @@ CHANGELOG
- The parameter `feed` for collectors is deprecated for 2.0 and has been replaced by the more consistent `name` (#1144).
- bug: allow path parameter for CollectorBot class.
- Handle errors better when the logger could not be initialized.
- `ParserBot`:
- For the csv parsing methods, `ParserBot.csv_params` is now used for all these methods.
- `ParserBot.parse_csv_dict` now saves the field names in `ParserBot.csv_fieldnames`.
- `ParserBot.parse_csv_dict` now saves the raw current line in `ParserBot.current_line`.
- `ParserBot.recover_line_csv_dict` now uses the raw current line.
- lib/message:
- Subitems in fields of type `JSONDict` (see below) can be accessed directly. E.g. you can do:
event['extra.foo'] = 'bar'
@@ -100,6 +110,7 @@ CHANGELOG
* you may now define more than one destination queues path the bot should pass the message to, see [Pipelines](https://github.com/certtools/intelmq/blob/develop/docs/User-Guide.md#pipeline-configuration) (#1088, #1190).
* the special path `"_on_error"` can be used to pass messages to different queues in case of processing errors (#1133).
- `lib/harmonization`: Accept `AS` prefix for ASN values (automatically stripped).
- added `intelmq.VAR_STATE_PATH` for variable state data of bots.
### Bots
- Removed print statements from various bots.
@@ -130,8 +141,11 @@ CHANGELOG
- changed feednames . Please refer to it's README for the exact changes.
- If the conversion function fails for a line, an error is raised and the offending line will be handled according to the error handling configuration.
Previously errors like these were only logged and ignored otherwise.
- add support for the feed `Accessible-Hadoop`
- add support for the feeds
- `Accessible-Hadoop` (#1231)
- `Accessible ADB` (#1285)
- Remove deprecated parameter `override`, use `overwrite` instead (#1071).
- The `raw` values now are exactly the input with quotes unchanged, the ParserBot methods are now used directly (#1011).
- The Generic CSV Parser `bots.parsers.generic.parser_csv`:
- It is possible to filter the data before processing them using the new parameters `filter_type` and `filter_text`.
- It is possible to specify multiple columns using `|` character in parameter `columns`.
@@ -165,7 +179,10 @@ CHANGELOG
- Added wait expert for sleeping
- Added domain suffix expert to extract the TLD/Suffix from a domain name.
- `bots.experts.maxmind_geoip`: New (optional) parameter `overwrite`, by default false. The current default was to overwrite!
- `intelmq.bots.experts.ripencc_abuse_contact`: Remove deprecated parameter `query_ripe_stat`, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1071).
- `intelmq.bots.experts.ripencc_abuse_contact`: Extend deprecated parameter compatibility `query_ripe_stat` until 2.0 because of a logic bug in the compatibility code, use `query_ripe_stat_asn` and `query_ripe_stat_ip` instead (#1071, #1291).
- `intelmq/bots/experts/asn_lookup/update-asn-data`: Errors produce proper output on stdout/stderr.
- `intelmq/bots/experts/maxmind_geoip/update-geoip-data`: Errors produce proper output on stdout/stderr.
- `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Errors produce proper output on stdout/stderr.
#### Outputs
- `bots.outputs.file`:
@@ -189,6 +206,7 @@ CHANGELOG
### Documentation
- Use Markdown for README again, as pypi now supports it.
- Developers Guide: Add instructions for pre-release testing.
### Packaging
- Add logcheck configuration to the packages.
@@ -217,15 +235,19 @@ CHANGELOG
- `bots.collectors.rt.collector_rt`: Log ticket id for downloaded reports.
#### Parsers
- `bots.parsers.shadowserver`: if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed.
- `bots.parsers.shadowserver`:
- if required fields do not exist in data, an exception is raised, so the line will be dumped and not further processed.
- fix a bug in the parsing of column `cipher_suite` in ssl poodle reports (#1288).
#### Experts
- Reverse DNS Expert: ignore all invalid results and use first valid one (#1264).
- `intelmq/bots/experts/tor_nodes/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289).
#### Outputs
### Documentation
- Bots: document redis cache parameters.
- Installation documentation: Ubuntu needs universe repositories.
### Packaging
- Dropped support for Ubuntu 17.10, it reached its End of Life as of 2018-07-19.
@@ -234,13 +256,17 @@ CHANGELOG
- Drop tests for Python 3.3 for the mode with all requirements, as some optional dependencies do not support Python 3.3 anymore.
- `lib.test`: Add parameter `compare_raw` (default: `True`) to `assertMessageEqual`, to optionally skip the comparison of the raw field.
- Add tests for RT collector.
- Add tests for Shadowserver Parser:
- SSL Poodle Reports.
- Helper functions.
### Tools
- `intelmqctl list` now sorts the output of bots and queues (#1262).
- `intelmqctl`: Correctly handle the corner cases with collectors and outputs for getting/sending messages in the bot debugger (#1263).
- `intelmqdump`: fix ordering of dumps in a file in runtime. All operations are applied to a sorted list (#1280).
### Contrib
- `cron-jobs/update-tor-nodes`: Use check.torproject.org as source as internet2.us is down (#1289).
### Known issues
View
108 README.md

This file was deleted.

Oops, something went wrong.
View
@@ -0,0 +1,117 @@
Welcome to IntelMQ!
===================
.. figure:: https://raw.githubusercontent.com/certtools/intelmq/master/docs/images/Logo_Intel_MQ.png
:alt: IntelMQ
IntelMQ
|Build Status| |codecov.io|
**IntelMQ** is a solution for IT security teams (CERTs, CSIRTs, abuse
departments,...) for collecting and processing security feeds (such as
log files) using a message queuing protocol. It's a community driven
initiative called **IHAP** (Incident Handling Automation Project) which
was conceptually designed by European CERTs/CSIRTs during several
InfoSec events. Its main goal is to give to incident responders an easy
way to collect & process threat intelligence thus improving the incident
handling processes of CERTs.
IntelMQ's design was influenced by
`AbuseHelper <https://github.com/abusesa/abusehelper>`__ however it was
re-written from scratch and aims at:
- Reducing the complexity of system administration
- Reducing the complexity of writing new bots for new data feeds
- Reducing the probability of events lost in all process with
persistence functionality (even system crash)
- Use and improve the existing Data Harmonization Ontology
- Use JSON format for all messages
- Integration of the existing tools (AbuseHelper, CIF)
- Provide easy way to store data into Log Collectors like
ElasticSearch, Splunk, databases (such as PostgreSQL)
- Provide easy way to create your own black-lists
- Provide easy communication with other systems via HTTP RESTFUL API
It follows the following basic meta-guidelines:
- Don't break simplicity - KISS
- Keep it open source - forever
- Strive for perfection while keeping a deadline
- Reduce complexity/avoid feature bloat
- Embrace unit testing
- Code readability: test with unexperienced programmers
- Communicate clearly
Table of Contents
-----------------
1. `How to Install <#how-to-install>`__
2. `Developers Guide <#developers-guide>`__
3. `User Guide <#user-guide>`__
4. `IntelMQ Manager <#intelmq-manager>`__
5. `Incident Handling Automation
Project <#incident-handling-automation-project>`__
6. `Data Harmonization <#data-harmonization>`__
7. `How to Participate <#how-to-participate>`__
8. `Licence <#licence>`__
How to Install
--------------
See `INSTALL <docs/INSTALL.md>`__.
Developers Guide
----------------
See `Developers Guide <docs/Developers-Guide.md>`__.
User Guide
----------
See `User Guide <docs/User-Guide.md>`__.
For support questions please use the intelmq-users mailing list:
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
IntelMQ Manager
---------------
Check out this graphical
`tool <https://github.com/certtools/intelmq-manager>`__ and easily
manage an IntelMQ system.
Incident Handling Automation Project
------------------------------------
- **URL:**
http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation
- **Mailing-list:** ihap@lists.trusted-introducer.org
Data Harmonization
------------------
IntelMQ use the Data Harmonization. Please read `this
document <docs/Data-Harmonization.md>`__ for more details.
How to participate
------------------
- Subscribe to the Intelmq-dev Mailing list:
https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev (for
developers)
- Watch out for our regular developers conf call
- IRC: server: irc.freenode.net, channel: #intelmq
- Via github issues
- Via Pull requests (please do read help.github.com first)
Licence
-------
This software is licensed under GNU Affero General Public License
version 3
.. |Build Status| image:: https://travis-ci.org/certtools/intelmq.svg?branch=master
:target: https://travis-ci.org/certtools/intelmq
.. |codecov.io| image:: https://codecov.io/github/certtools/intelmq/coverage.svg?branch=master
:target: https://codecov.io/github/certtools/intelmq?branch=master
@@ -18,7 +18,7 @@ _intelmqctl ()
#echo "posice: $COMP_CWORD $COMP_WORDS";
case $COMP_CWORD in
1)
opts="start stop restart reload run status clear list check enable disable";
opts="start stop restart reload run status log clear list check enable disable";
COMPREPLY=($(compgen -W "${opts} ${generic_pre} ${generic_post}" -- ${cur}));
return 0
;;
@@ -36,3 +36,13 @@ You can use a crontab-entry like this:
0 1 * * * /path/to/contrib/malware_name_mapping/download_mapping.py /opt/intelmq/var/lib/bots/modify/malware-family-names.conf && intelmqctl reload malware-family-name-expert
```
You can check the validity of the resulting rule file with `intelmqctl check`.
Applying the mapping to existing data
-------------------------------------
Using the script `apply_mapping_eventdb.py` you can apply the mapping to an existing PostgreSQL eventdb. See the help page for options:
```
apply_mapping_eventdb.py -h
```
It queries the database for all distinct malware names with the taxonomy "malicious code" and sets another column to the malware family name. The column names can be set using parameters.
Oops, something went wrong.

0 comments on commit 8093c06

Please sign in to comment.