Permalink
Browse files

v1.0 is now available

Former-commit-id: 9970d67
  • Loading branch information...
CNCS-PT committed Jun 18, 2015
1 parent 4955005 commit b0447aea601cc066cbbc3dc3a11e2bea609f3bdb
Showing with 2,991 additions and 3,407 deletions.
  1. +0 −9 CHANGELOG
  2. +62 −0 CHANGELOG.md
  3. +10 −1 COPYRIGHT
  4. +79 −35 README.md
  5. +13 −0 REQUIREMENTS
  6. +0 −128 TODO.md
  7. +0 −79 docs/README.md
  8. +3 −2 docs/User-Guide.md
  9. +53 −11 intelmq/bin/intelmqctl
  10. +97 −141 intelmq/bots/BOTS
  11. +0 −48 intelmq/bots/collectors/hpfeeds/collector.py
  12. 0 intelmq/{ → bots/collectors/http}/__init__.py
  13. +27 −0 intelmq/bots/collectors/http/collector_http.py
  14. +25 −0 intelmq/bots/collectors/http/collector_http_stream.py
  15. +71 −72 intelmq/bots/collectors/{url → http}/lib.py
  16. +7 −2 intelmq/bots/collectors/mail/{mail-attach.py → collector_mail_attach.py}
  17. +6 −2 intelmq/bots/collectors/mail/{mail-url.py → collector_mail_url.py}
  18. +0 −25 intelmq/bots/collectors/microsoft_dcu/README.md
  19. +0 −72 intelmq/bots/collectors/microsoft_dcu/collector.py
  20. +0 −15 intelmq/bots/collectors/url/collector.py
  21. +0 −1 intelmq/bots/collectors/xmpp/collector.py
  22. +30 −27 intelmq/bots/experts/abusix/{abusix.py → expert.py}
  23. +12 −8 intelmq/bots/experts/abusix/lib.py
  24. 0 intelmq/bots/experts/{asnlookup → asn_lookup}/README.md
  25. 0 intelmq/bots/{collectors/hpfeeds → experts/asn_lookup}/__init__.py
  26. +43 −38 intelmq/bots/experts/{asnlookup/asnlookup.py → asn_lookup/expert.py}
  27. +0 −52 intelmq/bots/experts/contactdb/contactdb.py
  28. +0 −5 intelmq/bots/experts/countrycodefilter/README.md
  29. +0 −33 intelmq/bots/experts/countrycodefilter/countrycodefilter.py
  30. +0 −92 intelmq/bots/experts/cymru/cymru.py
  31. 0 intelmq/bots/{collectors/microsoft_dcu → experts/cymru_whois}/__init__.py
  32. +83 −0 intelmq/bots/experts/cymru_whois/expert.py
  33. +16 −10 intelmq/bots/experts/{cymru → cymru_whois}/lib.py
  34. +3 −3 intelmq/bots/experts/deduplicator/README.md
  35. +31 −0 intelmq/bots/experts/deduplicator/expert.py
  36. +8 −0 intelmq/bots/experts/filter/README.md
  37. 0 intelmq/bots/{collectors/url → experts/filter}/__init__.py
  38. +42 −0 intelmq/bots/experts/filter/expert.py
  39. 0 intelmq/bots/experts/geoip/__init__.py
  40. +0 −53 intelmq/bots/experts/geoip/geoip.py
  41. +3 −3 intelmq/bots/experts/{geoip → maxmind_geoip}/README.md
  42. 0 intelmq/bots/{collectors/xmpp → experts/maxmind_geoip}/__init__.py
  43. +50 −0 intelmq/bots/experts/maxmind_geoip/expert.py
  44. +32 −31 intelmq/bots/experts/ripencc/{ripencc.py → expert.py}
  45. +4 −3 intelmq/bots/experts/ripencc/lib.py
  46. 0 intelmq/bots/experts/sanitizer/__init__.py
  47. +0 −91 intelmq/bots/experts/sanitizer/sanitizer.py
  48. +45 −44 intelmq/bots/experts/taxonomy/{taxonomy.py → expert.py}
  49. 0 intelmq/bots/outputs/debug/__init__.py
  50. +0 −17 intelmq/bots/outputs/debug/debug.py
  51. +2 −3 intelmq/bots/outputs/file/{file.py → output.py}
  52. 0 intelmq/bots/outputs/intelmailer/__init__.py
  53. +0 −24 intelmq/bots/outputs/intelmailer/intelmailer.py
  54. 0 intelmq/bots/outputs/logcollector/__init__.py
  55. +1 −4 intelmq/bots/outputs/mongodb/{mongodb.py → output.py}
  56. +7 −9 intelmq/bots/outputs/postgresql/{postgresql.py → output.py}
  57. 0 intelmq/bots/{experts/asnlookup → outputs/tcp}/__init__.py
  58. +6 −12 intelmq/bots/outputs/{logcollector/logcollector.py → tcp/output.py}
  59. 0 intelmq/bots/parsers/abusehelper/DO_NOT_USE_THIS_CODE
  60. +0 −1 intelmq/bots/parsers/abusehelper/__init__.py
  61. +0 −73 intelmq/bots/parsers/abusehelper/abusehelper.py
  62. 0 intelmq/bots/{experts/contactdb → parsers/alienvault}/__init__.py
  63. +79 −0 intelmq/bots/parsers/alienvault/parser.py
  64. +43 −37 intelmq/bots/parsers/arbor/parser.py
  65. +30 −27 intelmq/bots/parsers/bruteforceblocker/parser.py
  66. 0 intelmq/bots/parsers/certeu/__init__.py
  67. +0 −45 intelmq/bots/parsers/certeu/malicious-urls-parser.py
  68. +0 −45 intelmq/bots/parsers/dragonresearchgroup/parser-ssh.py
  69. +0 −45 intelmq/bots/parsers/dragonresearchgroup/parser-vnc.py
  70. +51 −0 intelmq/bots/parsers/dragonresearchgroup/parser_ssh.py
  71. +49 −0 intelmq/bots/parsers/dragonresearchgroup/parser_vnc.py
  72. +0 −49 intelmq/bots/parsers/dshield/parser.py
  73. +61 −0 intelmq/bots/parsers/dshield/parser_asn.py
  74. +0 −45 intelmq/bots/parsers/generic/parser.py
  75. +0 −33 intelmq/bots/parsers/hpfeeds/parser.py
  76. 0 intelmq/bots/{experts/countrycodefilter → parsers/hphosts}/__init__.py
  77. +53 −0 intelmq/bots/parsers/hphosts/parser.py
  78. +33 −28 intelmq/bots/parsers/malwaredomainlist/parser.py
  79. +0 −36 intelmq/bots/parsers/malwarepatrol/parser-dansguardian.py
  80. +42 −0 intelmq/bots/parsers/malwarepatrol/parser_dansguardian.py
  81. 0 intelmq/bots/parsers/microsoft_dcu/__init__.py
  82. +0 −349 intelmq/bots/parsers/microsoft_dcu/lib.py
  83. +0 −41 intelmq/bots/parsers/microsoft_dcu/parser.py
  84. +28 −25 intelmq/bots/parsers/openbl/parser.py
  85. +37 −29 intelmq/bots/parsers/phishtank/parser.py
  86. 0 intelmq/bots/parsers/shadowserver/__init__.py
  87. +0 −66 intelmq/bots/parsers/shadowserver/chargen-parser.py
  88. +0 −81 intelmq/bots/parsers/shadowserver/drone-parser.py
  89. +0 −76 intelmq/bots/parsers/shadowserver/microsoft-sinkhole.py
  90. +0 −66 intelmq/bots/parsers/shadowserver/qotd-parser.py
  91. +0 −67 intelmq/bots/parsers/shadowserver/snmp-parser.py
  92. 0 intelmq/bots/parsers/{generic → taichung_city_netflow}/__init__.py
  93. +70 −0 intelmq/bots/parsers/taichung_city_netflow/parser.py
  94. +0 −1 intelmq/bots/parsers/taichungcitynetflow/__init__.py
  95. +0 −57 intelmq/bots/parsers/taichungcitynetflow/parser.py
  96. +0 −42 intelmq/bots/parsers/torexitnode
  97. +51 −47 intelmq/bots/parsers/vxvault/parser.py
  98. +0 −176 intelmq/bots/utils.py
  99. +17 −0 intelmq/conf/defaults.conf
  100. +130 −0 intelmq/conf/harmonization.conf
  101. +6 −74 intelmq/conf/pipeline.conf
  102. +20 −48 intelmq/conf/runtime.conf
  103. +3 −75 intelmq/conf/startup.conf
  104. +6 −6 intelmq/conf/system.conf
  105. 0 intelmq/lib/__init__.py
  106. +0 −33 intelmq/lib/cache.py
  107. +0 −99 intelmq/lib/message.py
  108. +0 −66 intelmq/lib/pipeline.py
  109. +0 −62 intelmq/lib/utils.py
  110. 0 intelmq/tests/__init__.py
  111. 0 intelmq/tests/bots/__init__.py
  112. 0 intelmq/tests/bots/parsers/__init__.py
  113. 0 intelmq/tests/bots/parsers/dcu/__init__.py
  114. +0 −83 intelmq/tests/bots/parsers/dcu/lib.py
  115. +4 −0 scripts/prettyprint.sh
  116. +1 −0 scripts/prettyprint.txt
  117. +47 −0 scripts/vagrant/README.md
  118. +72 −0 scripts/vagrant/Vagrantfile
  119. +49 −0 scripts/vagrant/bootstrap.sh
  120. +0 −3 tests/README.md
  121. +18 −17 {intelmq/bots/experts/deduplicator → tests/error_generator}/deduplicator.py
  122. +74 −49 {intelmq/lib → tests/log-procedure}/bot.py
  123. +34 −0 tests/message-factory/code.py
  124. 0 tests/{ → old}/pipeline-rabbitmq.py
  125. 0 tests/{ → old}/pipeline-redis.py
  126. 0 tests/{ → old}/pipeline.py
  127. 0 tests/{ → old}/redis.conf
  128. +4 −0 tests/scripts/cleanup.sh
  129. +208 −0 tests/split-pipeline/bot.py
  130. +76 −0 tests/threads-test-poc/base.py
  131. +154 −0 tests/threads-test-poc/base2.py
  132. +13 −0 tests/threads-test-poc/notes.txt
  133. 0 tests/{threading-tests → threads-test}/bot.py
  134. 0 tests/{threading-tests → threads-test}/conf/BOTS
  135. 0 tests/{threading-tests → threads-test}/conf/pipeline.conf
  136. 0 tests/{threading-tests → threads-test}/conf/runtime.conf
  137. 0 tests/{threading-tests → threads-test}/conf/startup.conf
  138. 0 tests/{threading-tests → threads-test}/conf/system.conf
  139. 0 tests/{threading-tests → threads-test}/pipeline.py
  140. +122 −0 tests/translation_problems/harmonization.conf
  141. +330 −0 tests/translation_problems/harmonization.py
  142. 0 {intelmq/bots/parsers/hpfeeds → tests/translation_problems/malwaredomainlist}/__init__.py
  143. +54 −0 tests/translation_problems/malwaredomainlist/parser.py
  144. 0 {intelmq/bots/experts/cymru → tests/translation_problems/phishtank}/__init__.py
  145. +51 −0 tests/translation_problems/phishtank/parser.py
View

This file was deleted.

Oops, something went wrong.
View
@@ -0,0 +1,62 @@
CHANGELOG
==========
## 2015/06/03 (aaron)
* fixed the license to AGPL in setup.py
* moved back the docs/* files from the wiki repo to docs/. See #205.
* added python-zmq as a setup requirment in UserGuide . See #206
## When did this happen? (XXX FIXME)
* improvements in pipeline
FILE: lib/pipeline.py
- PipelineFactory to give possibility to easily add a new broker (Redis, ZMQ, etc..)
- Splitter feature: if this option is enable, will split the events in source queue to multiple destination queues
* add different messages support
FILE: lib/message.py
- the system is flexible to define a new type of message like 'tweet' without change anything in bot.py, pipeline.py. Just need to add a new class in message.py and harmonization.conf
* add harmonization support
FILE: lib/harmonization.py
FILE: conf/harmonization.conf
- in harmonization.conf is possible to define the fields of a specific message in json format.
- the harmonization.py has datatypes witch contains sanitize and validation methods that will make sure that the values are correct to be part of an event.
* Error Handling
- multiple parameters in configuration which gives possibility to define how bot will handle some errors. Example of parameters:
"error_procedure" - retry or pass in case of error
"error_retry_delay" - time in seconds to retry
"error_max_retries" - number of retries
"error_log_message" - log or not the message in error log
"error_log_exception" - log or not the exception in error log
"error_dump_message" - log or not the message in dump log to be fixed and re-insert in pipeline
* Exceptions
FILE: lib/exceptions.py
- custom exceptions for IntelMQ
* Defaults configrations
- new configuration file to specify the default parameters which will be apllied to all bots. Bots can overwrite the configurations.
* New bots/feeds
View
@@ -9,6 +9,15 @@ Copyright by:
Dalila Lima <dcrypt3d [ at ] gmail.com> -
Fyodor Y <fygrave [ at ] o0o.nu> -
Hélder Fernandes <helder.fernandes [ at ] fccn.pt> RCTS CERT
Krystian Kochanowski <krystian.kochanowski [ at ] gmail.com> -
Tiago Pedrosa < - > -
Josef Bernhart < - > -
ufoczek < - > -
robcza < - > -
Th4nat0s < - > -
Andre Pinheiro <ampp [ at ] dognaedis.com> Dognaedis
Bruno Teixeira <bteixeira [ at ] dognaedis.com> Dognaedis
Leandro Bragues <lbragues [ at ] dognaedis.com> Dognaedis
Ricardo Ferreira <rferreira [ at ] dognaedis.com> Dognaedis
This code is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE version 3.
View
114 README.md
@@ -1,35 +1,79 @@
![IntelMQ](http://s28.postimg.org/r2av18a3x/Logo_Intel_MQ.png)
**IntelMQ** is a solution for CERTs for collecting and processing security
feeds, pastebins, tweets using a message queue protocol.
It's a community driven initiative called **IHAP** (Incident Handling
Automation Project) which was conceptually designed
by European CERTs during several InfoSec events. Its main goal is to
give to incident responders an easy way to collect & process threat
intelligence thus improving the incident handling processes of CERTs.
IntelMQ's design was influenced by
[AbuseHelper](https://bitbucket.org/clarifiednetworks/abusehelper),
however it was re-written from scratch and aims at:
* Reduce the complexity of system administration
* Reduce the complexity of writing new bots for new data feeds
* Reduce the probability of events lost in all process with persistence functionality (even system crash)
* Use and improve the existing Data Harmonization Ontology
* Use JSON format for all messages
* Integration of the existing tools (AbuseHelper, CIF)
* Provide easy way to store data into Log Collectors like ElasticSearch, Splunk
* Provide easy way to create your own black-lists
* Provide easy communication with other systems via HTTP RESTFUL API
It follows the following basic meta-guidelines:
* Don't break simplicity - KISS
* Keep it open source - forever
* Strive for perfection while keeping a deadline
* Reduce complexity/avoid feature bloat
* Embrace unit testing
* Code readability: test with unexperienced programmers
* Communicate clearly
Visit [Wiki page](https://github.com/certtools/intelmq/wiki/).
![IntelMQ](http://s28.postimg.org/r2av18a3x/Logo_Intel_MQ.png)
**IntelMQ** is a solution for CERTs for collecting and processing security
feeds, pastebins, tweets using a message queue protocol.
It's a community driven initiative called **IHAP** (Incident Handling
Automation Project) which was conceptually designed
by European CERTs during several InfoSec events. Its main goal is to
give to incident responders an easy way to collect & process threat
intelligence thus improving the incident handling processes of CERTs.
IntelMQ's design was influenced by
[AbuseHelper](https://bitbucket.org/clarifiednetworks/abusehelper),
however it was re-written from scratch and aims at:
* Reduce the complexity of system administration
* Reduce the complexity of writing new bots for new data feeds
* Reduce the probability of events lost in all process with persistence functionality (even system crash)
* Use and improve the existing Data Harmonization Ontology
* Use JSON format for all messages
* Integration of the existing tools (AbuseHelper, CIF)
* Provide easy way to store data into Log Collectors like ElasticSearch, Splunk
* Provide easy way to create your own black-lists
* Provide easy communication with other systems via HTTP RESTFUL API
It follows the following basic meta-guidelines:
* Don't break simplicity - KISS
* Keep it open source - forever
* Strive for perfection while keeping a deadline
* Reduce complexity/avoid feature bloat
* Embrace unit testing
* Code readability: test with unexperienced programmers
* Communicate clearly
## Table of Contents
1. [How to Install](#how-to-install)
2. [Developers Guide](#dev-guide)
3. [IntelMQ Manager](#control-platform)
4. [Incident Handling Automation Project](#incident-handling-automation-project)
5. [Data Harmonization](#data-harmonization)
6. [Licence](#licence)
<a name="how-to-install"></a>
## How to Install
See [UserGuide](docs/User-Guide.md).
<a name="dev-guide"></a>
## Developers Guide
See [Developers Guide](docs/Developers-Guide.md).
<a name="control-platform"></a>
## IntelMQ Manager
Check the [tool](https://github.com/certtools/intelmq-manager) and manage easily IntelMQ system.
<a name="incident-handling-automation-project"></a>
## Incident Handling Automation Project
* **URL:** http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation
* **Mailing-list:** ihap@lists.trusted-introducer.org
<a name="data-harmonization"></a>
## Data Harmonization
IntelMQ use the Data Harmonization. Check the following
[document](docs/Data-Harmonization.md).
<a name="licence"></a>
## Licence
This software is licensed under GNU Affero General Public License version 3
View
@@ -0,0 +1,13 @@
python-dateutil==1.5
geoip2==0.5.1
dnspython==1.11.1
redis==2.10.3
pymongo==2.7.1
xmpppy==0.5.0rc1
imbox==0.5.5
unicodecsv==0.9.4
pytz==2012d
psutil==2.1.1
pyzmq==14.6.0
pydns==2.3.6
pycurl==7.19.0
View
128 TODO.md

This file was deleted.

Oops, something went wrong.
Oops, something went wrong.

0 comments on commit b0447ae

Please sign in to comment.