v1.0 is now available

Former-commit-id: 9970d67

CHANGELOG.md

@@ -0,0 +1,62 @@
+CHANGELOG
+==========
+
+## 2015/06/03 (aaron)
+
+  * fixed the license to AGPL in setup.py
+  * moved back the docs/* files from the wiki repo to docs/. See #205.
+  * added python-zmq as a setup requirment in UserGuide . See #206
+
+
+
+
+## When did this happen? (XXX FIXME)
+
+* improvements in pipeline
+  FILE: lib/pipeline.py
+
+  - PipelineFactory to give possibility to easily add a new broker (Redis, ZMQ, etc..)
+  - Splitter feature: if this option is enable, will split the events in source queue to multiple destination queues
+
+
+
+* add different messages support
+  FILE: lib/message.py
+
+  - the system is flexible to define a new type of message like 'tweet' without change anything in bot.py, pipeline.py. Just need to add a new class in message.py and harmonization.conf
+
+
+
+* add harmonization support
+  FILE: lib/harmonization.py
+  FILE: conf/harmonization.conf
+
+  - in harmonization.conf is possible to define the fields of a specific message in json format.
+  - the harmonization.py has datatypes witch contains sanitize and validation methods that will make sure that the values are correct to be part of an event.
+
+
+
+* Error Handling
+  - multiple parameters in configuration which gives possibility to define how bot will handle some errors. Example of parameters:
+  "error_procedure" - retry or pass in case of error
+  "error_retry_delay" - time in seconds to retry
+  "error_max_retries" - number of retries
+  "error_log_message" - log or not the message in error log
+  "error_log_exception" - log or not the exception in error log
+  "error_dump_message" - log or not the message in dump log to be fixed and re-insert in pipeline
+
+
+
+* Exceptions
+  FILE: lib/exceptions.py
+
+  - custom exceptions for IntelMQ
+
+
+
+* Defaults configrations
+  - new configuration file to specify the default parameters which will be apllied to all bots. Bots can overwrite the configurations.
+
+
+
+* New bots/feeds

COPYRIGHT

@@ -9,6 +9,15 @@ Copyright by:
    Dalila Lima <dcrypt3d [ at ] gmail.com> -
    Fyodor Y <fygrave [ at ] o0o.nu> -
    Hélder Fernandes <helder.fernandes [ at ] fccn.pt> RCTS CERT
-
+   Krystian Kochanowski <krystian.kochanowski [ at ] gmail.com> -
+   Tiago Pedrosa < - > -
+   Josef Bernhart < - > -
+   ufoczek < - > - 
+   robcza < - > - 
+   Th4nat0s < - > -
+   Andre Pinheiro <ampp [ at ] dognaedis.com> Dognaedis
+   Bruno Teixeira <bteixeira [ at ] dognaedis.com> Dognaedis
+   Leandro Bragues <lbragues [ at ] dognaedis.com> Dognaedis
+   Ricardo Ferreira <rferreira [ at ] dognaedis.com> Dognaedis
 
 This code is licensed under the GNU AFFERO GENERAL PUBLIC LICENSE version 3.

README.md

@@ -1,35 +1,79 @@
-![IntelMQ](http://s28.postimg.org/r2av18a3x/Logo_Intel_MQ.png)
-
-**IntelMQ** is a solution for CERTs for collecting and processing security 
-feeds, pastebins, tweets using a message queue protocol. 
-It's a community driven initiative called **IHAP** (Incident Handling 
-Automation Project) which was conceptually designed 
-by European CERTs during several InfoSec events. Its main goal is to 
-give to incident responders an easy way to collect & process threat 
-intelligence thus improving the incident handling processes of CERTs.
-
-IntelMQ's design was influenced by 
-[AbuseHelper](https://bitbucket.org/clarifiednetworks/abusehelper), 
-however it was re-written from scratch and aims at:
-
-* Reduce the complexity of system administration
-* Reduce the complexity of writing new bots for new data feeds
-* Reduce the probability of events lost in all process with persistence functionality (even system crash)
-* Use and improve the existing Data Harmonization Ontology
-* Use JSON format for all messages
-* Integration of the existing tools (AbuseHelper, CIF)
-* Provide easy way to store data into Log Collectors like ElasticSearch, Splunk
-* Provide easy way to create your own black-lists
-* Provide easy communication with other systems via HTTP RESTFUL API
-
-It follows the following basic meta-guidelines:
-
-* Don't break simplicity - KISS
-* Keep it open source - forever
-* Strive for perfection while keeping a deadline
- * Reduce complexity/avoid feature bloat
- * Embrace unit testing
- * Code readability: test with unexperienced programmers
-* Communicate clearly
-
-Visit [Wiki page](https://github.com/certtools/intelmq/wiki/).
+![IntelMQ](http://s28.postimg.org/r2av18a3x/Logo_Intel_MQ.png)
+
+**IntelMQ** is a solution for CERTs for collecting and processing security 
+feeds, pastebins, tweets using a message queue protocol. 
+It's a community driven initiative called **IHAP** (Incident Handling 
+Automation Project) which was conceptually designed 
+by European CERTs during several InfoSec events. Its main goal is to 
+give to incident responders an easy way to collect & process threat 
+intelligence thus improving the incident handling processes of CERTs.
+
+IntelMQ's design was influenced by 
+[AbuseHelper](https://bitbucket.org/clarifiednetworks/abusehelper), 
+however it was re-written from scratch and aims at:
+
+* Reduce the complexity of system administration
+* Reduce the complexity of writing new bots for new data feeds
+* Reduce the probability of events lost in all process with persistence functionality (even system crash)
+* Use and improve the existing Data Harmonization Ontology
+* Use JSON format for all messages
+* Integration of the existing tools (AbuseHelper, CIF)
+* Provide easy way to store data into Log Collectors like ElasticSearch, Splunk
+* Provide easy way to create your own black-lists
+* Provide easy communication with other systems via HTTP RESTFUL API
+
+It follows the following basic meta-guidelines:
+
+* Don't break simplicity - KISS
+* Keep it open source - forever
+* Strive for perfection while keeping a deadline
+ * Reduce complexity/avoid feature bloat
+ * Embrace unit testing
+ * Code readability: test with unexperienced programmers
+* Communicate clearly
+
+
+## Table of Contents
+
+1. [How to Install](#how-to-install)
+2. [Developers Guide](#dev-guide)
+3. [IntelMQ Manager](#control-platform)
+4. [Incident Handling Automation Project](#incident-handling-automation-project)
+5. [Data Harmonization](#data-harmonization)
+6. [Licence](#licence)
+
+
+<a name="how-to-install"></a>
+## How to Install
+
+See [UserGuide](docs/User-Guide.md).
+
+
+<a name="dev-guide"></a>
+## Developers Guide
+
+See [Developers Guide](docs/Developers-Guide.md).
+
+<a name="control-platform"></a>
+## IntelMQ Manager
+
+Check the [tool](https://github.com/certtools/intelmq-manager) and manage easily IntelMQ system.
+
+
+<a name="incident-handling-automation-project"></a>
+## Incident Handling Automation Project
+
+* **URL:** http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation
+* **Mailing-list:** ihap@lists.trusted-introducer.org
+
+
+<a name="data-harmonization"></a>
+## Data Harmonization
+
+IntelMQ use the Data Harmonization. Check the following 
+[document](docs/Data-Harmonization.md).
+
+<a name="licence"></a>
+## Licence
+
+This software is licensed under GNU Affero General Public License version 3

REQUIREMENTS

@@ -0,0 +1,13 @@
+python-dateutil==1.5
+geoip2==0.5.1
+dnspython==1.11.1
+redis==2.10.3
+pymongo==2.7.1
+xmpppy==0.5.0rc1
+imbox==0.5.5
+unicodecsv==0.9.4
+pytz==2012d
+psutil==2.1.1
+pyzmq==14.6.0
+pydns==2.3.6
+pycurl==7.19.0