Permalink
Browse files

Merge branch 'maintenance' into develop

  • Loading branch information...
wagner-certat committed Sep 25, 2018
2 parents e124490 + 46bdab0 commit 13e30af8a05d99e556636641ada5331cd022159c
View
@@ -54,6 +54,11 @@ CHANGELOG
------------------
### Core
- `lib/harmonization.py`: Change `parse_utc_isoformat` of `DateTime` class from private to public (related to #1322).
- `lib/utils.py`: Add new function `object_pair_hook_bots`.
### Development
- `bin/rewrite_config_files.py`: Fix ordering of BOTS file (#1327).
### Core
@@ -71,15 +76,19 @@ CHANGELOG
- `intelmq.bots.outputs.redis`: Fix sending password to redis server.
### Documentation
- FAQ: Explanation and solution on orphaned queues.
- Add or fix the tables of contents for all documentation files.
### Packaging
- Change the maintainer from Sasche Wilde to Sebastian Wagner (#1320).
### Tests
### Tools
- `intelmqctl check`: Shows more detailed information on orphaned queues.
### Contrib
- elasticsearch/elasticmapper: Add tlp field (#1308).
### Known issues
@@ -9,7 +9,7 @@ import sys
try:
from elasticsearch import Elasticsearch
except:
except ImportError:
print("[-] Please install ElasticSearch using the following command: 'pip install elasticsearch'.", file=sys.stderr)
sys.exit(-1)
@@ -84,7 +84,7 @@ def send_mapping(host, index, data):
try:
es = Elasticsearch([host], verify_certs=True)
response = es.indices.create(index=index, ignore=400, body=data)
except:
except Exception:
err = -1
return response, err
@@ -6,7 +6,7 @@ import sys
try:
import yaml
except:
except ImportError:
print("[-] Please install yaml using the following command: 'pip install pyyaml'.", file=sys.stderr)
sys.exit(-1)
@@ -41,7 +41,7 @@ def get_option_selected(options):
selection = int(selection)
if selection < len(options):
break
except:
except Exception:
print("Bad option.")
return selection
@@ -135,4 +135,4 @@ if __name__ == "__main__":
print("\n\n\n")
print("PIPELINE CONFIGURATION:\n")
print(json.dumps(pipeline_config, indent=4))
print(json.dumps(pipeline_config, indent=4))
View
@@ -1,9 +1,169 @@
# Bots Documentation
1. [Collectors](#collectors)
2. [Parsers](#parsers)
3. [Experts](#experts)
4. [Outputs](#outputs)
**Table of Contents:**
- [Bots Documentation](#bots-documentation)
- [General remarks](#general-remarks)
- [Initialization parameters](#initialization-parameters)
- [Common parameters](#common-parameters)
- [Collectors](#collectors)
- [Generic URL Fetcher](#generic-url-fetcher)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Generic URL Stream Fetcher](#generic-url-stream-fetcher)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Generic Mail URL Fetcher](#generic-mail-url-fetcher)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Generic Mail Attachment Fetcher](#generic-mail-attachment-fetcher)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Fileinput](#fileinput)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [MISP Generic](#misp-generic)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Request Tracker](#request-tracker)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Shodan Stream](#shodan-stream)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [TCP](#tcp)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [XMPP collector](#xmpp-collector)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Alien Vault OTX](#alien-vault-otx)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Blueliv Crimeserver](#blueliv-crimeserver)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Microsoft Azure](#microsoft-azure)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Microsoft Interflow](#microsoft-interflow)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Additional functionalities](#additional-functionalities)
- [Stomp](#stomp)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Twitter](#twitter)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Parsers](#parsers)
- [Generic CSV Parser](#generic-csv-parser)
- [Configuration parameters](#configuration-parameters)
- [Cymru CAP Program](#cymru-cap-program)
- [Information:](#information)
- [Cymru Full Bogons](#cymru-full-bogons)
- [Information:](#information)
- [Twitter](#twitter)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Shodan](#shodan)
- [Information](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Experts](#experts)
- [Abusix](#abusix)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [ASN Lookup](#asn-lookup)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Cymru Whois](#cymru-whois)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Domain Suffix](#domain-suffix)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Rule processing](#rule-processing)
- [Deduplicator](#deduplicator)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Field Reducer Bot](#field-reducer-bot)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Whitelist](#whitelist)
- [Blacklist](#blacklist)
- [Filter](#filter)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Generic DB Lookup](#generic-db-lookup)
- [Gethostbyname](#gethostbyname)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [IDEA](#idea)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [MaxMind GeoIP](#maxmind-geoip)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Modify](#modify)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Configuration File](#configuration-file)
- [Actions](#actions)
- [Examples](#examples)
- [Types](#types)
- [National CERT contact lookup by CERT.AT](#national-cert-contact-lookup-by-certat)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Reverse DNS](#reverse-dns)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [RFC1918](#rfc1918)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [RipeNCC Abuse Contact](#ripencc-abuse-contact)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Sieve](#sieve)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Taxonomy](#taxonomy)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Tor Nodes](#tor-nodes)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Url2FQDN](#url2fqdn)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Wait](#wait)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Outputs](#outputs)
- [File](#file)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Filename formatting](#filename-formatting)
- [Files](#files)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [MongoDB](#mongodb)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Installation Requirements](#installation-requirements)
- [PostgreSQL](#postgresql)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [Installation Requirements](#installation-requirements)
- [PostgreSQL Installation](#postgresql-installation)
- [REST API](#rest-api)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [SMTP Output Bot](#smtp-output-bot)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
- [TCP](#tcp)
- [Information:](#information)
- [Configuration Parameters:](#configuration-parameters)
## General remarks
View
@@ -1,16 +1,23 @@
## Table of Contents
1. [Overview](#overview)
2. [Rules for keys](#rules)
3. [Sections](#sections)
4. [Data types](#basicdatatypes)
5. [Fields List and data types](#fields-list-and-data-types)
6. [Type/Taxonomy Mapping](#mapping)
7. [Minimum required fields](#requirements)
**Table of Contents:**
- [Overview](#overview)
- [Rules for keys](#rules-for-keys)
- [Sections](#sections)
- [Feed](#feed)
- [Time](#time)
- [Source Identity](#source-identity)
- [Source Geolocation Identity](#source-geolocation-identity)
- [Source Local Identity](#source-local-identity)
- [Destination Identity](#destination-identity)
- [Destination Geolocation Identity](#destination-geolocation-identity)
- [Destination Local Identity](#destination-local-identity)
- [Extra values](#extra-values)
- [Fields List and data types](#fields-list-and-data-types)
- [Classification](#classification)
- [Minimum recommended requirements for events](#minimum-recommended-requirements-for-events)
<a name="overview"></a>
## Overview
All messages (reports and events) are Python/JSON dictionaries. The key names and according types are defined by the so called *harmonization*.
@@ -20,14 +27,11 @@ Every event **MUST** contain a timestamp field.
[IOC](https://en.wikipedia.org/wiki/Indicator_of_compromise) (Indicator of compromise) is a single observation like a log line.
<a name="rules"></a>
## Rules for keys
The keys can be grouped together in sub-fields, e.g. `source.ip` or `source.geolocation.latitude`. Thus, keys must match `^[a-z_](.[a-z0-9_]+)*$`.
<a name="sections"></a>
## Sections
As stated above, every field is organized under some section. The following is a description of the sections and what they imply.
@@ -72,12 +76,10 @@ Some sources report an internal (NATed) IP address.
### Extra values
Data which does not fit in the harmonization can be saved in the 'extra' namespace. All keys must begin with `extra.`, there are no other rules on key names and values. The values can be get/set like all other fields.
<a name="fields-list-and-data-types"></a>
## Fields List and data types
A list of allowed fields and data types can be found in [Harmonization-fields.md](Harmonization-fields.md)
<a name="mapping"></a>
## Classification
IntelMQ classifies events using three labels: taxonomy, type and identifier. This tuple of three values can be used for deduplication of events and describes what happened.
@@ -153,7 +155,6 @@ Example:
If you know of an IP address that connects to a zeus c&c server, it's about the infected device, thus type malware and identifier zeus. If you want to complain about the c&c server, it's type c&c and identifier zeus. The `malware.name` can have the full name, eg. 'zeus_p2p'.
<a name="requirements"></a>
## Minimum recommended requirements for events
Below, we have enumerated the minimum recommended requirements for an actionable abuse event. These keys should to be present for the abuse report to make sense for the end recipient. Please note that if you choose to anonymize your sources, you can substitute **feed** with **feed.code** and that only one of the identity keys **ip**, **domain name**, **url**, **email address** must be present. All the rest of the keys are **optional**.
Oops, something went wrong.

0 comments on commit 13e30af

Please sign in to comment.