Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 13 additions & 41 deletions docs/Bots.md
Original file line number Diff line number Diff line change
Expand Up @@ -486,58 +486,30 @@ pip3 install pymongo>=2.7.1

#### Configuration Parameters:

* `autocommit`: FIXME
The parameters marked with 'PostgreSQL' will be send
to libpq via psycopg2. Check the
[libpq parameter documentation] (https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS)
for the versions you are using.

* `autocommit`: [psycopg's autocommit mode](http://initd.org/psycopg/docs/connection.html?#connection.autocommit), optional, default True
* `connect_timeout`: PostgreSQL connect_timeout, optional, default 5 seconds
* `database`: PostgreSQL database
* `host`: PostgreSQL host
* `port`: PostgreSQL port
* `user`: PostgreSQL user
* `password`: PostgreSQL password
* `sslmode`: FIXME
* `autocommit`: FIXME
* `table`: FIXME
* `sslmode`: PostgreSQL sslmode
* `table`: name of the database table into which events are to be inserted

#### Installation Requirements

```
pip3 install psycopg2>=2.5.5
```
See [REQUIREMENTS.txt](../intelmq/bots/outputs/postgresql/REQUIREMENTS.txt)
from your installation.

#### PostgreSQL Installation

* Install PostgreSQL, at least version 9.4 is recommended.

```bash
> apt-get install postgresql-9.4 python-psycopg2 postgresql-server-dev-9.4
```

* Create a User and Database:

```shell
> su - postgres
> createuser intelmq -W
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n
Password:

> createdb -O intelmq --encoding='utf-8' intelmq-events
```

* Please note the --encoding='utf-8' in the line above! Without it, the output but will not be able to insert utf-8 data into the table.

* Depending on your setup adjust `/etc/postgresql/9.4/main/pg_hba.conf` to allow network connections for the intelmq user.

* Restart PostgreSQL.

* Generate `initdb.sql` by using the [psql_initdb_generator.py](https://github.com/certtools/intelmq/blob/master/intelmq/bin/intelmq_psql_initdb.py) tool which extracts all field names and data types from `Data-Harmonization.md`.

* Create the `events` table:

```bash
> psql intelmq-events < /tmp/initdb.sql # as intelmq user
> psql -U intelmq intelmq-events -W < /tmp/initdb.sql # as other user
```

See [outputs/postgresql/README.md](../intelmq/bots/outputs/postgresql/README.md)
from your installation.

* * *

Expand Down
28 changes: 25 additions & 3 deletions intelmq/bots/outputs/postgresql/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
You have two basic choices to run PostgreSQL:
1. on the same machine as intelmq, then you could use unix-sockets if available on your platform
2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right connection parameters to each psql or client call.

Make sure to consult your PostgreSQL documentation
about how to allow network connections and authentification in case 2.


# PostgreSQL Version
Any supported version of PostgreSQL should work
(v>=9.2 as of Oct 2016)[[1](https://www.postgresql.org/support/versioning/)].

If you use PostgreSQL server v >= 9.4, it gives you the possibility
to use the time-zone [formatting string](https://www.postgresql.org/docs/9.4/static/functions-formatting.html) "OF" for date-times
and the [GiST index for the cidr type](https://www.postgresql.org/docs/9.4/static/release-9-4.html#AEN120769). This may be useful depending on how
you plan to use the events that this bot writes into the database.

# How to install:

Use `intelmq_psql_initdb` to create initial sql-statements
Expand All @@ -13,15 +30,20 @@ the expert/certbund_contact bot.)
Therefore if still necessary: create the database-user
as postgresql superuser, which usually is done via the system user `postgres`:
```
createuser --encrypted --pwprompt intelmq
createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq
```

Create the new database:
```
createdb --owner=intelmq intelmq-events
createdb --encoding='utf-8' --owner=intelmq intelmq-events
```

Now initialise it as database-user `intelmq` (should ask for the password):
(The encoding parameter should ensure the right encoding on platform
where this is not the default.)

Now initialise it as database-user `intelmq` (in this example
a network connection to localhost is used, so you would get to test
if the user `intelmq` can authenticate):
```
psql -h localhost intelmq-events intelmq </tmp/initdb.sql
```