Reconciler

Description

The Reconciler is a "stand-alone" python tool to add context to generated Darwin alerts. It connects to sources and destinations to wait for new alerts, get additional context, add it to the alert, and add the alert to the configured destinations.

Managers

The Reconciler possess a set of "Managers", those managers' goal is to connect to sources and destinations to get information from/write results to. There are currently 2 managers developed, with different level of implementation/different capabilities:

• a Redis Manager
  • able to listen to alerts coming from a pubsub channel
  • able to poll a redis list to pop alerts (with a configurable polling frequency)
  • able to pop alerts from a list as soon as a message is received on the pubsub channel (in that case, pubsub messages are ignored, and only serve as triggers to pop from list)
  • able to get a context from a key
  • able to write reconciled alerts to Redis' list and/or pubsub channel
• a File Manager
  • able to write reconciled alerts to a file

Executable

Although the tool is able to be imported as a module, it is also developed to work as a stand-alone executable.

Parameters

The parameters can be accessed by providing the -h argument, and are:

• configuration_file [mandatory]: the path to the json configuration file (documented below)
• -v [optional]: the flag to increase verbosity (default is ERROR)
  • -v is for WARNING
  • -vv is for INFO
  • -vvv is for DEBUG
• -l logfile [optional]: the path to the log file to use to write logs to
• -c [optional, advanced]: tells the program the connected Redis are part of a cluster (behaviour documented below)

Configuration file

Configuration file allows to set the different managers for each step of the process (getting alerts, getting context, and writing alerts).

The different objects possible are:

• alerts_source [object, mandatory]
  • redis [object, mandatory]
    • unix [string, required if no ip]: the fullpath to the unix socket to connect to
    • ip [string, required if no unix]: the ip of the server to connect to
    • port [number, required if ip]: the port of the server to connect to
    • list [string]: the list to use when popping alerts
    • channel [string]: the channel to listen to when waiting for alerts
    • seconds_between_retries [number, default 1]: the number of seconds between 2 context queries
    • max_retries [number, default 1]: the number of times to try and query the context
• context_source [object, mandatory]
  • redis [object, mandatory]
    • unix [string, required if no ip]: the fullpath to the unix socket to connect to
    • ip [string, required if no unix]: the ip of the server to connect to
    • port [number, required if ip]: the port of the server to connect to
• alerts_destination [object, mandatory]
  • redis [object, mandatory if no file]
    • unix [string, required if no ip]: the fullpath to the unix socket to connect to
    • ip [string, required if no unix]: the ip of the server to connect to
    • port [number, required if ip]: the port of the server to connect to
    • list [string]: the list to use when adding reconciled alerts
    • channel [string]: the channel to publish to when adding reconciled alerts
  • file [object, mandatory if no redis]
    • path [string, required]: the path to the file to use when writing reconciled

example:

{
    "alerts_source": {
        "redis": {
            "unix": "/var/sockets/redis/redis.sock",
            "channel":"darwin.alerts",
            "list": "darwin_alerts",
            "max_retries": 2,
            "seconds_between_retries": 1
        }
    },
    "context_source": {
        "redis": {
            "ip": "127.0.0.1",
            "port": 6279
        }
    },
    "alerts_destination": {
        "file": {
            "path": "/tmp/reconciled_alerts.log"
        },
        "redis": {
            "ip": "192.168.1.123",
            "port": 6666,
            "list": "vlt.darwin.alerts"
        }
    }
}

Clustering

When working on high constraints environment, Redis server can be shared and clustered.

To allow correct behaviour and reliability of the tool, the argument -c/--clustering can be provided to tell the program to discover each Redis infrastructures, connect to masters, and remember replicas as fallbacks.

This allows the Redis Manager to adapt to redis instances becoming replicas, disconnecting or failing, by launching a new search on active connection AND fallbacks registered at the launch of the program. The tool will always try to discover a new master, and if one is found will save all its replicas as fallbacks to use in case the new master fails/becomes a replica.

This behaviour can cover almost all regular cases encountered, but not everyone: the tool is dependent on the state of the architecture when launched, and will basically not be able to redirect to a redis server added to the cluster AFTER Reconciler was launched.

If two or more servers are part of the cluster when the tool is launched, it will most likely be able to update to the updated architecture by using old servers that were already present at the time of start, but it won't if replicas were absent during startup. In this case a restart of the tool will most likely be necessary to ensure best reliability.

Module

This tool is designed to be available as an external module imported to an existing python codebase, there are some caveats when using and configuring the module, but documentation and example (through the main function) can be found in the source code