No description, website, or topics provided.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
axamd
debian
tests
.gitignore
COPYRIGHT
LICENSE
README.md
setup.py

README.md

Farsight Advanced Exchange Access (AXA) RESTful Interface

The Farsight AXA RESTful Interface adds a streaming HTTP interface on top of the AXA toolkit to enable developers of web-based applications to interface with Farsight's SRA (SIE Remote Access) and RAD (Realtime Anomaly Detector) servers.

Access is controlled via an API key that is passed as the X-API-Key HTTP header.

Advanced Exchange Access (AXA)

AXA is an in-house developed binary protocol for the real-time transit of Security Information Exchange (SIE) traffic.

SIE Remote Access (SRA)

SRA is Farsight Security's bespoke remote access service for SIE traffic. It transits SIE datagrams through an encrypted tunnel to the customer's network. More detail including client tool and C library downloads can be found at the AXA git repository.

Realtime Anomaly Detector (RAD)

RAD is the daughter service to SRA offering custom, server-side filtering of SIE data through the use of in-house developed "RAD Modules". With RAD, customers have access to complex filtering modules that provide services like real-time detection of DNS record corruption and brand infringement campaigns. RAD toolware is also available at the AXA git repository.

AXA Watch Format

To signify interest in a particular SIE channel, IP address, or domain, AXA uses the concept of "watches". These are summarized below:

Watch Type Syntax Notes
Channel ch=<#> Enable "all watch" for specified SIE channel number. Not supported in RAD mode.
IP Address ip=<ipv4 or ipv6 literal> Watch IP-based SIE traffic containing specified IP address. In CIDR notation.
Domain Name dns=<domain> Watch DNS-based SIE traffic containing specified domain name. Supports wildcards such as *. or *.example.com, but not www.* or *.host.*

Examples:

Watch Description
ch=212 All messages on channel 212.
ip=198.51.100.0/24 All messages referencing TEST-NET-2.
dns=*.example.com. All messages matching *.example.com.
dns=*. All messages with domain names.

Python Reference Implementation

Install axamd_client as per the following:

# python setup.py install

A full-featured client is installed as axamd_client. You may configure the client using /etc/axamd-client.conf, ~/.axamd-client.conf, or any file you specify on the command line. You may forego this and use only command line options if you prefer. Configuration files are YAML or JSON objects. Allowable keys are:

Option Type Description
server string URI to AXAMD server
apikey string Key for authentication
proxy string HTTP proxy
timeout number Socket timeout in seconds
retries integer Number of retries before giving up
retry-backoff number Progressively wait longer on each retry before the next attempt
sample-rate number Channel sampling rate (percent)
rate-limit integer Maximum packets per second
report-interval integer Seconds between emission of server accounting messages (packet statistics)

See client-config-schema.yaml for more details.

The client can be invoked from the command line as follows:

usage: axamd_client [-h] [--config CONFIG] [--number NUMBER]
                    [--duration DURATION] [--server SERVER] [--apikey APIKEY]
                    [--proxy PROXY] [--timeout TIMEOUT] [--retries RETRIES]
                    [--retry-backoff RETRY_BACKOFF] [--rate-limit PPS]
                    [--report-interval SECONDS] [--sample-rate PERCENTAGE]
                    [--list-channels] [--list-anomalies]
                    [--channels [CHANNEL [CHANNEL ...]]]
                    [--watches WATCH [WATCH ...]]
                    [--anomaly [MODULE [OPTIONS ...]]] [--debug] [--version]

Client for the AXA RESTful Interface

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG, -c CONFIG
                        Path to config file
  --number NUMBER, -n NUMBER
                        Return no more than N json messages and stop
  --duration DURATION, -d DURATION
                        Run for hh:mm:ss (or #w#d#h#m#s) duration and then stop.
  --server SERVER, -s SERVER
                        AXAMD server
  --apikey APIKEY, -k APIKEY
                        API key
  --proxy PROXY, -p PROXY
                        HTTP proxy
  --timeout TIMEOUT, -t TIMEOUT
                        Timeout for connections
  --retries RETRIES, -R RETRIES
                        Number of retries before giving up
  --retry-backoff RETRY_BACKOFF, -B RETRY_BACKOFF
                        Progressively backoff for each retry
  --rate-limit PPS, -l PPS
                        AXA rate limit
  --report-interval SECONDS, -i SECONDS
                        AXA report interval
  --sample-rate PERCENTAGE, -r PERCENTAGE
                        AXA sample rate (percentage)
  --list-channels       List available channels
  --list-anomalies      List available anomalies
  --channels [CHANNEL [CHANNEL ...]], -C [CHANNEL [CHANNEL ...]]
                        SRA mode: Channels to listen on
  --watches WATCH [WATCH ...], -W WATCH [WATCH ...]
                        SRA/RAD mode: Watch list
  --anomaly [MODULE [OPTIONS ...]], -A [MODULE [OPTIONS ...]]
                        RAD mode: Anomaly module and options
  --debug               Debug mode
  --version, -V         show program's version number and exit

Example usage:

 axamd_client --channels 204 --watches dns=farsightsecurity.com.
 axamd_client --channels 204 --watches dns=farsightsecurity.com. -c 5
 axamd_client --channels 204 --watches dns=farsightsecurity.com. -d 00:00:30

Note that -c and -d can be used together.

The client may also be used programmatically. It is compatible with both Python 2 and Python 3. The client raises axamd.ProblemDetails exceptions from server errors corresponding to the internet draft draft-ietf-appsawg-http-problem-03. For full details, invoke:

$ pydoc axamd.client

Example usage:

from axamd.client import Client, Anomaly
import json

apikey = '00000000-0000-0000-0000-00000000'
c = Client('https://axamd.sie-remote.net', apikey)

for line in c.sra(channels=[212], watches=['ch=212']):
    data = json.loads(line)

# or

import nmsg

# detect the string "mail" anywhere in the DNS label hierarchy using the literal and homoglyph matchers, ignore anything in the *.org TLD
for line in c.rad(anomalies=[Anomaly('brand_sentry', ['dns=*.'], 'brand=mail matcher=lit,hgl whitelist=*.org')],
        output_format='nmsg+json'):
    msg = nmsg.message.from_json(line)

While the distribution has several python dependencies, axamd.client.Client only depends on the Python requests module. Schema validation for parameters is enabled if the PyYAML and jsonschema modules are available.

nmsg is a Python module published by Farsight Security. See our software installation instructions for more details.

Protocol

The Farsight AXA RESTful Interface consists of four methods. Two of the methods, "Channel List" and "Anomaly List," are used to interrogate the server about which channels and anomaly modules the user is allowed to access. The remaining methods, "SRA Stream" and "RAD Stream," stream line-delimited JSON data per JSON-encoded parameters sent in a POST request.

Responses are encoded as JSON Text Sequences per RFC 7464. Each JSON text is preceded by one record separator character (ASCII 0x1e) and succeeded by one line break character (ASCII 0x20). In addition, AXAMD's implementation of RFC 7464 does not include line breaks within JSON texts so clients may use the line break character as the termination of the record and do not need to wait to receive a record separator.

The server reports problems using the protocol defined in the Problem Reports for HTTP APIs internet draft. See the Error Responses section for a detailed listing of all problem codes.

Channel List

  • URL

    /v1/sra/channels

  • METHOD

    GET

  • Success Response:

    Code: 200
    Content: { "ch14": "Darknet data" }

id: http://farsightsecurity.com/axamd-channels-output-schema#
$schema: http://json-schema.org/draft-04/schema#
title: axamd channels output schema
description: Schema describing the output of the AXAMD channels command, a mapping of channel numbers to names
type: object
patternProperties:
        '^ch[0-9]+$':
                description: Channel number and name
                type: string
  • Error Response:
Error type Reason
internal-server-error An internal error has occurred. Contact support, providing the logid from the response.
missing-api-key The X-API-Key header was not provided.
invalid-api-key The API key provided does not exist.
broken-api-key An internal error has occurred. Contact support, providing the logid from the response.
bad-request Unable to parse request. See the reason field in the response for details.

See Error Responses for more details.

  • Sample Call:

    curl --header 'X-API-Key: <elided>' $AXAMD_SERVER/v1/sra/channels

Anomaly List

  • URL

    /v1/rad/anomalies

  • METHOD

    GET

  • Success Response:

    Code: 200
    Content: { "brand_sentry": "Detect possible brand infringement." }

id: http://farsightsecurity.com/axamd-anomalies-output-schema#
$schema: http://json-schema.org/draft-04/schema#
title: axamd anomalies output schema
description: Schema describing the output of the AXAMD anomalies command, a mapping of anomaly module names to descriptions
type: object
patternProperties:
        '^ch[0-9]+$':
                description: Anomaly module name and description
                type: string
  • Error Response:
Error type Reason
internal-server-error An internal error has occurred. Contact support, providing the logid from the response.
missing-api-key The X-API-Key header was not provided.
invalid-api-key The API key provided does not exist.
broken-api-key An internal error has occurred. Contact support, providing the logid from the response.
bad-request Unable to parse request. See the reason field in the response for details.

See Error Responses for more details.

  • Sample Call:

    curl --header 'X-API-Key: <elided>' $AXAMD_SERVER/v1/rad/anomalies

SRA Stream

  • URL

    /v1/sra/stream

  • METHOD

    POST

  • Data Params

This method takes a JSON or YAML document conforming to the SRA Stream Parameters JSON Schema.

Field Name Type Required Description
channels array[integer] Yes Channel numbers to enable.
watches array[string] Yes Watch strings.
sample_rate number No Sampling rate (float over (0..1]) for the SRA server.
rate_limit integer No Maximum watch hits per second.
report_interval integer No Seconds between statistics messages.
output_format string No One of axa+json, nmsg+json.

See the AXA Watch Format for more details on watch syntax.

  • Success Response:

See the Stream Output Formats section.

  • Error Response:
Error type Reason
internal-server-error An internal error has occurred. Contact support, providing the logid from the response.
missing-api-key The X-API-Key header was not provided.
invalid-api-key The API key provided does not exist.
broken-api-key An internal error has occurred. Contact support, providing the logid from the response.
bad-request Unable to parse request. See the reason field in the response for details.

See Error Responses for more details.

  • Sample Call:

    curl --data '{ "channels":[212], "watches":["ch=212"] }' --header 'X-API-Key: <elided>' https://axamd.sie-remote.net/v1/sra/stream

RAD Stream

  • URL

    /v1/rad/stream

  • METHOD

    POST

  • Data Params

This method takes a JSON or YAML document conforming to the RAD Stream Parameters JSON Schema.

Field Name Type Required Description
anomalies object(module, watches, options) Yes Anomaly module definitions.
.. module string Yes Anomaly module name.
.. watches array[string] Yes Watch strings
.. options string No Anomaly module options.
sample_rate number No Sampling rate (float over (0..1]) for the SRA server.
rate_limit integer No Maximum watch hits per second.
report_interval integer No Seconds between statistics messages.
output_format string No One of axa+json, nmsg+json.

See the AXA Watch Format for more details on watch syntax.

  • Success Response:

See the Stream Output Formats section.

  • Error Response:
Error type Reason
internal-server-error An internal error has occurred. Contact support, providing the logid from the response.
missing-api-key The X-API-Key header was not provided.
invalid-api-key The API key provided does not exist.
broken-api-key An internal error has occurred. Contact support, providing the logid from the response.
bad-request Unable to parse request. See the reason field in the response for details.

See Error Responses for more details.

  • Sample Call:

    curl --data '{ "anomalies": [{ "module": "brand_sentry", "watches": ["dns=*."], "options": "brand=mail matcher=lit" }], "report_interval": 86400 }' --header 'X-API-Key: <elided>' 'https://axamd.sie-remote.net/v1/rad/stream

Stream Output Formats

You may specify any of three output formats for the two stream commands. The default, axa+json, is the native AXA format and supports all messages. The two nmsg formats only support watch and anomaly hits and do not include status messages at this time.

AXA JSON Messages

This is the default message format. Output consists of line-delimited, JSON-encoded AXA protocol strings. It can be explicitly requested using the parameter output_format=axa+json.

Each protocol message contains the following fields:

Field Name Type Description
tag "*" or integer Associates this message with a watch or anomaly module
op string Specifies which type of message this is.

For tags, "*" is for tagless messages, such as statistics. Otherwise, the tag number corresponds to the index of the watch or anomaly in the stream parameters, plus one.

The following example describes how watches are assigned for SRA streams:

{
   "watches": [
       "ip=192.0.2.0/24",
       "ip=198.51.100.0/24",
   ]
}

The watch ip=192.0.2.0/24 is assigned tag 1 and the watch ip=198.51.100.0/24 is assigned tag 2.

The following example describes how watches are assigned for RAD streams:

{
    "anomalies": [
        {
           "module": "ip_probe",
	   "watches": [
	       "ip=192.0.2.0/24",
	       "ip=198.51.100.0/24",
	   ]
        },
        {
           "module": "ip_probe",
	   "watches": [
	       "ip=203.0.113.0/24",
	   ]
        }
    ]
}

The watches ip=192.0.2.0/24 and ip=198.51.100.0/24 are assigned tag 1, and ip=203.0.113.0/24 is assigned tag 2.

There are four AXA op codes that are of interest to a user of the RESTful interface: MISSED and WATCH HIT for SRA users, RAD MISSED and ANOMALY HIT for RAD users.

  • MISSED
Field Name Type Description
missed integer The number of packets (SIE messages or raw IP packets) lost in the network between the source and the SRA server or dropped by the SRA server because it was too busy.
dropped integer The number of packets dropped by SRA client-server congestion.
rlimit integer The number of packets dropped by rate limiting.
filtered integer The total number of packets considered.
last_report integer The UNIX epoch of the previous report.
  • WATCH HIT (nmsg)

This is the watch hit that you will most often see. It is fired when SRA matches your watch against a nmsg message.

Field Name Type Description
channel string The channel number (ch#) on which this hit was observed.
nmsg object The JSON-encoded nmsg object. See NMSG JSON Messages.
field or field_idx string or integer The nmsg field (or index)that triggered this watch hit.
val_idx integer The value index within the nmsg field that triggered this watch hit.
vname or vid string or integer The NMSG vendor name (or ID).
mname or msgtype string or integer The NMSG module name (or ID).
time string Timestamp when the NMSG message was reported.
  • WATCH HIT (ip)

You will see this watch hit type when SRA matches your watch against an IP packet on an IP-only channel (such as the darknet channel).

Field Name Type Description
channel string The channel number (ch#) on which this hit was observed.
af string or integer The address family of src and dst.
src string Source address.
dst string Destination address.
ttl int IP time to live.
payload string base64-encoded packet payload.
proto string Network protocol.
src_port integer Network protocol src port.
dst_port integer Network protocol dst port.
flags array[string] TCP flags.
time string Timestamp when the IP packet was captured.

TCP flags are any of SYN, FIN, ACK, RST.

  • RAD MISSED
Field Name Type Description
sra_missed integer The number of packets (SIE messages or raw IP packets) lost in the network between the source and the SRA server or dropped by the SRA server because it was too busy.
sra_dropped integer The number of packets dropped by SRA client-server congestion.
sra_rlimit integer The number of packets dropped by rate limiting by the SRA server.
sra_filtered integer The total number of packets considered by the SRA server.
dropped integer The number of packets dropped by RAD client-server congestion.
rlimit integer The number of packets dropped by RAD rate limiting.
filtered integer The total number of packets considered by RAD modules.
last_report integer The UNIX epoch of the previous report
  • Anomaly Hit

Anomaly hits have all of the same fields of nmsg and IP watch hits, plus the following:

Field Name Type Description
an string Module that detected the anomaly.

Please consult the AXA json schema for full details about every protocol message type. Please do not use this schema to validate input. It is very large and computationally expensive to validate.

NMSG JSON Messages

The nmsg+json output format is useful if you intend to use the AXA RESTful interface solely as a transport protocol for nmsg messages. Messages can be loaded directly using the nmsg_message_from_json function (or nmsg.message.from_json in Python). It is requested using the parameter output_format=nmsg+json.

Field Name Type Description
mname string Module name.
vname string Vendor name.
time string Time when the message was recorded.
message object Module-specific fields.

Error Responses

The AXA RESTful interface implements the IETF draft draft-ietf-appsawg-http-problem-03 for all error responses. All responses have the following fields:

Field Description
type The problem type. See below.
title Human-readable title.
detail Human-readable error details. (optional)
status HTTP status code.
logid UUID identifying this transaction in log messages. Provide this to support for all problem-related queries.

Problem Types

All problem types below have the URI base https://www.farsightsecurity.com/axamd/problems. It is omitted for brevity.

Identifier Description
internal-server-error An internal error has occurred. Please contact support, providing the logid.
missing-api-key The X-API-Key header was not included in the request.
invalid-api-key The API Key is not provisioned.
broken-api-key An internal authentication error has occurred. Please contact support, providing the logid. Has an api-key field containing the damaged API key.
bad-request The request URI, method, or parameters are invalid. See the reason field for an explanation.
sra-not-enabled SIE Remote Access is not enabled for this account.
rad-not-enabled Realtime Anomaly Detector is not enabled for this account.
connection-error There was an internal error connecting to the SRA or RAD server. Please contact support, providing the logid.
channel-not-found The channel requested either does not exist or is not provisioned for this account. See the channel field for the channel number that caused the problem.
channel-error The SRA server returned an error when a channel was requested. See detail for a human-readable error message and the channel field for which channel caused the error.
watch-error The SRA or RAD server returned an error when a watch was requested. See detail for a human-readable error message. The anomaly field is populated in RAD mode with the anomaly module that the watch is connected with and the watch field for which watch caused the error.
anomaly-not-found The anomaly module requested either does not exist or is not provisioned for this account. See the anomaly field for the anomaly module name that caused the problem.
anomaly-error The RAD server returned an error when an anomaly was requested. See detail for a human-readable error message and anomaly for which anomaly module caused the error.
rate-limit-error The SRA or RAD server returned an error when setting the rate limit or report-interval. See the detail field for a human-readable error message and the limit or report-interval fields for the rate limit or report interval that caused the issue.
sample-rate-error The SRA or RAD server returned an error when setting the sample rate. See detail for a human-readable error message. The invalid sample rate is included in the rate field.