Home

Welcome to the epic-stream-processor wiki!

This wiki describes the usage and implementation details of the epic-stream-processor that is currently deployed on a single node at LWA-SV.

Installation

-TBD-

Usage

This server can be accessed on the CLI using the executable epic-stream-processor. It provides the following commands:

• start
• watch

$ epic-stream-processor --help
Usage: epic-stream-processor [OPTIONS] COMMAND [ARGS]...

  Epic Stream Processor.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  start  Start the epic stream processing server
  watch  Watch for the specified source in the incoming data.

Starting the server

The command start provides the following options

$ epic-stream-processor start --help
Usage: epic-stream-processor start [OPTIONS]

  Start the epic stream processing server

Options:
  -d, --db-name TEXT           DB name
  -p, --port INTEGER           DB port
  -U, --db-user TEXT           DB user
  -DB, --db-type [postgresql]  DB Type
  -h, --db-host TEXT           DB host (use sockets for local connections)
  --password TEXT              DB password. Defaults to peer authentication if
                               not set
  --help                       Show this message and exit.

The processor injects data into a Postgres database and requires a running instance. The processor auto-creates all the required tables if they don't exist. Refer to the internal docs for credentials. Example usage:

$ epic-stream-processor start -U epic -d epic_db -p 5433

The Postgres instance internally uses UNIX domain sockets (UDS) for communication on the same server and uses peer authentication. That means, it matches the db username with the user requesting the connection for authentication. Hence a password is generally not required. See Configuring Postgres for more details.

Adding a source to the watch list

The stream processor maintains a list of sources of interest in a postgres db. It uses this list to extract pixels from the incoming image and stores them the database (see Postgres schema for details on the schema). Any manual or automatic triggers must use this interface to add sources to the list. The watch command provides this functionality with the following options:

$ epic-stream-processor watch --help
Usage: epic-stream-processor watch [OPTIONS]

  Watch for the specified source in the incoming data.

  Any pixel data falling within the image FOV will be stored in a pgDB. For
  solar system sources specify the ra and dec as 0, 0.

Options:
  -n, --source-name TEXT          Name of the source  [required]
  -ra, --right-ascension TEXT     Right ascension (J2000, ICRS)  [required]
  -dec, --declination TEXT        Declination (J2000, ICRS)  [required]
  -fmt, --skypos-format [deg|hms]
                                  Format of the specified sky coordinates.
                                  Defaults to decimal degrees
  -wm, --watch-mode [continuous|timed]
                                  Watching mode for the source. Defaults to
                                  timed (1 week)
  -pt, --patch-type TEXT          Size of the aggregation window (nxn).
                                  Defaults to 3x3
  -R, --reason TEXT               Short description for the reason to monitor
                                  this source (e.g., Detect FRBs)  [required]
  -A, --author TEXT               Author's name  [required]
  -wb, --watch-begin TEXT         UTC time to begin watching the source.
                                  Defaults to now.         Use the format
                                  YYYY-MM-DDThh:mm:ss.ssss.
  -wd, --watch-duration TEXT      Duration of the watch. Defaults to one week.
                                  Provide duration in human readable form. For
                                  example,  as '6d23h59m59s9ms1us' or '7d'.
                                  This value is not used when watch_mode is
                                  set to continuous.
  --help                          Show this message and exit.

Internal design

The imaging data from the EPIC-imager is transported into the stream processor using UNIX domain sockets (UDS). Previous implementation used gRPC as the backend for communication