Kowalski: a multi-survey data archive and alert broker for time-domain astronomy

Kowalski is an API-driven multi-survey data archive and alert broker. Its main focus is the Zwicky Transient Facility.

Technical details

A schematic overview of the functional aspects of Kowalski and how they interact is shown below:

• A non-relational (NoSQL) database MongoDB powers the data archive, the alert stream sink, and the alert handling service.
• An API layer provides an interface for the interaction with the backend: it is built using a python asynchronous web framework, aiohttp, and the standard python async event loop serves as a simple, fast, and robust job queue. Multiple instances of the API service are maintained using the Gunicorn WSGI HTTP Server.
• A programmatic python client is also available to interact with Kowalski's API.
• Incoming and outgoing traffic can be routed through traefik, which acts as a simple and performant reverse proxy/load balancer.
• An alert brokering layer listens to Kafka alert streams and uses a dask.distributed cluster for distributed alert packet processing, which includes data preprocessing, execution of machine learning models, catalog cross-matching, and ingestion into MongoDB. It also executes user-defined filters based on the augmented alert data and posts the filtering results to a SkyPortal instance.
• Kowalski is containerized using Docker software and orchestrated with docker-compose allowing for simple and efficient deployment in the cloud and/or on-premise.

Interacting with a Kowalski instance

Kowalski is an API-first system. The full OpenAPI specs can be found here. Most users will only need the queries section of the specs.

The easiest way to interact with a Kowalski instance is by using a python client penquins.

Spin up your own kowalski

Clone the repo and cd to the cloned directory:

git clone https://github.com/dmitryduev/kowalski.git
cd kowalski

Use the kowalski.py utility to manage Kowalski.

Make sure the requirements to run the kowalski.py utility are met, e.g.:

pip install -r requirements.txt

Start up Kowalski using the default config/secrets (copying them over):

./kowalski.py up

Config file

You should cp config.defaults.yaml config.yaml instead of using the default config in a production setting. Make sure to choose strong passwords!

config.yaml contains the API and ingester configs, the supevisord config for the API and ingester containers, together with all the secrets, so be careful when committing code / pushing docker images.

Deployment scenarios

./kowalski.py up

Kowalski uses docker-compose under the hood and requires a docker-compose.yaml file. There are several available deployment scenarios:

• Bare-bones
• Bare-bones + broker for SkyPortal / Fritz
• Behind traefik

Bare-bones

Use docker-compose.defaults.yaml as a template for docker-compose.yaml. Note that the environment variables for the mongo service must match admin_* under kowalski.database in config.yaml.

Bare-bones + broker for SkyPortal / Fritz

Use docker-compose.fritz.defaults.yaml as a template for docker-compose.yaml. If you want the alert ingester to post (filtered) alerts to SkyPortal, make sure {"misc": {"broker": true}} in config.yaml.

Behind traefik

Use docker-compose.traefik.defaults.yaml as a template for docker-compose.yaml.

If you have a publicly accessible host allowing connections on port 443 and a DNS record with the domain you want to expose pointing to this host, you can deploy kowalski behind traefik, which will act as the edge router -- it can do many things including load-balancing and getting a TLS certificate from letsencrypt.

In docker-compose.yaml:

• Replace kowalski@caltech.edu with your email.
• Replace private.caltech.edu with your domain.

Run tests

./kowalski.py test

Shut down Kowalski

./kowalski.py down

Docs

OpenAPI specs are to be found under /docs/api once Kowalski is up and running.

Developer guidelines

How to contribute

Contributions to Kowalski are made through GitHub Pull Requests, a set of proposed commits (or patches).

To prepare, you should:

• Create your own fork the kowalski repository by clicking the "fork" button.

• Set up SSH authentication with GitHub.

• Clone (download) your copy of the repository, and set up a remote called upstream that points to the main Kowalski repository.

  git clone git@github.com:<yourname>/kowalski
  git remote add upstream git@github.com:dmitryduev/kowalski

Then, for each feature you wish to contribute, create a pull request:

1. Download the latest version of Kowalski, and create a new branch for your work.

   Here, let's say we want to contribute some documentation fixes; we'll call our branch rewrite-contributor-guide.

   git checkout master
   git pull upstream master
   git checkout -b rewrite-contributor-guide

2. Make modifications to Kowalski and commit your changes using git add and git commit. Each commit message should consist of a summary line and a longer description, e.g.:

   Rewrite the contributor guide
   
   While reading through the contributor guide, I noticed several places
   in which instructions were out of order. I therefore reorganized all
   sections to follow logically, and fixed several grammar mistakes along
   the way.
   

3. When ready, push your branch to GitHub:

   git push origin rewrite-contributor-guide

   Once the branch is uploaded, GitHub should print a URL for turning your branch into a pull request. Open that URL in your browser, write an informative title and description for your pull request, and submit it. There, you can also request a review from a team member and link your PR with an existing issue.

4. The team will now review your contribution, and suggest changes. To simplify review, please limit pull requests to one logical set of changes. To incorporate changes recommended by the reviewers, commit edits to your branch, and push to the branch again (there is no need to re-create the pull request, it will automatically track modifications to your branch).

5. Sometimes, while you were working on your feature, the master branch is updated with new commits, potentially resulting in conflicts with your feature branch. To fix this, please merge in the latest upstream/master branch:

   git merge rewrite-contributor-guide upstream/master

Developers may merge master into their branch as many times as they want to.

6. Once the pull request has been reviewed and approved by at least two team members, it will be merged into Kowalski.

Pre-commit hook

Install our pre-commit hook as follows:

pip install pre-commit
pre-commit install

This will check your changes before each commit to ensure that they conform with our code style standards. We use black to reformat Python code and flake8 to verify that code complies with PEP8.