stream_pub_sub.rst

Publish and Subscribe to a Kafka Stream

Publishing data to a stream and subscribing to a stream are handled independently and we describe each below.

Publish Data to a Kafka Topic

TOM Toolkit supports publishing data to a Kafka stream such as Hermes (an interface to HOPSKOTCH) and GCNClassicOverKafka.

When sharing photometry data via Hermes, the TOM publishes the data to be shared to a topic on the HOPSKOTCH Kafka stream. At this time, only photometry data is supported.

Configuring your TOM to Publish Data to a stream:

You will need to add a DATA_SHARING configuration dictionary to your settings.py that gives the credentials for the various streams with which you wish to share data.

# Define the valid data sharing destinations for your TOM.
DATA_SHARING = {
     'hermes': {
        'DISPLAY_NAME': os.getenv('HERMES_DISPLAY_NAME', 'Hermes'),
        'BASE_URL': os.getenv('HERMES_BASE_URL', 'https://hermes.lco.global/'),
        'CREDENTIAL_USERNAME': os.getenv('SCIMMA_CREDENTIAL_USERNAME',
                                          'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
        'CREDENTIAL_PASSWORD': os.getenv('SCIMMA_CREDENTIAL_PASSWORD',
                                          'set SCIMMA_CREDENTIAL_PASSWORD value in environment'),
        'USER_TOPICS': ['hermes.test', 'tomtoolkit.test']
     },
}

Subscribe to a Kafka Topic

TOM Toolkit allows a TOM to subscribe to a topic on a Kafka stream, ingesting messages from that topic and handling the data. This could involve simply logging the message or extracting the data from the message and saving it if it is properly formatted.

Configuring your TOM to subscribe to a stream:

First you will need to add tom_alertstreams to your list of INSTALLED_APPS in your settings.py.

INSTALLED_APPS = [
    ...
    'tom_alertstreams',
]

Then you will need to add an ALERT_STREAMS configuration dictionary to your settings.py. This gives the credentials for the various streams to which you wish to subscribe. Additionally, the TOPIC_HANDLERS section of the stream OPTIONS will include a list of handlers for each topic.

Some alert handlers are included as examples. Below we demonstrate how to connect to a Hermes Topic. You'll want to check out the tom-alertstreams README for more details.

ALERT_STREAMS = [
    {
        'ACTIVE': True,
        'NAME': 'tom_alertstreams.alertstreams.hopskotch.HopskotchAlertStream',
        'OPTIONS': {
            'URL': 'kafka://kafka.scimma.org/',
            'USERNAME': os.getenv('SCIMMA_CREDENTIAL_USERNAME', 'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
            'PASSWORD': os.getenv('SCIMMA_CREDENTIAL_PASSWORD', 'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
            'TOPIC_HANDLERS': {
                'tomtoolkit.test': 'tom_dataproducts.alertstreams.hermes.hermes_alert_handler',
            },
        },
    },
]