stream-logger

If you are beginning your journey with Senzing, please start with Senzing Quick Start guides.

You are in the Senzing Garage where projects are "tinkered" on. Although this GitHub repository may help you understand an approach to using Senzing, it's not considered to be "production ready" and is not considered to be part of the Senzing product. Heck, it may not even be appropriate for your application of Senzing!

Overview

The stream-logger.py python script consumes data from various stream sources (Kafka, RabbitMQ) and logs it. This is mostly used for testing the contents of of a queue. The senzing/stream-logger docker image is a wrapper for use in docker formations (e.g. docker-compose, kubernetes).

To see all of the subcommands, run:

$ ./stream-logger.py --help
usage: stream-logger.py [-h]
                        {kafka,rabbitmq,sleep,version,docker-acceptance-test}
                        ...

Log contents from a stream. For more information, see
https://github.com/senzing-garage/stream-logger

positional arguments:
  {kafka,rabbitmq,sleep,version,docker-acceptance-test}
                        Subcommands (SENZING_SUBCOMMAND):
    kafka               Read JSON Lines from Apache Kafka topic.
    rabbitmq            Read JSON Lines from RabbitMQ queue.
    sleep               Do nothing but sleep. For Docker testing.
    version             Print version of program.
    docker-acceptance-test
                        For Docker acceptance testing.

optional arguments:
  -h, --help            show this help message and exit

Contents

1. Expectations
2. Demonstrate using Command Line
   1. Install
   2. Run from command line
3. Demonstrate using Docker
   1. Configuration
   2. Run docker container
4. References

Legend

1. 🤔 - A "thinker" icon means that a little extra thinking may be required. Perhaps there are some choices to be made. Perhaps it's an optional step.
2. ✏️ - A "pencil" icon means that the instructions may need modification before performing.
3. ⚠️ - A "warning" icon means that something tricky is happening, so pay attention.

Expectations

• Space: This repository and demonstration require 6 GB free disk space.
• Time: Budget 40 minutes to get the demonstration up-and-running, depending on CPU and network speeds.
• Background knowledge: This repository assumes a working knowledge of:
  • Docker

Demonstrate using Command Line

Install

1. Install prerequisites:
   1. Debian-based installation - For Ubuntu and others
   2. RPM-based installation - For Red Hat, CentOS, openSuse and others.

Run from command line

1. Run command. Example:

   cd ${GIT_REPOSITORY_DIR}
   ./stream-logger.py version

Demonstrate using Docker

Configuration

Configuration values specified by environment variable or command line parameter.

• SENZING_DEBUG
• SENZING_KAFKA_BOOTSTRAP_SERVER
• SENZING_KAFKA_GROUP
• SENZING_KAFKA_TOPIC
• SENZING_LOG_LEVEL
• SENZING_MONITORING_PERIOD_IN_SECONDS
• SENZING_NETWORK
• SENZING_RABBITMQ_HOST
• SENZING_RABBITMQ_PASSWORD
• SENZING_RABBITMQ_PREFETCH_COUNT
• SENZING_RABBITMQ_QUEUE
• SENZING_RABBITMQ_USERNAME
• SENZING_RUNAS_USER
• SENZING_SLEEP_TIME_IN_SECONDS
• SENZING_SUBCOMMAND
• SENZING_THREADS_PER_PROCESS

Run docker container

1. ✏️ Set environment variables. Example:

   export SENZING_SUBCOMMAND=kafka
   export SENZING_KAFKA_BOOTSTRAP_SERVER=senzing-kafka:9092
   export SENZING_KAFKA_TOPIC=senzing-kafka-topic

2. Run docker container. Example:

   sudo docker run \
     --env SENZING_SUBCOMMAND="${SENZING_SUBCOMMAND}" \
     --env SENZING_KAFKA_BOOTSTRAP_SERVER="${SENZING_KAFKA_BOOTSTRAP_SERVER}" \
     --env SENZING_KAFKA_TOPIC="${SENZING_KAFKA_TOPIC}" \
     --interactive \
     --rm \
     --tty \
     senzing/stream-logger

References

1. Development
2. Errors
3. Examples
4. Related artifacts:
   1. DockerHub