Skip to content

Latest commit

 

History

History
303 lines (221 loc) · 11.5 KB

README.md

File metadata and controls

303 lines (221 loc) · 11.5 KB
Raw

docker-compose-rabbitmq-mssql

Synopsis

Using docker-compose, bring up a Senzing stack using RabbitMQ and a Microsoft SQL database.

Overview

This repository illustrates a reference implementation of Senzing using RabbitMQ as the queue and MSSQL as the underlying database.

The instructions show how to set up a system that:

  1. Reads JSON lines from a file on the internet and sends each JSON line to a message queue via the Senzing stream-producer.
    1. In this implementation, the queue is RabbitMQ.
  2. Reads messages from the queue and inserts into Senzing via the Senzing stream-loader.
    1. In this implementation, Senzing keeps its data in a Microsoft SQL database.
  3. Reads information from Senzing via Senzing API Server server.
  4. Views resolved entities in a web app.

The following diagram shows the relationship of the Docker containers in this Docker composition. Arrows represent data flow.

Image of architecture

Contents

  1. Prerequisites
  2. Demonstrate
  3. Advanced
  4. Related artifacts
  5. References

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Icons are used to signify additional actions by the user. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Expectations

  • Space: This repository and demonstration require 7 GB free disk space.
  • Time: Budget 2 hours to get the demonstration up-and-running, depending on CPU and network speeds.
  • Background knowledge: This repository assumes a working knowledge of:

Prerequisites

  1. docker - Minimum version: 20.10.16
  2. docker-compose - Minimum version: 1.29.0
  3. Create MsSQL compatible Docker images.

Demonstrate

  1. ✏️ Specify a new directory to hold demonstration artifacts on the local host. Example:

    export SENZING_DEMO_DIR=~/my-senzing
    1. ⚠️ macOS - File sharing must be enabled for SENZING_DEMO_DIR.
    2. ⚠️ Windows - File sharing must be enabled for SENZING_DEMO_DIR.

  2. Set environment variables. Example:

    export MSSQL_DIR=${SENZING_DEMO_DIR}/mssql
export RABBITMQ_DIR=${SENZING_DEMO_DIR}/rabbitmq
export SENZING_GID=$(id -g)
export SENZING_UID=$(id -u)
export SENZING_VAR_DIR=${SENZING_DEMO_DIR}/var

  3. Create directories. Example:

    mkdir -p ${MSSQL_DIR} ${RABBITMQ_DIR} ${SENZING_VAR_DIR}
chmod -R 777 ${SENZING_DEMO_DIR}

  4. Get versions of Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-versions-stable.sh \
    https://raw.githubusercontent.com/Senzing/knowledge-base/main/lists/docker-versions-stable.sh
source ${SENZING_DEMO_DIR}/docker-versions-stable.sh

  5. Download docker-compose.yaml. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-compose.yaml \
    "https://raw.githubusercontent.com/senzing-garage/docker-compose-demo/main/resources/mssql/docker-compose-rabbitmq-mssql.yaml"

  6. Bring up Senzing docker-compose stack. Example:

    cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose up

  7. Allow time for the components to be downloaded, start, and initialize.

    1. There will be errors in some Docker logs as they wait for dependent services to become available. docker-compose isn't the best at orchestrating Docker container dependencies.

View formation

Once the docker-compose formation is running, different aspects of the formation can be viewed.

Username and password for the following sites were either passed in as environment variables or are the default values seen in docker-compose-rabbitmq-mssql.yaml.

View Docker containers

  1. A good tool to monitor individual Docker logs is Portainer. When running, Portainer is viewable at localhost:9170.

View Senzing Entity Search WebApp

  1. Senzing Entity Search WebApp is viewable at localhost:8251.
  2. See additional tips for working with Senzing Entity Search WebApp.

View X-Term

The web-based Senzing X-term can be used to run Senzing command-line programs.

  1. Senzing X-term is viewable at localhost:8254.
  2. See additional tips for working with Senzing X-Term.

Use SSH

Instructions to use the senzing/sshd container are viewable in the senzing/docker-sshd repository

View Jupyter notebooks

  1. Change file permissions on database files. Example:

    sudo chmod 777 -R ${MSSQL_DIR}

  2. Jupyter Notebooks are viewable at localhost:9178.

  3. See additional tips for working with Jupyter Notebooks.

View Senzing API Server

View results from Senzing REST API server. The server supports the Senzing REST API.

  1. The OpenApi Editor with Servers value of http://localhost:8250 can be used to try the Senzing REST API.
  2. Example Senzing REST API request: localhost:8250/heartbeat
  3. See additional tips for working with Senzing API server.

View RabbitMQ

  1. RabbitMQ is viewable at localhost:15672.
    1. Defaults: username: user password: bitnami
  2. See additional tips for working with RabbitMQ.

View MSSQL

  1. MSSQL is viewable at localhost:9177.
    1. System: MS SQL (beta)
    2. Server: senzing-mssql
    3. Username: sa
    4. Password: Passw0rd
    5. Database: G2
  2. See additional tips for working with MSSQL.

Cleanup

When the docker-compose formation is no longer needed, it can be brought down and directories can be deleted.

  1. Bring down Docker formation. Example:

    cd ${SENZING_DEMO_DIR}
sudo docker-compose down

  2. Remove directories from host system. Example:

    sudo rm -rf ${SENZING_DEMO_DIR:-/tmp/nowhere/for/safety}

Advanced

The following topics discuss variations to the basic docker-compose demonstration.

  1. SSH port
  2. Set sshd password

Docker images

This docker formation brings up the following docker containers:

  1. bitnami/rabbitmq
  2. mcr.microsoft.com/mssql-tools
  3. mcr.microsoft.com/mssql/server:2019-GA-ubuntu-16.04
  4. senzing/adminer
  5. senzing/console
  6. senzing/entity-web-search-app
  7. senzing/init-container
  8. senzing/jupyter
  9. senzing/redoer
  10. senzing/senzing-poc-server
  11. senzing/senzing-tools
  12. senzing/sshd
  13. senzing/stream-loader
  14. senzing/stream-producer
  15. senzing/xterm

Configuration

Configuration values specified by environment variable or command line parameter.

Related artifacts

  1. DockerHub

References