ModelarDB-RS Pilot

Pilot for the ModelarDB architecture, including a manager node, an edge node, and a cloud node. The following repositories are included as git submodules:

1. ModelarDB-RS
2. Telegraf-Output-Apache-Arrow-Flight

Using the individual Docker environments that are included in each repository, this repository creates a single complete environment to demonstrate the features of the system.

Setup

Running the pilot can be done exclusively using Docker. Downloading Docker Desktop is recommended, to make maintenance of the created containers easier. Once Docker is set up, navigate to the ModelarDB-RS-Pilot folder. The Docker services for the pilot environment can be built and started using the following command:

docker-compose -p modelardb-pilot up

Note that -p modelardb-pilot is only used to name the project to make it easier to manage in Docker Desktop. Once created, the container can be started and stopped using Docker Desktop or by using the corresponding commands:

docker-compose -p modelardb-pilot start
docker-compose -p modelardb-pilot stop

Running the pilot environment starts the services required by ModelarDB to run the full environment with all features available. The following table describes the full list of services that are running in the pilot environment.

Name	Description
minio-server	MinIO server that functions as the remote object store.
create-bucket	MinIO client to initialize the demo bucket modelardata.
postgres	PostgreSQL database used for the cluster metadata database.
modelardb-manager	ModelarDB manager with an Apache Arrow Flight interface that can be accessed by sending requests to grpc://127.0.0.1:9998.
modelardb-edge	ModelarDB edge with an Apache Arrow Flight interface that can be accessed by sending requests to grpc://127.0.0.1:9999.
modelardb-cloud	ModelarDB cloud with an Apache Arrow Flight interface that can be accessed by sending requests to grpc://127.0.0.1:9997.
create-table	Python script in utility.py used to create the windmill table in ModelarDB after startup.
output-plugin-builder	Utility service used to build the binary for the Telegraf output plugin.
telegraf	Telegraf server used to ingest MQTT data into the ModelarDB edge node.

Configuration

When configuring the pilot environment, the modified images need to be recreated and the services need to be restarted.

When the pilot is started, the Telegraf server tries to connect to a local MQTT broker running at tcp://127.0.0.1:1883. To change this, change the servers setting in the [[inputs.mqtt_consumer]] section of telegraf.conf. Below servers the topics that are subscribed to can be changed. Currently, the Telegraf server only subscribes to /mqttstreamer/testtopic.

The Telegraf server ingests data into the windmill table that is created when the pilot is deployed. The table name can be configured by changing the table setting in sample.conf and modifying the CREATE MODEL TABLE statement in utility.py. The schema of the created table can also be configured in utility.py. More information on the CREATE MODEL TABLE syntax is provided in the ModelarDB-RS User documentation. Note that if the schema of the table is changed, an equivalent change must be made to telegraf.conf. Specifically, the columns of the table must match the columns specified under [inputs.mqtt_consumer.xpath.fields].

Finally, it is also possible to configure the ModelarDB edge node to transfer data more or less frequently to the MinIO remote object store. This is done by adjusting the MODELARDBD_UNCOMPRESSED_DATA_BUFFER_CAPACITY, MODELARDBD_COMPRESSED_RESERVED_MEMORY_IN_BYTES, and MODELARDBD_TRANSFER_BATCH_SIZE_IN_BYTES environment variables for the service modelardb-edge in docker-compose.yml.

Features

When the pilot is deployed, the different services are available to be accessed directly outside the deployment. The web interface for the MinIO object store is available at http://127.0.0.1:9001 and can be accessed by using minioadmin as the username and password.

The Apache Arrow Flight interface for the ModelarDB manager, edge, and cloud can be accessed using a Flight client such as PyArrow and connecting to grpc://127.0.0.1:9998 for the manager, grpc://127.0.0.1:9999 for the edge, and grpc://127.0.0.1:9997 for the cloud. Information on how to ingest data into the edge and query data from either the edge or cloud is available in the ModelarDB-RS Usage documentation. Note that when querying the ModelarDB edge, only data local in the edge is returned while when querying the ModelarDB cloud, only data in the MinIO object store is returned.

Since a Telegraf server is available, data can also be ingested into the ModelarDB edge by streaming data to the configured topic of the configured MQTT broker. The MQTT data needs to match the schema of the pilot table to be ingested into ModelarDB using Telegraf.