Prepare your stack

Now that our BDI platform is running, we can clone and configure our pilot stack.

There are a few differences between a pipeline running on the BDE stack and one running on a BDI platform, as some helper microservices are already integrated.

Indeed, the init-daemon, the loggers and the integrator-ui are already integrated in the platform and you just need to configure your stack to use their features.

Configuration

Ports:

As the BDI platform itself is exposed, please make sure not to use the same ports as it could prevent your stack from being usable.

The following ports are being used by the BDI platform by default:

• 80, by the integrator-ui
• 8890, by the Virtuoso endpoint
• 9200, by the elastic search
• 5601, by Kibana

Loggers:

The BDI platform comes integrated with loggers, which gives you access to your container's events, logs and stats out of the box, by just adding the proper configuration to your docker-compose.yml file.

my-service:
  labels:
    - "LOG=true"
    - "STATS=true"
    - "logging=true"

• All the containers that have a label LOG will be logged in the datastore.
• All the containers that have a label STATS will have their stats (taken from the docker daemon) logged into the database.
• All the containers that have a label logging will have their network traffic stats pushed into an Elasticsearch instance.

Init-daemon:

Instead of having a init-daemon instance per pipeline, the BDI platform comes equipped with its own, which means that once you have created the workflow sequence in the workflow builder, your services will be able to query the init-daemon to check whether they can start. Of course, you still have to make sure your services actually query the init-daemon for that information.

The required configuration to access the init-daemon is in two parts.

1. Your stack needs to register to the BDI platform init-daemon network (so that your services can reach it).
2. The services that need to reach the init-daemon should join its network.

An example below:

services:
  my-service-1:
    networks:
      - default
      - initdaemon
  my-service-2:
    networks:
      - default
      - initdaemon
networks: 
  initdaemon:
    external:
      name: bdiplatform_initdaemon

Once this is done, your services can reach the initdaemon as follows: curl http://initdaemon/canExecute?step=step_1

As the init-daemon is using "initdaemon" as its container-name, please make sure you do not use the same name for another service.

Creating stack

Browse to http://integrator-ui.big-data-europe.aksw.org/ and click "swarm-ui" in the headers. You can now create a new stack by pressing the corresponding button and by providing a name, an optional icon and a docker-compose file from either a git repository or already stored in the database.

---

Implementing Pilot on BDI Platform

1. Install the BDI Platform
2. Prepare your stack
3. Build your workflow
4. Start a new pipeline