Component: Topologies

A topology is an organization of nodes into a graph that determines the paths where the messages must travel. These nodes are of two types:

• Bolts. The nodes in a topology that receives input data from other nodes and emits new data into the topology. Read more at Component: Bolts.

• Spouts. The node in a topology that reads data from external sources and emits the data into the topology. Read more at Component: Spouts.

This page describes the available topologies that are currently present in the project.

Prerequisites

• Built project (see instructions in Home).
• Running Apache Kafka via Docker (see instructions in Home).
• Install Elasticsearch (see installation instructions).

NOTE: Before running the topologies, please address the environment variables settings that are required for running the chosen topology.

Running a Topology

The framework is designed for easy creation of document and text processing pipelines. The user can modify and/or create their own pipelines by defining the topology file in the /topologies folder. Afterwards, the topology can be started with the following command:

# navigate into the built distribution folder
cd ./dist
# start the pipeline
node ./pipeline -tn {topology-name} -tp {topology-path}

Where the topology-name is user defined (can be anything), while the topology-path is the relative path to the topology file.

This example

node ./pipeline -tn uuid.index-material-text -tp ../topologies/index-material-text

will run the topology which will receive text material metadata from the appropriate Kafka topic and process the materials without translating them for simple indexing in Elasticsearch.

NOTE: To add new spouts and bolts, please look at the list of available components in the qtopology documentation found here.

Available Topologies

• index-material-store-complete. Store the completely processed material in the indexing process into the database.
• index-material-store-incomplete. Store the incompletely processed materials in the indexing process into the database.
• index-material-text-translate. Process the text materials (includes translating the content).
• index-material-text. Process the text materials (without translating the content).
• index-material-video-translate. Process the video and audio materials (including transcriptions and translations).
• store-providers. Stores the OER providers in the database.
• store-recsys-selections. Stores the recommender engine selections into the database.
• store-ua-activity. Stores the user activity data collected via the X5GON Connect Service into the database.
• update-material-select. Selects the materials to be sent through the updating pipeline.
• update-material-store-complete. Stores the completely update materials into the database.
• update-material-text-translate. Update the text materials (including translating the content).
• update-material-video-translate. Updates the video and audio materials (including the transcriptions and translations).