-
Notifications
You must be signed in to change notification settings - Fork 1
Component: Topologies
A topology is an organization of nodes into a graph that determines the paths where the messages must travel. These nodes are of two types:
-
Bolts. The nodes in a topology that receives input data from other nodes and emits new data into the topology. Read more at Component: Bolts.
-
Spouts. The node in a topology that reads data from external sources and emits the data into the topology. Read more at Component: Spouts.
This page describes the available topologies that are currently present in the project.
- Built project (see instructions in Home).
- Running Apache Kafka via Docker (see instructions in Home).
- Install Elasticsearch (see installation instructions).
NOTE: Before running the topologies, please address the environment variables settings that are required for running the chosen topology.
The framework is designed for easy creation of document and text processing pipelines. The user can modify and/or create their own pipelines by defining the topology file in the /topologies folder. Afterwards, the topology can be started with the following command:
# navigate into the built distribution folder
cd ./dist
# start the pipeline
node ./pipeline -tn {topology-name} -tp {topology-path}
Where the topology-name
is user defined (can be anything), while the topology-path
is the relative
path to the topology file.
This example
node ./pipeline -tn uuid.index-material-text -tp ../topologies/index-material-text
will run the topology which will receive text material metadata from the appropriate Kafka topic and process the materials without translating them for simple indexing in Elasticsearch.
NOTE: To add new spouts and bolts, please look at the list of available components
in the qtopology
documentation found here.
- index-material-store-complete. Store the completely processed material in the indexing process into the database.
- index-material-store-incomplete. Store the incompletely processed materials in the indexing process into the database.
- index-material-text-translate. Process the text materials (includes translating the content).
- index-material-text. Process the text materials (without translating the content).
- index-material-video-translate. Process the video and audio materials (including transcriptions and translations).
- store-providers. Stores the OER providers in the database.
- store-recsys-selections. Stores the recommender engine selections into the database.
- store-ua-activity. Stores the user activity data collected via the X5GON Connect Service into the database.
- update-material-select. Selects the materials to be sent through the updating pipeline.
- update-material-store-complete. Stores the completely update materials into the database.
- update-material-text-translate. Update the text materials (including translating the content).
- update-material-video-translate. Updates the video and audio materials (including the transcriptions and translations).
This project has received funding from the European Union’s Horizon new policy 2020 research and innovation programme under grant agreement No 761758 (X5GON).