flows
An integration is called a flow. A flow defines the input to be read, the processing rules to be applied, and the output to be written.
Flows are a graphical set of instructions for how data is integrated (flows) from a source system to a target system.
Flows consist of a set of components (readers, processors or writers) connected together with component links.
A typical component (the rdbms-reader) component is shown below.
Each component has an input port and an output port (blue circles in the image above). The input and output ports are used to connect the component to other components with component links. Each port has a designation that dictates the type of message that can be sent into the component and the type of message that the component generates. The designations are:
-
R - A relational model based message
-
H - A hierarchical model based message
-
M - A model based message
-
T - A text based message
-
B - A binary based message
-
* - Any message type
Each component completes some action when an input message is received and sends one or more resultant messages when it is complete. The RDBMS-Reader as an example takes any input message type and produces model based message types. When an input message is received, the RDBMS-Reader executes a configured sql script and returns the resultant rows from execution of the script out as model based messages.
The following shows a simple flow that reads a table from a database and writes it to three different flat files, a delimited file, a fixed length file and an xml file.
In the flow above, each component completes a specific task as follows:
| Component | Type | Task |
|---|---|---|
Setup Person Database |
Sql Executor |
Executes a sql sript to set up the database and tables that will be used in the sample flow |
Person Table Reader |
RDBMS Reader |
Executes a sql statement to read the person table from the database created by the Setup Person Database component |
Delimited Formatter |
Format Delimited |
Formats the person data sent from the Person table reader in a delimited format |
Delimited File Writer |
Text File Writer |
Writes the delimited formatted data to a text file |
Fixed Length Formatter |
Format Fixed |
Formats the person data sent from the Person table reader in a fixed length format |
Fixed Length Writer |
Text File Writer |
Writes the delimited formatted data to a text file |
XML Formatter |
Format XML |
Formats the person data sent from the Person table reader in an xml format |
XML File Writer |
Text File Writer |
Writes the delimited formatted data to a text file |
When a flow executes, all components in the flow are initialized in parallel and begin waiting for messages to process. Any component with NO inbound component links receives a Control Message from the Metl framework telling it to perform its action.
The outbound messages from those components started by the Metl framework are fed to downstream components, and the flow runs until completion. In the example above, when the flow is run, all components are initialized and wait for inbound messages.
The Setup Person Database component receives a startup message, executes its sql script to create the initial database used in the sample flow, and then sends an output message to the Person Table Reader component which runs its sql statement to select the person records from the database. The output of the RDMBS reader is messages that contain rows which are sent to the Formatters, and so forth down the chain until all components are complete, and then the flow ends.
Data flows between components over the component links in a series of messages that are generated and consumed by each component. There are three different types of messages.
-
Model based message - A message that contains records in model format. I.E. data parsed and stored according to a given Metl Model
-
Text based message - A message that contains text based records. I.E. data in string format
-
Binary based message - A message that contains binary based data. I.E. A binary object
The difference in the message types, is the type of payload that is carried by each. A model based message has a payload that contains an array of model based records. Each model based record has attributes and values. Each value is tied to a specific attribute in the model.
A text based message has a payload that contains an array of strings (text data), and a binary based message has a payload of binary data.
To find all flows and components that contain (or call) a given flow through an embedded component, highlight the desired flow, then select File Where Used. This will display a screen showing all project - flow - components that have an embedded component that reference the highlighted flow. Selecting a row and clicking the 'Open Flow' button will open the flow in a new tab.