The airbyte-module (ABM) for Fybrik is a FybrikModule which makes use of Airbyte connectors.
ABM is both an Apache Arrow Flight and an HTTP server.
Airbyte is a data integration tool that focuses on extracting and loading data.
Airbyte has a vast catalog of connectors that support dozens of data sources and data destinations. These Airbyte connectors run in docker containers and are built in accordance with the Airbyte specification.
ABM is an arrow-flight server that enables applications to consume tabular data from a wide range of data sources.
Since Airbyte connectors are implemented as docker images and run as docker containers, the Airbyte Module does not require Airbyte as a prerequisite. To run the Airbyte Module, only docker is required.
Follow the instructions in the sample folder.
Follow the instructions in the helm folder.
If you would like to run a use case where the application has unrestricted access to a dataset, follow the instructions here.
However, if you are interested in a use case where the governance policies mandate that some of the dataset columns must be redacted, follow the instructions here. In this scenario, both the airbyte module and the arrow-flight-module are deployed. The airbyte module reads the dataset, whereas the arrow-flight-module transforms the dataset based on the governance policies.