Understanding the core components: airbyte-bootloader

[marcosmarxm]

Important

The airbyte-bootloader application runs at start up for Airbyte. It is responsible for making sure that the environment is upgraded and in a good state. e.g. It makes sure the internal database has been migrated to the correct version, added or removed unused columns or tables.

The code for the airbyte-bootloader is located here

The given Java code represents a class called Bootloader, which is responsible for performing various bootstrapping operations for the Airbyte environment. Let's go through the code step by step:

1. The class is annotated with @Singleton and @Slf4j, indicating that it is a Micronaut component and uses the Simple Logging Facade for Java (SLF4J) for logging.
2. The load() method is the main entry point of the class and performs the following bootstrapping operations:

   • Initializes the databases.

     Airbyte has two main databases the config and the jobs database which have several tables. We're talk more about them in next session.

   • Checks migration compatibility.

     The previous breaking version was 0.32.0-alpha described in our docs

   • Checks protocol version compatibility.

     Checks the compatibility of the protocol version used by Airbyte connectors. It validates the protocol version constraints, and if there are any issues, it throws an exception.

     The Airbyte Protocol currently all connectors are in the latest version of the protocol.

     The minimal version configured today is 0.0.0 (first release) and maximum is 0.3.0. This check make sure whatever you upgrade the connector follows the protocol Airbyte uses so it won't break after you upgrade.

   • Runs database migrations.

     Checks if upgrading from the current Airbyte version to a future version is a legal upgrade. It reads the initial Airbyte database version, compares it with the future version, and throws an exception if the upgrade is not allowed.

     Airbyte uses Flyway to make the migrations whatever there are changes in the airbyte-db . Flyway make the migration easy using code and testable SQL changes.

     Each migration is versioned in Github you check all migrations since v0.30.22 here

   • Creates a default workspace if none exists.

     This will be your user unique id needed to make API calls using the Configuration API

     The Workspace ID it's an random UUID.

   • Creates a default deployment if none exists.

     Any time Airbyte is started up with new volumes or persistence, it will be assigned a new deployment id. This is different from the lifecycle of the rest of the data layer which may be persisted across deployments

   • Sets the Airbyte version.

     Get the AIRBYTE_VERSION from the .env file and update the airbyte_metadata table.

   • Executes post-load tasks.

     it's used to release new features, currently there isn't any post-load task enabled.

   • Logs the completion of bootstrapping.

     Wooho it's done!

That's a high-level overview of the given Java code. It represents a bootloader component responsible for initializing and configuring various aspects of the Airbyte environment during startup.

Additional Resources

• The table airbyte_jobs_migrations stores all Flyway migration history and additional info
• The table airbyte_metadata keep history of each version of Airbyte and when it was upgraded. Almost the same as the airbyte_jobs_migrations but in higher level.

---

Please add any questions related to this component. The team will try to answer or help you.