🐙 octavia-cli: implement apply

[alafanechere]

What is octavia apply

The octavia apply command creates and updates Airbyte remote resources according to the local YAML configuration of these resources.

By default, running octavia apply will:

• Find all the available YAML configurations in your local project directory
• Deserialize the YAML into a BaseResource object
• Try to find if a remote resource exists on the targetted Airbyte instance matching the local one:
  • If the remote resource does not exist:
    1. Create the resource on the Airbyte instance.
    2. Save a local state.yaml file with the resource id, the checksum of the latest configuration applied, and the generation timestamp/
  • If the resource exists:
    1. Display a diff to the user between local and remote resources configuration, ask the user to validate
    2. Prompt user for validation of the diff
    3. Update the remote resource
    4. Save a local state.yaml file

Optional behavior:

• --force flag to not prompt the user with a diff validation and always run create or update.
• --file option to target specific configuration files (can be used multiple times-.

Demo

Previous actions made:

• octavia init
• octavia generate octavia generate source decd338e-5647-4c0b-adf4-da0e75f5a750 my_db
• Edited ./sources/my_db/configuration.yaml

Recommended reading order

1. octavia-cli/octavia_cli/apply/commands.py: Declares the logic of what happens when running an apply command.
2. octavia-cli/octavia_cli/apply/resources.py: Declares the resource objects, their interface, and how they interact with the backend + factory to create these resource objects. This PR implemented sources and destinations resources.
3. octavia-cli/octavia_cli/entrypoint.py: add the command to the entrypoint
4. octavia-cli/octavia_cli/apply/diff_helpers.py: Some helpers for diff computation with DeepDiff + checksum.
5. octavia-cli/tests/unit/test_apply/*: All code addition was unit tested, we maintain a 100% test coverage.

Refacto made

• File path for generated YAML: This PR creates a state.yaml file for all the managed resources. I decided to create a subfolder for each resource. These subfolders will contain configuration.yaml and state.yaml. (octavia-cli/octavia_cli/generate/renderer.py)
• The apply and generate commands require an initialized project directory. I implemented a decorator for these commands (octavia-cli/octavia_cli/check_context.py).
• Integration test for YAML generation of all connectors spec was failing: modify octavia-cli/tests/integration/test_generate/test_renderer.py to declare one test per connector, and not one test for all connectors -> eases the debugging to understand which connector broke the test.

What's missing

• Apply for connection resources (waiting for octavia generate connection to be ready)
• Check connection after resource creation or update (use check_connection_to_* and check_connection_to_*_for_update endpoints)

🚨 User Impact 🚨

• octavia apply command is now available
• The install instructions with docker are not valid anymore, will be fixed by 🐙 octavia-cli: package into a docker image #10708

[lmossman]

Had a few small comments but nothing major. Nice work on this! I think it looks great

[lmossman]

I think this will all work as expected if the remote Airbyte instance does not have any sources/destinations already that the user wants to interact with with this CLI.

However, what about the case where someone wants to use this CLI to manage sources/destinations that already exist in their Airbyte instance? Is there some way for them to export their existing sources/destinations to create these resources, so that they don't need to recreate them with this CLI? Maybe that functionality is coming in a later PR?

Another way I could see a user getting into this case is if they delete all of the octavia-cli resource files - they should have some way to regenerate them from their Airbyte instance.

[alafanechere]

You're 💯 right! This functionality will come with the octavia import feature. It's only possible to update resources created with the CLI at the moment.

[alafanechere]

Thank you for the review @lmossman . I implemented your suggestions. Sorry for the typos, in French resource is ressource 😄 .