Create doc on "performance mode"

[jlaneve]

As a follow-on to #800, we should create a document on how to tune Cosmos to be the most performant

[dosubot]

🤖

Hey @jlaneve, nice to see you diving into another challenge with us!

Sure, I can provide some guidance on creating a document that explains how to optimize the performance of Cosmos, following the changes made in pull request #800.

To optimize the performance of Cosmos, you can consider the following steps:

1. Deep Copying Configurations: In the migrate_to_new_interface function, the render_config and execution_config are deep copied to avoid changes affecting other DAGs or Task Groups that may reuse the same original configuration. This ensures that each DAG or Task Group has its own unique configuration, reducing the chance of conflicts and improving performance.

2. Argument Validation: The validate_arguments function checks for mutually exclusive selectors filters and deprecated arguments. This helps to avoid unnecessary computations and improves the performance of the system.

3. User Configuration Validation: The validate_initial_user_config and validate_adapted_user_config functions ensure that the user has set the fields as expected and all the necessary fields required by Cosmos to render the DAG are set. This helps to avoid unnecessary errors and improves the performance of the system.

4. Airflow Graph Building: The build_airflow_graph function is used to build the Airflow graph. This function is optimized to build the graph in the most efficient way possible.

The changes made in pull request #800 that are relevant to performance optimization are:

1. Deep copying of render_config and execution_config in the migrate_to_new_interface function. This ensures that changes do not affect other DAGs or Task Groups that may reuse the same original configuration, thus improving performance by avoiding unnecessary side effects.

2. The DbtToAirflowConverter class now directly uses ProjectConfig instead of creating a DbtProject. This reduces the overhead of creating an additional object and improves performance.

3. The DbtGraph is now loaded with the load method, which could potentially improve performance depending on the implementation of the load method.

4. The build_airflow_graph function is now used to build the Airflow graph. This could potentially improve performance depending on the implementation of the build_airflow_graph function.

Remember, performance optimization is a continuous process and it's important to monitor the performance of your system regularly to identify any potential bottlenecks. You can find more details in the cosmos/converter.py file.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• cosmos/converter.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.