Feat: Add support for YC managed Airflow

[petrikoro]

Add Support for YC Managed Airflow

Description

This PR introduces a new configuration for YC Managed Airflow, enabling the use of both username:password credentials and an IAM token for authentication.

Documentation for the YC Managed Airflow: https://yandex.cloud/en/docs/managed-airflow/operations/af-interfaces

Changes Made

• Added the YCAirflowSchedulerConfig class to support YC Managed Airflow configuration.

[CLAassistant]

All committers have signed the CLA.

[izeigerman]

It appears that the only difference from the vanilla AirflowSchedulerConfig configuration is a different name of the authorization header? Can we instead extend AirflowSchedulerConfig with ability to override it:

class AirflowSchedulerConfig(_BaseAirflowSchedulerConfig, BaseConfig):
    authorization_header_name: str = "Authorization"

It's fine if we still want a custom configuration for YC, but its implementation should become much simpler:

class YCAirflowSchedulerConfig(AirflowSchedulerConfig):
     authorization_header_name: str = "X-Cloud-Authorization"

Also does it need both the token and username / password?

[petrikoro]

Yes, both the token and username:password are required, as stated in the documentation: https://yandex.cloud/en/docs/managed-airflow/operations/af-interfaces#rest-api.

This is the primary reason I prefer to implement it as a separate class, rather than modifying the vanilla AirflowSchedulerConfig, since I don't want to override the authentication method (now it's either username:password or a token).

[izeigerman]

@petrikoro any chance you can add a section on YC AIrflow support to https://github.com/TobikoData/sqlmesh/blob/main/docs/integrations/airflow.md?

[petrikoro]

@petrikoro any chance you can add a section on YC AIrflow support to https://github.com/TobikoData/sqlmesh/blob/main/docs/integrations/airflow.md?

Updated docs, could you please take a look?

[izeigerman]

Sorry about the back and forth but it'd be helpful to mention which scheduler type to specify to use this. Eg. here's how we document this for the Cloud Composer: https://github.com/TobikoData/sqlmesh/blob/13cf28ab60972ef043fbe6c65ddc11908f949631/docs/reference/configuration.md#cloud-composer. We can then include a link to the config reference here.

[petrikoro]

Thank you! Somehow I missed the part about the configuration :D

I added information about the configuration in the latest commit and slightly adjusted the wording in docs/integrations/airflow.md

[izeigerman]

One last comment, should be good to go otherwise.

[izeigerman]

Thanks a lot for addressing comments!