Skip to content

Add Alibaba Cloud AnalyticDB Spark support #31738

@dcoliversun

Description

@dcoliversun

Description

Alibaba Cloud AnalyticDB for MySQL Data Lakehouse Edition (V3.0) uses the same Spark development method for batch applications and SQL applications.[1]

We want to provide operator/hook/sensor to enable Airflow users using AnalyticDB Spark.

reference:
[1] https://www.alibabacloud.com/help/en/analyticdb-for-mysql/latest/spark-offline-application-development

Use case/motivation

  1. implement custom operator/hook/sensor for AnalyticDB Spark.
  2. enable users to schedule AnalyticDB Spark Batch/SQL applications by DAGs. Such as follow.
  3. AnalyticDB Spark hooks provides log/Web UI url/application status for users.

Related issues

None

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions