Add a compatibility with Opensearch search for log management

[djadeau]

Description

Dear Airflow team,

Today, it is possible to configure Airflow to read logs of the tasks from ElasticSearch.

Is it possible to add the support of Opensearch as source of the task logs ?

Thank you.

Use case/motivation

Log sinks are often used in large companies. Concentrate logs in a log sinks is very practical for searches.

Opensearch and Elasticsearch have similar methods, so it should not create major change.

ElasticSearch has added some checks in the client library, so it is not possible anymore to Opensearch with the elasticsearch library.

Cloud providers such as OVHCloud or Scaleway, propose Opensearch as log manager as a service.

Opensearch is a full opensource solution contrary to Elasticsearch.

Related issues

No

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

@cjames23 is working on opensearch hook

[cjames23]

Sorry about the delay on this. If no one has made any of the Open Search capabilities I am almost complete with the hook and some cursory operators for creating indexes, documents and searching an index. I need to finish writing unit and system tests and documentation.

[cjames23]

@eladkal do you want to assign this to me with the two open PRs I have.

[cjames23]

OpenSearch provider was merged but does not have log integration yet for airflow task log integrations. Currently working on adding support for the AWS Managed version of OpenSearch and then I will begin working on the log integrations.

[erikvatt]

@cjames23 Have you started working on OpenSearch log integration?

[eladkal]

I think that #34693 has most of the needed logic for AWS integration it just require adjustments after the OpenSearch provider was implemented. Once this is completed we can start work on log integration.
Not sure though if @cjames23 is working on it

[ftforce]

Any updates on this?