Open search integrations #34693

cjames23 · 2023-09-30T03:28:04Z

This draft is to reduce the overall amount of code to have to review the first time and to get some eyes on an issue with the providers.yaml pre-commit check error. The intention here is a base for OpenSearch functionality. The Operators allow the use of either a high level or low level version of Open Search clients. The logo that you see is likely going to be replaced as it needs to be resized. Additional Unit Tests will be coming in a second revision.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…ment.

airflow/providers/amazon/provider.yaml

…opensearchqueryoperator unit test

Taragolis · 2023-09-30T11:28:37Z

airflow/providers/amazon/aws/hooks/opensearch.py

+class OpenSearchHook(AwsBaseHook):
+    """
+    This Hook provides a thin wrapper around the OpenSearch client.


AwsBaseHook/AwsGenericHook uses only for boto3 / botocore clients. Internals of this hook intend to use with this libraries and may work incorrectly if someone call public methods of AwsBaseHook/AwsGenericHook

Would it make sense then to create a base OpenSearch hook class in an OpenSearch providers package and use that as the base class for an AWS based Hook for OpenSearch? Most of the logic in this could be moved to a base hook for Open Search and then have the AWS Specific one just override the auth method used.

Yeah it how it works in general. There is some exceptions but most (if not all) of them created years ago. there is also exists hooks in Amazon Provider which not depends on AwsBaseHook, example:

RedshiftSQLHook

ChimeWebhookHook

AwsBaseHook it wrapper around one of the boto3 (botocore) clients, e.g.

osis

opensearch

opensearchserverless

Would there be a good reason to have any operators that do things like create domains in a DAG? With the above that seems like where an open search hook that is a sub class of AwsBaseHook would come in.

If you think it might be useful then I don't see any obstacles for add wrappers around this boto3 clients. For hook it is pretty simple, just create simple thin wrapper

airflow/airflow/providers/amazon/aws/hooks/sqs.py

Lines 23 to 40 in bd2f156

class SqsHook(AwsBaseHook):

"""

Interact with Amazon Simple Queue Service.

Provide thin wrapper around :external+boto3:py:class:`boto3.client("sqs") <SQS.Client>`.

Additional arguments (such as ``aws_conn_id``) may be specified and

are passed down to the underlying AwsBaseHook.

.. seealso::

- :class:`airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`

"""

def __init__(self, *args, **kwargs) -> None:

kwargs["client_type"] = "sqs"

super().__init__(*args, **kwargs)

And optionally add useful methods

Taragolis · 2023-09-30T11:30:29Z

airflow/providers/amazon/aws/hooks/opensearch.py

+    conn_type = "opensearch"
+    hook_name = "AWS Open Search Hook"
+
+    def __init__(self, *args: Any, open_search_conn_id: str, log_query: bool, **kwargs: Any):


Suggested change

def __init__(self, *args: Any, open_search_conn_id: str, log_query: bool, **kwargs: Any):

def __init__(self, *, open_search_conn_id: str, log_query: bool, **kwargs: Any):

I would recommend to accept in hook constructor keyword only arguments, positional arguments prevent to do some changes in the future without breaking changes.

Taragolis · 2023-09-30T11:33:00Z

airflow/providers/amazon/provider.yaml

I think that we should create support of OpenSearch (as product) in a separate provider. Otherwise this feature would be exclusively available only for AWS.

And after that we could add support of AWS OpenSearch (as managed service) into Amazon provider.

Working now on creating a providers package for OpenSearch and moving the main Hook logic there with the hook for AWS Open Search taking that as a base class and only doing overrides of the AWS Open Search specific requirements such as the auth and the extras requiring a region_name.

Lets have seperate PR for open search integration and depend PR for AWS related code on top of it.
It also requires to follow protocol of adding new provider
https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers in terms of mailing list thread

Okay let me revert the changes that added the new open search provider.

Changes are reverted. @eladkal @Taragolis What would be your preference on the approach for the new provider, I can get everything onto a new branch and cut a different PR and then mail the dev list for lazy consensus given open search is an apache licensed product, or do we need a full discussion and vote?

I don't think a discussion is needed it seems as it fits the criteria for lazy consensus vote as popular vendor neutral open source.

Draft PR #34705

Taragolis · 2023-09-30T11:34:08Z

generated/provider_dependencies.json

+      "mypy-boto3-appflow>=1.24.0",
+      "mypy-boto3-rds>=1.24.0",
+      "mypy-boto3-redshift-data>=1.24.0",
+      "mypy-boto3-s3>=1.24.0",


mypy packages moved out of the amazon-provider

Suggested change

"mypy-boto3-appflow>=1.24.0",

"mypy-boto3-rds>=1.24.0",

"mypy-boto3-redshift-data>=1.24.0",

"mypy-boto3-s3>=1.24.0",

airflow/providers/amazon/aws/hooks/opensearch.py

Taragolis · 2023-09-30T11:38:21Z

docs/integration-logos/aws/Amazon-OpenSearch-light.png

That is logo of OpenSearch as a product, logo of Amazon OpenSearch Service (ElasticSearch and OpenSearch)

you could take it from https://aws.amazon.com/architecture/icons/

… PR apache#34693

vincbeck · 2023-10-03T17:47:40Z

airflow/providers/amazon/aws/operators/opensearch.py

@@ -0,0 +1,189 @@
+#


Quite related to this comment thread but are these operators specific to the AWS managed service? Should not these operators be in the opensearch provider?

This PR is on hold right now until the other PR is ready. Once that is ready I will be updating this PR with pieces in place that handle the separate provider and really what my thought process here is going to be is that the particular Operators for the managed service just need to be sub classes of the Open Search provider ones and replace the hook with the one required for the managed service piece. I am open to other ideas here.

Sounds like very good to me 👍

eladkal · 2023-10-15T19:31:58Z

airflow/providers/amazon/aws/hooks/opensearch.py

+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class OpenSearchHook(AwsBaseHook):


I guess this hook needs to be modified after #34705 is merged?

I am working on that now, everything in the amazon provider will have a prefix of Aws. And this hook will be a child of the open search hook and potentially the AwsBaseHook as well to avoid rewriting the logic for getting a boto3 client.

cjames23 · 2023-10-16T19:42:08Z

Closing this PR in favor of actually just adding the auth required in the OpenSearch provider to simplify dependency management and code management. I will open a new PR in the coming days for adding the Auth piece to the OpenSearch provider

cjames23 added 3 commits September 29, 2023 03:19

Create Chime Notifier

9a18ce6

Rename chime notifier test file

6e45261

Create open search hook and operators for search, index, and add docu…

5df0196

…ment.

cjames23 requested review from eladkal and o-nikolas as code owners September 30, 2023 03:28

boring-cyborg bot added area:providers area:system-tests kind:documentation provider:amazon-aws AWS/Amazon - related issues labels Sep 30, 2023

cjames23 marked this pull request as draft September 30, 2023 03:28

cjames23 changed the title ~~[DRAFT] Open search integrations~~ Open search integrations Sep 30, 2023

cjames23 commented Sep 30, 2023

View reviewed changes

airflow/providers/amazon/provider.yaml Outdated Show resolved Hide resolved

cjames23 commented Sep 30, 2023

View reviewed changes

airflow/providers/amazon/provider.yaml Show resolved Hide resolved

cjames23 added 2 commits September 29, 2023 20:37

Merge branch 'apache:main' into open-search-integrations

1cbca1e

Fix system test, connection name, missing provider.yaml info and add …

851c3ab

…opensearchqueryoperator unit test

Taragolis reviewed Sep 30, 2023

View reviewed changes

cjames23 force-pushed the open-search-integrations branch from e595762 to 851c3ab Compare October 1, 2023 04:13

cjames23 added a commit to cjames23/airflow that referenced this pull request Oct 2, 2023

Create OpenSearch provider to enable AWS OpenSearch functionality see…

1abf3a5

… PR apache#34693

cjames23 mentioned this pull request Oct 2, 2023

Add Open Search Provider #34705

Merged

vincbeck reviewed Oct 3, 2023

View reviewed changes

Merge branch 'apache:main' into open-search-integrations

4d3edab

eladkal reviewed Oct 15, 2023

View reviewed changes

cjames23 closed this Oct 16, 2023

cjames23 deleted the open-search-integrations branch October 16, 2023 19:42

eladkal mentioned this pull request Mar 8, 2024

Add a compatibility with Opensearch search for log management #33619

Open

2 tasks

eladkal mentioned this pull request May 2, 2024

Add ability to interact with OpenSearch in AWS #39378

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Open search integrations #34693

Open search integrations #34693

cjames23 commented Sep 30, 2023

Taragolis Sep 30, 2023

cjames23 Sep 30, 2023

Taragolis Sep 30, 2023

cjames23 Oct 1, 2023

Taragolis Oct 3, 2023

Taragolis Sep 30, 2023

Taragolis Sep 30, 2023

cjames23 Sep 30, 2023

eladkal Sep 30, 2023 •

edited

Loading

cjames23 Sep 30, 2023

cjames23 Oct 1, 2023

eladkal Oct 1, 2023

cjames23 Oct 2, 2023

Taragolis Sep 30, 2023

Taragolis Sep 30, 2023

vincbeck Oct 3, 2023

cjames23 Oct 3, 2023

vincbeck Oct 3, 2023

eladkal Oct 15, 2023

cjames23 Oct 16, 2023

cjames23 commented Oct 16, 2023


	class SqsHook(AwsBaseHook):
	"""
	Interact with Amazon Simple Queue Service.

	Provide thin wrapper around :external+boto3:py:class:`boto3.client("sqs") <SQS.Client>`.

	Additional arguments (such as ``aws_conn_id``) may be specified and
	are passed down to the underlying AwsBaseHook.

	.. seealso::
	- :class:`airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
	"""

	def __init__(self, args, *kwargs) -> None:
	kwargs["client_type"] = "sqs"
	super().__init__(args, *kwargs)

	def __init__(self, args: Any, open_search_conn_id: str, log_query: bool, *kwargs: Any):
	def __init__(self, , open_search_conn_id: str, log_query: bool, *kwargs: Any):

		from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook


		class OpenSearchHook(AwsBaseHook):

Open search integrations #34693

Open search integrations #34693

Conversation

cjames23 commented Sep 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eladkal Sep 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjames23 commented Oct 16, 2023

eladkal Sep 30, 2023 •

edited

Loading