OpenSearch Support #880

AssafMentzer · 2021-08-31T20:11:08Z

Issue #, if available:
Elasticsearch support

Description of changes:
initialize code: only structure and _write functions signatures

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

awswrangler/elasticsearch/_write.py

…nto elasticsearch-support

jaidisido · 2021-09-14T19:58:56Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 4bfbfd6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking

Looks pretty good - left a few comments, mostly minor

awswrangler/opensearch/_write.py

awswrangler/opensearch/_read.py

awswrangler/opensearch/_utils.py

awswrangler/opensearch/_write.py

kukushking · 2021-09-27T17:48:16Z

tests/test_opensearch.py

+logging.getLogger("awswrangler").setLevel(logging.DEBUG)
+
+# TODO: create test_infra for opensearch
+OPENSEARCH_DOMAIN = 'search-es71-public-z63iyqxccc4ungar5vx45xwgfi.us-east-1.es.amazonaws.com'  # change to your domain


We usually use pytest fixtures that fetch the values from CloudFormation outputs, example

Yes, this was just temporary till I finish the test_infra which I now have.
checkout these files:

test_infra/scripts/delete-opensearch.sh

test_infra/scripts/deploy-opensearch.sh

test_infra/stacks/opensearch_stack.py

test_infra/app.py

tests/_utils.py

tests/test_opensearch.py

awswrangler/opensearch/_write.py

…nto elasticsearch-support

jaidisido · 2021-09-28T05:35:51Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 517a3a6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido · 2021-09-28T13:43:47Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 950231d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…nto elasticsearch-support

jaidisido · 2021-10-01T00:56:47Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: ed7a57c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido · 2021-10-01T01:21:23Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: aaf8943
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2021-10-01T16:25:59Z

@AssafMentzer I updated the role and reran the tests - results should be here soon

jaidisido · 2021-10-01T16:28:50Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 545e163
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido · 2021-10-04T17:38:14Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 6042ae4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido · 2021-10-04T20:14:23Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: c53cd6f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido · 2021-10-05T11:07:41Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 419a5ce
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido · 2021-10-06T00:50:04Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 29f892c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

maxispeicher

LGTM

maxispeicher · 2021-10-06T08:19:42Z

awswrangler/opensearch/_utils.py

+    return client.info().get("version", {}).get("number")
+
+
+def _get_version_major(client: OpenSearch) -> Any:


Is this needed for anything?

yes in opensearch/_write.py create_index method to support the deprecation of ES mapping types

maxispeicher · 2021-10-06T08:24:18Z

test_infra/pyproject.toml

 "aws-cdk.aws-rds" = "^1.115.0"
 "aws-cdk.aws-secretsmanager" = "^1.115.0"
 "aws-cdk.aws-ssm" = "^1.115.0"
+"aws-cdk.aws-opensearchservice" = "^1.124.0"


I think all cdk packages should have the same minimum version.

good point. made the change: commit

jaidisido · 2021-10-06T15:51:02Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-pDO66x4b9gEu
Commit ID: 827c3bf
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[skip ci] elasticsearch support init: structure and skeleton code

0a83644

kukushking approved these changes Sep 3, 2021

View reviewed changes

jaidisido approved these changes Sep 3, 2021

View reviewed changes

awswrangler/elasticsearch/_write.py Outdated Show resolved Hide resolved

Assaf Mentzer added 2 commits September 10, 2021 11:44

[skip ci] rename elasticsearch->opensearch

947119f

[skip ci] merge Assaf and Murali forks

4534d7a

jaidisido linked an issue Sep 14, 2021 that may be closed by this pull request

Add Support for Opensearch #891

Closed

jaidisido changed the title ~~[skip ci] elasticsearch support init: structure and skeleton code~~ [skip ci] opensearch support init: structure and skeleton code Sep 14, 2021

Assaf Mentzer added 3 commits September 14, 2021 11:41

[skip ci] fixed filter_path pandasticsearch issue

4e8f4e3

[skip ci] disable scan for now

7a010cd

Merge branch 'main' of https://github.com/awslabs/aws-data-wrangler i…

4bfbfd6

…nto elasticsearch-support

Assaf Mentzer added 7 commits September 15, 2021 10:49

[skip ci] path documentation

79e0a9a

[skip ci] add delete_index

f07e698

[skip ci] add delete_index

7d7318b

[skip ci] add index_json

6b90c93

[skip ci] add index_csv local path

73db6f5

[skip ci] add is_scroll to search (scan)

15d8aca

[skip ci] add search_by_sql

e01b1a0

kukushking reviewed Sep 27, 2021

View reviewed changes

Assaf Mentzer added 7 commits September 27, 2021 23:41

[skip ci] opensearch test infra

1e1fe37

[skip ci] index create/delete ignore exceptions

d574341

[skip ci] index_documents documents type

7bb6779

[skip ci] removed pandasticsearch dependency

75a2701

[skip ci] port typo

cea9abb

[skip ci] enforced_pandas_params

f6c7dd4

Merge branch 'main' of https://github.com/awslabs/aws-data-wrangler i…

517a3a6

…nto elasticsearch-support

Assaf Mentzer and others added 2 commits September 28, 2021 02:06

[skip ci] isort & black

030e21c

Added OpenSearch tutorial

950231d

Merge branch 'main' of https://github.com/awslabs/aws-data-wrangler i…

aaf8943

…nto elasticsearch-support

[skip ci] opensearch: indexing progressbar

545e163

[skip ci] opensearch.index_documents.max_retries default 5

6042ae4

opensearch: replace elasticsearch-py with opensearch-py low-level client

c53cd6f

Assaf Mentzer added 2 commits October 4, 2021 23:57

[skip ci] opensearch filter_path default value

5c5d717

[skip ci] opensearch tutorial

152c407

jaidisido changed the title ~~[skip ci] opensearch support init: structure and skeleton code~~ OpenSearch Support Oct 5, 2021

Merge branch 'main' into elasticsearch-support

419a5ce

jaidisido and others added 5 commits October 5, 2021 17:01

Merge branch 'main' into elasticsearch-support

808cf09

Minor - Pylint

53dff4b

[skip ci] opensearch: pylint f-string and file open encoding

c6e6d80

[skip ci] opensearch: pylint f-string

2307100

opensearch: add to CONTRIBUTING.md

29f892c

maxispeicher reviewed Oct 6, 2021

View reviewed changes

opensearch: update aws-cdk packages to have the same minimum version

827c3bf

jaidisido assigned AssafMentzer Oct 7, 2021

jaidisido added the feature label Oct 7, 2021

jaidisido added this to the 2.12.0 milestone Oct 7, 2021

jaidisido added minor release Will be addressed in the next minor release ready to release labels Oct 7, 2021

kukushking merged commit 8b20f2c into aws:main Oct 7, 2021

		return client.info().get("version", {}).get("number")


		def _get_version_major(client: OpenSearch) -> Any:

OpenSearch Support #880

OpenSearch Support #880

Uh oh!

Conversation

AssafMentzer commented Aug 31, 2021

Uh oh!

Uh oh!

jaidisido commented Sep 14, 2021

AWS CodeBuild CI Report

Uh oh!

kukushking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kukushking Sep 27, 2021

Choose a reason for hiding this comment

Uh oh!

AssafMentzer Sep 28, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jaidisido commented Sep 28, 2021

AWS CodeBuild CI Report

Uh oh!

jaidisido commented Sep 28, 2021

AWS CodeBuild CI Report

Uh oh!

jaidisido commented Oct 1, 2021

AWS CodeBuild CI Report

Uh oh!

jaidisido commented Oct 1, 2021

AWS CodeBuild CI Report

Uh oh!

kukushking commented Oct 1, 2021

Uh oh!

jaidisido commented Oct 1, 2021

AWS CodeBuild CI Report

Uh oh!

jaidisido commented Oct 4, 2021

AWS CodeBuild CI Report

Uh oh!

jaidisido commented Oct 4, 2021

AWS CodeBuild CI Report

Uh oh!

jaidisido commented Oct 5, 2021

AWS CodeBuild CI Report

Uh oh!

jaidisido commented Oct 6, 2021

AWS CodeBuild CI Report

Uh oh!

maxispeicher left a comment

Choose a reason for hiding this comment

Uh oh!

maxispeicher Oct 6, 2021

Choose a reason for hiding this comment

Uh oh!

AssafMentzer Oct 6, 2021

Choose a reason for hiding this comment

Uh oh!

maxispeicher Oct 6, 2021

Choose a reason for hiding this comment

Uh oh!

AssafMentzer Oct 6, 2021

Choose a reason for hiding this comment

Uh oh!

jaidisido commented Oct 6, 2021

AWS CodeBuild CI Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!