Aws connectors #358

SaiSharathReddy · 2024-04-02T23:11:16Z

feat: Adding utilities to connect AWS services like RDS

adding AWS Dataservices connectors utilities

Signed-off-by: Saisharath srsharathreddy@gmail.com

codecov · 2024-04-03T02:37:33Z

Codecov Report

Attention: Patch coverage is 97.14286% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 92.89%. Comparing base (db2cc4f) to head (497ba34).
Report is 3 commits behind head on main.

❗ Current head 497ba34 differs from pull request most recent head da865c1. Consider uploading reports for the commit da865c1 to get more accurate results

Files	Patch %	Lines
numalogic/connectors/rds/_base.py	90.32%	3 Missing ⚠️
numalogic/connectors/rds/_rds.py	92.30%	0 Missing and 1 partial ⚠️
...logic/connectors/utils/aws/boto3_client_manager.py	96.87%	0 Missing and 1 partial ⚠️
...malogic/connectors/utils/aws/sts_client_manager.py	95.83%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #358      +/-   ##
==========================================
+ Coverage   92.67%   92.89%   +0.21%     
==========================================
  Files          83       93      +10     
  Lines        4080     4290     +210     
  Branches      352      372      +20     
==========================================
+ Hits         3781     3985     +204     
- Misses        223      226       +3     
- Partials       76       79       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ab93

left some comments.

numalogic/connectors/rds/_base.py

numalogic/connectors/rds/db/MYSQLFetcher.py

pyproject.toml

tests/connectors/rds/test_base.py

numalogic/connectors/utils/aws/sts_client_manager.py

numalogic/connectors/rds/db/__init__.py

numalogic/connectors/utils/aws/exceptions.py

ab93

Some comments

pyproject.toml

numalogic/connectors/rds/_base.py

numalogic/connectors/rds/db/factory.py

numalogic/connectors/utils/aws/exceptions.py

numalogic/connectors/rds/db/mysql_fetcher.py

tests/connectors/rds/test_base.py

ab93 · 2024-04-11T18:31:58Z

numalogic/connectors/utils/aws/config.py

+    ATHENA = "athena"
+
+
+class DatabaseTypes(BaseEnum):


Suggested change

class DatabaseTypes(BaseEnum):

class DatabaseTypes(str, BaseEnum):

if you do this, then you can make a direct comparison

ab93 · 2024-04-11T18:32:25Z

numalogic/connectors/rds/db/mysql_fetcher.py

+    use cases.
+    """
+
+    database_type = DatabaseTypes.MYSQL.value


Suggested change

database_type = DatabaseTypes.MYSQL.value

database_type = DatabaseTypes.MYSQL

ab93 · 2024-04-11T18:33:01Z

numalogic/connectors/rds/db/factory.py

+            - UnRecognizedDatabaseTypeException: If the specified database type is not supported.
+
+        """
+        if database_type == DatabaseTypes.MYSQL.value:


Suggested change

if database_type == DatabaseTypes.MYSQL.value:

if database_type == DatabaseTypes.MYSQL:

Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

This reverts commit 20e09f8. Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: Saisharath Reddy Kondakindi <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

Useful for flattened vectored data --------- Signed-off-by: Avik Basu <ab93@users.noreply.github.com> Co-authored-by: Avik Basu <ab93@users.noreply.github.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

Signed-off-by: Avik Basu <ab93@users.noreply.github.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

SaiSharathReddy requested review from ab93, qhuai, nkoppisetty and s0nicboOm as code owners April 2, 2024 23:11

SaiSharathReddy marked this pull request as draft April 2, 2024 23:11

SaiSharathReddy force-pushed the aws_connectors branch from 5a7fee6 to 7f6eb72 Compare April 2, 2024 23:13

SaiSharathReddy force-pushed the aws_connectors branch 5 times, most recently from 577c0b1 to bd5edf4 Compare April 5, 2024 20:28

SaiSharathReddy marked this pull request as ready for review April 5, 2024 20:28

SaiSharathReddy force-pushed the aws_connectors branch 2 times, most recently from f54d76e to fdeaafc Compare April 8, 2024 21:11

ab93 reviewed Apr 8, 2024

View reviewed changes

SaiSharathReddy force-pushed the aws_connectors branch 3 times, most recently from 2e5bfc4 to 4e1d4cd Compare April 9, 2024 19:07

ab93 requested a review from shashank10456 April 9, 2024 21:52

SaiSharathReddy force-pushed the aws_connectors branch from dbf436a to 49e5bab Compare April 9, 2024 21:56

ab93 reviewed Apr 9, 2024

View reviewed changes

SaiSharathReddy force-pushed the aws_connectors branch 2 times, most recently from 87260aa to 330a6be Compare April 11, 2024 16:15

ab93 reviewed Apr 11, 2024

View reviewed changes

skondakindi added 5 commits April 11, 2024 13:11

adding AWS Dataservices connectors utilities

71b2874

Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

Revert "adding AWS Dataservices connectors utilities"

a7341e0

This reverts commit 20e09f8. Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

a5c649d

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

e303e9f

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

0a8dab4

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

skondakindi and others added 23 commits April 11, 2024 13:11

feat: Adding utilities to connect AWS services like RDS

779dcaa

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

4877462

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

5a12426

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

a7ee12a

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

37fe340

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

446b5fa

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

179db73

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: Adding utilities to connect AWS services like RDS

35b5ae9

adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

Fixing lint issues

c02396c

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

Fixing lint issues

c61ce0a

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

Fixing lint issues and PR comments

e379c83

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

Updated functionality as per comments.

aca25ad

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

moved RDS Config to AWS Utils

23d2118

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

added changes as per comments

fc3e51e

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

added changes as per comments

5af858f

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

added changes as per comments

c4811f1

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

added changes as per comments

4597e76

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: introduce stride in dataset (numaproj#360)

9f847f4

Useful for flattened vectored data --------- Signed-off-by: Avik Basu <ab93@users.noreply.github.com> Co-authored-by: Avik Basu <ab93@users.noreply.github.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

feat: initial support for flattened vector in backtest (numaproj#361)

464eb44

Signed-off-by: Avik Basu <ab93@users.noreply.github.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

fix: remove print (numaproj#363)

02cdd38

Signed-off-by: Avik Basu <ab93@users.noreply.github.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

added changes as per comments

b6b74f7

feat: Adding utilities to connect AWS services like RDS adding AWS Dataservices connectors utilities ---- Signed-off-by: Saisharath <srsharathreddy@gmail.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

SaiSharathReddy force-pushed the aws_connectors branch from 6cf1acf to b6b74f7 Compare April 11, 2024 20:11

Merge branch 'main' into aws_connectors

da865c1

ab93 approved these changes Apr 11, 2024

View reviewed changes

ab93 merged commit be29f02 into numaproj:main Apr 11, 2024
8 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Aws connectors #358

Aws connectors #358

SaiSharathReddy commented Apr 2, 2024

codecov bot commented Apr 3, 2024 •

edited

ab93 left a comment

ab93 left a comment

ab93 Apr 11, 2024

ab93 Apr 11, 2024

ab93 Apr 11, 2024

	class DatabaseTypes(BaseEnum):
	class DatabaseTypes(str, BaseEnum):

	database_type = DatabaseTypes.MYSQL.value
	database_type = DatabaseTypes.MYSQL

	if database_type == DatabaseTypes.MYSQL.value:
	if database_type == DatabaseTypes.MYSQL:

Aws connectors #358

Aws connectors #358

Conversation

SaiSharathReddy commented Apr 2, 2024

codecov bot commented Apr 3, 2024 • edited

Codecov Report

ab93 left a comment

Choose a reason for hiding this comment

ab93 left a comment

Choose a reason for hiding this comment

ab93 Apr 11, 2024

Choose a reason for hiding this comment

ab93 Apr 11, 2024

Choose a reason for hiding this comment

ab93 Apr 11, 2024

Choose a reason for hiding this comment

codecov bot commented Apr 3, 2024 •

edited