Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination Redshift: improve check performance #11580

Closed
Tracked by #11287
VitaliiMaltsev opened this issue Mar 30, 2022 · 6 comments
Closed
Tracked by #11287

Destination Redshift: improve check performance #11580

VitaliiMaltsev opened this issue Mar 30, 2022 · 6 comments
Assignees

Comments

@VitaliiMaltsev
Copy link
Contributor

Improve check method connection time if possible

@Phlair
Copy link
Contributor

Phlair commented Apr 17, 2022

@VitaliiMaltsev could you add some basic context stats to this issue please
e.g.

Currently seeing check times ~ x seconds / minutes

Would like to get check times down to ~ x seconds

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented May 9, 2022

@Phlair @misteryeo
Check method for Redshift Destination performance test results:


Master branch
Standard inserts ~ 3.2 seconds
S3 staging ~ 5.8 seconds

Buffering strategy
Standard inserts ~ 3.3 seconds
S3 staging ~ 5.8 seconds

AbstractJdbcDestination check improvement
Standard inserts ~ 2.4 seconds
S3 staging ~ 4.6 seconds

no additional implementation is required for this task. The performance of the check method for all JDBC destinations will be improved as soon as this PR is merged.

@misteryeo
Copy link
Contributor

Thanks @VitaliiMaltsev - it looks like there are outstanding comments on the PR so it's not ready to be merged?

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented May 12, 2022

Thanks @VitaliiMaltsev - it looks like there are outstanding comments on the PR so it's not ready to be merged?

@misteryeo yes, the PR is currently in Airbyte review status, outstanding comments were replied today

@grishick
Copy link
Contributor

@VitaliiMaltsev I've approved the PR

@VitaliiMaltsev
Copy link
Contributor Author

published and merged in scope of #12820

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: Done
Development

No branches or pull requests

6 participants