Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support AWS RDS MySQL as source #4561

Open
13 tasks
oeyh opened this issue May 22, 2024 · 0 comments
Open
13 tasks

Support AWS RDS MySQL as source #4561

oeyh opened this issue May 22, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@oeyh
Copy link
Collaborator

oeyh commented May 22, 2024

Is your feature request related to a problem? Please describe.
As a Data Prepper user, I would like to have an rds source to load existing data and stream change events from RDS MySQL databases.

Describe the solution you'd like
For export (loading existing data), we can create a snapshot, export it to S3 and read the data from S3
For stream (streaming change events), we can connect to MySQL's binary log stream to receive change events.

Describe alternatives you've considered (Optional)
Run SQL queries periodically through a JDBC driver to load existing and incremental data from the source database.

Additional context
The feature shares similar ideas with existing dynamodb source and documentdb source.

Tasks

  • Project setup, source configurations, skeleton code
  • Export implementation - create snapshot and export to S3
  • Export implementation- read exported data files in S3
  • Stream implementation
  • Checkpointing in both export and stream
  • Pipeline configuration transformation template
  • Support multiple entries in pipeline source (Support multiple sources for a pipeline #406)
  • Secrets manager integration
  • Secret rotation support
  • Add E2E acknowledge support
  • Add data type mapping
  • Add plugin metrics and aggregate metrics
  • Add integration tests
@oeyh oeyh added untriaged enhancement New feature or request and removed untriaged labels May 22, 2024
@oeyh oeyh self-assigned this May 22, 2024
@oeyh oeyh mentioned this issue May 28, 2024
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Development

No branches or pull requests

1 participant