Enhance Data Migration from RDS to TiDB

[overvenus]

Feature Request

Enhance Data Migration from RDS to TiDB, provides solutions for major data migration scenarios (should includes
comprehensive documentation and toolings).

Category

Document, Feature

Task List

• Discuss the experience and pain points of DM with existing Public Cloud RDS Users.
• Provide a solution for single database data migration,
  • Tools is not limited to DM, you can also use third-party tools/services.
  • The solution is about one-time full migration instead of long-term data synchronization service.
  • Available for user testing.
• Provide solutions for online data migration,
  • Study a few major data migration scenarios (such as AWS Aurora or GCP MySQL, RPO requirements, etc.)
  • Develop solutions for each of these.
• Integrate DM into operator in order to deploy on k8s, and facilitate user operation.

---

• To facilitate data migration for DBaaS,
  • 1. Restore from S3 compatible API? tidb-lightning#69 TiDB-Lightning should support loading data from S3/GCS.
  • 2. Lightning in SQL tidb-lightning#299 TiDB-Lightning via SQL.
  • 3. Support importing data in Apache Parquet format tidb-lightning#266 Support import data from Aurora snapshot (in Parquet format).
• 4. Support writing to cloud storage (S3, GCS, ...) dumpling#8 To facilitate data migration for RDS, dumpling should support dumping data to S3/GCS.
• DM 2.0 adapts to migrate data from RDS/Aurora to TiDB
• Research and test other data replication tools, e.g. AWS DMS

Progress Tracking

https://github.com/orgs/pingcap/projects/31

Time

GanttProgress: 100%
GanttStart: 2020-07-15
GanttDue: 2020-9-30 -> 2020-10-18

• Design: 5 * 4 features
• Develop: 5 * 4 features
• QA: 5 * 4 features
• Document: 10