Skip to content

dbtransfer-v0.3.1

Latest
Latest

Choose a tag to compare

View all tags
@aricy aricy released this 23 Dec 02:50
v0.3.1
7d34acf

发布说明

  1. 新功能
    Cassandra & ScyllaDB TTL 支持：
    新增三种 TTL 迁移模式：source（保留原始 TTL）、target（应用新的固定 TTL）、modify（在原始 TTL 基础上进行时间偏移）。
    灵活的数据过滤：
    在表映射配置中增加了 where 字段，允许用户自定义过滤条件（SQL/CQL），实现数据的选择性迁移。
    全局限速与流量控制：
    重构了限速机制，引入全局限速器和突发量（Burst）控制，确保在大规模并发迁移时，整体吞吐量严格符合预期设置。
    增强型断点管理：
    细化了断点续传配置，支持通过行数阈值（checkpoint_row_threshold）和时间间隔（checkpoint_interval）触发保存，大幅提升了在不稳定网络下的可靠性。
  2. 性能优化
    Cassandra 写入加速：
    引入了工作协程池进行并发批处理写入，显著提升了迁移速度。同时针对 ScyllaDB 最佳实践，默认优化了写入批次大小。
    指标统计改进：
    使用了更加精准的滑动平均算法计算实时迁移速率，并在日志中新增了“剩余行数”和“完成百分比”的动态预估。
  3. 修复与稳定性
    复杂数据类型：
    修复了 Cassandra 集合（List/Map/Set）及其嵌套类型的映射问题。
    增加了对自定义用户定义类型（UDT）和函数的自动依赖迁移支持。
    数据库兼容性：
    优化了 MySQL 和 PostgreSQL 的内部处理循环，解决了在高并发下统计信息不准及断点保存可能丢失的问题。

Release Notes

  1. New Features
    TTL Support for Cassandra & ScyllaDB:
    Introduced three TTL migration modes: source (replicate as-is), target (apply a new fixed value), and modify (adjust original TTL with a specified offset).
    Custom Data Filtering:
    Added a new where field in table mappings, allowing users to define custom SQL/CQL filters for selective data transfer.
    Global Rate Limiting & Burst Control:
    Refactored the throttling mechanism with a global rate limiter and intelligent burst control, ensuring total throughput across all threads stays within limits.
    Advanced Checkpointing:
    Enhanced reliability with granular checkpoint controls, including row-based thresholds (checkpoint_row_threshold) and time-based intervals (checkpoint_interval).
  2. Performance Enhancements
    High-Speed Cassandra Writing:
    Implemented a concurrent batch-writing worker pool to significantly increase throughput. Default batch sizes are now optimized per ScyllaDB best practices.
    Refined Metrics & Reporting:
    Upgraded the rate-calculation algorithm to use a moving average. Logs now include dynamic estimates for "Remaining Rows" and "Completion Percentage."
  3. Bug Fixes & Stability
    Complex Type Handling:
    Resolved issues with Cassandra collection types (List, Map, Set) and their nested structures.
    Added automated migration support for User-Defined Types (UDTs) and Functions.
    Database Stability Improvements:
    Optimized the internal migration loop for MySQL and PostgreSQL, fixing edge cases in rate-limit integration and checkpoint consistency during high-load operations.
Assets 6
Loading