Releases: Altinity/clickhouse-sink-connector
Release v2.1.0
Lightweight
Docker: altinity/clickhouse-sink-connector:2.1.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.1.0-lt/images/sha256-eb436979852212c7a0f624c1cc2f870ba4b173027be8e21654cdc9179b6d5590?context=explore
Kafka
Docker: altinity/clickhouse-sink-connector:2.1.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.1.0-kafka/images/sha256-65d4cbdcdfd159d6d7a1702635cd44da3135fa37dbab637a7a958e3339122b92?context=explore
What's Changed
Breaking Changes.
The configuration clickhouse.server.database
is now deprecated with the multiple database support.
By default the source MySQL/postgres database name will be used as the ClickHouse destination database name.
Changes
- Release 2.0.2 by @subkanthi in #510
- Added release notes for 2.0.2 by @subkanthi in #527
- add 1 second delay after query execution by @Selfeer in #537
- Update README.md by @subkanthi in #539
- Update Monitoring.md by @subkanthi in #541
- Change index_granularity to 8192 instead of 8198. by @subkanthi in #534
- Refactor TestFlows tests related to Lightweight by @Selfeer in #543
- Update config.yml to include database.server.id by @subkanthi in #544
- Update Troubleshooting.md by @subkanthi in #545
- Update Monitoring.md to include insert duration query and part log query by @subkanthi in #546
- Removed references to deduplication.policy in kafka configuration by @subkanthi in #547
- Use sequence number + timestamp in non-gtid mode for version column. by @subkanthi in #550
- Added logic to support multiple databases by @subkanthi in #535
- 523 handle scenario when records could be inserted with the same timestampnon gtid mode by @subkanthi in #552
- [528] Added logic to create view for replica_source_info table by @subkanthi in #549
- Replaced slf4j calls with log4j2 api calls by @subkanthi in #553
- Can't load table from Postgres to Clickhouse containing nullable numeric column by @ZlobnyiSerg in #529
- Kafka fixes for multiple database. by @subkanthi in #555
- Added integration test to perform updates on PK to verify incrementin… by @subkanthi in #554
- Enable postgres tests by @subkanthi in #556
- Grafana - Fix prometheus targets by @subkanthi in #558
- Fixed logic of creating sequence number based on debezium timestamp, … by @subkanthi in #557
- Removed excessive logging statements by @subkanthi in #565
- Changed CREATE VIEW to CREATE VIEW IF NOT EXISTS by @subkanthi in #567
- Fix alter drop column by @subkanthi in #560
- Force RMT to old version for Integration tests by @subkanthi in #571
- Changed from ts_ms to debezium ts_ms for adding sequence numbers by @subkanthi in #569
- Fixed alter table change column not null DDL query by @subkanthi in #573
- Changes to make sure the threads are exited when the CLI stop command… by @subkanthi in #525
- remove old broken tests by @Selfeer in #585
New Contributors
- @ZlobnyiSerg made their first contribution in #529
Full Changelog: 2.0.2...2.1.0
Release v2.0.2
Lightweight
Docker: altinity/clickhouse-sink-connector:2.0.2-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.2-lt/images/sha256-1c0db9877331a0aaceb2e0f2838db3a5156a3cfbfdb83a92235b6287b576c5d0?context=explore
Kafka
Docker: altinity/clickhouse-sink-connector:2.0.2-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.2-kafka/images/sha256-66e786bbd1230c26abdd0bad70b549abdb1040eab02a25b41c7c7217f51cf0bf?context=explore
What's Changed
- Added release notes for 2.0.1 by @subkanthi in #504
- Update README.md with latest release 2.0.1 by @subkanthi in #505
- 506 datetime limits were updated in jdbc libraryclickhouse java by @subkanthi in #508
- Update config.yml to set database.connectionTimezone by @subkanthi in #514
- Removed log4j 1.x dependencies by @subkanthi in #509
- Updated troubleshooting documentation by @subkanthi in #517
- Added sample application to create 5000 sample records. by @DougTidwell in #521
- Various fixes to the MySQL python commands by @aadant in #515
- ReplicatedReplacingmergetree support for kafka by @subkanthi in #522
- When MAX_QUEUE_SIZE is configured, the this.records.add(batch) method may throw an IllegalStateException and terminate the task. by @UsenPang in #512
- Release 2.0.2 by @subkanthi in #510
New Contributors
- @DougTidwell made their first contribution in #521
- @UsenPang made their first contribution in #512
Full Changelog: 2.0.1...2.0.2
Release v2.0.1
Lightweight
Docker: altinity/clickhouse-sink-connector:2.0.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.1-lt/images/sha256-99f38d1b5c34ac8bbcc86a86ca1eadca34e984c77ff1e2b3a54d5ee612029037?context=explore
What's Changed
- Updated README with 2.0.0 release assets by @subkanthi in #481
- 482 rearrange main readme documentation by @subkanthi in #483
- Doc updates for testflows by @Selfeer in #485
- update requirements by @Selfeer in #486
- Update documents for testflows by @Selfeer in #496
- Changed from ConcurrentLinkedQueue to LinkedBlockedQueue with the size configurable by @subkanthi in #491
- Added logic to auto create RRMT tables by @subkanthi in #492
- Separate testflows requirements by @Selfeer in #498
- update requirements by @Selfeer in #499
- update requirements by @Selfeer in #500
- Fix integration tests by @subkanthi in #502
- Release 2.0.1 by @subkanthi in #497
Known issues
- Error running DDL Query - when trying to create a new ReplicatedReplacingMergeTree table #503
Release v2.0.0
Lightweight
Docker: altinity/clickhouse-sink-connector:2.0.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.0-lt/images/sha256-62dee0cae362de871dc6daedef0e705174c0700e903af35299076aae2ab400cb?context=explore
Issues Closed
- #472: Fix old testflows tests
- #471: Testflows datetime tests fix
- #470: MySQL DATETIME values under year 100 are converted incorrectly to ClickHouse's DateTime64.
- #469: Testflows add sink logs
- #467: Testflows test updates
- #465: Add and update TestFlows Lightweight tests
- #464: Integration tests
- #463: the sink-connector lightweight should retry on failures
- #462: Creating a Table With PRIMARY KEY in MySQL With DATETIME Column Results in a Wrong Precision Value in ClickHouse
- #460: Update testflows tests
- #459: Values of the generated columns in MySQL are not transfered to the ClickHouse table
- #457: Testflows ci cd trigger
- #455: Testflows tests for renaming tables and is_deleted
- #454: Update testflows-sink-connector-lightweight.yml
- #453: Refactor DbWriter and DBKafkaOffsetWriter tests
- #452: Add junit test report in github actions
- #451: Constraints are not replicated on table creation but are replicated on new additions
- #450: Fix ddl parser bugs
- #449: Update README.md
- #448: Total records in one table doubled when i just update or create new one record
- #444: Update README.md
- #443: Added logic to store the record committer object and to acknowledge t…
- #442: Removed log4j 1.x and other unused dependencies
- #439: Added logic in auto.create.table to support table names with dash(hyphen)
- #438: new table in Postgresql
- #436: Added logic to launch debezium event loop in a separate thread, set t…
- #435: renaming tables when copying from postgres
- #434: Rename is_deleted column if source tables have column with the same name
- #433: 429 support for calculated columns(mysql generate)
- #432: Add clickhouse jdbc parameter config
- #431: Support source tables with is_deleted columns
- #430: Implement a fast table row count
- #429: Support for calculated columns
- #428: Add ability to pass JDBC parameters as configuration variables
- #427: Fix unjustified xfails
- #426: Add a python wrapper for mysqlsh
- #425: Refactored the overriden functions in Antlr parser to a separate clas…
- #422: CREATE TABLE TIMESTAMP PRECISION IS NOT MAPPED CORRECTLY to DATETIME64
- #420: [Fix] Export CLICKHOUSE_SINK_CONNECTOR_LT_IMAGE instead of SINK_LIGHTWEIGHT_VERSION
- #419: [Fix] add registry to the clickhouse_debezium_embedded
- #418: Ability to control the maximum batch size inserted to ClickHouse
- #417: Alter table add/drop column PostgreSql
- #416: Add --max_datetime_value to compare with CH DateTime64
- #414: Updated debezium version to 2.5.0.Beta1
- #413: sink connector cannot insert data into table named transaction
- #412: Allow for password less command lines
- #410: Update docker-compose-mysql.yml
- #409: Update README.md to include docker hub path
- #408: Change rest api parameter for binlog_position to match client
- #407: Added Integration test for testing debezium offset storage, fixed postgres yaml file
- #405: Added comments to sink connector lightweight configuration
- #404: Rework docker-compose-mysql.yml example used in the quikcstart guide
- #402: Restructure README and linked docs
- #399: Docker tag in Java tests
- #398: clickhouse-loader.py and *checksum.py security
- #396: PR from another fork
- #395: upload jar artifacts
- #394: Myro tk patch 1
- #393: Sanity run
- #392: Update docker upload, release pipeline, update testflows tests
- #391: Added step in docker-build.yml github workflow to set sink connector …
- #390: Add github step in workflow to update sink-connector jar version in lightweight pom.xml
- #389: Removed doc, docs folder, cleanup docker compose and update titles of…
- #388: Clean up documentation
- #387: Test PR from another fork
- #386: Configuration variable to disable replication.
- #383: Test PR from a different fork
- #382: Pull request workflow
- #381: Docker tag as environment variable
- #380: The sink-connector lightweight stops replicating
- #379: Fix datetime limits
- #378: testflow tests docker tag not using the version specified in the workflow file
- #376: Added tests to validate DATETIME values from datatypes.sql
- #375: Fix nullable fields and add clickhouse_datatime_timezone
- #374: Param source timezone (review)
- #373: Add logic to limit DateTime max
- #372: Update instructions to connect to cloud url.
- #367: Insert fails when its a DateTime64 column with timezone
- #366: Added additional test cases for version check and MergeTree tables, Add backticks to column names(fix for names with spaces))
- #362: Escape column names using backticks
- #361: Update README.md
- #358: Python utility
clickhouse_table_checksum
fails fordata_types
database - #356: Updated Debezium to 2.5.0.Alpha1(Latest release), Added mariadb docke…
- #355: Update README.md
- #353: Config variable to start the sink-connector without turning on replication
- #351: Include minimal description of variables in the config.yml file
- #349: Specify timezone for DateTime / DateTime64
- #347: Debezium offset configuration variables were updated which breaks offset storage
- #346: Fix integration tests
- #344: Fix for
UNKNOWN DATABASE
exception - #340: Choose the binary data type encoding in the sink-connector
- #338: clickhouse_loader.py : fixing temporal and binary data types
- #332: Support partitions when migrating to CH
- #328: Correctly handle temporal type Time zone.
- #326: Atomic update of the altinity_sink_connector.replica_source_info
- #321: CREATE TABLE DDL not transformed properly
- #319: java.util.ConcurrentModificationException from the DbWriter
- #290: Use debezium version 2.4
- #253: Better exception handling when STRUCT is empty
- #232: DateTime64 conversion logic
- #230: Offset acknowledgment - Add logic to persist/dump offset only after records are persisted to ClickHouse
- #193: Error parsing Gtid when values exceeds MAX INTEGER
- #126: 122 remove trailing zeros in timestamp and time columns
- #124: testflows & github action tests
- #85: Create row policy on auto create table to hide deleted rows
- #60: Manually set binlog position before starting streaming.
- #52: Kubernetes updates - Added redpanda 3 node cluster configuration and schema registry confl…
- #49: Document steps to connect to external MySQL server and instructions to load ontime dataset.
- #28: Support for JSON data type.
- #23: Map INT32 Kafka connect to INT256 CH type
- #16: Create sysbench script combine tests
- #14: Refactor logic to buffer records and flush.
- #1: Added initial commit of files from the original repo
Pull Requests Merged
- #464: Integration tests
- #472: Fix old testflows tests
- #471: Testflows datetime tests fix
- #469: Testflows add sink logs
- #450: Fix ddl parser bugs
- #467: Testflows test updates
- #465: Add and update TestFlows Lightweight tests
- #443: Added logic to store the record committer object and to acknowledge t…
- #460: Update testflows tests
- #457: Testflows ci cd trigger
- #455: Testflows tests for renaming tables and is_deleted
- #454: Update testflows-sink-connector-lightweight.yml
- #449: Update README.md
- #436: Added logic to launch debezium event loop in a separate thread, set t…
- #444: Update README.md
- #442: Removed log4j 1.x and other unused dependencies
- #439: Added logic in auto.create.table to support table names with dash(hyphen)
- #374: Param source timezone (review)
- #434: Rename is_deleted column if source tables have column with the same name
- #433: 429 support for calculated columns(mysql generate)
- #430: Implement a fast table row count
- #425: Refactored the overriden functions in Antlr parser to a separate clas…
- #432: Add clickhouse jdbc parameter config
- #426: Add a python wrapper for mysqlsh
- #405: Added comments to sink connector lightweight configuration
- #427: Fix unjustified xfails
- #414: Updated debezium version to 2.5.0.Beta1
- #420: [Fix] Export CLICKHOUSE_SINK_CONNECTOR_LT_IMAGE instead of SINK_LIGHTWEIGHT_VERSION
- #419: [Fix] add registry to the clickhouse_debezium_embedded
- #416: Add --max_datetime_value to compare with CH DateTime64
- #404: Rework docker-compose-mysql.yml example used in the quikcstart guide
- #408: Change rest api parameter for binlog_position to match client
- #412: Allow for password less command lines
- #410: Update docker-compose-mysql.yml
- #409: Update README.md to include docker hub path
- #407: Added Integration test for testing debezium offset storage, fixed postgres yaml file
- #379: Fix datetime limits
- #402: Restructure README and linked docs
- #399: Docker tag in Java tests
- #396: PR from another fork
- #394: Myro tk patch 1
- #393: Sanity run
- #381: Docker tag as environment variable
- #395: upload jar artifacts
- #392: Update docker upload, release pipeline, update testflows tests
- #391: Added step in docker-build.yml github workflow to set sink connector …
- #387: Test PR from another fork
- #389: Removed doc, docs folder, cleanup docker compose and update titles of…
- #382: Pull request workflow
- #383: Test PR from a different fork
- #376: Added tests to validate DATETIME values from datatypes.sql
- #375: Fix nullable fields and add clickhouse_datatime_timezone
- #366: Added additional test cases for version check and MergeTree tables, Add backticks to column names(fix for names with spaces))
- #356: Updated Debezium to 2.5.0.Alpha1(Latest release), Added mariadb docke…
- #372: Update instructions to connect to cloud url.
- #361: Update README.md
- #338: clickhouse_loader.py : fixing temporal and binary data types
- #344: Fix for
UNKNOWN DATABASE
exception - #355: Update README.md
- #346: Fix integration tests
Kafka version with support for passing JDBC parameters.
Kafka version with support for passing JDBC parameters.
Docker: altinity/clickhouse-sink-connector:latest
Sink connector Kafka(added binary encoding option)
Added option to write UUID/BLOB(Debezium.BYTES) datatype as raw bytes instead of encoding as Base64 or HEX strings.
pre_release_1.0.0
Config.yml
needs to be updated
From offset.storage.offset.storage.jdbc.offset.table.name
to offset.storage.jdbc.offset.table.name
From offset.storage.offset.storage.jdbc.offset.table.ddl
to offset.storage.jdbc.offset.table.ddl
- Fix for gtid overflow integer error.
clickhouse-debezium-embedded-1.1-SNAPSHOT.jar
Support for Postgres data types(Array)
What's Changed
- Added logic to ignore deletes by @subkanthi in #282
- Fix kakfa metadata in Kafka sink connector by @subkanthi in #286
- Docusarus docs by @subkanthi in #283
- Added support for postgresql data types(including Array). by @subkanthi in #297
- 296 postgresql data types add test cases and support by @subkanthi in #298
Full Changelog: release_0.7.0...release_0.9.0
release_0.7.0
What's Changed
- Added logic to set lsn through REST API. by @subkanthi in #275
Full Changelog: release_0.6.0...release_0.7.0
Sink connector client
What's Changed
- Changes to support yaml file configuration in Integration tests. by @subkanthi in #255
- Added logic to use yml file for Integration tests. by @subkanthi in #269
Full Changelog: release_0.5.0...release_0.6.0
Docker: registry.gitlab.com/altinity-public/container-images/clickhouse_debezium_embedded:2023-07-26