Skip to content

Releases: firebolt-db/firebolt-kafka-connector

v0.4.4

Choose a tag to compare

@asimrout-eng asimrout-eng released this 21 May 12:16

What's changed

Distribution

  • The Confluent Hub archive (firebolt-db-firebolt-kafka-connect-0.4.4.zip) is now published alongside the JARs on this release page. Use it with confluent-hub install, with Confluent Cloud Custom Connectors, or with self-hosted Kafka Connect (drop the unzipped contents into your plugin.path).

Security & hardening

  • Internal hardening of integration-test code and CI workflows. No behavioural change to the connector itself.

Compatibility

  • Tested against Kafka Connect 3.9.1 and 4.0.

Assets

  • firebolt-db-firebolt-kafka-connect-0.4.4.zip — Confluent Hub archive (preferred for confluent-hub install and Custom Connectors)
  • firebolt-kafka-connector-0.4.4.jar — uber-jar for self-hosted Kafka Connect (drop into plugin.path)
  • firebolt-kafka-connector-0.4.4-sources.jar — source archive

Full Changelog: v0.4.3...v0.4.4

v0.4.3

Choose a tag to compare

@github-actions github-actions released this 07 Apr 09:05

What's Changed

Full Changelog: v0.4.2...v0.4.3

v0.4.2

Choose a tag to compare

@github-actions github-actions released this 23 Mar 08:45

What's Changed

New Contributors

Full Changelog: v0.4.1...v0.4.2

v0.4.1

Choose a tag to compare

@github-actions github-actions released this 10 Mar 14:59

What's Changed

  • ci: fix connector creation in load tests (FIR-52465) by @goprean in #238
  • feat: add support for avro ingestion of integer (FIR-52468) by @goprean in #239
  • feat: add support for avro ingestion of bigint (FIR-52472) by @goprean in #240
  • feat: added support for numeric data type in avro (FIR-52473) by @goprean in #241
  • feat: add support for avro ingestion of real (FIR-52474) (#240) by @goprean in #242
  • feat: add support for avro ingestion of double precision (FIR-52475) by @goprean in #243
  • feat: add support for avro ingestion of boolean (FIR-52476) by @goprean in #244
  • feat: add support for avro ingestion of date (FIR-52477) by @goprean in #245
  • feat: add support for avro ingestion of timestamp (FIR-52478) by @goprean in #249
  • feat: added support for timestamptz with avro format (FIR-52479) by @goprean in #250
  • feat: added support for bytea data type in avro (FIR-52480) by @goprean in #251
  • feat: added support for text data type with avro format (FIR-52481) by @goprean in #252
  • Revert "ci: fix connector creation in load tests (FIR-52465) (#238)" by @goprean in #256
  • test: add a test with all data types with avro format (FIR-52512) by @goprean in #254
  • feat: added support for json data type with avro (FIR-52482) by @goprean in #253
  • test: added ability to run load tests using avro format (FIR-52510) by @goprean in #257

Full Changelog: v0.4.0...v0.4.1

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 17 Feb 12:22

What's Changed

  • ci: run load test continuously (FIR-51592) by @goprean in #232
  • feat: added support for json column data type (FIR-51880) by @goprean in #230
  • ci: capture some release notes summary in addition to version number (NoTicket) by @goprean in #234
  • ci: pass in the firebolt image tag as a github variable (NoTicket) by @goprean in #231
  • ci: parameterize the scenario and message count for continuous load testing (NoTicket) by @goprean in #236
  • test: run parquet binary ingestion for core too (FIR-51020) by @goprean in #235
  • refactor: use the manifest file from confluent (NoTicket) by @goprean in #237

Full Changelog: v0.3.1...v0.4.0

v0.3.1

Choose a tag to compare

@github-actions github-actions released this 11 Feb 11:34

What's Changed

  • ci: the version of the connector was missing from the slack message FIR-51418 by @goprean in #189
  • refactor: change the class names to binary to reflect the ingestion type FIR-51325 by @goprean in #181
  • feat: add schema support for parquet ingestion for integer FIR-51325 by @goprean in #176
  • feat: add schema support for parquet ingestion for bigint FIR-51325 by @goprean in #177
  • feat: add schema support for timestamp binary ingestion FIR-51325 by @goprean in #190
  • feat: add schema support for timestamptz binary ingestion FIR-51325 by @goprean in #191
  • feat: add schema support for real binary ingestion FIR-51325 by @goprean in #192
  • feat: add schema support for double binary ingestion FIR-51325 by @goprean in #193
  • feat: add schema support for decimal binary ingestion FIR-51325 by @goprean in #194
  • feat: add schema support for date binary ingestion FIR-51325 by @goprean in #195
  • feat: add schema support for text binary ingestion FIR-51325 by @goprean in #196
  • feat: add schema support for bytea binary ingestion FIR-51325 by @goprean in #197
  • feat: add schema support for boolean binary ingestion FIR-51325 by @goprean in #198
  • refactor(NoTicket): moved the prepared statement classes under ingestion/sql by @goprean in #199
  • build(NoTicket): bump org.sonarqube from 7.2.1.6560 to 7.2.2.6593 by @dependabot[bot] in #201
  • build(NoTicket): bump testcontainersVersion from 1.21.3 to 1.21.4 by @dependabot[bot] in #202
  • build(NoTicket): bump log4jVersion from 2.25.2 to 2.25.3 by @dependabot[bot] in #203
  • test: run parquet tests on cloud FIR-51519 by @goprean in #200
  • test: add support for schemaless load test run FIR-51592 by @goprean in #206
  • test: added load tests for table with 1000 columns FIR-51588 by @goprean in #204
  • test: randomize the payload of the kafka messages FIR-51592 by @goprean in #208
  • test: add one workflow to run multiple load tests combinations FIR-51592 by @goprean in #205
  • test: fix merge issue FIR-51592 by @goprean in #209
  • test: also need to set the partition min bytes FIR-51592 by @goprean in #210
  • test: increase timeout for load-test to 5hrs FIR-51592 by @goprean in #212
  • ci: do not truncate log messages FIR-51592 by @goprean in #213
  • test: combination of sql and 1000 columns causes out of memory exceptions on the connector by @goprean in #211
  • test: download the connector logs and upload them as artifacts to the job FIR-51592 by @goprean in #214
  • build(NoTicket): bump io.firebolt:firebolt-jdbc from 3.9.0 to 3.9.1 by @dependabot[bot] in #216
  • test: add scenario based testing FIR-51592 by @goprean in #207
  • test: missed the min fetch byte for scenario based FIR-51592 by @goprean in #218
  • test: set the max bytes to the received input FIR-51592 by @goprean in #219
  • build(NoTicket): bump io.firebolt:firebolt-jdbc from 3.9.1 to 3.10.0 by @dependabot[bot] in #220
  • build(NoTicket): bump org.apache.commons:commons-csv from 1.11.0 to 1.14.1 by @dependabot[bot] in #222
  • build(NoTicket): bump parquetVersion from 1.16.0 to 1.17.0 by @dependabot[bot] in #223
  • test: run scenario load tests with post processing FIR-51592 by @goprean in #225
  • build(NoTicket): bump jacksonVersion from 2.19.2 to 2.21.0 by @goprean in #226
  • test: added scenario2 load testing FIR-51592 by @goprean in #227

Full Changelog: v0.3...v0.3.1

v0.3

Choose a tag to compare

@github-actions github-actions released this 15 Dec 15:06

What's Changed

  • feat: add exactly once feature flag FIR-49600 by @goprean in #78
  • build(NoTicket): bump log4jVersion from 2.25.1 to 2.25.2 by @dependabot[bot] in #80
  • build(NoTicket): bump org.apache.commons:commons-lang3 from 3.18.0 to 3.19.0 by @dependabot[bot] in #84
  • feat(tests): allow json messages without registry FIR-49734 by @goprean in #85
  • feat(tests): add more integration tests for each supported data type by @goprean in #86
  • feat: allow consumption of kafka message without schema registry FIR-49734 by @goprean in #87
  • feat: fix the float serializartion/deserialization for large/small floats FIR-49832 by @goprean in #89
  • refactor: moved the tests and the convertes for schema into a schema package FIR-49860 by @goprean in #90
  • refactor: moved converters and tests to schema package FIR-49860 by @goprean in #91
  • build(NoTicket): bump okhttpVersion from 5.1.0 to 5.2.1 by @dependabot[bot] in #93
  • refactor: the way its decided which column converter to use by @goprean in #96
  • refactor: added the kafka message column value interface FIR-49860 by @goprean in #97
  • feat: added the ability to use post processing script - FIR-49943 by @goprean in #98
  • feat: allow the post-script execution at the end of a batch by @goprean in #99
  • build(NoTicket): bump org.apache.avro:avro from 1.12.0 to 1.12.1 by @dependabot[bot] in #102
  • build(NoTicket): bump org.sonarqube from 6.3.1.5724 to 7.0.0.6105 by @dependabot[bot] in #103
  • build(NoTicket): bump org.apache.commons:commons-text from 1.12.0 to 1.14.0 by @dependabot[bot] in #100
  • refactor: part5 refactor schema and schemaless base test classes (FIR-49860) by @goprean in #105
  • feat: FIR-49576 Implement MetadataService by @bogdantruta-firebolt in #79
  • test: Added more verifications to MetadataService unit tests FIR-49576 by @goprean in #107
  • Fir 49574 work for append only exactly once by @bogdantruta-firebolt in #106
  • build(NoTicket): bump org.sonarqube from 7.0.0.6105 to 7.0.1.6134 by @dependabot[bot] in #111
  • bug: allow saving a text column as proper json FIR-50187 by @goprean in #110
  • build(NoTicket): bump io.firebolt:firebolt-jdbc from 3.7.1 to 3.7.2 by @dependabot[bot] in #114
  • build(NoTicket): bump okhttpVersion from 5.2.1 to 5.3.0 by @dependabot[bot] in #116
  • bug: fix issue seen by customer when casting to int32 FIR-50549 by @goprean in #120
  • feat: use compression of the payload by default by @goprean in #121
  • bug: rollback the transaction if commit fails FIR-54476 by @goprean in #117
  • feat: optimize the null columns for all records in a batch FIR-50460 by @goprean in #118
  • Fir 50622 add property for optimizing inserts by @bogdantruta-firebolt in #124
  • feat: FIR-50334 allow setting of a post processing script file by @bogdantruta-firebolt in #123
  • ci: automate the release process FIR-50782 by @goprean in #125
  • refactor: prepare for adding a new ingestion service FIR-50864 by @goprean in #127
  • feat: add a new ingestion type config definition FIR-50864 by @goprean in #128
  • build(NoTicket): bump org.sonarqube from 7.0.1.6134 to 7.1.0.6387 by @dependabot[bot] in #135
  • build(NoTicket): bump org.apache.commons:commons-lang3 from 3.19.0 to 3.20.0 by @dependabot[bot] in #136
  • build(NoTicket): bump okhttpVersion from 5.3.0 to 5.3.2 by @dependabot[bot] in #137
  • feat: transform kafka messages to parquet file format FIR-50865 by @goprean in #126
  • refactor: use the decorator pattern for ingestion service with post-processing FIR-50864 by @goprean in #130
  • feat: add the binary ingestion service FIR-50865 by @goprean in #129
  • refactor: prepare for running integration tests against sql and binary ingestion FIR-50984 by @goprean in #131
  • feat: decide which ingestion service to use based on the ingestion.type definition FIR-50865 by @goprean in #132
  • feat: add consuming integer messages using parquet FIR-50866 by @goprean in #134
  • feat: support bigint ingestion with parquet FIR-50868 by @goprean in #138
  • feat: allow timestamp conversion to parquet FIR-50872 by @goprean in #139
  • feat: support conversion of timestamptz to parquet format FIR-50873 by @goprean in #141
  • feat: support ingestion of real data type using parquer format FIR-50874 by @goprean in #142
  • feat: allow ingestion of decimal data type using parquet FIR-50875 by @goprean in #143
  • feat: allow ingestion of double precision data using parquet FIR-50877 by @goprean in #144
  • feat: allow ingestion of date data type using parquet FIR-50876 by @goprean in #145
  • feat: allow text data type ingestion using parquet FIR-50867 by @goprean in #146
  • revert: FIR-51068 revert optimize imports config by @bogdantruta-firebolt in #149
  • feat: allow ingestion of boolean data type using parquet FIR-50871 by @goprean in #148
  • feat: FIR-51063 Change temporary http client to jdbc feature by @bogdantruta-firebolt in #147
  • feat: allow ingestion of bytea using parquet format FIR-50878 by @goprean in #150
  • refactor: extract the message generation and producing out of load test runner FIR-51114 by @goprean in #152
  • refactor: take out the verifier outside of the runner FIR-51114 by @goprean in #153
  • bug: allow parquet ingestion for tables with dashes in their name by @goprean in #154
  • refactor: use the message sequenceId in the TestFactory FIR-51114 by @goprean in #156
  • refactor: make sure we perform cleanup when laod tests fail FIR-51114 by @goprean in #158
  • ci: add the workflow to add a new connector plugin FIR-51190 by @goprean in #157
  • refactor: prepare load test to be run using binary ingestion FIR-51114 by @goprean in #155
  • docs: add some demo scripts FIR-51117 by @goprean in #151
  • bug: allow column names in table schema that are not valid names in avro FIR-51145 by @goprean in #159
  • bug: allow ingestion when column name and json attributes match by case insensitive equality FIR-51145 by @goprean in #160
  • build(NoTicket): bump org.sonarqube from 7.1.0.6387 to 7.2.0.6526 by @dependabot[bot] in #166
  • buil...
Read more

v0.2

Choose a tag to compare

@goprean goprean released this 03 Oct 09:23
47608b1

What's Changed

  • build(NoTicket): bump mockitoVersion from 5.18.0 to 5.19.0 by @dependabot[bot] in #39
  • (perf) FIR-48535 performance load testing against the connector by @goprean in #41
  • (perf) FIR-48535 Remove the runtime hook by @goprean in #42
  • (ci) NoTicket - fix the firebolt core directory permission by @goprean in #44
  • (deps) FIR-48534 add the commons-lang3 dependency by @goprean in #43
  • build(NoTicket): bump io.firebolt:firebolt-jdbc from 3.6.7 to 3.7.0 by @dependabot[bot] in #46
  • (feat) FIR-48534 batch records in smaller chunks if it exceeds the max org limit in firebolt by @goprean in #45
  • FIR-48534 use the exception type to detect when the payload is too large by @goprean in #47
  • (bug) FIR-48703 removing topic from topics definition attribute still process that topic by @goprean in #48
  • (feat) FIR-48781 allow certain strings to be converted to values in boolean column data type by @goprean in #49
  • Fir 48782 boolean failed conversion by @goprean in #50
  • (feat) FIR-48782 - records with int columns that cannot be converted move to the deadletter queue by @goprean in #51
  • (feat) FIR-48782 allow smaller integer types to be saved in bigint column by @goprean in #52
  • (feat) FIR-48782 - records with real data columns that cannot be converted move to the deadletter queue by @goprean in #53
  • (feat) FIR-48782 - records with double precision columns that cannot be converted move to the deadletter queue by @goprean in #54
  • build(NoTicket): bump org.sonarqube from 6.2.0.5505 to 6.3.1.5724 by @dependabot[bot] in #58
  • (feat) FIR-48782 - records with numeric columns that cannot be converted are moved to the deadletter queue by @goprean in #55
  • (feat) FIR-48782 - records with date columns that cannot be converted move to the deadletter queue by @goprean in #59
  • (feat) FIR-48782 records with timestamp columns that cannot be converted will be move to the deadletter queue by @goprean in #60
  • (feat) FIR-48782 records with timestamptz columns that cannot be converted will be move to the deadletter queue by @goprean in #61
  • (feat) FIR-48782 records with string columns that cannot be converted will be moved to the deadletter queue by @goprean in #62
  • (feat) FIR-48782 records with bytea columns that cannot be converted will be moved to the deadletter queue by @goprean in #63
  • build(NoTicket): bump org.projectlombok:lombok from 1.18.38 to 1.18.40 by @dependabot[bot] in #65
  • (feat) FIR-48916 Do not allow connecting without account, engine or database to Firebolt Cloud by @goprean in #64
  • (ci) FIR-49021 add a git workflow that lists all the custom plugins with firebolt in the name by @goprean in #68
  • feat: FIR-48541 add support for dead letter queue by @bogdantruta-firebolt in #66
  • refactor: FIR-48541 removing unused tests and reducing FireboltRecord list by @bogdantruta-firebolt in #69
  • build(NoTicket): bump org.projectlombok:lombok from 1.18.40 to 1.18.42 by @dependabot[bot] in #72
  • build(NoTicket): bump mockitoVersion from 5.19.0 to 5.20.0 by @dependabot[bot] in #73
  • build(NoTicket): bump com.google.guava:guava from 33.4.8-jre to 33.5.0-jre by @dependabot[bot] in #71
  • Fix the load tests running on latest connector by @goprean in #74
  • (feat) FIR-48662 group backend exceptions into recoverable and non-recoverable exceptions by @goprean in #75
  • feat: release version 0.2 (FIR-49394) by @goprean in #76
  • chore: Naming convention should be firebolt-db-firebolt-kafka-connector-.zip by @goprean in #77

New Contributors

Full Changelog: v0.1...v0.2