Stream Reactor 7.2.0

Enhancements

1. Automated Skip for Archived Objects:

   • The S3 source now seamlessly bypasses archived objects, including those stored in Glacier and Deep Archive. This enhancement improves efficiency by automatically excluding archived data from processing, avoiding the connector crashing otherwise

2. Enhanced Key Storage in Envelope Mode:

   • Changes have been implemented to the stored key when using envelope mode. These modifications lay the groundwork for future functionality, enabling seamless replay of Kafka data stored in data lakes (S3, GCS, Azure Data Lake) from any specified point in time.

Full Changelog: 7.1.0...7.2.0

Stream Reactor 7.1.0

Source Line-Start-End Functionality Enhancements

We've rolled out enhancements to tackle a common challenge faced by users of the S3 source functionality. Previously, when an external producer abruptly terminated a file without marking the end message, data loss occurred.

To address this, we've introduced a new feature: a property entry for KCQL to signal the handling of unterminated messages. Meet the latest addition, read.text.last.end.line.missing. When set to true, this property ensures that in-flight data is still recognized as a message even when EOF is reached but the end line marker is missing.

#Note

Upgrading from any version prior to 7.0.0, please see the release and upgrade notes for 7.0.0.

v7.0.0

This release brings changes which are not compatible with the previous version for S3, GCS and Azure sinks.A migration is required.
For migration details please follow the link here.

Data-lakes Sink Connectors

This release brings substantial enhancements to the data-lakes sink connectors, elevating their functionality and flexibility. The focal point of these changes is the adoption of the new KCQL syntax, designed to improve usability and resolve limitations inherent in the previous syntax.

Key Changes

New KCQL Syntax The data-lakes sink connectors now embrace the new KCQL syntax, offering users enhanced capabilities while addressing previous syntax constraints.
Data Lakes Sink Partition Name This update ensures accurate preservation of partition names by avoiding the scraping of characters like \ and /. Consequently, SMTs can provide partition names as expected, leading to reduced configuration overhead and increased conciseness.

KCQL Keywords Replaced

Several keywords have been replaced with entries in the PROPERTIES section for improved clarity and consistency:

WITHPARTITIONER: Replaced by PROPERTIES ('partition.include.keys'=true/false). When WITHPARTITIONER KeysAndValue is set to true, the partition keys are included in the partition path. Otherwise, only the partition values are included.
WITH_FLUSH_SIZE: Replaced by PROPERTIES ('flush.size'=$VALUE).
WITH_FLUSH_COUNT: Replaced by PROPERTIES ('flush.count'=$VALUE).
WITH_FLUSH_INTERVAL: Replaced by PROPERTIES ('flush.interval'=$VALUE).

Benefits

The adoption of the new KCQL syntax enhances the flexibility of the data-lakes sink connectors, empowering users to tailor configurations more precisely to their requirements. By transitioning keywords to entries in the PROPERTIES section, potential misconfigurations stemming from keyword order discrepancies are mitigated, ensuring configurations are applied as intended

Stream Reactor 6.3.1

This update specifically affects datalake sinks employing the JSON storage format. It serves as a remedy for users who have resorted to a less-than-ideal workaround: employing a Single Message Transform (SMT) to return a Plain Old Java Object (POJO) to the sink. In such cases, instead of utilizing the Connect JsonConverter to seamlessly translate the payload to JSON, reliance is placed solely on Jackson.

However, it's crucial to note that this adjustment is not indicative of a broader direction for future expansions. This is because relying on such SMT practices does not ensure an agnostic solution for storage formats (such as Avro, Parquet, or JSON).

Full Changelog: 6.3.1...6.3.1

Stream Reactor 6.3.0

Release notes

New Connector
The HTTP Sink is offered as beta. Please report any issues via GitHub issues.

Stream Reactor 6.2.0

Release notes

New Connector
The GCP Storage source is offered as beta. Please report any issues via GitHub issues.

Important
AWS S3 Source Partition search properties have changed. See the release notes for detailed information.

Stream Reactor 6.1.0

All Connectors:
In this release, all connectors have been updated to address an issue related to conflicting Antlr jars that may arise in specific environments.

AWS S3 Source:
Byte Array Support: Resolved an issue where storing the Key/Value as an array of bytes caused compatibility problems due to the connector returning java.nio.ByteBuffer while the Connect framework's ByteArrayConverter only works with byte[]. This update ensures seamless conversion to byte[] if the key/value is a ByteBuffer.

JMS Sink:
Fix for NullPointerException: Addressed an issue where the JMS sink connector encountered a NullPointerException when processing a message with a null JMSReplyTo header value.

JMS Source:
Fix for DataException: Resolved an issue where the JMS source connector encountered a DataException when processing a message with a JMSReplyTo header set to a queue.

AWS S3 Sink/GCP Storage Sink (beta)/Azure Datalake Sink (beta):
GZIP Support for JSON Writing: Added support for GZIP compression when writing JSON data to AWS S3, GCP Storage, and Azure Datalake sinks.

Stream Reactor 6.0.3

6.0.3

Stream Reactor 6.0.2

Improve suppport for handling GCP naming conventions

Stream Reactor 6.0.1

Three connectors were updated in this release:

• AWS S3
• Azure Datalake
• GCP Storage

The following enhancements were made:

• Removed check preventing nested paths being used in the sink.
• Avoid cast exception in GCP Storage connector when using Credentials mode.

Please Remember
The Azure Data Lake and GCP Storage sinks are offered as beta. Please report any issues via GitHub issues.

For the latest version of all other connectors please see Version 6.0.0