Skip to content

v0.11.9

Choose a tag to compare

View all tags
@mfelsche mfelsche released this 21 Oct 08:31
· 2795 commits to main since this release
v0.11.9
ef463d1

0.11.9

Fixes

  • Kafka onramp: Remove failing metadata fetch in order to verify topic existence. Instead detect subscription errors and stop the onramp in that case.

Details

The Kafka onramp used to fail non-reliably during startup when attempting to verify topic existence by fetching the metadata for all given topics. This was error prone. Additionally we hard-coded a timeout that was way too tight, especially for big clusters. We removed this check and instead error properly if we find out that a subscription did not work. This only works asynchronously, so the only way is to stop the source. The Tremor logs will show the error. Example:

2021-10-20T13:16:52.549478+02:00 INFO tremor_runtime::source::kafka - [Source::tremor://localhost/onramp/datain/simplees01/out] Subscribing to: ["test", "snot"]
2021-10-20T13:16:52.549752+02:00 INFO tremor_runtime::source::kafka - [Source::tremor://localhost/onramp/datain/simplees01/out] Subscription initiated...
2021-10-20T13:16:52.550037+02:00 INFO tremor_runtime::onramp - Onramp tremor://localhost/onramp/datain/simplees01/out started.
2021-10-20T13:16:52.550816+02:00 INFO tremor_runtime::source - [Source::tremor://localhost/onramp/datain/simplees01/out] Connecting tremor://localhost/pipeline/system::metrics/system/in as metrics pipeline.
2021-10-20T13:16:52.551138+02:00 INFO tremor_runtime::pipeline - [Pipeline::tremor://localhost/pipeline/system::passthrough/simplees01] Connecting tremor://localhost/onramp/datain/simplees01/out to 'in'
2021-10-20T13:16:52.551325+02:00 INFO tremor_runtime::system - Binding binding tremor://localhost/binding/default/simplees01
2021-10-20T13:16:52.567448+02:00 ERROR tremor_runtime::source::kafka - [Source::tremor://localhost/onramp/datain/simplees01/out] Subscription failed: UnknownTopicOrPartition (Broker: Unknown topic or partition). Stopping.
2021-10-20T13:16:52.567547+02:00 WARN tremor_runtime::source - [Source::tremor://localhost/onramp/datain/simplees01/out] Disconnected.
Assets 5
Loading