Common config for consumers and producers

[dmytro-landiak]

Pull Request description

#279

This PR introduces three new environment variables to provide more flexible and granular configuration for Kafka clients (producers and consumers). Previously, configuration options were limited or required individual property mapping. These new variables allow passing raw Kafka properties directly.

The new environment variables are:

• TB_KAFKA_DEFAULT_PRODUCER_CONSUMER_CONFIG
• TB_KAFKA_DEFAULT_CONSUMER_CONFIG
• TB_KAFKA_DEFAULT_PRODUCER_CONFIG

Configuration Precedence & Order of Usage

The configuration is applied in a specific hierarchical order, where specific configurations override general ones.

1. Base Layer: First, TB_KAFKA_DEFAULT_PRODUCER_CONSUMER_CONFIG is applied. This is useful for shared properties (e.g., bootstrap.servers, security.protocol, SASL config).
2. Specific Layer (Overrides): Next, depending on the client type, either TB_KAFKA_DEFAULT_PRODUCER_CONFIG or TB_KAFKA_DEFAULT_CONSUMER_CONFIG is applied.

Logic:

• For a Producer: TB_KAFKA_DEFAULT_PRODUCER_CONSUMER_CONFIG + TB_KAFKA_DEFAULT_PRODUCER_CONFIG
• For a Consumer: TB_KAFKA_DEFAULT_PRODUCER_CONSUMER_CONFIG + TB_KAFKA_DEFAULT_CONSUMER_CONFIG

Note: If a property exists in both the shared config and the specific config, the specific config value takes precedence.

Also, specific configurations, such as TB_KAFKA_MSG_ALL_ADDITIONAL_* and other topic-specific settings, take precedence over TB_KAFKA_DEFAULT_CONSUMER_CONFIG and TB_KAFKA_DEFAULT_PRODUCER_CONFIG.

General checklist

• You have reviewed the guidelines document.
• Labels that classify your pull request have been added.
• The milestone is specified and corresponds to fix version.
• Description references specific issue.
• Description contains human-readable scope of changes.
• Description contains brief notes about what needs to be added to the documentation.
• No merge conflicts, commented blocks of code, code formatting issues.
• Changes are backward compatible or upgrade script is provided.

Front-End feature checklist

• Screenshots with affected component(s) are added. The best option is to provide 2 screens: before and after changes;
• If you change the widget or other API, ensure it is backward-compatible or upgrade script is present.

Back-End feature checklist

• Added corresponding unit and/or integration test(s). Provide written explanation in the PR description if you have failed to add tests.
• If new dependency was added: the dependency tree is checked for conflicts.