Change listeners configuration to list to allow for more listeners #3603

scholzj · 2020-09-04T00:20:35Z

Type of change

Refactoring

Description

This PR implements the Strimzi Proposal no 5 and brings the possibility to configure listeners as a list and allow to configure more listeners at the same time.

Compared to the proposal, the internal listener type is now internal since service seemed a bit too confusing (all listeners use some services). Since all listeners now use the same API objects, there is now a new class for validation to check all kinds of different parameters and ensure only the compatible fields are used. But this also allows to use a lot of new configuration combinations which were not possible in the past because they were not implemented. That includes for example advertised host and port configuration for all listeners etc. I also merged the override and configuration sections which were confusing and had a big overlap in the past API. Additionally, I added option to use fully qualified service names in internal listeners as requested by some users (has to be enabled - #2656).

There is also a convertor which converts the old API to the new API for backwards compatibility. There are also several methods which take care of using the right names for the old listeners to not change listener names in Kafka or service names in Kube since that would change all addresses etc.

PS: Sorry for the big PR ... I really was trying to find ways how to split it into smaller PRs. But I do not think it was really possible here.

Checklist

Write tests
Make sure all tests pass
Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
Update CHANGELOG.md

scholzj · 2020-09-04T07:59:56Z

/azp run acceptance

azure-pipelines · 2020-09-04T08:00:52Z

Azure Pipelines successfully started running 1 pipeline(s).

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/ArrayOrObjectKafkaListeners.java

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/KafkaListenerType.java

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/ListenersConvertor.java

...perator/src/main/java/io/strimzi/operator/cluster/model/KafkaBrokerConfigurationBuilder.java

cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaCluster.java

tombentley · 2020-09-04T08:30:50Z

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/ArrayOrObjectKafkaListeners.java

+        public void serialize(ArrayOrObjectKafkaListeners value, JsonGenerator generator, SerializerProvider provider) throws IOException {
+            if (value != null) {
+                if (value.listValue != null)    {
+                    generator.writeObject(value.listValue);


We write an object for the list value?

Yes. This is one of the chaotic things where Object here is Java Object and not YAML / JSON Object :-(.

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/GenericKafkaListener.java

cluster-operator/src/main/java/io/strimzi/operator/cluster/model/ListenersValidator.java

.../java/io/strimzi/operator/cluster/operator/assembly/KafkaAssemblyOperatorCustomCertTest.java

scholzj · 2020-09-04T15:13:56Z

/azp run acceptance

azure-pipelines · 2020-09-04T15:14:15Z

Pull request contains merge conflicts.

scholzj · 2020-09-04T18:28:18Z

/azp run acceptance

azure-pipelines · 2020-09-04T18:28:28Z

Azure Pipelines successfully started running 1 pipeline(s).

scholzj · 2020-09-04T21:42:28Z

/azp run acceptance

azure-pipelines · 2020-09-04T21:42:38Z

Azure Pipelines successfully started running 1 pipeline(s).

scholzj · 2020-09-05T00:31:21Z

/azp run regression

azure-pipelines · 2020-09-05T00:31:31Z

Azure Pipelines successfully started running 1 pipeline(s).

ppatierno

Good work I had a first pass.

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/ArrayOrObjectKafkaListeners.java

cluster-operator/src/main/java/io/strimzi/operator/cluster/model/ListenersUtils.java

scholzj · 2020-09-05T19:46:42Z

/azp run acceptance

azure-pipelines · 2020-09-05T19:46:54Z

Azure Pipelines successfully started running 1 pipeline(s).

scholzj · 2020-09-05T22:34:03Z

/azp run regression

azure-pipelines · 2020-09-05T22:34:13Z

Azure Pipelines successfully started running 1 pipeline(s).

scholzj · 2020-09-06T12:17:12Z

@strimzi-ci run tests

strimzi-ci · 2020-09-06T16:15:04Z

❌ Test Summary ❌

TEST_PROFILE: acceptance
EXCLUDED_GROUPS: networkpolicies
TEST_CASE: *ST
TOTAL: 50
PASS: 49
FAIL: 1
SKIP: 0
BUILD_NUMBER: 1514
BUILD_ENV: oc cluster up

Re-run command:
@strimzi-ci run tests false profile=acceptance testcase=io.strimzi.systemtest.bridge.HttpBridgeKafkaExternalListenersST#testTlsAuthWithWeirdUsername

scholzj · 2020-09-06T16:22:33Z

@strimzi-ci run tests false profile=acceptance testcase=io.strimzi.systemtest.bridge.HttpBridgeKafkaExternalListenersST#testTlsAuthWithWeirdUsername

strimzi-ci · 2020-09-06T17:22:26Z

✔️ Test Summary ✔️

TEST_PROFILE: acceptance
EXCLUDED_GROUPS: networkpolicies,flaky
TEST_CASE: io.strimzi.systemtest.bridge.HttpBridgeKafkaExternalListenersST#testTlsAuthWithWeirdUsername
TOTAL: 1
PASS: 1
FAIL: 0
SKIP: 0
BUILD_NUMBER: 1515
BUILD_ENV: oc cluster up

scholzj · 2020-09-06T21:22:57Z

@strimzi-ci run tests false profile=acceptance testcase=io.strimzi.systemtest.kafka.BackwardsCompatibleListenersST

scholzj · 2020-09-06T21:24:59Z

@tombentley @samuel-hawker @ppatierno @Frawless I changed this from Draft to regular PR and I think this should now be ready for the final review. Thanks

Signed-off-by: Jakub Scholz <www@scholzj.com>

scholzj · 2020-09-07T17:58:56Z

/azp run acceptance

azure-pipelines · 2020-09-07T17:59:06Z

Azure Pipelines successfully started running 1 pipeline(s).

scholzj · 2020-09-07T17:59:12Z

@strimzi-ci run tests

Frawless

Great work!

.azure/regression-pipeline.yaml

ppatierno

Great work!

CHANGELOG.md

Signed-off-by: Jakub Scholz <www@scholzj.com>

...ava/io/strimzi/api/kafka/model/listener/arraylistener/GenericKafkaListenerConfiguration.java

scholzj · 2020-09-07T22:44:35Z

/azp run regression

azure-pipelines · 2020-09-07T22:44:46Z

Azure Pipelines successfully started running 1 pipeline(s).

scholzj · 2020-09-07T22:45:05Z

/azp run acceptance

azure-pipelines · 2020-09-07T22:45:16Z

Azure Pipelines successfully started running 1 pipeline(s).

strimzi-ci · 2020-09-07T23:49:47Z

✔️ Test Summary ✔️

TEST_PROFILE: acceptance
EXCLUDED_GROUPS: networkpolicies
TEST_CASE: *ST
TOTAL: 47
PASS: 47
FAIL: 0
SKIP: 0
BUILD_NUMBER: 1524
BUILD_ENV: oc cluster up

❗ Test Failures ❗

io.strimzi.systemtest.security.oauth.OauthTlsST

scholzj added this to the 0.20.0 milestone Sep 4, 2020

samuel-hawker reviewed Sep 4, 2020

View reviewed changes

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/ArrayOrObjectKafkaListeners.java Outdated Show resolved Hide resolved

samuel-hawker reviewed Sep 4, 2020

View reviewed changes

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/ListenersConvertor.java Outdated Show resolved Hide resolved

samuel-hawker reviewed Sep 4, 2020

View reviewed changes

tombentley reviewed Sep 4, 2020

View reviewed changes

scholzj force-pushed the make-it-possible-to-alternate-multiple-types-in-api-schema branch from ffc8c7f to 02b4b0c Compare September 4, 2020 18:01

ppatierno reviewed Sep 5, 2020

View reviewed changes

api/src/main/java/io/strimzi/api/kafka/model/listener/v2/ArrayOrObjectKafkaListeners.java Outdated Show resolved Hide resolved

cluster-operator/src/main/java/io/strimzi/operator/cluster/model/ListenersUtils.java Show resolved Hide resolved

strimzi deleted a comment from azure-pipelines bot Sep 6, 2020

scholzj requested a review from Frawless September 6, 2020 21:24

scholzj added 13 commits September 7, 2020 19:00

Refactor listeners to allow full array of listeners

57f0166

Signed-off-by: Jakub Scholz <www@scholzj.com>

Review comments

da2892a

Signed-off-by: Jakub Scholz <www@scholzj.com>

Fix some STs

07c26c2

Signed-off-by: Jakub Scholz <www@scholzj.com>

Fix another system test

eb9dd45

Signed-off-by: Jakub Scholz <www@scholzj.com>

Review comments and ST fixes

981c2ba

Signed-off-by: Jakub Scholz <www@scholzj.com>

Another bunch of fixes

6dc1232

Signed-off-by: Jakub Scholz <www@scholzj.com>

Fix the stupid renaming

a8b596e

Signed-off-by: Jakub Scholz <www@scholzj.com>

More unit tests and CHANGELOG

9044ef0

Signed-off-by: Jakub Scholz <www@scholzj.com>

Fix load balancer source range test

8b3f5a0

Signed-off-by: Jakub Scholz <www@scholzj.com>

MAke sure serialization works and write backwards compatibility STs

0130915

Signed-off-by: Jakub Scholz <www@scholzj.com>

Increase regression test pipeline timout and fix OCP system test

d8ba7ac

Signed-off-by: Jakub Scholz <www@scholzj.com>

Fix typo

931b8a7

Signed-off-by: Jakub Scholz <www@scholzj.com>

Review comments

c606555

Signed-off-by: Jakub Scholz <www@scholzj.com>

scholzj force-pushed the make-it-possible-to-alternate-multiple-types-in-api-schema branch from 7250109 to c606555 Compare September 7, 2020 17:11

Frawless approved these changes Sep 7, 2020

View reviewed changes

.azure/regression-pipeline.yaml Show resolved Hide resolved

ppatierno approved these changes Sep 7, 2020

View reviewed changes

see-quick reviewed Sep 7, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Fix CHANGELOG typo and split Azure regression builds into 5 jobs

e6aa3b4

Signed-off-by: Jakub Scholz <www@scholzj.com>

see-quick approved these changes Sep 7, 2020

View reviewed changes

...ava/io/strimzi/api/kafka/model/listener/arraylistener/GenericKafkaListenerConfiguration.java Show resolved Hide resolved

scholzj merged commit e786ce9 into strimzi:master Sep 8, 2020

scholzj deleted the make-it-possible-to-alternate-multiple-types-in-api-schema branch September 8, 2020 06:57

see-quick mentioned this pull request Sep 25, 2020

[MO] - [system test] -> test suite of multiple listeners #3651

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change listeners configuration to list to allow for more listeners #3603

Change listeners configuration to list to allow for more listeners #3603

scholzj commented Sep 4, 2020 •

edited

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

tombentley Sep 4, 2020

scholzj Sep 4, 2020

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

scholzj commented Sep 5, 2020

azure-pipelines bot commented Sep 5, 2020

ppatierno left a comment

scholzj commented Sep 5, 2020

azure-pipelines bot commented Sep 5, 2020

scholzj commented Sep 5, 2020

azure-pipelines bot commented Sep 5, 2020

scholzj commented Sep 6, 2020

strimzi-ci commented Sep 6, 2020

scholzj commented Sep 6, 2020

strimzi-ci commented Sep 6, 2020

scholzj commented Sep 6, 2020

scholzj commented Sep 6, 2020

scholzj commented Sep 7, 2020

azure-pipelines bot commented Sep 7, 2020

scholzj commented Sep 7, 2020

Frawless left a comment

ppatierno left a comment

scholzj commented Sep 7, 2020

azure-pipelines bot commented Sep 7, 2020

scholzj commented Sep 7, 2020

azure-pipelines bot commented Sep 7, 2020

strimzi-ci commented Sep 7, 2020

Change listeners configuration to list to allow for more listeners #3603

Change listeners configuration to list to allow for more listeners #3603

Conversation

scholzj commented Sep 4, 2020 • edited

Type of change

Description

Checklist

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

tombentley Sep 4, 2020

Choose a reason for hiding this comment

scholzj Sep 4, 2020

Choose a reason for hiding this comment

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

scholzj commented Sep 4, 2020

azure-pipelines bot commented Sep 4, 2020

scholzj commented Sep 5, 2020

azure-pipelines bot commented Sep 5, 2020

ppatierno left a comment

Choose a reason for hiding this comment

scholzj commented Sep 5, 2020

azure-pipelines bot commented Sep 5, 2020

scholzj commented Sep 5, 2020

azure-pipelines bot commented Sep 5, 2020

scholzj commented Sep 6, 2020

strimzi-ci commented Sep 6, 2020

❌ Test Summary ❌

scholzj commented Sep 6, 2020

strimzi-ci commented Sep 6, 2020

✔️ Test Summary ✔️

scholzj commented Sep 6, 2020

scholzj commented Sep 6, 2020

scholzj commented Sep 7, 2020

azure-pipelines bot commented Sep 7, 2020

scholzj commented Sep 7, 2020

Frawless left a comment

Choose a reason for hiding this comment

ppatierno left a comment

Choose a reason for hiding this comment

scholzj commented Sep 7, 2020

azure-pipelines bot commented Sep 7, 2020

scholzj commented Sep 7, 2020

azure-pipelines bot commented Sep 7, 2020

strimzi-ci commented Sep 7, 2020

✔️ Test Summary ✔️

❗ Test Failures ❗

scholzj commented Sep 4, 2020 •

edited