MGDSTRM-9181: Have kas ingress controller utilise the openshift-router's dynamic config manager. #850
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k-wall
changed the title
k-wall
changed the title
shawkins
reviewed
Jan 13, 2023
k-wall
commented
Jan 14, 2023
shawkins
reviewed
Jan 16, 2023
|
It's quite difficult to give the responsibility to create the blueprints to fleetshard. It is the admin route that the problem. We have the possibility that developer and standard profiles can have different rate limits, and to support kas-installer, we support edge/passthrough and http/https. These requirements made me push the responsibility for creating the blueprint to I've used a pattern where the operator will create the "required" blueprints as the user requests the first kafka. So expect to see an initial haproxy reload when the first kafka instance is created. You should see no more after that. I don't like the fact that blueprints will accumulate if we ever change the annotations but haven't got a reasonable way to identify stale ones. |
rareddy
reviewed
Jan 16, 2023
SamBarker
approved these changes
Jan 16, 2023
operator/src/main/java/org/bf2/operator/managers/IngressControllerManager.java
Show resolved
Hide resolved
operator/src/main/java/org/bf2/operator/managers/IngressControllerManager.java
Outdated
Show resolved
Hide resolved
operator/src/main/java/org/bf2/operator/managers/IngressControllerManager.java
Outdated
Show resolved
Hide resolved
operator/src/main/java/org/bf2/operator/managers/IngressControllerManager.java
Outdated
Show resolved
Hide resolved
operator/src/main/java/org/bf2/operator/operands/AbstractKafkaCluster.java
Outdated
Show resolved
Hide resolved
…'s dynamic config manager.
…Cluster.java Co-authored-by: Sam Barker <sam@quadrocket.co.uk>
SamBarker
reviewed
Jan 16, 2023
operator/src/main/java/org/bf2/operator/managers/IngressControllerManager.java
Outdated
Show resolved
Hide resolved
SamBarker
reviewed
Jan 16, 2023
There was a problem hiding this comment.
Running locally I'm seeing
023-01-17 12:44:49,899 DEBUG [org.bf2.ope.man.StrimziBundleManager] (QuarkusQuartzScheduler_Worker-12) Strimzi bundle Subscription periodic check
2023-01-17 12:44:50,098 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:44:50,743 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:44:51,379 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:44:51,584 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:44:52,007 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:44:52,224 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:44:52,457 INFO [org.bf2.ope.man.IngressControllerManager] (QuarkusQuartzScheduler_Worker-9) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:44:52,647 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:44:52,862 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:44:53,061 INFO [org.bf2.ope.man.IngressControllerManager] (QuarkusQuartzScheduler_Worker-9) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:44:53,268 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:44:54,771 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:44:55,420 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:44:56,051 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:44:56,261 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:44:56,887 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:44:57,101 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:44:57,735 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:44:57,954 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:44:58,571 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:44:58,781 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:44:59,626 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:00,464 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:01,312 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:02,148 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:02,778 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:02,990 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:03,631 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:03,835 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:04,465 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:04,671 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:05,299 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:06,071 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:06,823 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:07,663 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:08,503 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:09,134 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:09,344 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:10,001 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:10,201 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:10,827 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:11,033 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:11,662 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:11,872 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:12,716 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:13,563 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:14,411 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:15,234 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:15,865 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:16,076 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:16,713 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:16,923 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
2023-01-17 12:45:17,555 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:17,759 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1c
2023-01-17 12:45:18,391 INFO [org.bf2.ope.man.IngressControllerManager] (OkHttp https://api.sbarker-osd-410.felr.s1.devshift.org:6443/...) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:18,597 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas
2023-01-17 12:45:19,449 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1b
2023-01-17 12:45:20,278 INFO [org.bf2.ope.man.IngressControllerManager] (CachedSingleThreadScheduler-1415400893-pool-13-thread-1) Updating the resource limits/container command for Deployment openshift-ingress/router-kas-us-east-1a
Almost continuously which seems suspicious.
|
@showuon and I have verified the fix in dev (using OSD 4.10.15). The continuous update loop mentioned above was because I forgot to stop the on cluster version of fleetshard when spooling up the local copy using quarkus:dev |
showuon
reviewed
Jan 17, 2023
operator/src/main/java/org/bf2/operator/managers/IngressControllerManager.java
Show resolved
Hide resolved
showuon
reviewed
Jan 17, 2023
operator/src/main/java/org/bf2/operator/managers/IngressControllerManager.java
Show resolved
Hide resolved
…llerManager.java Co-authored-by: Sam Barker <sam@quadrocket.co.uk>
showuon
reviewed
Jan 17, 2023
showuon
reviewed
Jan 17, 2023
There was a problem hiding this comment.
Overall LGTM, except the max-dynamic-servers config needs to modify to 6. Besides, there's a test failure seems to be related to the change:
Error: IngressControllerManagerWithProfilesTest.testIngressControllerReplicaCountsWithoutCollocation:63 expected: <2> but was: <4>
Error: Errors:
Error: IngressControllerManagerWithProfilesTest.testIngressControllerNodePlacement:48 NullPointer
It isn't related these changes, it's an unresolved issue we have with quarkus deterministically using mocks - for some reason it doesn't always use what you try to setup in the test. |
Good to know. Then I don't have any other comments. Thanks. |
showuon
approved these changes
Jan 17, 2023
operator/src/test/java/org/bf2/operator/managers/IngressControllerManagerTest.java
Show resolved
Hide resolved
|
Kudos, SonarCloud Quality Gate passed! |
biswassri
pushed a commit
that referenced
this pull request
Jan 17, 2023
…r's dynamic config manager. (#850) * MGDSTRM-9181: Have kas ingress controller utilise the openshift-route's dynamic config manager. * give fleetshard the responsibility to create blueprints for bootstrap/admin routes Co-authored-by: Sam Barker <sam@quadrocket.co.uk>
biswassri
pushed a commit
that referenced
this pull request
Jan 17, 2023
…r's dynamic config manager. (#850) * MGDSTRM-9181: Have kas ingress controller utilise the openshift-route's dynamic config manager. * give fleetshard the responsibility to create blueprints for bootstrap/admin routes Co-authored-by: Sam Barker <sam@quadrocket.co.uk>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enables the (unsupported) dynamic config manager option on openshift-ingress. This is done as otherwise, haproxy reconfigurations (which occur every time new kafka instance is provisioned or deprovisioned) cause all kafka connections to all instances to be dropped. This is too disruptive for applications.
Work based on ideas from @shawkins and @showuon .