Support multi-cluster deployments (Central + SecuredCluster on separate clusters)#105
Support multi-cluster deployments (Central + SecuredCluster on separate clusters)#105AlexVulaj wants to merge 3 commits into
Conversation
AlexVulaj
force-pushed
the
multi-cluster-support
branch
4 times, most recently
from
49fa237 to
a0ffb80
Compare
|
This will be very helpful for Infra and e2e testing. In Infra/Automation, we've had a long-term todo of adding multi-cluster capability to the demo (ansible scripts) and repeated asks/investigations on various teams have looked at multi-cluster deploys for automated testing. |
AlexVulaj
force-pushed
the
multi-cluster-support
branch
from
31a0cfc to
ce1d010
Compare
AlexVulaj
force-pushed
the
multi-cluster-support
branch
from
ce1d010 to
1817bee
Compare
|
Holding this PR until we can have further discussions around the UX of the feature. |
AlexVulaj
force-pushed
the
multi-cluster-support
branch
from
1817bee to
d81f08c
Compare
AlexVulaj
force-pushed
the
multi-cluster-support
branch
2 times, most recently
from
99c5975 to
7755974
Compare
AlexVulaj
force-pushed
the
multi-cluster-support
branch
from
7755974 to
954384a
Compare
|
Ready for re-review @porridge @mclasmeier |
|
|
||
| if deploySettings.SecuredCluster.CentralEndpoint != "" { | ||
| if !components.IncludesSensor() { | ||
| return errors.New("securedCluster.centralEndpoint can only be used when deploying secured-cluster") |
There was a problem hiding this comment.
I'm not sure about this kind of validation.
I have always thought that roxie just takes what it needs from the provided config. Even if it has a full configuration for component A and only deployment of component B is requested, the config should be usable unmodified.
This validation step would prevent the user from using a single, merged configuration, no?
| // SecuredClusterConfig holds deployment settings for the SecuredCluster component. | ||
| type SecuredClusterConfig struct { | ||
| Namespace string `yaml:"namespace,omitempty"` | ||
| CentralEndpoint string `yaml:"centralEndpoint,omitempty"` |
There was a problem hiding this comment.
Thought about this since yesterday and I keep coming back to the question: why not simply use spec["centralEndpoint"]? This is already a field in the CR and if we want to keep roxie a relative light-weight layer on top of our operator-based installation, we could naturally just use the existing spec field.
|
|
||
| 1. Deploy Central on the hub cluster: | ||
| ```bash | ||
| MAIN_IMAGE_TAG=4.9.2 ./roxie deploy central |
There was a problem hiding this comment.
For the (new) docs, I think we should prever roxie deploy -t <version> over MAIN_IMAGE_TAG, which was primarily added for backwards compat reasons with deployment scripts.
| ROX_ADMIN_PASSWORD=<admin-password> \ | ||
| ROX_CA_CERT_FILE=<path-to-ca-cert> \ | ||
| ./roxie deploy secured-cluster \ | ||
| --set securedCluster.centralEndpoint=<central-loadbalancer-ip>:443 |
There was a problem hiding this comment.
The external endpoint is also contained in some environment variable written by roxie, right?
| --set securedCluster.centralEndpoint=<central-loadbalancer-ip>:443 | ||
| ``` | ||
|
|
||
| > **Note:** The Central endpoint is printed during deployment. If you are in the roxie subshell, |
There was a problem hiding this comment.
Ah, there it comes.
Maybe move this a bit up, because the above instructions as written direct the user to this mode?
Maybe also mention that for some use cases (automation), --envrc might be the better fit?
porridge
left a comment
porridge
left a comment
There was a problem hiding this comment.
FTR I don't have anything to add over what @mclasmeier wrote.
4 participants
Summary
Roxie currently assumes Central and SecuredCluster are deployed on the same Kubernetes cluster. Many ACS customers and internal teams run a hub + spoke architecture where Central lives on one cluster and one or more SecuredClusters are deployed to separate clusters. This PR adds multi-cluster support so that roxie can deploy a SecuredCluster that points to a Central running on a different cluster.
A new
centralEndpointconfig property is added toSecuredClusterConfig, settable via config file or--set:or via config file:
When
centralEndpointis set, roxie uses it as the SecuredCluster CR'scentralEndpointinstead of the internal cluster DNS name. Authentication to Central for CRS generation uses the existingROX_ADMIN_PASSWORDandROX_CA_CERT_FILEenvironment variables. WhencentralEndpointis not set, behavior is completely unchanged — the internalcentral.<namespace>.svc:443endpoint is used, preserving full backward compatibility.Testing
securedCluster.centralEndpointpointing to cluster A — both deployed and connected successfully, sensor logs confirmed connection to remote CentralCloses #99