-
Notifications
You must be signed in to change notification settings - Fork 116
Multi-namespace test #693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Multi-namespace test #693
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| multi-namespace.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,188 @@ | ||
| --- | ||
| vas: | ||
| multi-namespace-skmo: | ||
| stages: | ||
| - name: namespace-configuration # stage 0 | ||
| path: examples/va/multi-namespace/namespace | ||
| wait_conditions: | ||
| - >- | ||
| oc -n default wait ns openstack2 | ||
| --for jsonpath='{.status.phase}'=Active | ||
| --timeout=5m | ||
| values: | ||
| - name: namespace-values | ||
| src_file: values.yaml | ||
| build_output: namespace.yaml | ||
|
|
||
| - name: nncp-configuration # stage 1 | ||
| path: examples/va/multi-namespace/control-plane/networking/nncp | ||
| wait_conditions: | ||
| # We don't wait for these NNCPs at this stage, because we'll wait for | ||
| # both namespaces in the next stage so that they can deploy in parallel | ||
| # to save time | ||
| - >- | ||
| oc -n default wait ns openstack2 | ||
| --for jsonpath='{.status.phase}'=Active | ||
| --timeout=5m | ||
| values: | ||
| - name: network-values | ||
| src_file: values.yaml | ||
| build_output: nncp.yaml | ||
|
|
||
| - name: nncp-configuration2 # stage 2 | ||
| path: examples/va/multi-namespace/control-plane2/networking/nncp | ||
| wait_conditions: | ||
| - >- | ||
| oc -n openstack wait nncp | ||
| -l osp/nncm-config-type=standard | ||
| --for jsonpath='{.status.conditions[0].reason}'=SuccessfullyConfigured | ||
| --timeout=5m | ||
| values: | ||
| - name: network-values2 | ||
| src_file: values.yaml | ||
| build_output: nncp2.yaml | ||
|
|
||
| - name: network-configuration # stage 3 | ||
| path: examples/va/multi-namespace/control-plane/networking | ||
| wait_conditions: | ||
| - >- | ||
| oc -n metallb-system wait pod | ||
| -l app=metallb -l component=speaker | ||
| --for condition=Ready | ||
| --timeout=5m | ||
| values: | ||
| - name: network-values | ||
| src_file: nncp/values.yaml | ||
| build_output: network.yaml | ||
|
|
||
| - name: network-configuration2 # stage 4 | ||
| path: examples/va/multi-namespace/control-plane2/networking | ||
| wait_conditions: | ||
| - >- | ||
| oc -n metallb-system wait pod | ||
| -l app=metallb -l component=speaker | ||
| --for condition=Ready | ||
| --timeout=5m | ||
| values: | ||
| - name: network-values2 | ||
| src_file: nncp/values.yaml | ||
| build_output: network2.yaml | ||
|
|
||
| - name: control-plane # stage 5 | ||
| path: examples/va/multi-namespace-skmo/control-plane | ||
| wait_conditions: | ||
| # Just confirm the central OSCP CR exists in the API server. | ||
| # The pre_stage_run of stage 6 (prepare-leaf.yaml) will use | ||
| # Ansible retry logic to wait for Keystone and openstackclient | ||
| # readiness before proceeding, allowing parallel deployment. | ||
| - >- | ||
| oc -n openstack wait osctlplane controlplane | ||
| --for jsonpath='{.metadata.name}'=controlplane | ||
| --timeout=5m | ||
| values: | ||
| - name: service-values | ||
| src_file: service-values.yaml | ||
| - name: network-values | ||
| src_file: ../../multi-namespace/control-plane/networking/nncp/values.yaml | ||
| build_output: ../control-plane.yaml | ||
|
|
||
| - pre_stage_run: # stage 6 | ||
| - name: Prepare SKMO leaf prerequisites in regionZero | ||
| type: playbook | ||
| source: "skmo/prepare-leaf.yaml" | ||
| inventory: "${HOME}/ci-framework-data/artifacts/zuul_inventory.yml" | ||
| name: control-plane2 # stage 6 | ||
| path: examples/va/multi-namespace-skmo/control-plane2 | ||
| wait_conditions: | ||
| - >- | ||
| oc -n openstack wait osctlplane controlplane --for condition=Ready | ||
| --timeout=60m | ||
| - >- | ||
| oc -n openstack2 wait osctlplane controlplane --for condition=Ready | ||
| --timeout=60m | ||
| values: | ||
| - name: service-values | ||
| src_file: service-values.yaml | ||
| - name: network-values2 | ||
| src_file: ../../multi-namespace/control-plane2/networking/nncp/values.yaml | ||
|
vakwetu marked this conversation as resolved.
|
||
| build_output: ../control-plane2.yaml | ||
| post_stage_run: | ||
| - name: Trust leaf region CA in central region | ||
| type: playbook | ||
| source: "skmo/trust-leaf-ca.yaml" | ||
| inventory: "${HOME}/ci-framework-data/artifacts/zuul_inventory.yml" | ||
| - name: Ensure central control plane uses custom CA bundle | ||
| type: playbook | ||
| source: "skmo/ensure-central-ca-bundle.yaml" | ||
| inventory: "${HOME}/ci-framework-data/artifacts/zuul_inventory.yml" | ||
| - name: Configure barbican-keystone-listener transport URL for leaf region | ||
| type: playbook | ||
| source: "skmo/configure-leaf-listener.yaml" | ||
| inventory: "${HOME}/ci-framework-data/artifacts/zuul_inventory.yml" | ||
|
|
||
| - name: edpm-nodeset # stage 7 | ||
| path: examples/va/multi-namespace/edpm/nodeset | ||
| wait_conditions: | ||
| # We don't wait for this namespace's OpenStackDataPlaneNodeSet at | ||
| # this stage, because we'll wait for both namespaces in the next | ||
| # stage so that they can deploy in parallel to save time | ||
| - >- | ||
| oc -n default wait ns openstack2 | ||
| --for jsonpath='{.status.phase}'=Active | ||
| --timeout=5m | ||
| values: | ||
| - name: edpm-nodeset-values | ||
| src_file: values.yaml | ||
| build_output: nodeset.yaml | ||
|
|
||
| - pre_stage_run: # stage 8 | ||
| - name: Get OpenStackDataPlaneServices for openstack2 namespace | ||
| type: playbook | ||
| source: "../../playbooks/multi-namespace/ns2_osdp_services.yaml" | ||
| inventory: "${HOME}/ci-framework-data/artifacts/zuul_inventory.yml" | ||
| name: edpm-nodeset2 | ||
| path: examples/va/multi-namespace/edpm2/nodeset | ||
| wait_conditions: | ||
| - >- | ||
| oc -n openstack wait | ||
| osdpns openstack-edpm --for condition=SetupReady | ||
| --timeout=10m | ||
| - >- | ||
| oc -n openstack2 wait | ||
| osdpns openstack-edpm --for condition=SetupReady | ||
| --timeout=10m | ||
| values: | ||
| - name: edpm-nodeset2-values | ||
| src_file: values.yaml | ||
| build_output: nodeset2.yaml | ||
|
|
||
| - name: edpm-deployment # stage 9 | ||
| path: examples/va/multi-namespace/edpm | ||
| wait_conditions: | ||
| # We don't wait for this namespace's OpenStackDataPlaneDeployment at | ||
| # this stage, because we'll wait for both namespaces in the next | ||
| # stage so that they can deploy in parallel to save time | ||
| - >- | ||
| oc -n default wait ns openstack2 | ||
| --for jsonpath='{.status.phase}'=Active | ||
| --timeout=5m | ||
| values: | ||
| - name: edpm-deployment-values | ||
| src_file: values.yaml | ||
| build_output: deployment.yaml | ||
|
|
||
| - name: edpm-deployment2 # stage 10 | ||
| path: examples/va/multi-namespace/edpm2 | ||
| wait_conditions: | ||
| - >- | ||
| oc -n openstack wait | ||
| osdpns openstack-edpm --for condition=Ready | ||
| --timeout=60m | ||
| - >- | ||
| oc -n openstack2 wait | ||
| osdpns openstack-edpm --for condition=Ready | ||
| --timeout=60m | ||
| values: | ||
| - name: edpm-deployment2-values | ||
| src_file: values.yaml | ||
| build_output: deployment2.yaml | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| namespace.yaml | ||
| nncp.yaml | ||
| nncp2.yaml | ||
| networking.yaml | ||
| networking2.yaml | ||
| control-plane.yaml | ||
| control-plane2.yaml | ||
| nodeset.yaml | ||
| nodeset2.yaml | ||
| deployment.yaml | ||
| deployment2.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # Single Keystone Multi-region OpenStack (SKMO) | ||
|
|
||
| This is a collection of CR templates that represent a validated Red Hat | ||
| OpenStack Services on OpenShift deployment with the following | ||
| characteristics: | ||
|
|
||
| - 3 master/worker combo-node OpenShift cluster | ||
| - Two OpenStack control planes deployed in separate namespaces: | ||
| - **Central region** (`openstack` namespace, `regionOne`): hosts the | ||
| Keystone identity service and the Horizon dashboard. The horizon | ||
| dashboard provides a "single plane of glass" for the central and | ||
| leaf regions. | ||
| - **Leaf region** (`openstack2` namespace, `regionTwo`): all services | ||
| authenticate against the central Keystone identity service | ||
| (`externalKeystoneAPI: true`). Neither keystone nor horizon pods | ||
| are present in the leaf region. | ||
| - Barbican is enabled in both regions. | ||
| - **Central region** The barbican-keystone-listener listens to the | ||
| central region's rabbitmq service using a unique pool_name: | ||
| (`pool_name = barbican-listener-regionOne`) | ||
| - **Leaf Region** The barbican-keystone-listener is configured to | ||
| listen to the central region's rabbitmq service, rather than the | ||
| leaf region rabbitmq service. It uses a unique pool_name: | ||
| (`pool_name = barbican-listener-regionTwo`) | ||
| - Mutual CA trust: each region's root CA certificates are added to the | ||
| other region's custom CA bundle | ||
| - Network isolation over a single NIC using OVN | ||
| - 3-replica Galera database and RabbitMQ per region | ||
| - RabbitMQ memory reduced to 2Gi per instance for compact clusters | ||
|
|
||
| ## Considerations | ||
|
|
||
| 1. These CRs are validated for the overall functionality of the OSP cloud | ||
| deployed, but they nonetheless require customization for the particular | ||
| environment in which they are utilized. In this sense they are | ||
| _templates_ meant to be consumed and tweaked to fit the specific | ||
| constraints of the hardware available. | ||
|
|
||
| 2. The CRs are applied against an OpenShift cluster in _stages_. That is, | ||
| there is an ordering in which each grouping of CRs is fed to the | ||
| cluster. It is _not_ a case of simply taking all CRs from all stages | ||
| and applying them all at once. | ||
|
|
||
| 3. [kustomize](https://kustomize.io/) is used to generate control plane | ||
| CRs dynamically. The `values.yaml` and `service-values.yaml` files in | ||
| each stage directory must be updated to fit your environment. kustomize | ||
| version 5 or newer is required. | ||
|
|
||
| 4. The `control-plane2/skmo-values.yaml` file contains SKMO-specific | ||
| configuration such as the leaf region name, Keystone endpoint URLs, | ||
| and CA bundle secret names. These **must** be customized before | ||
| deployment. | ||
|
|
||
| 5. The two control planes are deployed in parallel (stages 5 and 6) to | ||
| reduce overall deployment time. The automation hooks handle the | ||
| inter-region setup (transport URLs, CA trust) automatically. | ||
|
|
||
| ## Customization | ||
|
|
||
| Before deploying, update the following files: | ||
|
|
||
| ### Central region (`control-plane/`) | ||
|
|
||
| - `service-values.yaml`: Set the Barbican keystone listener pool name | ||
| for the central region (`barbican-listener-regionOne` by default). | ||
|
|
||
| ### Leaf region (`control-plane2/`) | ||
|
|
||
| - `service-values.yaml`: Set the Barbican keystone listener pool name | ||
| for the leaf region, Keystone `externalKeystoneAPI` URL, and any | ||
| other leaf-specific service configuration. | ||
| - `skmo-values.yaml`: Set the following values for your environment: | ||
| - `leafRegion`: The region name for the leaf control plane (e.g. `regionTwo`) | ||
| - `keystoneInternalURL` / `keystonePublicURL`: The central Keystone | ||
| endpoint URLs reachable from the leaf region | ||
| - `leafAdminUser` / `leafAdminProject`: Admin credentials for the leaf | ||
| region in the central Keystone | ||
|
|
||
| ## Stages | ||
|
|
||
| All stages must be executed in the order listed below. See | ||
| `automation/vars/multi-namespace-skmo.yaml` for the automation | ||
| configuration used in CI. | ||
|
|
||
| 1. [Install the OpenStack K8S operators and their dependencies](../../common/) | ||
| 2. Configure node network (`nncp`) for the central namespace | ||
| 3. Configure node network (`nncp`) for the leaf namespace | ||
| 4. Deploy networking (NAD, MetalLB, etc.) for the central namespace | ||
| 5. Deploy networking (NAD, MetalLB, etc.) for the leaf namespace | ||
| 6. **Deploy the central control plane** (`control-plane/`). The central | ||
| Keystone and Barbican services are configured here. The automation | ||
| waits only for the CR to exist, allowing the leaf control plane to | ||
| deploy in parallel. | ||
| 7. **Deploy the leaf control plane** (`control-plane2/`). Before | ||
| deployment, automation hooks: | ||
| - Register the leaf region and admin user in the central Keystone | ||
| - Create a cross-region RabbitMQ `TransportURL` for the leaf Barbican | ||
| keystone listener | ||
| - Copy the transport URL secret to the leaf namespace | ||
| - Add the central region's root CA to the leaf CA bundle | ||
|
|
||
| After deployment, automation hooks: | ||
| - Add the leaf region's root CA to the central CA bundle | ||
| - Configure the central control plane to use the custom CA bundle | ||
| - Patch the leaf control plane with the cross-region transport URL | ||
| for `barbican-keystone-listener` |
43 changes: 43 additions & 0 deletions
43
examples/va/multi-namespace-skmo/control-plane/kustomization.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| --- | ||
| apiVersion: kustomize.config.k8s.io/v1alpha1 | ||
| kind: Component | ||
|
|
||
| components: | ||
| - ../../multi-namespace/control-plane | ||
|
|
||
| patches: | ||
| - target: | ||
| kind: ConfigMap | ||
| name: service-values | ||
| path: service-values.yaml | ||
| - target: | ||
| group: core.openstack.org | ||
| version: v1beta1 | ||
| kind: OpenStackControlPlane | ||
| name: controlplane | ||
| patch: |- | ||
| - op: replace | ||
| path: /spec/rabbitmq/templates/rabbitmq/resources/requests/memory | ||
| value: 2Gi | ||
| - op: replace | ||
| path: /spec/rabbitmq/templates/rabbitmq/resources/limits/memory | ||
| value: 2Gi | ||
| - op: replace | ||
| path: /spec/rabbitmq/templates/rabbitmq-cell1/resources/requests/memory | ||
| value: 2Gi | ||
| - op: replace | ||
| path: /spec/rabbitmq/templates/rabbitmq-cell1/resources/limits/memory | ||
| value: 2Gi | ||
|
|
||
| replacements: | ||
|
fultonj marked this conversation as resolved.
|
||
| - source: | ||
| kind: ConfigMap | ||
| name: service-values | ||
| fieldPath: data.barbican.barbicanKeystoneListener.customServiceConfig | ||
| targets: | ||
| - select: | ||
| kind: OpenStackControlPlane | ||
| fieldPaths: | ||
| - spec.barbican.template.barbicanKeystoneListener.customServiceConfig | ||
| options: | ||
| create: true | ||
15 changes: 15 additions & 0 deletions
15
examples/va/multi-namespace-skmo/control-plane/service-values.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| --- | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: service-values | ||
| annotations: | ||
| config.kubernetes.io/local-config: "true" | ||
| data: | ||
| tls: | ||
| caBundleSecretName: custom-ca-certs | ||
| barbican: | ||
| barbicanKeystoneListener: | ||
| customServiceConfig: | | ||
| [keystone_notifications] | ||
| pool_name = barbican-listener-regionOne |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| nncp2.yaml | ||
| networking2.yaml | ||
| control-plane2.yaml |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fultonj What do you think about this? See https://github.com/openstack-k8s-operators/architecture/pull/693/changes#diff-4f3f430f947b93ad5cc30274ea06d33908e6c1af52012f9955e60f2429cd6550R86 to see how it's used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving this sets precedence that
..traversal insrc_fileis an acceptable pattern.I think that's OK. I think it's good to promote DRY instead of requiring the same files to be copied (avoid drift).
An alternative is to do a symlink, but I think that would make things more messy. Let's just be clear that there will be cases where from other VAs/DTs could be used.