Skip to content

New blog post about how to sequence subscriptions #491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 8, 2024
Merged

New blog post about how to sequence subscriptions #491

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a5c4116
New blog post about how to sequence subscriptions
Nov 7, 2024
4e92ccb
Minor formatting and remove a stray sentence
Nov 7, 2024
b4fac6f
Clarify support for resource hooks
Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
333 changes: 333 additions & 0 deletions content/blog/2024-11-07-clustergroup-sequencing.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,333 @@
---
date: 2024-11-07
title: Additional Sequencing Capabilities in clusterGroup
summary: How to sequence subscriptions in the Validated Patterns framework
author: Martin Jackson
blog_tags:
- patterns
- how-to
- sequencing
- subscriptions
---
:toc:

== Preamble

Ideally, all subscriptions installed in Kubernetes and OpenShift will envision
all potential conflicts, and will deal gracefully with situations when other
artifacts installed on the cluster require reconfiguration or different behavior.

Architecturally, we have always said that we prefer eventual consistency, by which
we mean that in the presence of error conditions the software should be prepared to
retry until it can achieve the state it wants to be in. In practice that means we
should be able to install a number of artifacts at the same time, and the artifacts
themselves should be able to achieve the declarative state expressed in their installation.
But some software does not work this way (even if its stated goal and intention is to
work this way) and it is advantageous to be able to impose order of events to create
better situations for installation. For example, even well-crafted software can be
subject to different kinds of timing problems, as we will illustrate below.

Because of this, we have introduced a set of capabilties to the Validated Patterns
clusterGroup chart to enforce sequencing on the objects in a declarative and
Kubernetes-native way.

In this blog post, we will explore the various options that are available as of
today in the Validated Patterns framework for enforcing sequencing for subscriptions.
Inside applications, Validated Patterns has supported these primitives since the first
release of Medical Diagnosis, and will continue to do so.

Since the focus of Validated Patterns on OpenShift is the OpenShift GitOps Operator, these
capabilities rely on the use of resource hooks, described in the upstream docs https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/[here].

Within the framework, we support resource hooks in three ways:

1. Supporting annotations directly on subscription objects at the clusterGroup level
2. Exposing a new optional sequenceJob attribute on subscriptions that uses resource hooks
3. Exposing a new top-level clusterGroup object, extraObjects, that allows users full control of creating their own
resource hooks.

These features are available in version 0.9.8 and later of the Validated Patterns clustergroup chart.

== Race Conditions: The Problem

Timing issues are one of the key problems of distributed systems. One of the biggest categories of timing problems
is https://en.wikipedia.org/wiki/Race_condition[race conditions]. In our context, let's say subscription A reacts to a
condition in subscription B. Subscription A only checks on that condition during installation time. Thus,
subscription A's final state depends on when, exactly, subscription B was installed. Even if A and B were both installed
at the same time, the normal variance of things like how quickly the software was downloaded could result in different
parts of the installation being run at different times, and potentially different results.

The ideal solution to this problem is for subscription B to always be watching for the presence of subscription A, and
reconfiguring itself if it sees subscription A being installed. But if subscription B does not want to do this, or
for some reason cannot do this, or even if a better fix is committed but not yet available, we want to have a set of
practical solutions in the Validated Patterns framework.

So, in the absence of an ideal fix - where all subscriptions are prepared to deal with all possible race outcomes -
a very effective way of working around the problem is enforce ordering or sequencing of actions.

== The Specific Race Condition that led to this feature

The specific case that gave rise to the development of this feature is a race condition between OpenShift Data
Foundation (ODF) and OpenShift Virtualization (OCP-Virt), which will be fixed in a future release of OCP-Virt. The
condition results when OCP-Virt, on installation, discovers a default storageclass, but then subsequently ODF is
installed, which OCP-Virt has specific optimizations for, related to how images are managed for VM guests to be
created. One way to workaround the race condition is to ensure that ODF completes creating its storageclasses before
the OCP-Virt subscription is allowed to install.

== Sync-Waves and ArgoCD/OpenShift GitOps Resource Hooks

The way that resource hooks are designed to work is by giving ordering hints, so that ArgoCD knows what order to
apply resources in. The mechanism is described in the ArgoCD upstream docs https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/[here]. When sync-waves are in use, all resouces in the same sync-wave have to be "healthy" before
resources in the numerically next sync-wave are synced. This mechanism gives us a way of having ArgocD help us enforce
order with objects that it manages.

== Solution 1: Sync-Waves for Subscriptions in clusterGroup

The Validated Patterns framework now allows Kubernetes https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/[annotations] to be added directly to subscription objects in the clusterGroup. ArgoCD uses annotations
for Resource Hooks. The clustergoup chart now passes any annotations attached to subscriptions through to the
subscription object(s) that the clustergroup chart creates. For example:

[source,yaml]
----
openshift-virtualization:
name: kubevirt-hyperconverged
namespace: openshift-cnv
channel: stable
annotations:
argocd.argoproj.io/sync-wave: "10"

openshift-data-foundation:
name: odf-operator
namespace: openshift-storage
annotations:
argocd.argoproj.io/sync-wave: "5"
----

will result in a subscription object that includes the annotations:

[source,yaml]
----
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
annotations:
argocd.argoproj.io/sync-wave: "10"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"operators.coreos.com/v1alpha1","kind":"Subscription","metadata":{"annotations":{"argocd.argoproj.io/sync-wave":"10"},"labels":{"app.kubernetes.io/instance":"ansible-edge-gitops-hub"},"name":"kubevirt-hyperconverged","namespace":"openshift-cnv"},"spec":{"channel":"stable","installPlanApproval":"Automatic","name":"kubevirt-hyperconverged","source":"redhat-operators","sourceNamespace":"openshift-marketplace"}}
creationTimestamp: "2024-11-07T14:24:31Z"
generation: 1
labels:
app.kubernetes.io/instance: ansible-edge-gitops-hub
operators.coreos.com/kubevirt-hyperconverged.openshift-cnv: ""
name: kubevirt-hyperconverged
namespace: openshift-cnv
resourceVersion: "46763"
uid: e9b3892c-9383-41ca-9e8f-ae7be82f012f
spec:
channel: stable
installPlanApproval: Automatic
name: kubevirt-hyperconverged
source: redhat-operators
sourceNamespace: openshift-marketplace

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
annotations:
argocd.argoproj.io/sync-wave: "5"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"operators.coreos.com/v1alpha1","kind":"Subscription","metadata":{"annotations":{"argocd.argoproj.io/sync-wave":"5"},"labels":{"app.kubernetes.io/instance":"ansible-edge-gitops-hub"},"name":"odf-operator","namespace":"openshift-storage"},"spec":{"installPlanApproval":"Automatic","name":"odf-operator","source":"redhat-operators","sourceNamespace":"openshift-marketplace"}}
creationTimestamp: "2024-11-07T14:21:12Z"
generation: 1
labels:
app.kubernetes.io/instance: ansible-edge-gitops-hub
operators.coreos.com/odf-operator.openshift-storage: ""
name: odf-operator
namespace: openshift-storage
resourceVersion: "56652"
uid: 2d9f026f-50e6-4fc1-ad11-8a6a2a636017
spec:
installPlanApproval: Automatic
name: odf-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
----

With this configuration, any objects created with sync-waves lower than "10" must be healthy before the objects in
sync-wave "10" sync. In particular, the odf-operator subscription must be healthy before the kubevirt-hyperconverged
subscription will sync. Similarly, if we defined objects with higher sync-waves than "10", all the resources with
sync-waves higher than "10" will wait until the resources in "10" are healthy. If the subscriptions in question wait
until their components are healthy before reporting they are healthy themselves, this might be all you need to do.
In the case of this particular issue, it was not enough. But because all sequencing in ArgoCD requires the use of
sync-wave annotations, adding the annotation to the subscription object will be necessary for using the other
solutions.

== Solution 2: The `sequenceJob` attribute for Subscriptions in clusterGroup

In this situation, we have a subscription that installs an operator, but it is not enough for just the subscriptions
to be in sync-waves. This is because the subscriptions install operators, and it is the action of the operators
themselves that we have to sequence. In many of these kinds of situations, we can sequence the action by looking for
the existence of a single resource. The new `sequenceJob` construct in subscriptions allows for this kind of
relationship by creating a Job at the same sync-wave precedence as the subscription, and looking for the existence
of a single arbitrary resource in an arbitrary namespace. The Job then waits for that resource to appear, and when
it does, it will be seen as "healthy" and will allow future sync-waves to proceed.

In this example, the ODF operator needs to have created a storageclass so that the OCP-Virt operators can use it as
virtualization storage. If it does not find the kind of storage it wants, it will use the default storageclass
instead, which may lead to inconsistencies in behavior. We can have the Validated Patterns framework create a
mostly boilerplate job to look for the needed resource this way:

[source,yaml]
----
openshift-virtualization:
name: kubevirt-hyperconverged
namespace: openshift-cnv
channel: stable
annotations:
argocd.argoproj.io/sync-wave: "10"

openshift-data-foundation:
name: odf-operator
namespace: openshift-storage
sequenceJob:
resourceType: sc
resourceName: ocs-storagecluster-ceph-rbd
annotations:
argocd.argoproj.io/sync-wave: "5"
----

Note the addition of the `sequenceJob` section in the odf-operator subscription block. This structure will result
in the following Job being created alongside the subscriptions:

[source,yaml]
----
apiVersion: batch/v1
kind: Job
metadata:
annotations:
argocd.argoproj.io/hook: Sync
argocd.argoproj.io/sync-wave: "5"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"batch/v1","kind":"Job","metadata":{"annotations":{"argocd.argoproj.io/hook":"Sync","argocd.argoproj.io/sync-wave":"5"},"labels":{"app.kubernetes.io/instance":"ansible-edge-gitops-hub"},"name":"odf-operator-sequencejob","namespace":"openshift-operators"},"spec":{"completions":1,"parallelism":1,"template":{"spec":{"containers":[{"command":["/bin/bash","-c","while [ 1 ];\ndo\n oc get sc ocs-storagecluster-ceph-rbd \u0026\u0026 break\n echo \"sc ocs-storagecluster-ceph-rbd not found, waiting...\"\n sleep 5\ndone\necho \"sc ocs-storagecluster-ceph-rbd found, exiting...\"\nexit 0\n"],"image":"quay.io/hybridcloudpatterns/imperative-container:v1","name":"odf-operator-sequencejob"}],"restartPolicy":"OnFailure"}}}}
creationTimestamp: "2024-11-07T16:27:26Z"
generation: 1
labels:
app.kubernetes.io/instance: ansible-edge-gitops-hub
name: odf-operator-sequencejob
namespace: openshift-operators
resourceVersion: "201283"
uid: 3084075d-bc1f-4e23-b44d-a13c5d184a6a
spec:
backoffLimit: 6
completionMode: NonIndexed
completions: 1
manualSelector: false
parallelism: 1
podReplacementPolicy: TerminatingOrFailed
selector:
matchLabels:
batch.kubernetes.io/controller-uid: 3084075d-bc1f-4e23-b44d-a13c5d184a6a
suspend: false
template:
metadata:
creationTimestamp: null
labels:
batch.kubernetes.io/controller-uid: 3084075d-bc1f-4e23-b44d-a13c5d184a6a
batch.kubernetes.io/job-name: odf-operator-sequencejob
controller-uid: 3084075d-bc1f-4e23-b44d-a13c5d184a6a
job-name: odf-operator-sequencejob
spec:
containers:
- command:
- /bin/bash
- -c
- |
while [ 1 ];
do
oc get sc ocs-storagecluster-ceph-rbd && break
echo "sc ocs-storagecluster-ceph-rbd not found, waiting..."
sleep 5
done
echo "sc ocs-storagecluster-ceph-rbd found, exiting..."
exit 0
image: quay.io/hybridcloudpatterns/imperative-container:v1
imagePullPolicy: IfNotPresent
name: odf-operator-sequencejob
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: OnFailure
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
----

Since the job is created in sync-wave "5" (which it inherits from the subscription it is attached to by default, though
you can specify a different sync-wave if you prefer), this job must complete before sync-wave "10" starts. So the
storageclass `ocs-storagecluster-ceph-rbd` must exist before OCP-Virt starts deploying, ensuring that it will be able
to "see" and use that storageclass as its default virtualization storage class.

Each subscription is permitted one sequenceJob. Each sequenceJob may have the following attributes:

* *syncWave*: Defaults to the subscription's syncwave from annotations.
* *resourceType*: Resource kind for the resource to watch for.
* *resourceName*: Name of the resource to watch for.
* *resourceNamespace*: Namespace to watch for the resourceType and resourceName in.
* *hookType*: Any of the permissible ArgoCD Resource Hook types. Defaults to "Sync".
* *image*: Image of the container to use for the job. Defaults to the Validated Patterns imperative image.
* *command*: Command to run inside the container, if the default is not suitable. This also enables you to specify multiple resources to watch for in the same job, or to look for a different condition altogether.
* *disabled*: Set this to true in an override if you wish to disable the sequenceJob for some reason (such as running on
a different version of OpenShift or running on a different cloud platform).

If the sequenceJob is not sufficient for your sequencing needs, we have a more generic interface that you can use
that places no restrictions on the objects you can add, so you can use it to create different kinds of conditions.

== Solution 3: The `extraObjects` attribute in clusterGroup

The most open-ended solution to the sequencing problem involves defining arbitrary objects under the `extraObjects`
key for the clustergroup. Here is how you could do that using the example we have been using so far:

[yaml,source]
----
extraObjects:
wait-for-virt-storageclass:
apiVersion: batch/v1
kind: Job
metadata:
name: wait-for-virt-storageclass
annotations:
argocd.argoproj.io/hook: Sync
argocd.argoproj.io/sync-wave: "5"
spec:
parallelism: 1
completions: 1
template:
spec:
restartPolicy: OnFailure
containers:
- name: wait-for-storage-class
image: quay.io/hybridcloudpatterns/imperative-container:v1
command:
- /bin/bash
- -c
- |
while [ 1 ];
do
oc get sc ocs-storagecluster-ceph-rbd && break
echo "Storage class ocs-storagecluster-ceph-rbd not found, waiting..."
sleep 5
done
echo "Storage class ocs-storagecluster-ceph-rbd found, exiting"
exit 0
----

Note that each extraObject has a key and value, and the value will be passed almost unaltered as a Kubernetes manifest.
The special key `disabled` can be used to disable a specific, named extraObject from being created in subsequent
overrides.

== Conclusion

Here is hoping that you do not have sequencing problems to solve in your OpenShift deployments. But if you do, we
hope you will find this feature in Validated Patterns useful. Please let us know, one way or the other, or if you
find other uses, especially for the `extraObjects` feature.