feat(23570): Add controller for workspace backup #1530

tolusha · 2025-11-07T10:13:30Z

Is it time when backup is started or finished?
To be honest, I don't understand the value of this field

This value is used to determine whether a workspace was stopped after the last backup cron run. We don't want to do a backup of a workspace if it already has a valid backup and it is still stopped. I compare this timestamp with the time when the workspace was stopped, and based on that, I determine if the backup is necessary.

If it is related to a specific workspace, then how do we handle multiple of them?

It is specific to a whole cluster as it is defined on operator config level, not individual workspaces. This value tracks when the last cron job was executed, not the backup Job itself. And using this value, we are able to determine if the workspace was stopped after the last round and needs a backup.

cc @dkwon17
What do you think of adding a specific annotation to a DW about lastbackuptime instead of having a status?

I thought about it a bit more, maybe the backup job pod can update the DW with an annotation with backup time if the backup and push to registry was both successful? Since it would let DWO know which DevWorkspaces have successfully backed up or not. From the user/admin perspective, I guess they could also view the image registry to tell which were successfully backed up.

Maybe it can be done in a different PR though, IMHO I don't think it's crucial right now cc @cgruver @ibuziuk

A benefit I see of having an annotation set to each DW, is in this scenario:

There's some kind of error when backing up and pushing the backup images (ex. image registry is down)

Assuming that my-devworkspace wasn't started before the next execution of the backup cron job, the DWO will not create a job to backup my-devworkspace since DWO assumes that it was already backed up. DWO only knows the last "global" DWO backup time, and would assume that my-devworkpace was already backed up:
https://github.com/allda/devworkspace-operator/blob/6af5fb855a0512cff8f90c9a7d128c17269eb587/controllers/backupcronjob/backupcronjob_controller.go#L234

If the DWO knew which DevWorkspaces were successfully/unsucessfully backed up, DWO would be able to know that my-devworkspace wasn't backed up, and would be able to create another job to try backing it up once more

annotation with backup time is a very good idea, smth. similar to idling - could be done as a separate PR though

-Original file line number
+Diff line change
@@ Expand Up / @@ -73,6 +73,17 @@ jobs: @@
               quay.io/devfile/project-clone:sha-${{ steps.git-sha.outputs.sha }}
             file: ./project-clone/Dockerfile
+        - name: Build and push
+          uses: docker/build-push-action@0a97817b6ade9f46837855d676c4cca3a2471fc9 #v4.2.1
+          with:
+            context: ./project-backup
+            push: true
+            platforms: linux/amd64, linux/arm64, linux/ppc64le, linux/s390x
+            tags: |
+              quay.io/devfile/project-backup:next
+              quay.io/devfile/project-backup:sha-${{ steps.git-sha.outputs.sha }}
+            file: ./project-backup/Dockerfile
       build-next-olm-imgs:
         runs-on: ubuntu-latest
         needs: build-next-imgs
@@ Expand Down Expand Up / @@ -147,6 +158,7 @@ jobs: @@
               export TAG="sha-${{ needs.build-next-imgs.outputs.git-sha }}"
               export DEFAULT_DWO_IMG="quay.io/devfile/devworkspace-controller:$TAG"
               export PROJECT_CLONE_IMG="quay.io/devfile/project-clone:$TAG"
+              export PROJECT_BACKUP_IMG="quay.io/devfile/project-backup:$TAG"
               # Next builds are not rolled out unless the version is incremented. We want to use semver
               # prerelease tags to make sure each new build increments on the previous one, e.g.
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -130,3 +130,6 @@ jobs: @@
         -
           name: Check if project-clone dockerimage build is working
           run: docker build -f ./project-clone/Dockerfile .
+        -
+          name: Check if project-backup containerimage build is working
+          run: docker build -f ./project-backup/Containerfile project-backup/

-Original file line number
+Diff line change
@@ Expand Up @@
     export DWO_BUNDLE_IMG ?= quay.io/devfile/devworkspace-operator-bundle:next
     export DWO_INDEX_IMG ?= quay.io/devfile/devworkspace-operator-index:next
     export PROJECT_CLONE_IMG ?= quay.io/devfile/project-clone:next
+    export PROJECT_BACKUP_IMG ?= quay.io/devfile/project-backup:next
     export PULL_POLICY ?= Always
     export DEFAULT_ROUTING ?= basic
     export KUBECONFIG ?= ${HOME}/.kube/config
@@ Expand Down Expand Up / @@ -128,6 +129,7 @@ _print_vars: @@
     	@echo "    DWO_BUNDLE_IMG=$(DWO_BUNDLE_IMG)"
     	@echo "    DWO_INDEX_IMG=$(DWO_INDEX_IMG)"
     	@echo "    PROJECT_CLONE_IMG=$(PROJECT_CLONE_IMG)"
+    	@echo "    PROJECT_BACKUP_IMG=$(PROJECT_BACKUP_IMG)"
     	@echo "    PULL_POLICY=$(PULL_POLICY)"
     	@echo "    ROUTING_SUFFIX=$(ROUTING_SUFFIX)"
     	@echo "    DEFAULT_ROUTING=$(DEFAULT_ROUTING)"
@@ Expand Down Expand Up / @@ -369,6 +371,7 @@ help: Makefile @@
     	@echo 'Supported environment variables:'
     	@echo '    DWO_IMG                    - Image used for controller'
     	@echo '    PROJECT_CLONE_IMG          - Image used for project-clone init container'
+    	@echo '    PROJECT_BACKUP_IMG         - Image used for project-backup workspace backup container'
     	@echo '    NAMESPACE                  - Namespace to use for deploying controller'
     	@echo '    KUBECONFIG                 - Kubeconfig which should be used for accessing to the cluster. Currently is: $(KUBECONFIG)'
     	@echo '    ROUTING_SUFFIX             - Cluster routing suffix (e.g. $$(minikube ip).nip.io, apps-crc.testing)'
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -72,6 +72,47 @@ type CleanupCronJobConfig struct { @@
     	Schedule string `json:"schedule,omitempty"`
     }
+    type RegistryConfig struct {
+    	// A registry where backup images are stored. Images are stored
+    	// in {path}/${DEVWORKSPACE_NAMESPACE}/${DEVWORKSPACE_NAME}:latest
+    	// +kubebuilder:validation:Required
+    	Path string `json:"path,omitempty"`
+    	// AuthSecret is the name of a Kubernetes secret of
+    	// type kubernetes.io/dockerconfigjson.
+    	// The secret is expected to be in the same namespace the workspace is running in.
+    	// If secret is not found in the workspace namespace, the operator will look for the secret
+    	// in the namespace where the operator is running in.
+    	// as the DevWorkspaceOperatorCongfig.
+    	// The secret must contain "controller.devfile.io/watch-secret=true" label so that it can be
+    	// recognized by the operator.
+    	// +kubebuilder:validation:Optional
+    	AuthSecret string `json:"authSecret,omitempty"`
+    }
+    type OrasConfig struct {
+    	// ExtraArgs are additional arguments passed to the oras CLI
+    	// +kubebuilder:validation:Optional
+    	ExtraArgs string `json:"extraArgs,omitempty"`
+    }
+    type BackupCronJobConfig struct {
+    	// Enable determines whether backup CronJobs should be created for workspace PVCs.
+    	// Defaults to false if not specified.
+    	// +kubebuilder:validation:Optional
+    	Enable *bool `json:"enable,omitempty"`
+    	// RegistryConfig defines the registry configuration where backup images are stored.
+    	// +kubebuilder:validation:Required
+    	Registry *RegistryConfig `json:"registry,omitempty"`
+    	// OrasConfig defines additional configuration options for the oras CLI used to
+    	// push and pull backup images.
+    	OrasConfig *OrasConfig `json:"oras,omitempty"`
+    	// Schedule specifies the cron schedule for the backup cron job.
+    	// For example, "0 1 * * *" runs daily at 1 AM.
+    	// +kubebuilder:default:="0 1 * * *"
+    	// +kubebuilder:validation:Optional
+    	Schedule string `json:"schedule,omitempty"`
+    }
     type RoutingConfig struct {
     	// DefaultRoutingClass specifies the routingClass to be used when a DevWorkspace
     	// specifies an empty `.spec.routingClass`. Supported routingClasses can be defined
@@ Expand Down Expand Up / @@ -189,6 +230,8 @@ type WorkspaceConfig struct { @@
     	RuntimeClassName *string `json:"runtimeClassName,omitempty"`
     	// CleanupCronJobConfig defines configuration options for a cron job that automatically cleans up stale DevWorkspaces.
     	CleanupCronJob *CleanupCronJobConfig `json:"cleanupCronJob,omitempty"`
+    	// BackupCronJobConfig defines configuration options for a cron job that automatically backs up workspace PVCs.
+    	BackupCronJob *BackupCronJobConfig `json:"backupCronJob,omitempty"`
     	// PostStartTimeout defines the maximum duration the PostStart hook can run
     	// before it is automatically failed. This timeout is used for the postStart lifecycle hook
     	// that is used to run commands in the workspace container. The timeout is specified in seconds.
@@ Expand Down Expand Up / @@ -331,14 +374,26 @@ type ConfigmapReference struct { @@
     	Namespace string `json:"namespace"`
     }
+    type OperatorConfigurationStatus struct {
+    	// Conditions represent the latest available observations of the OperatorConfiguration's state
+    	Conditions []metav1.Condition `json:"conditions,omitempty"`
+    	// LastBackupTime is the timestamp of the last successful backup. Nil if
+    	// no backup is configured or no backup has yet succeeded.
+    	LastBackupTime *metav1.Time `json:"lastBackupTime,omitempty"`
+    }
     // DevWorkspaceOperatorConfig is the Schema for the devworkspaceoperatorconfigs API
     // +kubebuilder:object:root=true
+    // +kubebuilder:subresource:status
     // +kubebuilder:resource:path=devworkspaceoperatorconfigs,scope=Namespaced,shortName=dwoc
     type DevWorkspaceOperatorConfig struct {
     	metav1.TypeMeta   `json:",inline"`
     	metav1.ObjectMeta `json:"metadata,omitempty"`
     	Config *OperatorConfiguration `json:"config,omitempty"`
+    	// Status represents the current status of the DevWorkspaceOperatorConfig
+    	// automatically managed by the DevWorkspace Operator.
+    	Status *OperatorConfigurationStatus `json:"status,omitempty"`
     }
     // DevWorkspaceOperatorConfigList contains a list of DevWorkspaceOperatorConfig
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -31,7 +31,7 @@ @@
     set -e
     # List of environment variables that will be replaced by envsubst
-    SUBST_VARS='$NAMESPACE $DWO_IMG $PROJECT_CLONE_IMG $ROUTING_SUFFIX $DEFAULT_ROUTING $PULL_POLICY'
+    SUBST_VARS='$NAMESPACE $DWO_IMG $PROJECT_CLONE_IMG $PROJECT_BACKUP_IMG $ROUTING_SUFFIX $DEFAULT_ROUTING $PULL_POLICY'
     SCRIPT_DIR=$(cd "$(dirname "$0")"; pwd)
     DEPLOY_DIR="$SCRIPT_DIR/../../deploy/"
@@ Expand All / @@ -58,6 +58,11 @@ Arguments: @@
           '--use-defaults' is passed; otherwise, the value of the PROJECT_CLONE_IMG
           environment variable is used. If unspecifed, the default value of
           'quay.io/devfile/project-clone:next' is used.
+      --project-backup-image
+          Image to use for the project backup workspace. Used only when
+          '--use-defaults' is passed; otherwise, the value of the PROJECT_BACKUP_IMG
+          environment variable is used. If unspecifed, the default value of
+          'quay.io/devfile/project-backup:next' is used.
       --split-yaml
           Parse output file combined.yaml into a yaml file for each record
           in combined yaml. Files are output to the 'objects' subdirectory
@@ Expand Down Expand Up / @@ -96,6 +101,10 @@ while [[ "$#" -gt 0 ]]; do @@
           PROJECT_CLONE_IMG=$2
           shift
           ;;
+          --project-backup-image)
+          PROJECT_BACKUP_IMG=$2
+          shift
+          ;;
           --split-yamls)
           SPLIT_YAMLS=true
           ;;
@@ Expand All / @@ -118,6 +127,7 @@ if $USE_DEFAULT_ENV; then @@
       export NAMESPACE=devworkspace-controller
       export DWO_IMG=${DEFAULT_DWO_IMG:-"quay.io/devfile/devworkspace-controller:next"}
       export PROJECT_CLONE_IMG=${PROJECT_CLONE_IMG:-"quay.io/devfile/project-clone:next"}
+      export PROJECT_BACKUP_IMG=${PROJECT_BACKUP_IMG:-"quay.io/devfile/project-backup:next"}
       export PULL_POLICY=Always
       export DEFAULT_ROUTING=basic
       export DEVWORKSPACE_API_VERSION=a6ec0a38307b63a29fad2eea945cc69bee97a683
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(23570): Add controller for workspace backup #1530

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tolusha Nov 7, 2025

Uh oh!

Allda Nov 11, 2025

Uh oh!

tolusha Nov 12, 2025

Uh oh!

Allda Nov 14, 2025

Uh oh!

tolusha Dec 1, 2025

Uh oh!

dkwon17 Dec 6, 2025

Uh oh!

dkwon17 Dec 6, 2025

Uh oh!

ibuziuk Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat(23570): Add controller for workspace backup #1530

Uh oh!

feat(23570): Add controller for workspace backup #1530

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!