Add drift detection and periodic resync for OpenStack resources by eshulman2 · Pull Request #759 · k-orc/openstack-resource-controller

eshulman2 · 2026-04-19T10:45:21Z

Summary

Implements periodic resync and external deletion handling for the OpenStack Resource Controller. When enabled, ORC periodically reconciles resources with OpenStack to detect configuration drift and external deletions, automatically recreating managed resources that were deleted outside of ORC while surfacing terminal errors for imported resources.

Changes

API Changes

Added resyncPeriod field to the code generator spec template for per-resource resync configuration
Added lastSyncTime field to the code generator status template to track successful reconciliation timestamps
Added GetResyncPeriod(), GetLastSyncTime(), and IsImported() methods to APIObjectAdapter interface

CLI & Manager

Added --default-resync-period CLI flag with default value of 0 (disabled)
Added ResyncConfigurable interface to propagate global default to all controllers

Core Resync Logic

Created internal/controllers/generic/resync/ package with:
- DetermineResyncPeriod(): resolves effective period (per-resource → global → disabled)
- CalculateJitteredDuration(): wraps wait.Jitter with 20% positive-only jitter
- ShouldScheduleResync(): guards against scheduling for disabled/terminal/pending-requeue states
Modified shouldReconcile() to trigger reconciliation when resync period has elapsed
Modified reconcileNormal() to schedule jittered requeue after successful reconciliation

External Deletion Handling

Modified GetOrCreateOSResource() to detect 404s when fetching by status.id
Returns typed ExternallyDeleted signal via ReconcileStatus for managed non-imported resources
Restored safety guard for unexpected nil returns from actuators
Imported/unmanaged resources return terminal error
Added ClearStatusID() to status writer for clearing ID before recreation

Status Updates

Modified UpdateStatus() to set lastSyncTime only on successful reconciliation
Fixed Available condition: terminal errors now set False instead of Unknown

Code Generation

Updated api.template and adapter.template with new fields and methods
Regenerated all resources, CRDs, deep copy, and client code

Documentation

Created website/docs/user-guide/drift-detection.md
Updated enhancements/drift-detection.md with implementation status

Commit structure

API fields + templates — hand-written type and template changes
Regenerate — all generated code (CRDs, adapters, apply configs, OpenAPI, docs)
Resync scheduling — scheduler, controller integration, CLI flag, all controller wiring
E2E tests for resync — kuttl test suites + integration tests
External deletion — detection, recreation, condition fix, unit + E2E tests
Documentation — user guide and enhancement doc
Design update — update enhancement proposal with typed ExternallyDeleted signal

Design decisions

Typed ExternallyDeleted signal: The original proposal used a (nil, nil) return from GetOrCreateOSResource to signal external deletion. This was replaced with a typed ExternallyDeleted flag on ReconcileStatus, which avoids ambiguity with actuator bugs that might also return nil and restores a safety guard for unexpected nil returns from GetOSResourceByID.
wait.Jitter reuse: CalculateJitteredDuration wraps k8s.io/apimachinery/pkg/util/wait.Jitter rather than reimplementing the same [base, base*1.2) jitter range.
Removed dead CommonStatus/CommonOptions types: These types were declared but never embedded — the code generator inlines the fields directly. They were removed to avoid misleading readers.

Testing

Unit tests for resync package, shouldReconcile, status updates, external deletion (all policy/import combinations)
6 kuttl E2E test suites: resync-period, resync-disabled, resync-terminal-error, resync-jitter, external-deletion, external-deletion-import
go build, go vet, make lint all pass

eshulman2

there are some issues especially with docs

mandre · 2026-04-23T11:17:58Z

Adding the hold label because this is an experiment.

eshulman2 · 2026-05-31T11:18:28Z

@mandre I revised the PR myself removing any indication of internal references I think it is ready for review

dlaw4608

Hey @eshulman2 , I ran it locally following the drift-detection.md and it works well, also ran some of the E2E tests, I left a few comments, please let me know what you think?

winiciusallan

Hi @eshulman2, thanks for working on this. This is such a nice feature to have in ORC.

I've left a few comments along all the changes. Let me know what you think.

winiciusallan · 2026-06-18T21:21:26Z


+	// Schedule a resync requeue when the effective resync period is configured,
+	// there is no terminal error, and no other requeue is already pending.
+	// Positive-only jitter of [0%, +20%] is applied to spread load across


This could be another effort, but I would like to leave the idea here: Would it be good to allow the operator/user to configure the jitter interval?

I don't see any specific reason or use case to make this configurable, any specific use case in mind for adding this?

@mandre any strong opinions on that?

I don't see why one would want to configure this. Let's reconsider if someone has a use case.

github-actions · 2026-06-28T13:10:27Z

Failed to assess the semver bump. See logs for details.

mandre

Just a first review for now, I'll need another pass.

mandre · 2026-07-03T17:11:04Z

+      CUR3=$(kubectl get network.openstack.k-orc.cloud network-resync-jitter-3 \
+        -n ${NAMESPACE} -o jsonpath='{.status.lastSyncTime}')
+
+      # All three must have updated their lastSyncTime, demonstrating that


It would be interesting to verify that the interval between init and current is different for each resource.

I agree this would be useful to check, but I’m concerned that requiring all intervals to differ could be flaky because the jitter distribution is probabilistic and two resources may legitimately get the same value. I updated the test to verify that all three resources resync and that at least one of the three observed intervals differs. There is still a statistical chance this could flake if all three land on the same observed interval, but it is much lower, and the stricter jitter behavior is covered by the scheduler unit tests.

mandre · 2026-07-03T17:36:05Z

-		if osResource != nil {
-			log.V(logging.Verbose).Info("Got existing OpenStack resource", "ID", actuator.GetResourceID(osResource))
+		if osResource == nil {
+			return nil, progress.WrapError(fmt.Errorf("GetOSResourceByID returned nil resource with no error for ID %q", *resourceID))


Is this fixing an existing bug?

It’s not fixing a known bug. It is a defensive check for an invalid actuator response: GetOSResourceByID should not return nil with no error for a concrete status ID. I’m fine dropping it if we want to keep this PR scoped to external deletion/resync behavior.

mandre · 2026-07-06T09:20:46Z

+	// string, e.g. "10m", "1h". Set to "0s" to disable periodic resync for
+	// this resource.
+	// +optional
+	ResyncPeriod *metav1.Duration `json:"resyncPeriod,omitempty"` //nolint:kubeapilinter // metav1.Duration is appropriate for user-facing duration config


I was initially on the fence to add a ResyncPeriod for roleassignment since the roleassignment is a relationship resource, it doesn't have an ID, but on second thought it's useful to re-create the role assignment in case it's deleted out of band.

Now that raises a new question, should we do the same for router interface?

I agree we should probably add it for RouterInterface too. It has the same relationship-resource problem as RoleAssignment: if the OpenStack router interface is deleted out of band, resync is what lets ORC notice and recreate it. The main complication is that RouterInterface currently marks the whole spec immutable. To support per-object resync configuration properly, we would need to change that validation so the actual relationship fields remain immutable, while spec.resyncPeriod can be changed like on the generated resources. I don’t see a strong technical reason not to do that, but it is an API behavior change, but as this is a large code/behavioral change I think we can justify it as part of a new version if must.

I'm going to add it.

mandre

Good. Let's now squash the commits into logical chunks.

Introduce per-resource resync configuration and last sync status across the API surface. The new spec field lets users opt individual resources into periodic reconciliation while status.lastSyncTime records the last successful OpenStack sync. Regenerate CRDs, OpenAPI, apply configurations, and controller adapters so the new fields are available consistently to controllers, clients, and generated documentation.

Wire the manager-level default resync period into every generated controller and make it part of the controller contract. Controllers resolve the effective period from the per-resource value first, then the global default, with zero disabling periodic resync. Add the generic scheduler and status update logic that records successful syncs, preserves pending timers after spurious events, applies positive-only jitter to spread load, and warns when the global default is set very low.

Represent externally deleted OpenStack resources with a typed ReconcileStatus instead of overloading ordinary errors. Managed resources can then clear status.id and re-enter the normal creation path, while unmanaged or imported resources keep terminal error behavior. Update the generic get-or-create and condition handling paths so external deletion is explicit, status updates remain coherent, and invalid actuator responses are surfaced as errors.

Extend the resync behavior to custom relationship controllers that do not follow the generated generic controller path. RoleAssignment can now periodically verify and recreate missing assignments, and RouterInterface gains matching resync and lastSyncTime handling. Reuse the generic ShouldReconcile predicate to keep skip/resync decisions consistent across generic controllers, RoleAssignment, and RouterInterface while preserving relationship-specific reconciliation flow.

Add focused unit coverage for resync period resolution, jittered scheduling, lastSyncTime updates, external deletion state propagation, and generic reconcile skip behavior. Add API validation and KUTTL scenarios covering RouterInterface resync mutability, network resync scheduling, disabled resync behavior, terminal-error exclusion, and managed/imported external deletion outcomes.

Update the enhancement proposal and user guide to describe periodic drift detection, global and per-resource configuration, jitter, terminal-error behavior, and external deletion semantics. Refresh generated godoc and navigation so the new reconcile status and generic controller interfaces are reflected in the development documentation.

mandre · 2026-07-08T15:09:49Z

Thanks @eshulman2, this is an important feature and I'm glad it's finally merging 🎉

github-actions Bot added the semver:minor Backwards-compatible change label Apr 19, 2026

eshulman2 force-pushed the forge/aisos-376 branch from 5072ca0 to 3ebabf9 Compare April 19, 2026 14:43

eshulman2 changed the title ~~[AISOS-376] Implementation for AISOS-376~~ [AISOS-376] Add drift detection and periodic resync for OpenStack resources Apr 20, 2026

eshulman2 commented Apr 20, 2026

View reviewed changes

mandre added the hold Do not merge label Apr 23, 2026

eshulman2 changed the title ~~[AISOS-376] Add drift detection and periodic resync for OpenStack resources~~ Add drift detection and periodic resync for OpenStack resources May 25, 2026

eshulman2 force-pushed the forge/aisos-376 branch 4 times, most recently from 9138a1d to da9fb9b Compare May 31, 2026 08:28

dlaw4608 reviewed Jun 3, 2026

View reviewed changes

Comment thread .gitignore Outdated

eshulman2 force-pushed the forge/aisos-376 branch from 2f59e67 to 03223a3 Compare June 9, 2026 06:42

eshulman2 requested a review from dlaw4608 June 9, 2026 06:42

eshulman2 force-pushed the forge/aisos-376 branch from 03223a3 to e29ca26 Compare June 9, 2026 07:08

dlaw4608 reviewed Jun 9, 2026

View reviewed changes

eshulman2 requested a review from dlaw4608 June 10, 2026 13:24

dlaw4608 reviewed Jun 11, 2026

View reviewed changes

Comment thread website/docs/user-guide/drift-detection.md Outdated

eshulman2 requested a review from dlaw4608 June 11, 2026 12:10

winiciusallan reviewed Jun 19, 2026

View reviewed changes

eshulman2 requested a review from winiciusallan June 24, 2026 12:06

winiciusallan reviewed Jun 24, 2026

View reviewed changes

Comment thread internal/controllers/generic/reconciler/resource_actions.go Outdated

Comment thread internal/controllers/generic/reconciler/resource_actions_test.go Outdated

github-actions Bot removed the semver:minor Backwards-compatible change label Jun 28, 2026

eshulman2 force-pushed the forge/aisos-376 branch from bf78f06 to 00de750 Compare June 28, 2026 13:36

github-actions Bot added the semver:minor Backwards-compatible change label Jun 28, 2026

eshulman2 requested a review from winiciusallan June 30, 2026 09:43

mandre reviewed Jul 3, 2026

View reviewed changes

Comment thread enhancements/drift-detection.md Outdated

mandre reviewed Jul 3, 2026

View reviewed changes

Comment thread enhancements/drift-detection.md Outdated

mandre reviewed Jul 3, 2026

View reviewed changes

Comment thread enhancements/drift-detection.md Outdated

eshulman2 force-pushed the forge/aisos-376 branch from 2b53bfb to da6353f Compare July 5, 2026 08:22

eshulman2 requested a review from mandre July 6, 2026 08:40

mandre reviewed Jul 6, 2026

View reviewed changes

eshulman2 requested a review from mandre July 7, 2026 16:26

mandre reviewed Jul 8, 2026

View reviewed changes

eshulman2 requested a review from mandre July 8, 2026 12:36

mandre approved these changes Jul 8, 2026

View reviewed changes

eshulman2 added 6 commits July 8, 2026 17:18

eshulman2 force-pushed the forge/aisos-376 branch from d0fc696 to 3e115ae Compare July 8, 2026 14:34

winiciusallan approved these changes Jul 8, 2026

View reviewed changes

eshulman2 force-pushed the forge/aisos-376 branch from 3e115ae to a8901c5 Compare July 8, 2026 15:04

mandre approved these changes Jul 8, 2026

View reviewed changes

mandre removed the hold Do not merge label Jul 8, 2026

mandre enabled auto-merge July 8, 2026 15:09

mandre added this pull request to the merge queue Jul 8, 2026

Merged via the queue into k-orc:main with commit ef6119d Jul 8, 2026
10 of 11 checks passed

eshulman2 deleted the forge/aisos-376 branch July 9, 2026 06:17

Uh oh!

Conversation

eshulman2 commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

API Changes

CLI & Manager

Core Resync Logic

External Deletion Handling

Status Updates

Code Generation

Documentation

Commit structure

Design decisions

Testing

Uh oh!

eshulman2 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mandre commented Apr 23, 2026

Uh oh!

eshulman2 commented May 31, 2026

Uh oh!

Uh oh!

dlaw4608 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

winiciusallan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 28, 2026

Uh oh!

mandre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eshulman2 commented Apr 19, 2026 •

edited

Loading

eshulman2 left a comment •

edited

Loading

eshulman2 Jul 7, 2026 •

edited

Loading