Summary
Operator sometimes makes a one-shot HTTP call to a Tenant’s RustFS admin/S3 API during initial provisioning. If the Tenant pod/service is still starting, the call fails and provisioning is not retried. Result: canned policies, users and buckets are never created and the Tenant stays Blocked.
Repro (quick)
- Fresh cluster, deploy operator + examples/provisioning-tenant.yaml (or let Flux apply quickly).
- Operator reconciles before the Tenant S3 admin endpoint is ready → admin API call fails (conn refused / timeout) → provisioning items are left failed/pending and are not retried.
Root cause (brief)
The provisioning path calls RustFS admin endpoints once via RustfsAdminClient. Transient failures (network/timeouts/5xx) are treated as final and provisioning is marked attempted, preventing later retries when the Tenant becomes ready. This may sometimes be triggered by transient network interruptions during pod startup (e.g., Cilium network policy reprogramming), but I haven’t confirmed that yet.
Suggested fix (short)
Treat transient admin errors as retryable: requeue the reconcile with backoff (Action::requeue_after) or perform a bounded exponential-backoff retry before declaring failure. Only mark provisioning permanently failed for non-retryable errors (invalid policy, missing credsSecret, etc.). Add a small helper to classify transient vs permanent RustfsClientError.
Tests to add (short)
- Unit: mock admin client so first N calls fail then succeed; assert reconcile retries and provisioning becomes Ready.
- E2E: delay Tenant S3 readiness and confirm operator reprovisions after service becomes available.
Offer to help
I’d like to implement this and open a PR if maintainers agree (proposing an is_transient_client_error helper + requeue_after/backoff). Please let me know if that approach is acceptable or if you prefer a different retry policy.
Summary
Operator sometimes makes a one-shot HTTP call to a Tenant’s RustFS admin/S3 API during initial provisioning. If the Tenant pod/service is still starting, the call fails and provisioning is not retried. Result: canned policies, users and buckets are never created and the Tenant stays Blocked.
Repro (quick)
Root cause (brief)
The provisioning path calls RustFS admin endpoints once via RustfsAdminClient. Transient failures (network/timeouts/5xx) are treated as final and provisioning is marked attempted, preventing later retries when the Tenant becomes ready. This may sometimes be triggered by transient network interruptions during pod startup (e.g., Cilium network policy reprogramming), but I haven’t confirmed that yet.
Suggested fix (short)
Treat transient admin errors as retryable: requeue the reconcile with backoff (Action::requeue_after) or perform a bounded exponential-backoff retry before declaring failure. Only mark provisioning permanently failed for non-retryable errors (invalid policy, missing credsSecret, etc.). Add a small helper to classify transient vs permanent RustfsClientError.
Tests to add (short)
Offer to help
I’d like to implement this and open a PR if maintainers agree (proposing an is_transient_client_error helper + requeue_after/backoff). Please let me know if that approach is acceptable or if you prefer a different retry policy.