fix(storage): fix purge infinite loop and garbled errors on object-locked buckets

[natalie-o-perret]

Problem

exo storage purge hangs forever on buckets with object lock enabled.

3 bugs in DeleteObjectVersions (pkg/storage/sos/object.go):

1. Infinite loop: when DeleteObjects returns HTTP 200 with per-object errors (objects are locked), the Deleted slice is empty. No progress check meant the loop kept re-listing and re-submitting the same objects forever.

2. Garbled errors: types.Error has *string fields. Formatting with %v printed raw pointer addresses instead of the actual message:

   Error happened: delete error: {<nil> 0xc332882c390 0xc332882c3b0 0xc332882c3a0}
   

3. Skipped pages: ListObjectVersions never forwarded KeyMarker/VersionIdMarker, so only the first page of versions was processed.

Fix

• Exit early when no objects were successfully deleted (all locked).
• Paginate ListObjectVersions with KeyMarker/VersionIdMarker.
• Format delete errors with aws.ToString() on each *string field.
• Fix a select-race in the purge command handler (cmd/storage_purge.go).

Before

=== RUN   TestDeleteObjectVersions_HappyPath
--- PASS: TestDeleteObjectVersions_HappyPath (0.00s)
=== RUN   TestDeleteObjectVersions_ComplianceLock
panic: test timed out after 15s
        running tests:
                TestDeleteObjectVersions_ComplianceLock (15s)

goroutine 10 [runnable]:
github.com/exoscale/cli/pkg/storage/sos_test.drainDeleteObjectVersions(...)
        pkg/storage/sos/object_test.go:620 +0x110
github.com/exoscale/cli/pkg/storage/sos_test.TestDeleteObjectVersions_ComplianceLock(...)
        pkg/storage/sos/object_test.go:707 +0x197

FAIL    github.com/exoscale/cli/pkg/storage/sos 15.104s

After

=== RUN   TestDeleteObjectVersions_HappyPath
--- PASS: TestDeleteObjectVersions_HappyPath (0.00s)
=== RUN   TestDeleteObjectVersions_ComplianceLock
--- PASS: TestDeleteObjectVersions_ComplianceLock (0.00s)
=== RUN   TestDeleteObjectVersions_Pagination
--- PASS: TestDeleteObjectVersions_Pagination (0.00s)
PASS    github.com/exoscale/cli/pkg/storage/sos 0.005s

Tests

• Unit tests for the happy path, compliance-lock exit, and pagination (object_test.go).
• Integration test against a live SOS bucket with GOVERNANCE object lock (tests/integ/with-api/storage_purge_object_lock_test.go).

End-to-end repro script (boto3, no aws CLI required)

#!/usr/bin/env python3
# Requirements: pip install boto3
# Env: EXOSCALE_API_KEY, EXOSCALE_API_SECRET, EXOSCALE_DEFAULT_ZONE
# Binary: make build  (or set EXO_BIN=path/to/exo)

import os
import random
import re
import subprocess
import time

import boto3
from botocore.config import Config

PURGE_TIMEOUT = 10

api_key    = os.environ["EXOSCALE_API_KEY"]
api_secret = os.environ["EXOSCALE_API_SECRET"]
zone       = os.environ["EXOSCALE_DEFAULT_ZONE"]
endpoint   = os.environ.get("EXOSCALE_SOS_ENDPOINT", f"https://sos-{zone}.exo.io")
exo_bin    = os.environ.get("EXO_BIN", "bin/exo")

s3 = boto3.client(
    "s3",
    endpoint_url=endpoint,
    aws_access_key_id=api_key,
    aws_secret_access_key=api_secret,
    region_name=zone,
    config=Config(signature_version="s3v4"),
)

bucket = f"repro-purge-lock-{random.randint(0, 999999):06d}"
print(f"bucket: {bucket}")

s3.create_bucket(Bucket=bucket, ObjectLockEnabledForBucket=True)

def cleanup():
    print("\ncleaning up...")
    try:
        resp = s3.list_object_versions(Bucket=bucket)
        for v in resp.get("Versions", []) + resp.get("DeleteMarkers", []):
            s3.delete_object(
                Bucket=bucket,
                Key=v["Key"],
                VersionId=v["VersionId"],
                BypassGovernanceRetention=True,
            )
    except Exception as e:
        print(f"  warning: {e}")
    try:
        s3.delete_bucket(Bucket=bucket)
        print(f"  bucket {bucket} deleted")
    except Exception as e:
        print(f"  warning: {e}")

s3.put_object_lock_configuration(
    Bucket=bucket,
    ObjectLockConfiguration={
        "ObjectLockEnabled": "Enabled",
        "Rule": {"DefaultRetention": {"Mode": "GOVERNANCE", "Days": 1}},
    },
)

for name, body in [("hello.txt", b"hello world\n"), ("world.txt", b"world\n")]:
    s3.put_object(Bucket=bucket, Key=f"test/{name}", Body=body)
    print(f"uploaded: test/{name}")

print(f"\nrunning: {exo_bin} storage purge sos://{bucket}/ (timeout={PURGE_TIMEOUT}s)")

start = time.monotonic()
try:
    result = subprocess.run(
        [exo_bin, "-z", zone, "storage", "purge", "--force", f"sos://{bucket}/"],
        capture_output=True,
        text=True,
        timeout=PURGE_TIMEOUT,
    )
    elapsed = time.monotonic() - start
    output = result.stdout + result.stderr
    print(f"exited after {elapsed:.1f}s (rc={result.returncode})")
    print(output or "  (empty)")
    if re.search(r"0x[0-9a-fA-F]{8,}", output):
        print("FAIL: raw pointer addresses in output (bug #2)")
    else:
        print("OK: no pointer addresses")
except subprocess.TimeoutExpired:
    print(f"FAIL: did not exit after {time.monotonic() - start:.0f}s -- infinite loop (bug #1)")
finally:
    cleanup()

Note

GitHub Copilot leveraged to:

• Sketch the Python script used for (integration) testing-purposes
• Draft Go (unit) tests
• Rephrase this PR description

[natalie-o-perret]

[SC-176386]