Skip to content

v0.0.8

Choose a tag to compare

@rsdoiel rsdoiel released this 16 Sep 18:18
· 87 commits to main since this release

Archive SQL Backup to S3 now copies every backup in the directory and verifies each copy, trimming only the ones old enough to delete. Previously one age-filtered set drove the upload, the verification and the deletion alike, so a dump too recent to trim was never copied at all -- a directory with nothing old enough reported 'No files match the age threshold. Nothing to do.' having archived nothing, which is exactly the case where the newest dump most needs to be in S3. A HeadObject pre-pass skips what is already archived, a blank retention answer means copy but delete nothing, and a per-file upload failure now warns and continues rather than aborting the run.

OpenSearch snapshots now include Invenio's audit logs and job logs. The allowlist previously carried only the data-stream form of the audit log, so on an instance restored from a snapshot -- which is every instance after a migration -- the historical audit log was backed up and every audit record written since go-live was not. Both index forms are now listed, along with job logs in both forms. Note the restore-side consequence: a restore now deletes and replaces a target's matching audit-log and job-log indices where it previously left them alone.

Restoring an OpenSearch snapshot no longer leaves the target unable to archive. The S3 sync runs as root on the host, so it wrote the entire snapshot repository root-owned, while OpenSearch runs as uid 1000 inside its container and could only write the top-level directory. Nothing restored the ownership, and because repository registration only ever verifies a write into that top-level directory, both the registration and the subsequent snapshot request reported success -- the failure surfaced only later, as every shard failing to write. The restore now sets ownership across the whole repository after syncing and before registering, and once a restore is verified it deletes the local snapshot through the OpenSearch API and deregisters the repository, leaving the instance ready to archive instead of accumulating stale snapshots. Nothing destructive runs before that verification succeeds. Relatedly, a snapshot that fails after being accepted is no longer reported as a bare 404 for a snapshot clasm created seconds earlier: it now says that creation failed after acceptance and points at the search container's own log, where the real cause is, and a missing snapshot is tolerated briefly first in case a busy cluster simply has not caught up.

Note on verification: the SQL archive change and the log-index change were both verified against real AWS. The OpenSearch repository-ownership and cleanup changes are unit-tested but NOT yet verified against real AWS -- the instance that exhibited the bug was repaired by hand and can no longer reproduce it, so live verification is deferred to the next real migration.

Full Changelog: v0.0.7...v0.0.8