chore(deps): Bump the orch-python group in /microceph-orch with 2 updates

.github/dependabot.yml

@@ -0,0 +1,50 @@
+# Dependabot version updates.
+#
+# Dependabot updates dependencies that already declare a version; it never
+# *introduces* a pin. docs/requirements.txt is therefore pinned (==) so that it
+# falls under management; tests/robot/requirements.txt keeps its existing >=
+# floors, which Dependabot raises via versioning-strategy: increase below.
+version: 2
+updates:
+  # Go modules: microceph/go.mod (+ go.sum).
+  - package-ecosystem: gomod
+    directory: /microceph/
+    schedule:
+      interval: daily
+    open-pull-requests-limit: 10
+    groups:
+      go-dependencies: # bundle all Go bumps into a single PR
+        patterns: ["*"]
+
+  # Python requirements.txt files (docs build + robot tests).
+  - package-ecosystem: pip
+    directories: # plural form takes a list (and globs); `directory` does not
+      - /docs # docs/requirements.txt
+      - /tests/robot # tests/robot/requirements.txt
+    schedule:
+      interval: weekly
+    open-pull-requests-limit: 10
+    versioning-strategy: increase
+    groups:
+      python-requirements:
+        patterns: ["*"]
+
+  # microceph-orch: PEP 621 pyproject.toml + uv.lock (native uv ecosystem).
+  - package-ecosystem: uv
+    directory: /microceph-orch
+    schedule:
+      interval: weekly
+    open-pull-requests-limit: 10
+    groups:
+      orch-python:
+        patterns: ["*"]
+
+  # GitHub Actions referenced in .github/workflows/*.yml.
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: weekly
+    open-pull-requests-limit: 10
+    groups:
+      github-actions: # one PR for all action bumps
+        patterns: ["*"]

ISSUE_adopt-vm-readiness.md

@@ -0,0 +1,28 @@
+# Flaky: Adopt test with cephadm fails when the LXD VM agent never comes up
+
+Part of #773 (Reduce Flaky Tests). Data: CI Health Report #717.
+Affected job: **Tests / Adopt test with cephadm** — 10.8% over the last 60 days on `main` (largest non-doc failer).
+
+Related flaky issue (same root cause, different harness): the DSL VM-readiness flake — #<insert issue number here>.
+
+## What happens (observed behaviour)
+
+`create_cephadm_vm()` launches the VM once and waits ~200 s; when the agent never responds it `exit 1`s with no relaunch:
+
+```
+Error: LXD VM agent isn't currently running
+...
+Timeout waiting for machine
+```
+
+## Steps to reproduce
+
+Intermittent on `main`. Observed instances:
+
+- https://github.com/canonical/microceph/actions/runs/25006847909/job/73232057308
+- https://github.com/canonical/microceph/actions/runs/25122463102/job/73626737457
+- https://github.com/canonical/microceph/actions/runs/25343841361/job/74307903663
+
+## What were you expecting to happen?
+
+A VM whose agent is briefly slow should be relaunched/retried, not fail adoption setup outright.

ISSUE_disk-list-getstorage.md

@@ -0,0 +1,31 @@
+# Flaky: `microceph disk list` fails on a udevd /dev/disk/by-id TOCTOU race
+
+Part of #773 (Reduce Flaky Tests). Data: CI Health Report #717.
+Affected job: **Tests / DSL WAL-DB cleanup tests** — 9.1% over the last 60 days on `main`.
+
+Same issue as #738
+
+## What happens (observed behaviour)
+
+After an OSD remove, the suite polls `microceph disk list`, which intermittently errors:
+
+```
+internal error: unable to fetch available disks: failed listing storage devices:
+Failed to find "/dev/disk/by-id/.#scsi-0QEMU_QEMU_HARDDISK_...": lstat ...: no such file or directory
+```
+
+LXD's `resources.GetStorage()` enumerates `/dev/disk/by-id` and races udevd, which atomically replaces symlinks via a `.#`-prefixed temp entry — visible in `readdir` then gone on `lstat`. The daemon `/1.0/resources` endpoint called `GetStorage()` with no retry (PR #739 only added a retry on the OSD-add path).
+
+## Steps to reproduce
+
+Intermittent on `main`. Observed instance:
+
+- https://github.com/canonical/microceph/actions/runs/27990395702/job/82841425600
+
+## What were you expecting to happen?
+
+`disk list` succeeds; a transient, self-healing udev symlink rename should not surface as a user-facing error.
+
+## Fix
+
+Routes the `/resources` endpoint through a shared `ceph.GetStorageWithRetry` helper 

ISSUE_doc-linkcheck.md

@@ -0,0 +1,28 @@
+# Flaky: documentation Link check fails on transient external-site errors
+
+Part of #773 (Reduce Flaky Tests). Data: CI Health Report #717.
+Affected job: **Main Documentation Checks / Link check / Check links in the documentation** — ~70% over the last 60 days on `main`.
+
+## What happens (observed behaviour)
+
+The Sphinx linkchecker marks valid external links as broken when `canonical.com` / `ubuntu.com` return transient **429** (rate limiting), **502/503**, or read-timeouts, failing the docs build. These are infra hiccups, not broken links.
+
+## Steps to reproduce
+
+Intermittent on `main`. Observed instances (note 28164041735 failed on its retry too):
+
+- https://github.com/canonical/microceph/actions/runs/28164041735/job/83411400793 (attempt 1)
+- https://github.com/canonical/microceph/actions/runs/28164041735/job/83498826379 (attempt 2)
+- https://github.com/canonical/microceph/actions/runs/27600225197/job/81599195573
+- https://github.com/canonical/microceph/actions/runs/27559551401/job/81467561175
+- https://github.com/canonical/microceph/actions/runs/26900691284/job/79351940258
+- https://github.com/canonical/microceph/actions/runs/26865736450/job/79228885623
+- https://github.com/canonical/microceph/actions/runs/26801388372/job/79008627366
+
+## What were you expecting to happen?
+
+The link check passes when the target URLs are valid; only genuinely dead links should fail the build.
+
+## Fix
+
+Addressed by PR #<insert PR number here> — raises `linkcheck_timeout`/`linkcheck_retries`/`linkcheck_rate_limit_timeout` so transient errors clear (no `linkcheck_ignore` additions, so real broken links are still caught).

ISSUE_dsl-vm-readiness.md

@@ -0,0 +1,34 @@
+# Flaky: DSL functional tests fail when the LXD VM agent never comes up
+
+Part of #773 (Reduce Flaky Tests). Data: CI Health Report #717.
+Affected job: **Tests / DSL functional tests** — 11.5% over the last 60 days on `main`.
+
+Related flaky issue (same root cause, different harness): the Adopt VM-readiness flake.
+
+## What happens (observed behaviour)
+
+An LXD VM launches (`lxc launch` returns 0) but its agent never responds within the readiness window, so the harness times out:
+
+```
+[dsl-functest] FAIL: Timed out waiting for VM '...' to be ready
+Error: LXD VM agent isn't currently running
+```
+
+The old `setup_dsl_test()` retried only `lxc launch`, so the post-launch readiness wait called `fail()` and aborted the whole case.
+
+## Steps to reproduce
+
+Intermittent on `main`. Observed instances:
+
+- https://github.com/canonical/microceph/actions/runs/27663269867/job/81811941445
+- https://github.com/canonical/microceph/actions/runs/27734216347/job/82047574030
+
+## What were you expecting to happen?
+
+A VM whose agent is briefly slow should be relaunched/retried, not fail the test case outright.
+
+## Fix
+
+Addressed by PR #<insert PR number here> — wraps launch + attach + readiness wait in a 3-attempt relaunch loop (300 s per attempt; a healthy VM is ready in ~30 s, so waiting longer would not recover a dead-agent boot).
+
+Follow-up: the Robot `launch_outer_test_vm` outer-VM setup has the same agent-never-registers gap (it retries only the `lxc launch` command, then waits on the agent and fails) and could get the same relaunch treatment.

ISSUE_lxc-file-push.md

@@ -0,0 +1,27 @@
+# Flaky: Test maintenance mode fails on a transient lxc file push socket reset
+
+Part of #773 (Reduce Flaky Tests). Data: CI Health Report #717.
+Affected job: **Tests / Test maintenance mode** — 2.7% over the last 60 days on `main`.
+
+Related flaky issue (companion harness-retry fix): the snap-store outage flake — #<insert issue number here>.
+
+## What happens (observed behaviour)
+
+A single `lxc file push` during suite setup fails with a forkfile socket reset and cascades the whole suite to FAIL (6/7 tests had passed):
+
+```
+Failed to push script to node-wrk1: Error: error receiving version packet from server:
+read unix @->/proc/self/fd/22/forkfile.sock: read: connection reset by peer
+```
+
+## Steps to reproduce
+
+Intermittent on `main`. Observed instance:
+
+- https://github.com/canonical/microceph/actions/runs/27827205271/job/82354673524
+
+## What were you expecting to happen?
+
+A momentary VM-agent socket reset on `lxc file push` should be retried, not abort suite setup.
+
+

ISSUE_snap-store.md

@@ -0,0 +1,25 @@
+# Flaky: transient snap-store 408/5xx
+
+Part of #773 (Reduce Flaky Tests). Data: CI Health Report #717.
+Affected job: **Tests / Regression test for sequential join mon host refresh** — 2.7% over the last 60 days on `main`.
+
+Related flaky issue (companion harness-retry fix): the lxc file push socket-reset flake — #<insert issue number here>.
+
+## What happens (observed behaviour)
+
+A snap operation in suite setup fails when the Snap Store returns 408/5xx, so every test in the suite fails with "Parent suite setup failed":
+
+```
+error: cannot perform the following tasks:
+- Fetch and check assertions for snap "snapd" (...) (cannot get nonce from store: store server returned status 408)
+```
+
+## Steps to reproduce
+
+Intermittent on `main`. Observed instance:
+
+- https://github.com/canonical/microceph/actions/runs/27827205271/job/82354673528
+
+## What were you expecting to happen?
+
+A transient Snap Store outage during `snap install` / `snap refresh` should be retried, not abort suite setup. A genuine error (e.g. bad channel) should still fail immediately.

PR_BODY_adopt-vm-readiness.md

@@ -0,0 +1,31 @@
+# Description
+
+The **Adopt test with cephadm** job is the largest non-doc CI failer on `main` (10.8% over 60 days, per #717), failing with the same root cause as the DSL tests: an LXD VM launches but its agent never responds (`Error: LXD VM agent isn't currently running`). `create_cephadm_vm()` launched once, waited ~200 s, then `exit 1` with no relaunch.
+
+This wraps launch + volume create/attach + readiness wait in a 3-attempt retry loop (delete the VM and its `$name-{1,2,3}` volumes between attempts, exit 1 only after the last). Because three attempts can take ~600 s of waiting, the Robot caller's `create_cephadm_vm` step timeout is raised 600 s → 1200 s so the loop isn't SIGKILLed before the final attempt.
+
+This is the companion to the **DSL VM-readiness retry** (PR #<insert PR number here>) — same failure class, different harness. CI evidence: [25006847909](https://github.com/canonical/microceph/actions/runs/25006847909/job/73232057308), [25122463102](https://github.com/canonical/microceph/actions/runs/25122463102/job/73626737457), [25343841361](https://github.com/canonical/microceph/actions/runs/25343841361/job/74307903663).
+
+Fixes #773
+Fixes #<insert issue number here>
+Relates to #717
+
+## Type of change
+
+- [x] Clean code (code refactor, test updates; does not introduce functional changes)
+
+## How has this been tested?
+
+`bash -n` and `shellcheck -S error` are clean on `adoptutils.sh`; `cephadm_adopt_tests.robot` parses. The 3 × 200 s readiness budget plus relaunch overhead fits under the raised 1200 s step timeout. Full integration requires a built snap + LXD KVM and runs in CI.
+
+## Contributor checklist
+
+Please check that you have:
+
+- [x] self-reviewed the code in this PR
+- [x] added code comments, particularly in less straightforward areas
+- [x] checked and added or updated relevant documentation
+- [ ] added or updated HTML meta descriptions for any new or modified documentation pages (see [#643](https://github.com/canonical/microceph/pull/643))
+- [ ] verified that page title and headings accurately represent page content for new or modified documentation pages
+- [ ] checked and added or updated relevant release notes
+- [ ] added tests to verify effectiveness of this change

PR_BODY_disk-list-getstorage.md

@@ -0,0 +1,35 @@
+# Description
+
+The daemon `/1.0/resources` endpoint called LXD's `resources.GetStorage()` once with no retry. `GetStorage()` enumerates `/dev/disk/by-id` and hits a udevd TOCTOU race (a `.#`-prefixed temp symlink is seen in `readdir` then vanishes on `lstat`), so `microceph disk list` fails intermittently. This surfaced in the **DSL WAL-DB cleanup** suite (9.1% over 60 days, per #717):
+
+```
+failed listing storage devices: Failed to find "/dev/disk/by-id/.#scsi-...": lstat ...: no such file or directory
+```
+
+PR #739 added this retry to the OSD-add path but not to the daemon `/resources` endpoint that `disk list` hits directly. This extracts a shared `ceph.GetStorageWithRetry` helper (3 attempts, linear backoff), routes both `cmdResourcesGet` and `OSDManager.getStorageWithRetry` through it, and adds unit tests.
+
+CI evidence: [run 27990395702](https://github.com/canonical/microceph/actions/runs/27990395702/job/82841425600).
+
+Fixes #773
+Fixes #779
+Relates to #717
+
+## Type of change
+
+- [x] Bug fix (non-breaking change which fixes an issue)
+
+## How has this been tested?
+
+New `ceph/storage_test.go` tests inject the exact CI error string and assert the retry absorbs it (and that it exhausts/returns correctly); the existing OSD-path `TestGetStorageWithRetry` still passes after the delegation refactor.
+
+## Contributor checklist
+
+Please check that you have:
+
+- [x] self-reviewed the code in this PR
+- [x] added code comments, particularly in less straightforward areas
+- [x] checked and added or updated relevant documentation
+- [ ] added or updated HTML meta descriptions for any new or modified documentation pages (see [#643](https://github.com/canonical/microceph/pull/643))
+- [ ] verified that page title and headings accurately represent page content for new or modified documentation pages
+- [ ] checked and added or updated relevant release notes
+- [x] added tests to verify effectiveness of this change

PR_BODY_doc-linkcheck.md

@@ -0,0 +1,36 @@
+# Description
+
+The documentation **Link check** job is the most frequent CI failure on `main` (~70% over 60 days, per the CI Health Report #717): external sites (`canonical.com`, `ubuntu.com`) intermittently return 429 / 502 / read-timeouts that Sphinx marks as broken links, even though the links are valid.
+
+This hardens the linkchecker against those transient infra errors, without hiding real broken links (no `linkcheck_ignore` additions):
+
+- `linkcheck_timeout` 30 → 60
+- `linkcheck_retries` 3 → 5
+- `linkcheck_rate_limit_timeout` (default 300) → 600
+
+A genuinely dead URL still returns BROKEN on every retry and still fails the build. CI evidence (incl. a retried-then-still-failed attempt): [run 28164041735 att1](https://github.com/canonical/microceph/actions/runs/28164041735/job/83411400793), [att2](https://github.com/canonical/microceph/actions/runs/28164041735/job/83498826379). Full list in the tracking issue.
+
+Fixes #773
+Fixes #<insert issue number here>
+Relates to #717
+
+## Type of change
+
+- [x] Clean code (code refactor, test updates; does not introduce functional changes)
+- [x] Documentation update (change to documentation only)
+
+## How has this been tested?
+
+`docs/conf.py` parses and the three options resolve to `60` / `5` / `600` with `linkcheck_ignore` unchanged. All three are valid Sphinx linkcheck builder options.
+
+## Contributor checklist
+
+Please check that you have:
+
+- [x] self-reviewed the code in this PR
+- [x] added code comments, particularly in less straightforward areas
+- [x] checked and added or updated relevant documentation
+- [ ] added or updated HTML meta descriptions for any new or modified documentation pages (see [#643](https://github.com/canonical/microceph/pull/643))
+- [ ] verified that page title and headings accurately represent page content for new or modified documentation pages
+- [ ] checked and added or updated relevant release notes
+- [ ] added tests to verify effectiveness of this change

PR_BODY_dsl-vm-readiness.md

@@ -0,0 +1,31 @@
+# Description
+
+The **DSL functional tests** fail intermittently (11.5% over 60 days, per #717) when an LXD VM launches but its agent never responds within the readiness window (`Error: LXD VM agent isn't currently running`). The old `setup_dsl_test()` retried only `lxc launch`, so `wait_for_dsl_vm()` then called `fail()` and aborted the case.
+
+A healthy VM reaches "ready" in ~30 s, so an agent still absent after minutes is a failed boot that only a fresh boot recovers — waiting longer does not help (the wait already ran for the full window before this change). This wraps launch + attach + readiness-wait in a 3-attempt relaunch loop and adds a non-fatal `wait_for_vm_command_nonfatal()`. Each attempt waits up to **300 s** (~10× the healthy ready time), so a dead boot is recycled in ~5 min instead of 10. `create_dsl_volumes` stays outside the loop (volumes persist), and the delete-and-relaunch only runs *between* attempts, so a final exhausted failure leaves the VM intact for the `EXIT` trap's diagnostics.
+
+This extends the launch-command retry that `launch_dsl_vm` already does, and shares the `LXD VM agent isn't currently running` root cause with the **Adopt VM-readiness retry** (PR #<insert PR number here>), which applies the same pattern to a different harness. Note: the Robot `launch_outer_test_vm` outer-VM setup has the same gap (it retries only the launch command, then waits on the agent and fails) and could get the same relaunch treatment as a follow-up. CI evidence: [run 27663269867](https://github.com/canonical/microceph/actions/runs/27663269867/job/81811941445), [run 27734216347](https://github.com/canonical/microceph/actions/runs/27734216347/job/82047574030).
+
+Fixes #773
+Fixes #781
+Relates to #717
+
+## Type of change
+
+- [x] Clean code (code refactor, test updates; does not introduce functional changes)
+
+## How has this been tested?
+
+`bash -n` and `shellcheck -S error` are clean. Re-entrancy verified (volumes persist; only the VM instance is deleted/relaunched). The ~30 s healthy ready time was measured from a passing run, confirming the 300 s per-attempt budget keeps ~10× headroom; 3 × 300 s fits well under the suite's 14400 s wrapper timeout. Full integration requires a built snap + LXD KVM and runs in CI.
+
+## Contributor checklist
+
+Please check that you have:
+
+- [x] self-reviewed the code in this PR
+- [x] added code comments, particularly in less straightforward areas
+- [x] checked and added or updated relevant documentation
+- [ ] added or updated HTML meta descriptions for any new or modified documentation pages (see [#643](https://github.com/canonical/microceph/pull/643))
+- [ ] verified that page title and headings accurately represent page content for new or modified documentation pages
+- [ ] checked and added or updated relevant release notes
+- [ ] added tests to verify effectiveness of this change