Skip to content

Fix multiline wait swallowing errors, wire delete exclude through#352

Merged
umputun merged 2 commits into
umputun:masterfrom
paskal:fix/wait-delete-exclude
Jul 4, 2026
Merged

Fix multiline wait swallowing errors, wire delete exclude through#352
umputun merged 2 commits into
umputun:masterfrom
paskal:fix/wait-delete-exclude

Conversation

@paskal

@paskal paskal commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Multiline wait swallowed its error

Previously, a multiline wait command that timed out was reported as successful whenever the remote temp-dir cleanup succeeded: the teardown defer assigned its result to the named return unconditionally, overwriting the timeout error with nil (single-line waits were unaffected as they have no teardown). The defer now only surfaces the teardown error when there is no primary error, mirroring Script. A new wait multiline failed test fails on the old code and passes now.

delete ignored documented exclude patterns

Previously, the runner dropped the exclude field of delete/mdelete commands and passed only recur to the executor, so files the playbook asked to keep were deleted. Wiring it through surfaced several latent executor issues, all fixed here:

  • isExcludedSubPath only protected the immediate parent directory of an excluded path, so exclusions nested deeper than one level were removed together with their ancestor directory; it now protects any ancestor of an excluded path.
  • Remote recursive delete failed when an exclusion pattern matched nothing, leaving skipped ancestor directories behind and then erroring on the non-empty root; it now removes the whole tree when no exclusion actually matched, same as the local executor.
  • Glob directory patterns like dir*/* matched the files inside but not the directory itself, so local delete removed the whole directory before visiting the excluded children; the directory rule is now glob-aware.
  • Exclusion matching now uses forward-slash semantics on all platforms, so documented /-separated patterns also work with Windows local paths.

The sudo + exclude combination is rejected with an explicit error (for mdelete before any location is deleted), since sudo deletion runs a plain rm that cannot apply exclusion patterns; README notes this limitation.

Known cosmetic limitation: when an exclusion pattern of a mixed set matches nothing (or a wildcard pattern like */keep.log matches only some directories), empty ancestor directories can be left behind. No data is lost or wrongly deleted in these cases.

Split out of #350.

@paskal paskal requested a review from umputun as a code owner July 3, 2026 20:16

@umputun umputun left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verified both fixes independently: the new wait test indeed fails on master, and the exclusion rewrite checks out except for the items below.

  1. the glob branch in isExcluded excludes plain files matching the dir-glob prefix: da*/* now excludes a file named data.txt (checked old vs new: false -> true). The helper is shared with copy/sync/upload, so their exclusion behavior silently changes too. Both walkers have IsDir at hand, pls thread it through so dir*/* only protects directories
  2. zero-match exclude list still deletes the whole tree silently (pkg/executor/remote.go:267-270, matches local). A typo'd keep-pattern deserves a [WARN] no exclusion pattern matched anything under ... before the tree goes away
  3. exclude on a non-recursive delete is silently ignored without sudo but hard-errors with sudo. Worth making it symmetric, reject exclude without recur the same way
  4. normalizeSlashes kills backslash escapes in patterns (foo\*bar no longer matches a literal foo*bar). Fine as a contract for a deployment tool, just add a README line that backslashes are treated as path separators and escapes are unsupported
  5. the sudo+exclude error doesn't name the offending location, worth including it for mdelete with several entries

nits: isExcluded deserves the same godoc its sibling got, exclude patterns get re-normalized on every segment iteration (normalize once before the loop), the inner path.Match shadow can be flattened with trimmed, found := strings.CutSuffix(ex, "/*"), and README gained "honour" where the rest uses American spelling.

btw the runtime placement of the sudo+exclude guard is right, task-level sudo propagates after config validate, so no change needed there.

@paskal paskal force-pushed the fix/wait-delete-exclude branch from 66a93bc to 4f78b2a Compare July 4, 2026 00:07
@paskal

paskal commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 4f78b2a — all five points plus the nits.

  1. dir-glob only protects directories. Threaded isDir through isExcluded and gated the dir/*-prefix rule on it, so da*/* no longer excludes a plain file like data.txt. Passed the real IsDir at every walker/entry call site (delete walkers, sync walks, remote props) and false at the file-only glob loops (upload/download, unmatched-files). Added a unit case and a local-delete case where a same-prefix file (dir9.txt) is deleted while dir1/keep.txt survives; verified it fails without the gate.
  2. No silent full-tree delete. Both local and remote recursive delete now log [WARN] no exclude pattern matched anything under ... before removing the tree when nothing matched.
  3. Symmetric exclude+recur. exclude now requires recur and is rejected without it for both sudo and non-sudo, via one shared validation.
  4. README. Added notes: exclude requires recur; backslashes are treated as path separators so glob escapes are unsupported; a dir/* pattern protects the directory too; a non-matching pattern removes the whole tree (with the WARN).
  5. Errors name the location. Both the requires-recur and sudo+exclude errors include the offending path, and mdelete validates every entry before deleting any.

Nits: isExcluded got the godoc its sibling has, patterns are normalized once before the segment loop, and honourhonor in the README.

The empty-ancestor-dirs case for a partially-matching mixed exclude set is documented in the PR body as a known cosmetic limitation (no data lost or wrongly deleted). CI green.

@umputun umputun left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confirmed all five round-1 items and the nits are addressed, verified the isDir fix empirically (da*/* no longer catches data.txt, dir exclusion intact), full suite green here including container tests. The gating introduced two new issues though:

  1. the remote zero-match [WARN] fires on every recursive remote delete, exclude or not (pkg/executor/remote.go:269-273). Local is protected by the len(excl) == 0 early return, remote has no such guard, and that path includes the temp-dir teardown after every multiline script/wait, so every deployment log gets the warning. Pls gate it on a non-empty exclude list, and a test asserting no warn for exclude-free deletes would pin it
  2. remote Upload (remote.go:80) and findMatchedFiles (remote.go:670) hardcode isDir=false, but glob matches can be directories. exclude: ["subdir/*"] used to skip dir subdir on master and still does in Local.Upload (it stats each match), while remote now feeds the dir into sftpUpload and fails. Pls stat the matches and pass the real value, os.Stat for upload (globs are local paths) and sftp Stat for download

also README changed but site/docs-src/index.md wasn't regenerated, needs make prep-site (it still carries the round-1 wording incl. "honour").

minor, non-blocking: Local.Upload now stats before the exclusion check, so an excluded broken symlink fails the whole upload instead of being skipped. On stat error worth trying the match with isDir=false and skipping if excluded.

paskal added 2 commits July 4, 2026 01:47
Previously, a multiline wait command that timed out was reported as
successful when the remote temp-dir cleanup succeeded, because the
teardown defer overwrote the returned error with nil.

Previously, the delete command ignored its documented exclude patterns
and removed files the playbook asked to keep. Wiring it through:

- isExcludedSubPath now protects any ancestor of an excluded path, not
  just the immediate parent, so exclusions nested deeper than one level
  survive.
- Remote recursive delete removes the whole tree when no exclude pattern
  matched, matching the local executor, and logs a WARN in that case so
  a mistyped pattern is not silently destructive.
- The dir-glob rule (a 'dir/*' pattern also protecting the directory it
  names) is now gated on the entry being a directory, threaded through
  all isExcluded call sites, so a pattern like 'dir*/*' no longer
  excludes a plain file such as 'data.txt'.
- Exclusion matching uses forward-slash semantics on all platforms, so
  documented patterns work with windows local paths too.
- exclude requires recur and is rejected without it, symmetric between
  sudo and non-sudo; sudo+exclude is still rejected. Both errors name the
  offending location, and mdelete validates all entries before deleting
  any of them.

The wait teardown fix mirrors Script: the teardown error is only
surfaced when there is no primary error.
- Gate the remote no-match delete WARN on a non-empty exclude list, so an
  exclude-free recursive delete (e.g. temp-dir cleanup after every
  multiline script or wait) no longer logs a spurious warning; local was
  already guarded by its len(excl)==0 early return.
- Remote Upload and findMatchedFiles now stat each glob match and pass the
  real isDir to isExcluded, so a directory exclude like 'subdir/*' skips a
  matched directory instead of feeding it into sftpUpload/Download and
  failing. Local.Upload already stats; it now also checks exclusion before
  reporting a stat error so an excluded broken symlink is skipped rather
  than failing the whole upload.
- Regenerate site/docs-src/index.md from README (make prep-site).

Tests: no-warn for exclude-free recursive delete, warn plus full removal
for a non-matching exclude, and a glob upload that excludes a matched
directory (fails without the stat).
@paskal paskal force-pushed the fix/wait-delete-exclude branch from 4f78b2a to 1e9cc15 Compare July 4, 2026 00:58
@paskal

paskal commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Round-2 addressed in 1e9cc15 (rebased on current master).

  1. Remote WARN no longer fires on exclude-free deletes. Gated the "no exclude pattern matched" warning on len(exclude) > 0, so recursive deletes without excludes — including the temp-dir cleanup after every multiline script/wait — stay quiet. Local was already covered by its len(excl) == 0 early return. Added a test asserting no warn for an exclude-free recursive delete, plus one asserting the warn (and full removal) when a pattern matches nothing.
  2. Glob matches that are directories are handled. remote.Upload now os.Stats each match (local paths) and findMatchedFiles uses sftpClient.Stat (remote paths), passing the real isDir, so exclude: ["subdir/*"] skips a matched directory instead of feeding it into sftpUpload/Download and failing. Added a glob-upload test with an excluded directory that fails without the stat.
  3. Minor: Local.Upload now checks exclusion before surfacing a stat error, so an excluded broken symlink is skipped rather than failing the whole upload (same ordering applied to the remote paths).
  4. Regenerated site/docs-src/index.md via make prep-site — it now carries the current delete/exclude wording (and "honor").

Full executor + runner suites green here including container tests; the dir-exclude upload fix negative-checked (fails without the stat). CI green.

@umputun umputun left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both blocking items are sorted, verified: the remote warn is gated on len(exclude) > 0 with the whole-tree fallback still firing for exclude-free deletes, and Upload/Download now stat each match (os.Stat local, sftp Stat remote) so dir-glob excludes skip a matched directory. The new remote_test.go cases pin all of it. LGTM.

two minor leftovers, non-blocking, up to you:

  • the filepath.Rel error paths wrap err (nil there) instead of the local e, so a rel failure would print %!w(<nil>) and lose the cause. Three spots: local.go:99, remote.go:78, remote.go:677. Pre-existing, adjacent to what you touched
  • the download-side dir-glob fix (findMatchedFiles) has no test of its own. The Upload one covers the symmetric path, but a mirror test excluding a matched directory on download would pin it directly

happy to see these in a follow-up or here, whatever you prefer.

@umputun umputun merged commit bcba7ca into umputun:master Jul 4, 2026
2 checks passed
@paskal paskal deleted the fix/wait-delete-exclude branch July 4, 2026 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants