You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
pos3 console-script entry point with ls, download, upload subcommands
(#10).
pos3 ls <prefix> [-r] [--profile NAME] lists objects, one full s3:// URL
per line on stdout.
pos3 download <url> [--local PATH] [--delete] [--exclude PATTERN]... [--profile NAME]
prints only the resulting local path to stdout; progress and logs go to
stderr, so data_dir=$(pos3 download s3://bucket/dataset/) is safe.
pos3 upload <url> [--local PATH] [--delete] [--exclude PATTERN]... [--profile NAME]
is one-shot (no background loop or interval). Source defaults to the cache
path pos3 download would have produced; errors if the source doesn't
exist.
--delete defaults OFF for both download and upload (the Python API
defaults to True; the CLI is more conservative for interactive use).
--profile is supported alongside the URL form s3://<profile>@bucket/...;
the URL form wins on conflict, matching the Python precedence.
-n / --dry-run on download and upload prints the planned per-file
actions to stdout (in aws s3 sync --dryrun style) and performs no
transfers, no deletes, and no directory creation.
download and upload require an s3:// URL. The Python API's
local-path passthrough still works in code; the CLI rejects non-S3
inputs with a clear error so a typo can't silently succeed. ls is
unchanged and still accepts both forms.
pos3.TransferPlan dataclass plus module-level pos3.plan_download(remote, ...)
and pos3.plan_upload(remote, ...) wrappers: compute the set of (source, destination) copies and target deletes a real call would
perform, without performing any of them. Same calling pattern as pos3.download / pos3.upload — use them inside a with pos3.mirror():
block. The CLI's -n / --dry-run is implemented on top of these.
TransferError and TransferPlan are now in pos3.__all__.
Changed
Per-object transfer failures now raise pos3.TransferError instead of
being logged and swallowed. Previously, if any worker in a download or
upload batch failed, the error was sent to the logger and the call
returned normally — data_dir = pos3.download(...) could return a path
to a partial cache, and pos3 download exited 0 after a failed S3 GET.
Both now propagate. The new exception exposes .operation and .failures (list of the underlying per-worker exceptions). The CLI
catches it and exits 1 with the failure on stderr. Background
interval syncs (upload(..., interval=N)) are best-effort: a TransferError from one tick is logged and the daemon continues so
the next interval can retry. Only the final sync on context exit and
any one-shot call propagate. Cleanup on error — when the mirror() body is unwinding with an exception, a TransferError from
the sync_on_error=True cleanup sync is logged but swallowed, so the
original application exception remains the visible cause.
pos3.mirror() no longer creates the cache root directory eagerly on
context entry. The leaf directory is still created on demand when a
file is actually downloaded, so the visible behavior of download() is
unchanged. Dry-run and plan_* paths are now genuinely side-effect
free on the local filesystem.