v0.9.14

Added

• MemPath.relative_to(other, *, walk_up=False). It raised
  NotImplementedError; it now answers as PurePosixPath.relative_to does,
  on the path's own names (no .. applied): other is a MemPath of the same
  backend or a str, and the result is a relative MemPath on that backend.
  ValueError when the path is not below other, when only one of them is
  rooted, or when the backends differ. walk_up=True gives the ..-prefixed
  form on every Python version, not only 3.12 and later. Code that caught
  NotImplementedError around it can call it directly.
• copy(on_error=) and copy_into(on_error=) take rm()'s error
  predicate. on_error(error, path) -> bool (or a bool) is asked once for
  each child of a recursive copy that fails, with path the source entry:
  True skips it and the copy goes on, False raises the error. The same
  function used to mean opposite things to the two methods (lambda *a: False
  made rm() raise and copy() suppress every error), and a predicate
  written for one raised TypeError in the other. ignore_error= keeps its
  notify-only callable; use on_error= for a predicate. Passing both raises
  TypeError before anything is copied. sftp: (the asyncssh concurrent walk
  included), s3:, gs: and az: accept it, and PathContract gains a test
  that gives one predicate to rm() and copy().
• Path.info, and Python 3.14's pathlib.Path.copy() onto any Path.
  Every Path (MemPath, UriPath, SftpPath, ...) has info, on every
  Python version: an object with exists(), is_dir(), is_file() (each with
  follow_symlinks=True) and is_symlink(), the shape of 3.14's
  pathlib.types.PathInfo. It calls stat() at most once following links and
  once not, and remembers the answers on that object only, so a new path.info
  asks again. LocalPath keeps pathlib's own on 3.14. A new hook,
  Path._copy_from(source, follow_symlinks=True, preserve_metadata=False),
  lets pathlib.Path(f).copy(MemPath("/x")) work on 3.14, where it raised
  AttributeError: ... has no attribute '_copy_from': it overwrites a file,
  raises on an existing directory, and with preserve_metadata=True copies the
  permission bits and the modification time where the class can set them.
  Path.copy() of this library is unchanged and does not call it.
• FileStat.size_known, and listings that know a type but not a size.
  A listing can now say "this is a file" without inventing its size: a stat
  with size_known=False answers is_dir(), is_file(), exists(),
  walk(), glob() and rm(recursive=True), while stat() of the listed
  child asks the server for the real size and time, and PathSyncer, the
  progress total of copy() and quick_check never read the placeholder.
  gitlab: blobs, an Apache index with sizes like 1.2K and the stdlib
  bulleted-list index (http:) used to give no hint at all, so walking a tree
  cost a request per file; it now costs one per directory. An exact size from a
  listing that has one stays a full hint. A scheme of your own may yield
  FileStat(is_dir=False, size_known=False) from _scandir(); code that reads
  a listing's st_size or st_mtime should check size_known.
• PathSyncer(follow_mounts=False) and uripath sync --no-follow-mounts.
  A mount point found below the target root (a mounted volume, a bind mount)
  is still synced into by default, as rsync does. With follow_mounts=False
  it is an error for that entry instead: OSError(EXDEV) naming it, offered to
  ignore_error with event TypeMismatch, and nothing behind it is listed,
  written or removed, in a dry run too. A bind mount aliases another directory
  as a junction does, and rm(recursive=True) already stops at one; this lets
  a sync with remove_missing=True refuse it. A junction is replaced under
  either value, and the root target is used as given.
• touch() moves an existing file's time, and a sync leaves its copies with
  their source's time. The generic touch() of an existing file did nothing;
  it now sets the access and modification times to now through a new _utime()
  hook (_utime(*, atime=None, mtime=None), seconds since the epoch, None
  leaves that time alone), implemented by LocalPath, FileUri, MemPath,
  SftpPath on both client libraries (whole seconds) and FtpPath on a server
  whose FEAT lists MFMT (modification time only). A class that does not
  implement it raises NotImplementedError from _utime() and touch()
  leaves the file as it was, as before. PathContract gains
  supports_utime (default True; set it False on a test class whose
  backend cannot set a time) and a test behind it.
• PathSyncer(preserve_mtime=True). A file the sync copies is given the
  modification time its source reported, where the target's class implements
  _utime(); a source that reports no time, a target that cannot set one or
  refuses it, keep the time the copy gave the file, and the sync reports
  nothing. Directories and links are left alone. quick_check (default on for
  a pair with a non-local side) could never succeed after this library's own
  copy, so a remote tree was checksummed, and so downloaded, again by every
  later run; the second run of an unchanged tree now reads no file.
• The names the built-in schemes take from the core are documented, and the
  helpers among them are public. The shipped API header gains "For path
  implementers" and "For scheme implementers" sections, each hook with its
  signature and one line of contract (_init, _from_parsed_parts,
  _format_parsed_parts, _make_child_relpath, _rename_target,
  _pop_stat_hint, _same_location, the backend-state hooks, _anchor_segments
  and the rest). Twelve helpers that were underscore names now have public ones:
  pathlib_next.path.check_follow and contains,
  pathlib_next.utils.archive.detect_format,
  pathlib_next.utils._commit.CommitOnClose, pathlib_next.uri.same_authority
  and DerivedBackend, and in pathlib_next.uri.source compose_uri,
  compose_host, percent_encode, is_drive, SAFE_PATH and ERRORS. The
  underscore spellings (_check_follow, _contains, _encode, ...) still
  import and are the same objects, so nothing has to change; a scheme written
  outside this package can use the documented names instead. path.__all__
  now lists check_follow and contains.
• glob(recurse_symlinks=True) and rglob(recurse_symlinks=True) work.
  They raised NotImplementedError on every class. ** now descends through
  symlinks to directories, as pathlib 3.13 does, and the keyword is accepted
  on every Python version. A loop ends where pathlib 3.13 would follow it
  until the system refuses: a link to a directory the descent is already
  inside (st_dev/st_ino) is listed as an entry and not entered, and a
  backend whose stat has no identity follows at most 40 links in one descent.
  utils.glob.glob() and utils.glob.select() take the keyword too. The
  default stays False; a caller that caught the NotImplementedError to
  fall back to its own walk can drop the fallback.
• checksum() and supported_checksums() on s3:, gs: and az:. An
  object-store path now answers NativeChecksum from the digest the store
  keeps, with one request and no byte of the content. S3Path gives the plain
  MD5 that the ETag is for an object one request wrote and that KMS or a
  customer key does not encrypt (checksum("md5")), and the stored full-object
  crc32, crc32c, crc64nvme, sha1 or sha256; GsPath gives md5Hash
  and crc32c; AzPath gives the Content-MD5 property. A multipart ETag
  (...-N), a composite checksum, a GCS object with a content encoding or no
  md5Hash, a blob without Content-MD5 and any other algorithm raise
  NotImplementedError, never a guess; a missing path is FileNotFoundError
  and a prefix directory IsADirectoryError. PathSyncer uses them without
  being told: the second sync() of an unchanged tree of 100 s3: objects
  to another prefix sent 200 GetObject before and sends none. Azure's
  Content-MD5 is a property a client can set and the service does not verify
  against the content in every case.
• copy() of a file within one object store is one request of the store.
  s3:/gs: (inside a bucket, or between buckets that one client reaches: the
  same supplied backend, or both paths' default) and az: (inside a
  container) issue CopyObject, copy_blob or the blob copy instead of
  reading the file and writing it back; copy(recursive=True) and move() of
  a prefix do so for each file. A file copy on s3: took 6 requests (a
  GetObject and a PutObject among them) and takes 4, of which one is the
  copy. The checks copy() always made run first and in the same order (the
  same file, a target that is a directory or holds keys, an existing target
  without overwrite=True); an existing target is replaced by the copy and no
  longer deleted first, so it is intact when the copy fails; progress is
  called once with the total; a file named as the root of a
  recursive=True copy is still read and written; a store that answers "not
  implemented" gets the streamed copy. The copy keeps the object's content type
  and metadata, which the streamed copy did not.
• gs: and az: read and write an object in pieces. Reading asks for
  ranges of 4 MiB (gs:) or 512 KiB (az:, schemes.az.READ_CHUNK: the SDK
  holds about nine times a response, assign a larger number to trade memory
  for requests) as the stream is read, instead of downloading the whole
  object first; the stream stays seekable and seek() asks nothing until the
  next read. Writing spools in memory up to 4 MiB and then in a temp file, as
  s3: does, and uploads on close: a resumable upload in 2 MiB requests on
  gs:, blocks on az:. A read is one version of the object: the first
  request reports its generation (gs:) or ETag (az:) and every later
  request of that stream requires it, so an object replaced while it is
  being read raises OSError (errno.ESTALE, EIO where there is none:
  the object changed while it was being read) instead of returning the
  bytes of two objects, and one deleted meanwhile is FileNotFoundError;
  each open is its own snapshot, and seek(n, 2) measures that version.
  A 32 MiB object is read and written with a peak of under 9 MiB of Python
  memory against the SDK test servers. Nothing to change; a program that
  held a gs: or az: file object open for a long time now issues a
  request per chunk read, and catches OSError if the object may be
  replaced under it.
• ArchiveUri.batch() holds the changes to a zip and writes them in one
  replacement. with zip_path.batch(): makes the writes, mkdir(),
  unlink(), rmdir(), rename() and rm() inside the block change a
  pending member list that every path of the archive reads, and replaces
  the archive file once when the outermost block ends: the members, their
  order and their metadata are what the same changes made one by one
  leave. An exception in the block leaves the file byte for byte as it
  was, and so does a file another process replaced meanwhile
  (OSError(EBUSY)). A block inside another joins it, and another
  thread's use of the archive waits for the block. A tar, a non-local
  zip and a nested archive raise NotImplementedError, as their writes
  do. Each change patches the pending member list instead of deriving it
  again: a block of any number of renames, deletes or overwrites derives the
  list twice. A new hook, Path._write_batch() (documented for path implementers, a block
  that does nothing by default), is what copy(recursive=True) and
  PathSyncer.sync() enter around their writes to their target.
• Symbolic links in zip: and tar: archives are links. A tar
  symlink and a zip entry whose Unix mode says link are listed as links:
  is_symlink() is True, stat(follow_symlinks=False) is a link,
  readlink() (new on archives, a LinkTarget) is the text it stores,
  and stat(), exists(), is_dir(), open() and iterdir() follow it
  inside the archive. A zip link used to read as a file holding its own
  text, a tar link as the target's content (a loop was a
  RecursionError, a link to a directory IsADirectoryError), and
  ln/x below a link was not found. A link that is absolute, leaves the
  archive or points at nothing is now FileNotFoundError naming the link
  and a loop, or more than 40 links in one lookup, OSError(ELOOP).
  walk() and glob() list a link once and do not enter it by default;
  a listing that does follow links (walk(follow_symlinks=True),
  glob(recurse_symlinks=True), copy(recursive=True), PathSyncer)
  follows one directory link per path and never a link to the directory
  that holds it or to one above it, which answers OSError(ELOOP) to a
  following stat() and is an error for that entry in a copy or a sync
  (pass ignore_error= or follow_symlinks=False), so a few links cannot
  make a recursive operation visit more entries than the members and
  links times one more than the directory links. rm() removes the entry
  and never what it points at, and
  copy(follow_symlinks=False) and a PathSyncer that keeps links
  recreate it. A program that read a zip link's text with read_bytes()
  now gets its target's content: use readlink() for the text. Writing
  through a link, and creating one, raise NotImplementedError. A tar
  hard link is unchanged.

Changed

• A sync leaves each copy with its source's modification time
  (preserve_mtime=True, the default). A copy used to carry the time it was
  written at. Pass preserve_mtime=False to keep that. quick_check now
  compares two times at the resolution of the coarser side: in whole seconds
  when either has no fractional part (SFTP, FTP and the object stores report
  seconds, a local file system fractions), else to a microsecond; times one
  second apart still differ. A pair that is local on both sides still skips the
  quick check.
• move() asks a backend about each end once. A move that falls back to
  copy and delete stat-ed the source and the target several times each before
  it started to copy (the same-entry check, move()'s own checks, then
  copy()'s). It now takes one stat of each and hands the answer to the
  same-entry check and to the copy() it calls, for that call only; nothing is
  kept on a path. On an object store each stat is a request pair, so a
  directory move makes fewer requests. Every refusal move() made is made as
  before, in the same order. Nothing to change in a caller or a subclass; a
  class that overrides samefile() is still asked.
• S3Path and GsPath built without backend= share one default backend
  per process. Each used to build a backend of its own, and with it a client
  and a connection pool, as soon as it needed one; a loop that builds a path
  per key paid for one each. They now use one lazily built, locked backend, as
  AzPath does per account, so two separately built paths use one client and
  the credentials, region and endpoint that the environment gave the first
  request are theirs. A supplied backend is unaffected, and so are
  same_filesystem(), pickling and copy.copy. To keep a client of its own,
  pass backend=S3Backend(...)/GsBackend(...). A supplied AzBackend is
  used for whatever account the URI names; the header now says so.
• Copying a tree into a zip and syncing onto one rewrite the archive
  once. copy(recursive=True) and PathSyncer.sync() replaced the
  archive file for every member and directory they wrote (71 times for a
  60-file tree, 6 s for 300 small files); they now run inside
  batch() and replace it once. A copy or a sync that raises now leaves
  the archive as it was: the members written before the error used to
  stay, so a program that read them back after catching the error finds
  the archive unchanged. A dry run opens no batch. Nothing to change in a
  caller that does not depend on those partial results.
• SftpPath.replace() is one atomic request. It sends
  posix-rename@openssh.com where the server has it, so the file at the
  target is replaced without being removed first, and a refusal that names
  its cause (a permission, a missing source) leaves both entries as they
  were. move(overwrite=True) onto an existing file of the same server
  uses it: it was a removal followed by a rename, with a window in which
  the target did not exist, and a failed rename lost the target. A server
  without the extension, or one that refuses it with a bare failure, gets
  the removal and rename as before; a directory at the target is checked by
  Path.replace() first, as before. Nothing to change in a caller.
• stat() of a github: file asks with HEAD. It was a GET of the
  contents object, which carries the file base64-encoded, so every stat(),
  exists() and is_file() of a file downloaded it. The size now comes from
  the Content-Length of a HEAD asked for the raw media type, and a path
  that does not exist is answered by that request too. A directory other
  than the root of the repository now costs two requests (the HEAD, then
  the listing as before), which counts against the API's rate limit; a reply
  that is not the API's own, or a server that refuses HEAD, is followed by
  the GET as before. Nothing to change in a caller.
• The dates of an HTML index are documented as UTC. An index page
  prints them with no zone; http: has always read them as UTC, so the
  st_mtime of a listed entry can be off by the server's offset. The
  divergences page and the header now say so; quick_check=False or a
  comparison of sizes avoids it.

Fixed

• github: no longer reads a proxy's HTML page as the file. A raw
  read answered 200 with Content-Type: text/html and no
  X-GitHub-Media-Type header (a captive portal, a gateway's login page)
  returned the page as the file's content; it is now OSError(EIO) naming
  the path. A reply that carries the header, and so an HTML file of the
  repository, still reads. The rule has not been checked against the live
  service. Nothing to change in a caller.
• A predicate that raises ends rm() and PathSyncer with its own
  exception. Each directory above the failing entry offered that exception
  back to the predicate as one more failure, so a predicate that raised for
  one kind of error could swallow its own exception a level up. It now
  propagates as it is and is not offered again (also in the asyncssh
  concurrent rm()). Nothing to change for a caller.
• On Python 3.12 a literal pattern component typed in another case yields
  the stored name. On a case-insensitive filesystem glob("SUB") selected
  SUB where pathlib 3.12 selects the entry as the directory lists it (sub),
  because 3.12 reads every literal from the listing. native=True (the
  default) follows 3.12 there; other versions and native=False still join a
  literal as typed. Nothing to change for a caller.
• From Python 3.13 glob("name/") tests the literal as pathlib does, with
  os.path.lexists(name + os.sep), so the platform decides what a link is
  there: on Windows a dangling directory symlink is selected, on macOS a link
  to a file is. LocalPath selected nothing for either. Linux and earlier
  versions are unchanged.
• GsPath.rm(recursive=True) deletes in batches. It sent one DELETE per
  blob; it now queues them in batch requests of 100 (a prefix of 250 blobs is
  3 requests), offers each key the batch refused to ignore_error once with its
  own path, and falls back to one request per key for a chunk the client cannot
  batch. A blob that is already gone no longer raises, as on s3: and az:.
  Nothing to change for a caller.

Full Changelog: v0.9.13...v0.9.14