Repository navigation
Releases: jose-pr/pathapi
Release list
v0.1.2
Added
max_workersfor recursivecopy(),rm()andPathSyncer.copy()(and
copy_into()),rm()andPathSyncer(...)takemax_workers=None.None, the
default, is one thread, exactly as before. An int of 2 or more hands the files
and links of each directory to a pool of at most that many threads, which
helps a backend that pays a round trip per entry. It applies to a class that sets
the new class attribute_threadsafe = True(LocalPathandFTPPathdo); any other
class, including every class of another distribution until it sets the attribute,
runs serially whatever was asked. Callbacks (progress,on_error,ignore_error,
the synchook, follow policies) run one at a time but on pool threads, in no
fixed order, and the error that ends the operation is the first to happen. A value
below 1 isValueError, one that is not an int isTypeError.PathContract
has asupports_concurrent_useswitch for the matching test.AsyncPathandAsyncFile: a path for use from an event loop.AsyncPath(path)
wraps anyPath; the pure members (name,parent,/,with_*(),match(), ...)
are plain and giveAsyncPath, every method that does I/O is a coroutine of the same
name and arguments,iterdir(),glob(),rglob()andwalk()return async iterators,
andopen()returns an object that is awaited or used withasync withand gives an
AsyncFile(read,readline,write,seek, ...,aclose()). They are exported
frompathapiandpathapi.path. Each call runs the wrapped path's sync method on a
worker thread (asyncio.to_thread()), so every backend can be used from a loop and the
loop is not blocked; cancelling anawaitabandons the wait and the worker finishes the
call. Recursivecopy(),rm()andmove()are one such call, somax_workers=is the
sync fan-out.supports()andcapabilities()of anAsyncPathanswer for the wrapped
path. A subclass gets its own class for every path derived from it, built by the
classmethodAsyncPath._adopt(path)(cls(path)), which a subclass with another
constructor overrides.AsyncHTTPClient: the HTTP client for an event loop.pathapi.http.AsyncHTTPClient
is the sibling ofHTTPClient: the same constructor arguments, redirect and
Authorizationrules,timeout,deadline, reply limit and errors, withrequest()
andredirect()as coroutines,aclose()andasync with. It returns an
AsyncHTTPResponse(await read(),iter_bytes()as an async iterator,await aclose()). It sends through an async transport, an object withasync send()and
async aclose(). The default one,ThreadedTransport, runs the synchronous transport
PATHAPI_HTTP_TRANSPORTnames on worker threads, so every synchronous transport works
from a loop; a native one is declared in the entry-point group
pathapi.http_async_transportsand chosen byPATHAPI_HTTP_ASYNC_TRANSPORT(same
rules and messages as the synchronous registry;async_transport_names(),
get_async_transport()anddefault_async_transport()).- The async half of the transport suite.
python -m pathapi.http.testing --http-async-transports NAMEholds an async transport to the promises of the
synchronous suite (test_async_transport.py,test_async_client.py). Without the
option the async tests run overbuiltinwhen--http-transportsis left out too,
and are not collected when it names a transport: a distribution that runs
--http-transports NAMEgets the tests it got before. pathapi.testing.AsyncPathContract. The pytest contract ofAsyncPathover a
backend: it takes therootfixture ofPathContractand checks each coroutine
against the sync path's result, withasyncio.run()and no extra test dependency. Its
class attributeasync_path_class(defaultAsyncPath) names the class under test, so a
subclass ofAsyncPathis tested with the paths it derives.
Changed
- An unknown HTTP transport name is told where to look, not which package to
install.get_transport()andPATHAPI_HTTP_TRANSPORTraiseValueErrorfor a
name no installed distribution declares in the grouppathapi.http_transports,
naming the group and suggestingpathapi-NAMEas a name to look for. The error is
aValueErrorthat is also anImportError, so code that caught either still
does.requestsis no longer special:transport_names()lists it only when an
installed distribution declares it. The refusal of a proxy that is nothttp://
points at a transport of that group that supports it. Install the distribution
that provides the transport.
Fixed
- A
nanor infinite HTTP timeout is refused withValueError.timeout=float("nan")
passed the check, and the built-in transport then failed insocket.settimeout()after
it had opened the connection, which leaked the socket. Pass a positive finite number of
seconds, orNone. move(overwrite=True)no longer destroys the old target when the move fails.
An existing target was removed before the source was renamed or copied over it, so
a failing rename or copy lost the target. The target now stays until the source
has taken its place: a local file is replaced atomically,rename()is tried
over a file elsewhere, a backend that refuses has the target renamed aside and put
back on failure, and a copy and delete finishes the new file or tree beside the
target first. A directory target is swapped in by rename where its class has one.
If you list the directory while a move runs, you may see a hidden
.NAME.<hex>.pathapi-tmpsibling.copy(follow_symlinks=False, overwrite=True)of a link no longer destroys the old
target when the new link cannot be made. The existing target was unlinked before
symlink_to(), so a refused link lost it. Where the target's class can rename, the
link is now made in a hidden sibling and renamed over the target; a class that
cannot rename still unlinks first. A target class that cannot make links now raises
NotImplementedErrorbefore anything is removed.
Full Changelog: v0.1.1...v0.1.2
v0.1.1
Added
deadlinefor a whole HTTP request.HTTPClient(deadline=...)and the
deadline=keyword ofrequest()andredirect()take the seconds a request may
take, redirects and the reading of the body included (None, the default, is no
limit). Each wait's timeout is held to the time that remains andTimeoutErroris
raised once it has passed;timeoutstill bounds each wait. A read already blocked
can overrun the deadline by at most its timeout. A value that is not a number is
TypeError, one that is not positiveValueError. It works over every transport.
Fixed
copy(overwrite=True)no longer destroys the old target when the copy fails.
An existing file was removed before the new content was written, so a source that
failed half way left no file, or a partial one. The old file is kept until the
new content is complete: it is written to a hidden.NAME.<hex>.pathapi-tmp
sibling and renamed over it, and a class whose write stream commits on close is
written to directly. A class that cannot rename writes the sibling and then writes
it over the target. If you list the directory while a copy runs, you may see that
sibling.pathlib.Path(src).copy(target)(3.14) onto aPathof this library refuses
unsafe child names._copy_from()joined every listed name onto the target; a
name that is not one component (.., a separator, on a Windows target a drive or
a trailing dot) now raisesValueErrorbefore anything is written for it, as
copy(recursive=True)does. A source that lists such names needs a copy that
skips them.- The shipped transport suite names the file of its tests wherever it is run
from.python -m pathapi.http.testing(andrun()) roots pytest at the suite's
own directory unless--rootdiris given, so node ids keep their file when the
checkout and the installed package are on different drives.
Full Changelog: v0.1.0...v0.1.1
v0.1.0
Published as pathlib-next up to 0.10.0; from 0.1.0 as pathapi. The released
sections below keep the name they were published under.
Added
supports()andcapabilities(): which protocols a path class implements.
pathapi.protocols.supports(obj_or_class, *protocols)is true when every
primitive of every protocol named is something other than its default stub, and
capabilities(obj_or_class)lists the protocols that hold. Ask once instead of
calling and catchingNotImplementedError. The answer is advisory: a primitive may
still refuse an argument. A class whose answer depends on its connection defines
_supports(protocol) -> bool | None.- The operations of
Pathare protocols you can mix in.Create(mkdir,
touch),Listing(iterdir,glob,rglob,walk),Remove(unlink,rmdir,
rm),Readlink(readlink),Symlink(symlink_to),Identity(samefile),Copy,Move
and the pure-path descriptionNamedare inpathapi.protocols, and
Pathis built from them. A read-only backend can be a class built fromPathname,
Listing,StatandBinaryOpenwith noPath:glob,checksum.md5
andPathSyncertake it as a source. Chmodis split, and so is every pair of a reader and a writer.Chown
(chown()),Owned(owner(),group()),Utime(_utime()),Readlinkand
Symlinkare protocols of their own, sosupports(path, Chmod)asks forchmod()
alone and an archive supportsReadlinkand notSymlink.Chmodstill inherits
ChownandUtime, sochown,_chownand_utimestay reachable from it.owner(),group(),readlink(),hardlink_to(),absolute()andresolve()
on everyPath. A backend supplies_user_name()and_group_name(),
readlink(),_hardlink_to()and_cwd();resolve()follows links through
readlink(), at most 40, and is lexical where a class has none.LocalPathkeeps
pathlib's own.- The contract suites' switches
supports_listing,supports_mkdir,
supports_delete,supports_utimeandsupports_renamedefault toNone, which
askssupports()aboutListing,Create,Remove,UtimeandMove, so a class
without a protocol skips those tests with no switch set and the skip names the
protocol. Set one toTrueorFalseon the test class to override the answer. providers.primitivemarks a default of a primitive whose body raises an error of its
own, sosupports()reads it as absent.providers.derived_from(*primitives)marks a method that only wraps other
primitives, so a base class can defineiterdir()over_listdir()and
supports()still says a subclass without_listdir()cannot list.providers.notimplementedtakes a reason:@notimplemented("why")marks a stub
thatsupports()reads as absent and whoseNotImplementedErrorcarries the
reason; the bare form is unchanged.
Changed
-
The distribution, the import package and the repository are
pathapi. Version
numbering restarts at 0.1.0.pathlib-next0.10.1 is a final release that depends on
pathapiand re-exports it under the old import name with aDeprecationWarning.
What to do: installpathapiand change the names in this table.Before After pip install pathlib-nextpip install pathapiimport pathlib_next,pathlib_next.<module>import pathapi,pathapi.<module>entry-point group pathlib_next.http_transportspathapi.http_transportspathlib-next-requests(pathlib_next_requests)pathapi-requests(pathapi_requests)PATHLIB_NEXT_HTTP_TRANSPORTPATHAPI_HTTP_TRANSPORTthe pathlib-next/<version>user agent, the.pathlib-next-tmpsuffix of aPathSyncertemporary siblingpathapi/<version>,.pathapi-tmpthe pathlib_next.syncandpathlib_next.httploggerspathapi.sync,pathapi.httpbenchmark result schema pathlib_next.bench/1pathapi.bench/1 -
copy()over an existing file does not callunlink()on a target whose class has no
Remove(it replaces the content throughopen()), andmove()raises
NotImplementedErrorbefore copying when the source cannot be removed and has no
rename()to use. Before, the first raised from theunlink()and the second left
the file in two places. What to do: nothing, unless you relied on either. -
Every
Pathnow has areadlink(), which raisesNotImplementedErrorwhere the class
cannot read links, sohasattr(path, "readlink")no longer tells the two apart. What
to do: asksupports(path, Readlink). -
A write stream that commits on close (
CommitOnClose, and any stream built on
DiscardOnError), or the text file over one, that is garbage collected without being
closed is now discarded with aResourceWarninginstead of committed, as awith
block that raised already was. Before, the collector closed it and so uploaded or
rewrote what the program never finished. What to do: close the file (usewith);
a stream of your own onDiscardOnErrorcan override_holds_uncommitted()to stay
silent when it has nothing to commit. -
_supports(protocol)may make at most one request per backend, cached for the
backend's lifetime, and never raises (it answersNonewhen the request fails).
Full Changelog: v0.10.0...v0.1.0
pathlib-next-v0.10.1
The last release under the name pathlib-next: the project is now pathapi. This
release contains no implementation of its own.
Changed
pathlib-nextdepends onpathapi>=0.1.0,<0.2, andimport pathlib_nextre-exports
it after aDeprecationWarning: every public name, and every submodule
(pathlib_next.protocols,pathlib_next.http, ...) as the same module object as its
pathapicounterpart, soisinstanceagrees across the two names. The package
shipspy.typed;PATHLIB_NEXT_HTTP_TRANSPORTis copied toPATHAPI_HTTP_TRANSPORT
when only the old one is set. What to do: installpathapiand importpathapi;
README.mdhas the table of names.pathapi0.1.0 is 0.10.0 plus the protocols of
pathapiand the changes listed in its changelog.
v0.10.0
pathlib-next 0.10.0 is the core alone: the path classes, pathlib_next.http,
FTPPath and the archive paths, with no dependency, no extra and no console
script. The URI layer, every scheme that needs a library, and the uripath
command are distributions of their own that depend on it. Under Changed, the
migration comes in this order: the table of names, then every change of
behaviour with what to do about it.
Added
pathlib_next.http: HTTP requests on the standard library, with a
transport you can replace, and no URI.import pathlib_next.httpneeds no
extra and no third-party library (it loadshttp.clientandssl;
import pathlib_nextstill loads neither).HTTPClient(transport=None, *, headers=None, timeout=(10, 60), user_agent=None)sendsrequest(method, url, *, data=None, headers=None, stream=False, allow_redirects=None, timeout, auth=None)through a transport and follows redirects:GET,
HEADandOPTIONSup to 30, withAuthorizationdropped for good by a hop
to another scheme, host or port; every other method is sent again once after
a 307 or 308 to the same origin and is otherwise anOSError(EIO)naming the
status. A transport is any object withsend(method, url, *, headers, body=None, timeout, stream=False)that returns anHTTPResponse, and a
close();HTTPTransport(*, ssl_context=None, verify=True, proxies=None)is
the built-in one onhttp.client(TLS verified by default,proxies=as an
object indexed with the request URL, one idle connection per thread and
server). The package also holds the helpers a path built on it needs:
raise_for_status,translate_errors(a failed request as theOSError
typepathlibraises),read_limited,json_bodyanddrop_body(a reply
read whole is held toMAX_REPLY_BYTES, 8 MiB after decoding, and the
setting is assignable),peer_text,split_userinfo,server_sizeand
is_redirect.PATHLIB_NEXT_HTTP_TRANSPORTnames the transport a client
builds when it is given none:builtin, or the name of one an installed
distribution declares in the entry-point grouppathlib_next.http_transports
(requests, from thepathlib-next-requestsdistribution;transport_names()
lists them without importing any andget_transport(name)loads one).
prepare_request()makes the checks of the transport contract for a
transport of your own.pathlib_next.http.testing: the suite that holds a transport to the
promises of the built-in one. With the distribution that provides the
transport andpytestinstalled,python -m pathlib_next.http.testing --http-transports requests -qruns the tests that depend on the transport
against it, over loopback servers scripted at the byte level. The test
files ship in the wheel. It is testing support and not part of the stable
API: its contents, including test names, fixtures and helper signatures, may
change in any release, andpathlib_next.httpnever imports it.pathlib_next.ftp:FTPPathandFTPBackend, an FTP tree that needs no
URI and no extra.FTPPath("/pub/x", host="h", user="u", tls=True)builds
the path's ownFTPBackend;FTPPath("/pub/x", backend=backend)shares one
connection cache between paths built separately (or plugs in a
BaseFTPBackendof your own), and giving both is aTypeError. Nothing
connects until the first operation.FTPBackend(host, port=21, *, user, password, tls, timeout, ssl_context, verify)keeps a control connection per
thread, is a context manager, andclose()ends every connection it holds.
str()is the remote path,repr()names the endpoint and never shows a
password, and two paths are one tree when they share a backend or name the
same host, port, account and security.pathlib_next.FTPPathis imported on
first use, soimport pathlib_nextstill loads neitherftplibnorssl.pathlib_next.archive:ZipPath,TarPathandArchivePath, the members
of an archive file that is anyPath, with no URI and no extra.
ZipPath("d/x", archive="a.zip")takes the archive as aPath, or astr
oros.PathLike(aLocalPath); the segments are the member path, read
from the archive root.ZipPathandTarPathfix the format,ArchivePath
reads it from the extension or the first bytes on first use, and building a
path reads nothing. One handle serves each archive file and format, so paths
built separately see each other's writes. A zip that is a local file is
written in place (write_bytes,mkdir,unlink,rmdir,rename,
rm(recursive=True),batch()); every other archive is read-only, in
memory or on any otherPath.str()is the member path,repr()names the
archive, andas_uri()iszip:<archive uri>!/<member>.
ArchivePath.backend_for()gives the handle to pass asarchive=.
pathlib_next.ZipPath,TarPathandArchivePathare imported on first use.pathlib_next.providers.TreePath, the baseFTPPathand the archive
paths are built on, for a path whose names are POSIX segments below the
root of a tree a backend serves.pathlib_next.providers.name_stem(),name_suffix()and
name_suffixes(). One path component split into stem and suffix as
pathlibof the running interpreter splits it, for a class that computes
those itself and must agree withPathname.pathlib_next.exceptions:PathError, the base of the exceptions the
package defines.NonRelativePatternError(an absolute glob pattern) is
now aPathErroras well as aNotImplementedErrorand aValueError, so
everyexceptclause that caught it still does. I/O failures stay the
OSErrortypespathlibraises.PathError,NonRelativePatternErrorand
MemPathare exported frompathlib_next.pathlib_next.mempath,pathlib_next.syncandpathlib_next.testing
declare__all__, sofrom pathlib_next.sync import *publishes
PathSyncer,SyncEventandPathAndStatand nothing else, and a type
checker reads those names as the module's exports.pathlib_next.providers.DiscardOnError, a mixin for a write stream that
commits when it is closed: its__exit__callsdiscard()when thewith
block raised, then closes.CommitOnCloseinherits it and has a
discard(); see the entry under Changed.
Changed
-
Where every name went. Change each import and each install as the table
says. There is no alias for an old name: the old import raises
ModuleNotFoundErrororImportError, and a name that is not in the table
was never public. The URI layer, the schemes and the command line are the
distributionsuripath,sftppath,objpath,repopathand
pathlib-next-requests, which each document their own changes of behaviour;
the core's own moves come first.Was Is now pathlib_next.utils.sync(PathSyncer,SyncEvent,PathAndStat)pathlib_next.syncpathlib_next.utils.glob(glob,select,full_match,parse_pattern, ...)pathlib_next.globpathlib_next.utils.glob.NonRelativePatternErrorpathlib_next.NonRelativePatternError(defined inpathlib_next.exceptions)pathlib_next.utils.checksum(md5,sha256,stream,native)pathlib_next.checksumpathlib_next.utils.md5,pathlib_next.utils.sha256pathlib_next.checksum.md5,pathlib_next.checksum.sha256pathlib_next.utils.archive,pathlib_next.utils.make_archive,pathlib_next.utils.unpack_archivepathlib_next.archive.make_archive,pathlib_next.archive.unpack_archivepathlib_next.utils.archive.detect_format(also_detect_format)pathlib_next.archive.detect_formatpathlib_next.utils.stat.FileStatpathlib_next.FileStatpathlib_next.utils.as_modepathlib_next.providers.parse_modepathlib_next.utils.as_ownerpathlib_next.providers.parse_ownerpathlib_next.utils.UNCHANGEDpathlib_next.providers.UNCHANGEDpathlib_next.utils.as_error_handler,as_error_predicate,OfferedOncepathlib_next.providers.as_error_handler,as_error_predicate,OfferedOncepathlib_next.utils.is_safe_child_name,is_windows_flavouredpathlib_next.providers.is_safe_child_name,is_windows_flavouredpathlib_next.utils.notimplementedpathlib_next.providers.notimplementedpathlib_next.utils.LRUpathlib_next.providers.LRUpathlib_next.utils._commit.CommitOnClose(also_CommitOnClose)pathlib_next.providers.CommitOnClosepathlib_next.path.check_follow,contains,copy_error_predicate(also_check_follow,_contains)pathlib_next.providers.check_follow,contains,copy_error_predicatepathlib_next.utils.sizeof_fmt,pathlib_next.utils.parsedateremoved: no replacement is public the package pathlib_next.urithe distribution uripath, import nameuripathpathlib_next.Uri,pathlib_next.UriPath(andpathlib_next.uri.Uri,UriPath,UriLike)uripath.URI,uripath.URIPath,uripath.URILikePath,Pathname,FileStatimported frompathlib_next.urithe same names in pathlib_nextpathlib_next.uri.source.Sourceuripath.Source(alsouripath.source.Source)pathlib_next.uri.source.compose_uri,compose_host,percent_encode,is_drive,SAFE_PATH,ERRORSthe same names in uripath.sourcepathlib_next.uri.same_authorityuripath.source.same_authoritypathlib_next.uri.DerivedBackenduripath.schemes.DerivedBackendpathlib_next.uri.query.Query;Query(text),Query.decode()uripath.Query, no longer astr;Query.parse(text),Query.to_pairs()Source.from_str(text, strict);Source.as_str(sanitize=True),as_str(sanitize=False)Source.parse(text, *, strict=True);str(source),source.reveal()`pathlib_next.uri.s...
v0.9.14
Added
MemPath.relative_to(other, *, walk_up=False). It raised
NotImplementedError; it now answers asPurePosixPath.relative_todoes,
on the path's own names (no..applied):otheris aMemPathof the same
backend or astr, and the result is a relativeMemPathon that backend.
ValueErrorwhen the path is not belowother, when only one of them is
rooted, or when the backends differ.walk_up=Truegives the..-prefixed
form on every Python version, not only 3.12 and later. Code that caught
NotImplementedErroraround it can call it directly.copy(on_error=)andcopy_into(on_error=)takerm()'s error
predicate.on_error(error, path) -> bool(or a bool) is asked once for
each child of a recursive copy that fails, withpaththe source entry:
Trueskips it and the copy goes on,Falseraises the error. The same
function used to mean opposite things to the two methods (lambda *a: False
maderm()raise andcopy()suppress every error), and a predicate
written for one raisedTypeErrorin the other.ignore_error=keeps its
notify-only callable; useon_error=for a predicate. Passing both raises
TypeErrorbefore anything is copied.sftp:(the asyncssh concurrent walk
included),s3:,gs:andaz:accept it, andPathContractgains a test
that gives one predicate torm()andcopy().Path.info, and Python 3.14'spathlib.Path.copy()onto anyPath.
EveryPath(MemPath,UriPath,SftpPath, ...) hasinfo, on every
Python version: an object withexists(),is_dir(),is_file()(each with
follow_symlinks=True) andis_symlink(), the shape of 3.14's
pathlib.types.PathInfo. It callsstat()at most once following links and
once not, and remembers the answers on that object only, so a newpath.info
asks again.LocalPathkeeps pathlib's own on 3.14. A new hook,
Path._copy_from(source, follow_symlinks=True, preserve_metadata=False),
letspathlib.Path(f).copy(MemPath("/x"))work on 3.14, where it raised
AttributeError: ... has no attribute '_copy_from': it overwrites a file,
raises on an existing directory, and withpreserve_metadata=Truecopies the
permission bits and the modification time where the class can set them.
Path.copy()of this library is unchanged and does not call it.FileStat.size_known, and listings that know a type but not a size.
A listing can now say "this is a file" without inventing its size: a stat
withsize_known=Falseanswersis_dir(),is_file(),exists(),
walk(),glob()andrm(recursive=True), whilestat()of the listed
child asks the server for the real size and time, andPathSyncer, the
progress total ofcopy()andquick_checknever read the placeholder.
gitlab:blobs, an Apache index with sizes like1.2Kand the stdlib
bulleted-list index (http:) used to give no hint at all, so walking a tree
cost a request per file; it now costs one per directory. An exact size from a
listing that has one stays a full hint. A scheme of your own may yield
FileStat(is_dir=False, size_known=False)from_scandir(); code that reads
a listing'sst_sizeorst_mtimeshould checksize_known.PathSyncer(follow_mounts=False)anduripath sync --no-follow-mounts.
A mount point found below the target root (a mounted volume, a bind mount)
is still synced into by default, asrsyncdoes. Withfollow_mounts=False
it is an error for that entry instead:OSError(EXDEV)naming it, offered to
ignore_errorwith eventTypeMismatch, and nothing behind it is listed,
written or removed, in a dry run too. A bind mount aliases another directory
as a junction does, andrm(recursive=True)already stops at one; this lets
a sync withremove_missing=Truerefuse it. A junction is replaced under
either value, and the root target is used as given.touch()moves an existing file's time, and a sync leaves its copies with
their source's time. The generictouch()of an existing file did nothing;
it now sets the access and modification times to now through a new_utime()
hook (_utime(*, atime=None, mtime=None), seconds since the epoch,None
leaves that time alone), implemented byLocalPath,FileUri,MemPath,
SftpPathon both client libraries (whole seconds) andFtpPathon a server
whoseFEATlistsMFMT(modification time only). A class that does not
implement it raisesNotImplementedErrorfrom_utime()andtouch()
leaves the file as it was, as before.PathContractgains
supports_utime(defaultTrue; set itFalseon a test class whose
backend cannot set a time) and a test behind it.PathSyncer(preserve_mtime=True). A file the sync copies is given the
modification time its source reported, where the target's class implements
_utime(); a source that reports no time, a target that cannot set one or
refuses it, keep the time the copy gave the file, and the sync reports
nothing. Directories and links are left alone.quick_check(default on for
a pair with a non-local side) could never succeed after this library's own
copy, so a remote tree was checksummed, and so downloaded, again by every
later run; the second run of an unchanged tree now reads no file.- The names the built-in schemes take from the core are documented, and the
helpers among them are public. The shipped API header gains "For path
implementers" and "For scheme implementers" sections, each hook with its
signature and one line of contract (_init,_from_parsed_parts,
_format_parsed_parts,_make_child_relpath,_rename_target,
_pop_stat_hint,_same_location, the backend-state hooks,_anchor_segments
and the rest). Twelve helpers that were underscore names now have public ones:
pathlib_next.path.check_followandcontains,
pathlib_next.utils.archive.detect_format,
pathlib_next.utils._commit.CommitOnClose,pathlib_next.uri.same_authority
andDerivedBackend, and inpathlib_next.uri.sourcecompose_uri,
compose_host,percent_encode,is_drive,SAFE_PATHandERRORS. The
underscore spellings (_check_follow,_contains,_encode, ...) still
import and are the same objects, so nothing has to change; a scheme written
outside this package can use the documented names instead.path.__all__
now listscheck_followandcontains. glob(recurse_symlinks=True)andrglob(recurse_symlinks=True)work.
They raisedNotImplementedErroron every class.**now descends through
symlinks to directories, as pathlib 3.13 does, and the keyword is accepted
on every Python version. A loop ends where pathlib 3.13 would follow it
until the system refuses: a link to a directory the descent is already
inside (st_dev/st_ino) is listed as an entry and not entered, and a
backend whose stat has no identity follows at most 40 links in one descent.
utils.glob.glob()andutils.glob.select()take the keyword too. The
default staysFalse; a caller that caught theNotImplementedErrorto
fall back to its own walk can drop the fallback.checksum()andsupported_checksums()ons3:,gs:andaz:. An
object-store path now answersNativeChecksumfrom the digest the store
keeps, with one request and no byte of the content.S3Pathgives the plain
MD5 that the ETag is for an object one request wrote and that KMS or a
customer key does not encrypt (checksum("md5")), and the stored full-object
crc32,crc32c,crc64nvme,sha1orsha256;GsPathgivesmd5Hash
andcrc32c;AzPathgives theContent-MD5property. A multipart ETag
(...-N), a composite checksum, a GCS object with a content encoding or no
md5Hash, a blob withoutContent-MD5and any other algorithm raise
NotImplementedError, never a guess; a missing path isFileNotFoundError
and a prefix directoryIsADirectoryError.PathSynceruses them without
being told: the secondsync()of an unchanged tree of 100s3:objects
to another prefix sent 200GetObjectbefore and sends none. Azure's
Content-MD5is a property a client can set and the service does not verify
against the content in every case.copy()of a file within one object store is one request of the store.
s3:/gs:(inside a bucket, or between buckets that one client reaches: the
same supplied backend, or both paths' default) andaz:(inside a
container) issueCopyObject,copy_blobor the blob copy instead of
reading the file and writing it back;copy(recursive=True)andmove()of
a prefix do so for each file. A file copy ons3:took 6 requests (a
GetObjectand aPutObjectamong them) and takes 4, of which one is the
copy. The checkscopy()always made run first and in the same order (the
same file, a target that is a directory or holds keys, an existing target
withoutoverwrite=True); an existing target is replaced by the copy and no
longer deleted first, so it is intact when the copy fails;progressis
called once with the total; a file named as the root of a
recursive=Truecopy is still read and written; a store that answers "not
implemented" gets the streamed copy. The copy keeps the object's content type
and metadata, which the streamed copy did not.gs:andaz:read and write an object in pieces. Reading asks for
ranges of 4 MiB (gs:) or 512 KiB (az:,schemes.az.READ_CHUNK: the SDK
holds about nine times a response, assign a larger number to trade memory
for requests) as the stream is read, instead of downloading the whole
object first; the stream stays seekable andseek()asks nothing until the
next read. Writing spools in memory up to 4 MiB and then in a temp file, as
s3:does, and uploads on close: a resumable upload in 2 MiB requests on
gs:, blocks onaz:. A read ...
v0.9.13
Added
copy_into(),move_into()andreplace()on everyPath. pathlib 3.14
has them and the generic classes did not (MemPath,UriPath,DavPath,
SftpPath, ... raisedAttributeError;LocalPathhadcopy_into()and
move_into()on 3.14 only).copy_into(target_dir, ...)copies under the
same name withcopy()'s keywords and defaults,move_into(target_dir, *, overwrite=False)moves withmove()'s, and both return the new path.
replace(target)ismove(target, overwrite=True)and returnstarget; a
directory is replaced only when empty (OSError(ENOTEMPTY)otherwise, and
nothing is removed). Astrdirectory or target stays on the same backend.
infois not added.Pathname.is_absolute()has a default: the path has a root. It raised
NotImplementedErrorforMemPathand everyPathnamesubclass that did not
override it.- Generic paths can be ordered.
sorted(),<,<=,>and>=raised
TypeErrorforMemPath,Uri,UriPathand every customPath.
Pathnamenow orders two paths of the same exact type by the key its==
uses (the segments; the URI text forUri), and returnsNotImplemented
for anything else, aspathlibdoes across flavours. A subclass with its
own__eq__overrides_order_key()to match it.LocalPathkeeps
pathlib's ordering. pathlib_next.__version__and__all__on the core modules. The root has
__version__, the installed distribution's version ("0+unknown"for a
source tree that is not installed). The root,path,fspath,protocols
(withfs,ioandchecksum),utils,utils.statandutils.glob
declare__all__with the namessrc/pathlib_next/AGENTS.mddocuments for
them. Nothing documented moved.InsecureTransportWarning. ARepoBackend(github:,gitlab:,git:)
that is about to send anAuthorizationheader (its token, or one passed as
headers=) to a plainhttp://API root on a host other than loopback now
warns with thisUserWarningsubclass (importable from
pathlib_next.uri.schemes._gitrepo,.githuband.gitlab); the request is
still sent. Use anhttps://api_base, or filter the warning for a
trusted network.SftpAuthenticationErrorandSftpHostKeyError. Importable from
pathlib_next.uri.schemes.sftp. A refused login is the first (a
PermissionError,EACCES) and a host key that is unknown, changed or
refused the second (aConnectionError,ECONNABORTED), on both SFTP
backends. CatchOSError, or these two to tell a login from a host key.ArchiveUri.refresh(). A remote archive (zip:http://...!/m) is read once
and kept while any path to it lives, so a change at its source was never seen
by a program that held a path.path.refresh()forgets what is held of the
archive, for every path to it, and the next use reads it again. A local
archive is checked against its file on every use and needs none.GsBackend(timeout=, retry=). Both go to every call aGsPathmakes
(timeoutin seconds or a(connect, read)pair,retrya
google.api_core.retry.RetryorNonefor none). Without them the SDK's
defaults apply, and an unreachable endpoint kept every call waiting for about
two minutes. A customBaseGsBackendadds options throughcall_options().schemes.az.COPY_POLL_TIMEOUT. How longAzPath.rename()waits for the
server-side copy (300 seconds); set it before the call to change it.pathlib_next.testing: five capability switches and eight tests. For a
contract class that subclassesPathContract, the new tests are:
a payload of about 290 KiB holding every byte value and runs of CR/LF, written
whole and in pieces and read back whole and in parts; names holding a space,
#,%,+,&,=and mixed case, stored, listed, renamed and removed as
written;stat()of a longer file and of a
directory, and a numericst_mtime;_scandir()agreeing withstat()for every child; nothing
existing below a file; partialread(n); andrename()onto a non-empty
directory raising. The switches, each documented in the module and in the
guide:supports_mkdir,supports_deleteandsupports_move(default
True: a backend with no directories, delete or move, such as TFTP, sets them
Falseinstead of marking tests as expected failures),supports_unusual_names
(defaultTrue), andsupports_question_mark_names(defaultFalse: a
Windows file system cannot store a name holding?). A backend that kept
passing the old suite and now fails a new test was altering bytes, sizes or
names: fix it, or set the switch that names the gap. With
supports_delete = False,copy(overwrite=True)is no longer exercised;
enforces_directory_hierarchy = Falsealso skips the rename-onto-a-directory
test.
Changed
from pathlib_next import *publishes only the documented names. It also
broughtP,PNand the__future__nameannotations, and
from pathlib_next.protocols import *andfrom pathlib_next.utils import *
brought the modules they import.P,PN,K,Vand the rest stay
importable from the module that defines them; a script that relied on
from pathlib_next import *for one of them imports it by name.- The constructors of
Path,LocalPath,MemPathandFileStatpass mypy
and pyright at their defaults. Both readPath("x"),LocalPath("x"),
MemPath("/a")andFileStat(...)as constructing an abstract class (the
protocol stubs had empty bodies,FileStat's fields were not declared, and
Pathname's abstract members counted against the barePath, which builds a
LocalPath). The stubs now raiseNotImplementedErrorthrough a call,
FileStatdeclares its fields,Path.__new__is annotated, and a checker
treatsPathname's abstract members as ordinary methods. Nothing changes at
run time: ABC enforcement still refuses a subclass that omits one. A consumer
can drop a# type: ignore[abstract]on those four constructions.
FileStatLike.st_mtimeis annotatedfloat, which is what every backend
reports. - On Python 3.14
LocalPath.copy_into()andmove_into()are this
library's, not stdlib's. Stdlib's overwrote an existing file, copied
directories without being asked andLocalPathanswered differently from
every other backend. They now behave asMemPath's do: an existing target
raisesFileExistsErrorunlessoverwrite=True, a directory needs
recursive=True, and the new path is returned. Passoverwrite=True, recursive=Truefor the stdlib behaviour.LocalPath.replace()stays
stdlib'sos.replace()on every version. with_stem("")raises on 3.13 and later when the name has a suffix, as
pathlibdoes there (MemPath("/a/c.txt").with_stem("")was/a/.txt).
Before 3.13 it is unchanged.walk()lists a Windows junction infilenamesand does not enter it
unlessfollow_symlinks=True, as it already did for a symlink to a directory
and aspathlib.Path.walk()does on 3.14. It listed the junction in
dirnamesand descended: a junction to its own parent was walked 64 levels
deep, until the path was too long. A caller that wants the target's files
passesfollow_symlinks=True. The header anddocs/divergences.mdno longer
say thatLocalPathprotects a following walk from a loop; neither does
pathlib.utils.as_mode()andutils.as_owner()refuse what cannot be a mode or an
owner.as_mode(-1)returned-1,as_mode("07777777777777")a number
wider than any mode, andas_owner(True, False)was(1, 0)on 3.14 and
(True, False)on 3.9. Aboolis nowTypeError, a mode outside
0..0o177777(ast_modewith its type bits still passes, asos.chmod
accepts it;stat.S_IMODE()drops them) and an id outside0..2**32-1
(-1stays "unchanged")ValueError.UriPath(url)type-checks under pyright.UriPath.__new__annotated
schemesmapasdictwith a default ofNone, which pyright rejects for
every construction (UriPath("sftp://h/x")); it is now an optional mapping of
scheme to class.ReadPathContracthas a pickle test and asupports_pickleswitch. A
contract subclass checks thatpickle.loads(pickle.dumps(path))is equal to
the path, hashes alike, and for a URI path is placed on the same filesystem.
An implementation whose paths cannot pickle setssupports_pickle = False
on its test class and the test skips.Uri(0),Uri(5)and other falsy non-text arguments raiseTypeError.
OnlyNoneand empty text are the empty URI;Uri(0)orUri([])used to
be silently the empty URI, andUri(5)said it gotNoneType. The message
now names the type that was passed. PassNoneor""for an empty part.- An
http:directory index larger than 8 MiB raises.iterdir(),
walk()andglob()read the whole index and parse it at about a second per
megabyte with no bound, so a server decided how long a listing took. A body
overpathlib_next.uri.schemes.http.MAX_LISTING_BYTES(8 MiB after decoding)
is nowOSError(EFBIG), checked againstContent-Lengthbefore the body
is read. Assign a larger number to that module constant to list a bigger
index. - Listing an
http:subdirectory andstat()of a redirecting URL send fewer
requests. A directory without a trailing slash is now asked for with it
first (oneGETinstead of aGETand the redirect it earned, per
directory of awalk()), falling back to the path as given on a 404, and
stat()follows theLocationof the first answer instead of repeating the
request that was redirected. - The header and the schemes guide say that only
Authorizationis dropped
from a redirectedGET/HEAD.with_session(headers={"X-Api-Key": ...})
has always been sent to the new host...
v0.9.12
Added
Path._node_key(), an override hook answering where a path's node
lives:(namespace, names), the object two paths share when they resolve
names in one tree and the node's normalized position in it.copy()and
move()compare two paths that both answer it across classes.MemPath
and the archive paths override it; any other type keeps comparing two paths
of one type with==, so nothing needs to change.PathContractcallsrm(recursive=True)withfollow_symlinks=and
follow_binds=(test_rm_recursive_accepts_the_follow_policies, and
test_rm_rejects_a_policy_that_is_not_one). An implementation that
overridesrm()must accept both keywords, asPath.rm()does; its contract
subclass fails these two tests until it does. No fixture or capability
attribute changed.
Changed
- The scheme extras now declare version ranges.
uritools,requests,
paramiko,boto3,google-cloud-storage,azure-storage-blob,
azure-identityandasyncsshhad no lower bound (asyncsshhad only the
Python 3.9 cap) and no upper bound, andnetimpshad only a floor. The
ranges are nowuritools>=5.0,<7,netimps>=0.4.0,<0.5(was>=0.3.1),
requests>=2.22.0,<3,paramiko>=5.0,<6,asyncssh>=2.20.0,<3(below 2.22
on Python 3.9),boto3>=1.36.0,<2,google-cloud-storage>=3.0,<4,
azure-storage-blob>=12.4.0,<13andazure-identity>=1.0,<2. An
environment that pins one of them below its floor must raise the pin; one
that needs a new major version must wait for a release that admits it. copy(preserve_metadata=True)drops the setuid, setgid and sticky bits
between two different classes. A mode read from a tar member, a zip entry
or a remote server was applied verbatim, so04755in an archive made a
setuid file on the local disk. A copy between two classes now applies the
permission bits without those three; a copy within one class keeps them.
Runchmod()afterwards where the bit is wanted.move()callsrename()only onto a path of the same class. The default
ofPath._rename_compatible()was True, so aPathsubclass with a
rename()of its own was handed a path of another store and ran the rename
inside its own (aMemPathsubclass moved onto aLocalPathlooked for the
local path in memory). The target of another class now gets copy + delete.
A subclass whoserename()handles other classes overrides
_rename_compatible();UriPathandLocalPathdecide for themselves as
before.
Fixed
- A
..or.name in a remote listing no longer escapes the tree. The
s3:,gs:,az:,github:andgitlab:listings and the default
UriPathlisting yielded a name such as..,.,a/bor an empty one as
a child (http:,dav:andsftp:already skipped them), and theftp:
listing yielded one holding NUL. Joined onto the directory,..named its
parent, sowalk()left the directory it was given and never ended,
rm(recursive=True)deleted the parent tree, and
PathSyncer(remove_missing=True)removed the parent or copied a sibling
tree into the destination. All of those listings now skip such a name, and
walk(),rm(recursive=True)andPathSynceralso refuse one that a custom
scheme's_scandir()yields, before building any child from it:walk()
leaves it out ofdirnames/filenamesand, whenon_erroris given, calls
it with aValueError;rm(recursive=True)offers aValueErrorto
ignore_error(error, directory)once per entry and removes nothing for it;
PathSyncerraisesValueErrorthroughignore_errorfor the source's and
the target's names alike. A custom_scandir()should skip such names
itself (utils.is_safe_child_name()decides). An object key with a.or
..segment is no longer listed. - Names Windows rewrites are refused for a Windows destination.
utils.is_safe_child_name(name, windows=True)accepted a name ending in a
dot or a space and the reserved device names. Windows storesreport.as
report, socopy(recursive=True)of a tree holding both merged them into
one file with no error, and a memberNULof an archive was written to the
device.
Withwindows=Truethe function now also refuses a trailing dot or space
andCON,PRN,AUX,NUL,COM1-COM9andLPT1-LPT9, in any case,
with or without an extension (nul.txt);windows=Falseis unchanged.
copy(recursive=True)andPathSyncerreport such a name as a
ValueErrorper entry for a Windows-flavoured target, and
unpack_archive()skips such a member for a Windows destination with a
UserWarning. Rename a source file with such a name, or extract it on
another platform. - A failed
open("a")onhttp:no longer empties the remote file. In the
defaultappend_mode="rewrite", a failure of theGETthat reads the
current content (a 500, a 403, a timeout, a body cut short) raised to the
caller, and then the half-built stream was garbage-collected and uploaded
an empty body: the file was replaced by a zero-byte one.open("a")now
raises and sends nothing, in both append modes; a write stream whose
constructor did not finish never uploads. - A write through a redirect no longer reports success without storing the
bytes.http:anddav:followed a redirect forPUT,PATCH,
DELETE,MKCOLandMOVEthe way a browser does. A 301 re-sent the
PUTwithout its body and replaced the file with an empty one; a 302 or
303 turned the request into aGET, sowrite_bytes()returned having
written nothing andunlink(),mkdir()andrename()returned having
done nothing. Those requests now go out withallow_redirects=False
(anallow_redirectsinwith_session()does not apply to them). A 307
or 308 to the same scheme, host and port is sent once more with the same
method and body; any other 3xx, a redirect to another origin, a second
redirect and a body that cannot be sent twice raiseOSError(EIO)
naming the status and theLocation, without its userinfo. A server that
redirects writes must be addressed by its final URL. open("x")onhttp:anddav:no longer replaces a file whose
existence check failed, and patch-mode append no longer overwrites the start
of a file whose size is unknown.open("x")read any error of its
existence check (a 5xx, a 403, a 429, a timeout) as "missing" and sent the
PUT, replacing the file. It now raises that error and sends nothing, and
the upload carriesIf-None-Match: *, which a server that honours it
answers with 412, nowFileExistsError. Withappend_mode="patch", a
HEADreply withoutContent-Lengthwas read as size 0 and the appended
bytes were written at offset 0 (Content-Range: bytes 0-2/*);open("a")
now raisesOSErrornaming the path and sends noPATCH.stat()still
reportsst_size0 for such a reply. A server whose existence check fails
for a missing file needsopen("w")instead ofopen("x"); one that does
not state the size of its files needsappend_mode="rewrite".- A session or token no longer follows a path to another endpoint, and the
mirroring idiom keeps the destination's backend. Since 0.9.7, a path that
crossed endpoints (UriPath(base, "http://other/d/"),base.joinpath(Uri(...), "x"),base / Uri(...)) keptbase's backend for the nextstrjoin,
with_name()orcopy()/move()destination written as a URL, until its
first read, so anHttpPath.with_session(auth=...)session or a
github://TOKEN@...token was sent to the other host. And
dst_root / src.relative_to(src_root)dropped the backend given to
dst_rootand built the default one, so the copy ran on the ambient S3
client, or on an HTTP session without its headers. A derived path now takes
the backend its source holds only when it is on the same scheme, userinfo,
host and port, the rightmost joined path on the result's endpoint
supplies it, and the sourceless result ofrelative_to()holds none. A
path that crosses endpoints needs its ownwith_session()or
with_backend(). - Deriving a path no longer builds a backend.
parent,parents,
with_name(),with_suffix(),with_query(),relative_to(),
UriPath(base, x)and a join with aUriorPurePathbuilt the source's
backend: the firstparentof ansftp:path imported asyncssh, and
raisedImportErrorwhere neither SSH library is installed. They now copy
the backend the source already holds, or leave the new path without one
until its first I/O. Paths derived before any I/O therefore
build their own backend instead of sharing one made for the source; read
path.backendfirst, or passbackend=, to share a connection. - A backend that cannot be weakly referenced no longer turns off the
same-file guard. When_initbackend()returned adict, anobject()or
an instance of a__slots__class without"__weakref__", two separately
built paths to one URL were judged to be on different filesystems and
a.copy(b, overwrite=True)onto itself was not refused. A path now records
that it built its backend and passes the record on, so such a backend reads
as derived and the copy is refused. A backend given throughbackend=or
with_backend()still reads as supplied. - Dot segments are removed from a joined path, as the constructor removes
them. From 0.9.7 to 0.9.11,Uri("http://h/d/") / "../x"(astrjoin
onto a path ending in a slash) addressedhttp://h/d/x, because..was
resolved against the empty last segment, whileUri("http://h/d/../x")is
http://h/x, and a join with aUriargument (/ Uri("../x"),
Uri(base, "../x")) kepthttp://h/d/../xunresolved. All of them now give
http://h/x. The dot segments of the whole joined path are removed, so
Uri("http://h/d/x") / ".."...
v0.9.11
Added
- Native checksums on the asyncssh SFTP backend.
SftpPath.checksum()
andsupported_checksums()now send thecheck-file-handleextension on
AsyncsshSftpBackend(the backend chosen when asyncssh is installed), as
they already did on the paramiko backend. Against a server that implements
it (ProFTPD's mod_sftp, for example),PathSyncercompares server-side
digests instead of reading both files. OpenSSH does not implement it: the
refusal costs one request per connection, and files are compared by
streaming as before. A nativechecksum()is not bounded by the backend's
timeout, because the server hashes the whole file before it answers.
Changed
- The
uriextra now requiresnetimps>=0.3.1(was>=0.2.0). An
environment that pins an older netimps must raise that pin.
Full Changelog: v0.9.10...v0.9.11
v0.9.10
Added
Path._same_filesystem(other), an override hook answering whether two
paths of one type resolve their segments in the same place. Path equality
ignores that, andcopy(),move()andPathSyncernow ask it before
calling two paths the same file, nested, or overlapping. A path type whose
instances can front different hosts or stores should override it; the
default keeps the previous behaviour.pathlib_next.testing.PathContract
gains two tests: a file copied onto a separately built spelling of itself
is refused with its content intact, and paths under one root share a
filesystem.
Changed
- The built-in
Base*Backendclasses (s3,gs,az,ftp,sftp,
github/gitlab) are weakly referenceable, which is how aUriPath
tells a backend it derived for itself from one it was given. A custom
backend class using__slots__without"__weakref__"still works; it is
simply always read as supplied.
Fixed
- A path type that fronts several hosts can transfer between them again.
Since 0.9.4,copy()/move()raisedOSError(EINVAL)"Source and target
are the same file" andPathSyncerraised "source and target overlap" for
/app.confon one host onto/app.confon another, whenever the type kept
its connection under any attribute other than the private_backendthe
guards looked for. Such a type now overrides_same_filesystem(). - Two separately built URIs to the same URL are recognised as the same
file. Each derived its own backend on first use, which the guards read as
two hosts:UriPath(url).copy(UriPath(url), overwrite=True)was not
refused on any scheme withoutst_dev/st_ino(S3 and WebDAV among
them), and neither was a sync of a URL onto itself. Only two backends the
caller supplied now tell equal URIs apart. One consequence: a path given a
backend and a bare path to the same URL are now the same file for
copy()/move(), as they already were forPathSyncer. copy(recursive=True)'s "into itself" check now honours two supplied
backends, like the other two guards; it refused a copy between two hosts
whose URIs nested.
Full Changelog: v0.9.9...v0.9.10