Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 24 additions & 20 deletions Doc/library/struct.rst
Original file line number Diff line number Diff line change
Expand Up @@ -227,32 +227,32 @@ platform-dependent.
+--------+--------------------------+--------------------+----------------+------------+
| ``c`` | :c:expr:`char` | bytes of length 1 | 1 | |
+--------+--------------------------+--------------------+----------------+------------+
| ``b`` | :c:expr:`signed char` | integer | 1 | \(1), \(2) |
| ``b`` | :c:expr:`signed char` | int | 1 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``B`` | :c:expr:`unsigned char` | integer | 1 | \(2) |
| ``B`` | :c:expr:`unsigned char` | int | 1 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``?`` | :c:expr:`_Bool` | bool | 1 | \(1) |
+--------+--------------------------+--------------------+----------------+------------+
| ``h`` | :c:expr:`short` | integer | 2 | \(2) |
| ``h`` | :c:expr:`short` | int | 2 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``H`` | :c:expr:`unsigned short` | integer | 2 | \(2) |
| ``H`` | :c:expr:`unsigned short` | int | 2 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``i`` | :c:expr:`int` | integer | 4 | \(2) |
| ``i`` | :c:expr:`int` | int | 4 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``I`` | :c:expr:`unsigned int` | integer | 4 | \(2) |
| ``I`` | :c:expr:`unsigned int` | int | 4 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``l`` | :c:expr:`long` | integer | 4 | \(2) |
| ``l`` | :c:expr:`long` | int | 4 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``L`` | :c:expr:`unsigned long` | integer | 4 | \(2) |
| ``L`` | :c:expr:`unsigned long` | int | 4 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``q`` | :c:expr:`long long` | integer | 8 | \(2) |
| ``q`` | :c:expr:`long long` | int | 8 | \(2) |
+--------+--------------------------+--------------------+----------------+------------+
| ``Q`` | :c:expr:`unsigned long | integer | 8 | \(2) |
| ``Q`` | :c:expr:`unsigned long | int | 8 | \(2) |
| | long` | | | |
+--------+--------------------------+--------------------+----------------+------------+
| ``n`` | :c:type:`ssize_t` | integer | | \(3) |
| ``n`` | :c:type:`ssize_t` | int | | \(2), \(3) |
+--------+--------------------------+--------------------+----------------+------------+
| ``N`` | :c:type:`size_t` | integer | | \(3) |
| ``N`` | :c:type:`size_t` | int | | \(2), \(3) |
+--------+--------------------------+--------------------+----------------+------------+
| ``e`` | :c:expr:`_Float16` | float | 2 | \(4), \(6) |
+--------+--------------------------+--------------------+----------------+------------+
Expand All @@ -268,7 +268,7 @@ platform-dependent.
+--------+--------------------------+--------------------+----------------+------------+
| ``p`` | :c:expr:`char[]` | bytes | | \(8) |
+--------+--------------------------+--------------------+----------------+------------+
| ``P`` | :c:expr:`void \*` | integer | | \(5) |
| ``P`` | :c:expr:`void \*` | int | | \(2), \(5) |
+--------+--------------------------+--------------------+----------------+------------+

.. versionchanged:: 3.3
Expand Down Expand Up @@ -342,27 +342,31 @@ Notes:
The ``'p'`` format character encodes a "Pascal string", meaning a short
variable-length string stored in a *fixed number of bytes*, given by the count.
The first byte stored is the length of the string, or 255, whichever is
smaller. The bytes of the string follow. If the string passed in to
smaller. The bytes of the string follow. If the byte string passed in to
:func:`pack` is too long (longer than the count minus 1), only the leading
``count-1`` bytes of the string are stored. If the string is shorter than
``count-1`` bytes of the string are stored. If the byte string is shorter than
``count-1``, it is padded with null bytes so that exactly count bytes in all
are used. Note that for :func:`unpack`, the ``'p'`` format character consumes
``count`` bytes, but that the string returned can never contain more than 255
``count`` bytes, but that the :class:`!bytes` object returned can never contain more than 255
bytes.
When packing, arguments of types :class:`bytes` and :class:`bytearray`
are accepted.

(9)
For the ``'s'`` format character, the count is interpreted as the length of the
bytes, not a repeat count like for the other format characters; for example,
byte string, not a repeat count like for the other format characters; for example,
``'10s'`` means a single 10-byte string mapping to or from a single
Python byte string, while ``'10c'`` means 10
separate one byte character elements (e.g., ``cccccccccc``) mapping
to or from ten different Python byte objects. (See :ref:`struct-examples`
for a concrete demonstration of the difference.)
If a count is not given, it defaults to 1. For packing, the string is
If a count is not given, it defaults to 1. For packing, the byte string is
truncated or padded with null bytes as appropriate to make it fit. For
unpacking, the resulting bytes object always has exactly the specified number
of bytes. As a special case, ``'0s'`` means a single, empty string (while
unpacking, the resulting :class:`!bytes` object always has exactly the specified number
of bytes. As a special case, ``'0s'`` means a single, empty byte string (while
``'0c'`` means 0 characters).
When packing, arguments of types :class:`bytes` and :class:`bytearray`
are accepted.

(10)
For the ``'F'`` and ``'D'`` format characters, the packed representation uses
Expand Down
24 changes: 3 additions & 21 deletions Lib/shutil.py
Original file line number Diff line number Diff line change
Expand Up @@ -1317,27 +1317,9 @@ def _unpack_zipfile(filename, extract_dir):
if not zipfile.is_zipfile(filename):
raise ReadError("%s is not a zip file" % filename)

zip = zipfile.ZipFile(filename)
try:
for info in zip.infolist():
name = info.filename

# don't extract absolute paths or ones with .. in them
if name.startswith('/') or '..' in name:
continue

targetpath = os.path.join(extract_dir, *name.split('/'))
if not targetpath:
continue

_ensure_directory(targetpath)
if not name.endswith('/'):
# file
with zip.open(name, 'r') as source, \
open(targetpath, 'wb') as target:
copyfileobj(source, target)
finally:
zip.close()
with zipfile.ZipFile(filename) as zip:
zip._ignore_invalid_names = True
zip.extractall(extract_dir)

def _unpack_tarfile(filename, extract_dir, *, filter=None):
"""Unpack tar/tar.gz/tar.bz2/tar.xz/tar.zst `filename` to `extract_dir`
Expand Down
67 changes: 65 additions & 2 deletions Lib/test/test_shutil.py
Original file line number Diff line number Diff line change
Expand Up @@ -2136,8 +2136,6 @@ def test_make_zipfile_rootdir_nodir(self):
def check_unpack_archive(self, format, **kwargs):
self.check_unpack_archive_with_converter(
format, lambda path: path, **kwargs)
self.check_unpack_archive_with_converter(
format, FakePath, **kwargs)
self.check_unpack_archive_with_converter(format, FakePath, **kwargs)

def check_unpack_archive_with_converter(self, format, converter, **kwargs):
Expand Down Expand Up @@ -2194,6 +2192,71 @@ def test_unpack_archive_zip(self):
with self.assertRaises(TypeError):
self.check_unpack_archive('zip', filter='data')

def test_unpack_archive_zip_badpaths(self):
srcdir = self.mkdtemp()
zipname = os.path.join(srcdir, 'test.zip')
abspath = os.path.join(srcdir, 'abspath')
with zipfile.ZipFile(zipname, 'w') as zf:
zf.writestr(abspath, 'badfile')
zf.writestr(os.sep + abspath, 'badfile')
zf.writestr('/abspath', 'badfile')
zf.writestr('C:/abspath', 'badfile')
zf.writestr('D:\\abspath', 'badfile')
zf.writestr('E:abspath', 'badfile')
zf.writestr('F:/G:/abspath', 'badfile')
zf.writestr('//server/share/abspath', 'badfile')
zf.writestr('\\\\server2\\share\\abspath', 'badfile')
zf.writestr('../relpath', 'badfile')
zf.writestr(os.pardir + os.sep + 'relpath2', 'badfile')
zf.writestr('good/file', 'goodfile')
zf.writestr('good..file', 'goodfile')

dstdir = os.path.join(self.mkdtemp(), 'dst')
unpack_archive(zipname, dstdir)
self.assertTrue(os.path.isfile(os.path.join(dstdir, 'good', 'file')))
self.assertTrue(os.path.isfile(os.path.join(dstdir, 'good..file')))
self.assertFalse(os.path.exists(abspath))
self.assertFalse(os.path.exists(os.path.join(dstdir, 'abspath')))
self.assertFalse(os.path.exists(os.path.join(dstdir, 'G_')))
self.assertFalse(os.path.exists(os.path.join(dstdir, 'server')))
if os.name != 'nt':
self.assertTrue(os.path.isfile(os.path.join(dstdir, 'C:', 'abspath')))
self.assertTrue(os.path.isfile(os.path.join(dstdir, 'D:\\abspath')))
self.assertTrue(os.path.isfile(os.path.join(dstdir, 'E:abspath')))
self.assertTrue(os.path.isfile(os.path.join(dstdir, 'F:', 'G:', 'abspath')))
self.assertTrue(os.path.isfile(os.path.join(dstdir, '\\\\server2\\share\\abspath')))
if os.pardir == '..':
self.assertFalse(os.path.exists(os.path.join(dstdir, '..', 'relpath')))
self.assertFalse(os.path.exists(os.path.join(dstdir, 'relpath')))
else:
self.assertTrue(os.path.isfile(os.path.join(dstdir, '..', 'relpath')))
self.assertFalse(os.path.exists(os.path.join(dstdir, os.pardir, 'relpath2')))
self.assertFalse(os.path.exists(os.path.join(dstdir, 'relpath2')))

dstdir2 = os.path.join(self.mkdtemp(), 'dst')
os.mkdir(dstdir2)
with os_helper.change_cwd(dstdir2):
unpack_archive(zipname, '')
self.assertTrue(os.path.isfile(os.path.join('good', 'file')))
self.assertTrue(os.path.isfile('good..file'))
self.assertFalse(os.path.exists(abspath))
self.assertFalse(os.path.exists('abspath'))
self.assertFalse(os.path.exists('C_'))
self.assertFalse(os.path.exists('server'))
if os.name != 'nt':
self.assertTrue(os.path.isfile(os.path.join('C:', 'abspath')))
self.assertTrue(os.path.isfile('D:\\abspath'))
self.assertTrue(os.path.isfile('E:abspath'))
self.assertTrue(os.path.isfile(os.path.join('F:', 'G:', 'abspath')))
self.assertTrue(os.path.isfile('\\\\server2\\share\\abspath'))
if os.pardir == '..':
self.assertFalse(os.path.exists(os.path.join('..', 'relpath')))
self.assertFalse(os.path.exists('relpath'))
else:
self.assertTrue(os.path.isfile(os.path.join('..', 'relpath')))
self.assertFalse(os.path.exists(os.path.join(os.pardir, 'relpath2')))
self.assertFalse(os.path.exists('relpath2'))

def test_unpack_registry(self):

formats = get_unpack_formats()
Expand Down
21 changes: 16 additions & 5 deletions Lib/zipfile/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1410,6 +1410,7 @@ class ZipFile:

fp = None # Set here since __del__ checks it
_windows_illegal_name_trans_table = None
_ignore_invalid_names = False

def __init__(self, file, mode="r", compression=ZIP_STORED, allowZip64=True,
compresslevel=None, *, strict_timestamps=True, metadata_encoding=None):
Expand Down Expand Up @@ -1890,21 +1891,31 @@ def _extract_member(self, member, targetpath, pwd):

# build the destination pathname, replacing
# forward slashes to platform specific separators.
arcname = member.filename.replace('/', os.path.sep)

if os.path.altsep:
arcname = member.filename
if os.path.sep != '/':
arcname = arcname.replace('/', os.path.sep)
if os.path.altsep and os.path.altsep != '/':
arcname = arcname.replace(os.path.altsep, os.path.sep)
# interpret absolute pathname as relative, remove drive letter or
# UNC path, redundant separators, "." and ".." components.
arcname = os.path.splitdrive(arcname)[1]
drive, root, arcname = os.path.splitroot(arcname)
if self._ignore_invalid_names and (drive or root):
return None
if self._ignore_invalid_names and os.path.pardir in arcname.split(os.path.sep):
return None
invalid_path_parts = ('', os.path.curdir, os.path.pardir)
arcname = os.path.sep.join(x for x in arcname.split(os.path.sep)
if x not in invalid_path_parts)
if os.path.sep == '\\':
# filter illegal characters on Windows
arcname = self._sanitize_windows_name(arcname, os.path.sep)
arcname2 = self._sanitize_windows_name(arcname, os.path.sep)
if self._ignore_invalid_names and arcname2 != arcname:
return None
arcname = arcname2

if not arcname and not member.is_dir():
if self._ignore_invalid_names:
return None
raise ValueError("Empty filename.")

targetpath = os.path.join(targetpath, arcname)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Fix vulnerability in :func:`shutil.unpack_archive` for ZIP files on Windows
which allowed to write files outside of the destination tree if the patch in
the archive contains a Windows drive prefix. Now such invalid paths will be
skipped. Files containing ".." in the name (like "foo..bar") are no longer
skipped.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Fixed string table and sample record bounds checks in :mod:`!_remote_debugging`
when decoding certain ``.pyb`` inputs on 32-bit builds. Patch by Maurycy
Pawłowski-Wieroński.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update bundled `libexpat <https://libexpat.github.io/>`_ to version 2.8.0.
36 changes: 18 additions & 18 deletions Misc/sbom.spdx.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading