[SPARK-57962][PYTHON] Guard against path traversal in install_spark tar extraction

[peter-toth]

What changes were proposed in this pull request?

install_spark() in python/pyspark/install.py downloads a Spark release archive and extracts it by rewriting each tar member's name with os.path.relpath(member.name, package_name + os.path.sep) and calling tar.extract(member, dest). os.path.relpath does not strip .. segments (it can produce them), so a crafted archive member could resolve to a path outside dest (a "zip slip" / path traversal) when extracted.

This change extracts the loop into a helper _extract_tar that, before extracting each member, resolves its destination (os.path.realpath(os.path.join(dest, member.name))) and rejects any member that would land outside dest. The behavior for legitimate archives is unchanged.

tarfile's filter='data' (PEP 706) rejects such members natively and would replace this manual check, but it is only generally available from Python 3.12.0 (backported to 3.11.4+), so we keep the explicit check while Spark still supports Python 3.11, and leave a note to revisit once the minimum supported Python is >= 3.12.

Why are the changes needed?

install_spark() runs at pip install pyspark time (when PYSPARK_HADOOP_VERSION/PYSPARK_HIVE_VERSION is set) and downloads from an Apache mirror or a user-supplied PYSPARK_RELEASE_MIRROR. Extracting archive members without a containment check means a member path containing .. would be written outside the intended destination directory. Adding an explicit resolved-path check is a straightforward robustness improvement that works on all supported Python versions.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New unit test test_extract_tar in python/pyspark/tests/test_install_spark.py that (1) extracts a benign archive with the top-level package directory stripped, and (2) asserts a member whose path escapes the destination directory is rejected and nothing is written.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

[dongjoon-hyun]

Could you double-check this of the PR description, @peter-toth ? AFAIK, data filter is backported to Python 3.11.4, 3.10.12, 3.9.12. Given that, this PR had better use data filter.

the built-in tarfile extraction filter='data' is only available on 3.12+, while Spark supports 3.11+

[peter-toth]

Thanks @dongjoon-hyun, you're right - I confirmed tarfile.data_filter is present in 3.11.13, and per PEP 706 it was backported to 3.11.4 / 3.10.12 / 3.9.17, so it's available on all supported versions. I've switched to filter='data' (which also covers absolute paths, unsafe links, and permission bits) and corrected the PR description.

[dongjoon-hyun]

+1, LGTM. Thank you, @peter-toth .

[gaogaotiantian]

We can't do this unconditionally just because the latest 3.11.x supports it. For python users, it's very common to stay at a lower micro level (Python does not auto-upgrade). For those users, this will become an installation failure. They wouldn't know how to fix it. They would just blame spark.

We need to either

1. Wait until all the supported python version supports it (we don't have restriction on micro versions)
2. Check python version first before using the extra argument or has a fallback for the exception

[dongjoon-hyun]

We can't do this unconditionally just because the latest 3.11.x supports it. For python users, it's very common to stay at a lower micro level (Python does not auto-upgrade). For those users, this will become an installation failure. They wouldn't know how to fix it. They would just blame spark.

We need to either

1. Wait until all the supported python version supports it (we don't have restriction on micro versions)
2. Check python version first before using the extra argument or has a fallback for the exception

In general, you are right. However, I'm not sure that is still a valid assumption for this improvement JIRA issue for Apache Spark 5.0.0 scheduled in 2027. Could you elaborate the mentioned comment environments a little more, please, @gaogaotiantian ?

IIRC, Python 3.11.4, 3.10.12, 3.9.17 were released on 2023-06-06. It's already over 3 years and one month ago.

[gaogaotiantian]

If this is for 5.0.0 rather than 4.3.0, it might be much safer. 3.11 will be retired on Oct. 2027.

My concern is that for some Python users, they download their minor Python version (3.11) once and never upgrade it. It's not that uncommon for Python users to stay at a certain version. For those users, when they upgrade pyspark version, they'll hit an error about tarfile and be very confused. From their point of view, they just upgraded pyspark version which worked perfectly with their current environment, and it failed to work after the upgrade.

This may not impact a huge user base, but it's still unnecessary. I think a simple version check sys.version_info >= (3, 11, 4) would do a lot. We can either fallback to the tarfile method without the parameter, or raise a more explicit error asking the user to upgrade their python version, rather than fail with a confusing tarfile exception about not having the parameter.

[dongjoon-hyun]

May I ask if it's only your imagination or you are trying to refer some real production environments?

My concern is that for some Python users, they download their minor Python version (3.11) once and never upgrade it.

To be honest with you, I don't think it's a usable or real production system, @gaogaotiantian . If you are aware of any systems, you had better inform them to upgrade Python first instead of trying install new Spark there. That's the Python and Apache Spark community recommendation.

For example, let's say that they are using Python 3.11.3 (April 5, 2023) in their production environment. They have the following issues.

Correctness/Crash/Hang Patches

Python 3.11.4 (June 6, 2023)

• gh-104615 — Compiler generated wrong ordering of assignments in code like a, a = x, y. Silently produces incorrect results — the most dangerous class of bug.
• gh-103590 — try-except* wrongly wrapped a single exception in an ExceptionGroup, breaking exception-handling logic.
• gh-104035 — Slotted frozen dataclasses ignored user-defined getstate/setstate, silently breaking custom pickling.
• gh-102818 — sys.setprofile/sys.settrace duplicated or omitted frames in tracebacks when the callback raised.
• gh-103987 — mmap could access memory-mapped files after invalidation (crash).
• gh-104265 — Instantiating _csv.Reader/_csv.Writer directly crashed the interpreter (3.10 regression).
• gh-104536 — Race condition in multiprocessing process cleanup causing spurious AttributeError in process.close().
• gh-96522 — Potential deadlock in pty.spawn().
• gh-87474 — File descriptor leaks in subprocess.Popen.

Python 3.11.5 (August 24, 2023)

• gh-106092 — Segfault from a use-after-free in frame deallocation (trashcan mechanism).
• gh-105840 — Crashes when specializing function calls with too many defaults.
• gh-105588 — Crashes when compiling malformed ast nodes.
• gh-104432 — Unaligned memory access in grp/socket C APIs (visible on ARM macOS).
• gh-94777 — ProcessPoolExecutor hung forever if a child process crashed while data was being written to the call queue.
• gh-106684 — asyncio.StreamWriter leaked memory when not explicitly closed.
• gh-105375 — Exceptions could be silently overwritten in builtins, pickle, _datetime, _ssl, array, _ctypes, and zoneinfo.
• gh-105332 — Reverted enum pickling from by-name back to by-value (unpickling compatibility regression).
• gh-103171 — Reverted an undocumented behavior change in runtime-checkable typing protocols.

Python 3.11.6 (October 2, 2023)

• gh-105829 — concurrent.futures.ProcessPoolExecutor deadlock.
• gh-109593 — Deadlock in the multiprocessing resource tracker on reentrant calls.
• gh-107219 — Race condition in concurrent.futures when a process was terminated abruptly.
• gh-108987 — _thread.start_new_thread() race during interpreter finalization.
• gh-104372 — subprocess with vfork() blocked other threads; GIL now released while waiting for child exec.
• gh-108520 — multiprocessing.SemLock initialization regression affecting nested processes (locks passed between processes).
• gh-110038 — KqueueSelector.select() missed ready events when an fd was registered for both read and write (affects asyncio on BSD/macOS).
• gh-109631 — re functions (findall, split, sub) could not be interrupted by Ctrl-C.
• gh-110036 — multiprocessing on Windows: Popen.terminate() mishandled PermissionError.

Python 3.11.7 (December 4, 2023)

• gh-112618 — typing.Annotated caching bug: Annotated[str, True] and Annotated[str, 1] were treated as identical (silent type-metadata corruption).
• gh-112509 — typing.TypedDict keys could appear in both required_keys and optional_keys.
• gh-110196 — Pickling ipaddress.IPv6Address silently lost the scope_id.
• gh-102388 — Out-of-bounds read in iso2022_jp_3/iso2022_jp_2004 codecs.
• gh-73561 — http.client sent the IPv6 scope suffix in the Host: header (protocol violation).
• gh-112625 — Use-after-free: bytearray.join() could read freed memory if the bytearray was cleared during iteration.
• gh-111942 — Crashes in io.TextIOWrapper with invalid arguments.
• gh-109786 — Crashes and reference leaks when re-entering itertools.pairwise.next().
• gh-110378 — contextmanager()/asynccontextmanager() failed to close generators that yielded more than once.
• gh-109216 — Memory leak in the BUILD_MAP bytecode instruction.

Python 3.11.8 (February 6, 2024)

• gh-105967 — zlib.crc32() and binascii.crc32() returned incorrect checksums on multi-gigabyte inputs on macOS. Directly relevant to data-integrity pipelines.
• gh-104522 — OSError from subprocess wrongly attributed failures to cwd (misleading error attribution).
• Crashes / races / hangs:
• gh-106905 — Race condition in multithreaded ast.parse() (global parser state shared across threads).
• gh-113566 — 3.11-specific crash when repr() of an asyncio.Future was requested after module GC.
• gh-89811 — Bytecode-specialization crash (missing type-version validity check).
• gh-79325 — Infinite recursion in tempfile.TemporaryDirectory() cleanup on Windows.
• gh-114887 — Regression: asyncio.create_datagram_endpoint() rejected raw (non-datagram) sockets.
• gh-113214 — AttributeError during asyncio SSL aborts in SSL-over-SSL scenarios.
• gh-113280 / gh-109534 — Socket and reference leaks in ssl/asyncio SSL handshake error paths.

Python 3.11.9 (April 2, 2024)

• gh-96497 — Name mangling of class variables resolved incorrectly in assignment expressions (walrus) inside comprehensions.
• gh-116957 — configparser left values in an invalid state (list instead of str) after duplicate-key errors.
• gh-114563 — Three Decimal.format bugs: memory leak with z option, wrong output with F + z, wrong # alternate form.
• gh-95782 — io.BufferedReader/BufferedRandom tell()/seek() could return negative offsets.
• gh-115243 — Crash in collections.deque.index() under concurrent modification.
• gh-114572 — Race in ssl.SSLContext.cert_store_stats()/get_ca_certs() with thread-shared contexts.
• gh-114763 — importlib.util.LazyLoader race during concurrent attribute access.
• gh-117178 — Regression in lazy loading of self-referential modules.
• gh-115618 — Incorrect refcount decrease for None in property.getter()/setter()/deleter().

CVE Patches

Python 3.11.4 (June 6, 2023)

• CVE-2023-24329 — urllib.parse.urlsplit() allowed URLs beginning with C0 control characters or spaces to bypass scheme/host blocklist checks (SSRF / filter bypass). (gh-102153)
• CVE-2007-4559 mitigation (PEP 706) — tarfile.extractall() / shutil.unpack_archive() gained the filter= argument to block path-traversal tar members. The safe-extraction API does not exist before 3.11.4. (gh-102953)
• gh-99889 — uu.decode() directory traversal: a crafted input could write files to arbitrary paths when no output file was specified.
• gh-104049 — http.server.SimpleHTTPRequestHandler disclosed the local on-disk path in generated directory indexes.
• gh-103142 — Bundled OpenSSL upgraded to 1.1.1u in binary installers (rolls up multiple OpenSSL CVEs).

Python 3.11.5 (August 24, 2023)

• CVE-2023-40217 — ssl.SSLSocket TLS handshake bypass: unencrypted data sent before the handshake could be treated as post-handshake encrypted data, bypassing certificate verification. (gh-108310)
• CVE-2023-41105 — os.path.normpath() silently truncated paths at embedded null bytes, allowing path-validation bypass. This is a regression that exists only in 3.11.0–3.11.4. (gh-106242)

Python 3.11.8 (February 6, 2024)

• CVE-2023-6597 — tempfile.TemporaryDirectory cleanup dereferenced symlinks, allowing modification of permissions on files outside the temp directory. (gh-91133)
• CVE-2024-0450 — zipfile "quoted-overlap" zip-bomb: overlapping entries now raise BadZipFile instead of enabling resource-exhaustion attacks. (gh-109858)
• gh-113659 — .pth files with hidden-file names are now skipped (arbitrary-code-execution hardening at interpreter startup).

Python 3.11.9 (April 2, 2024)

• CVE-2023-52425 — Bundled libexpat updated to 2.6.0; new APIs to control Expat reparse deferral (XML denial-of-service). (gh-115398, gh-115399)
• gh-114572 — ssl.SSLContext.cert_store_stats() / get_ca_certs() race condition when the context is shared across threads (memory corruption / crash).
• gh-115243 — Crash in collections.deque.index() under concurrent modification.

Python 3.11.10 (September 7, 2024)

• CVE-2024-28757, CVE-2024-45490, CVE-2024-45491, CVE-2024-45492 — Bundled libexpat upgraded to 2.6.3. (gh-123678, gh-116741)
• CVE-2024-4030 — On Windows, os.mkdir(mode=0o700) did not actually restrict the directory to the current user. (gh-118486)
• CVE-2024-4032 — ipaddress misclassified some private/global IPv4 and IPv6 ranges. (gh-113171)
• CVE-2024-6232 — ReDoS (quadratic regex complexity) when parsing crafted tarfile headers. (gh-121285)
• CVE-2024-6923 — email did not quote newlines in message headers, enabling header injection. (gh-121650)
• CVE-2024-7592 — Quadratic-complexity DoS when parsing crafted cookies in http.cookies. (gh-123067)
• CVE-2024-8088 — Infinite loop in zipfile.Path when reading crafted zip archives. (gh-123270)
• CVE-2023-27043 (follow-up) — email.utils.getaddresses() / parseaddr() gained a strict= parameter to reject malformed addresses. (gh-102988)
• Plus hardening: socket.socketpair() fallback authentication (gh-122133), missing audit events (gh-121957).

Python 3.11.11 (December 3, 2024)

• CVE-2024-9287 — venv activation scripts did not quote paths, allowing command injection via a crafted venv path. (gh-124651)
• CVE-2024-50602 — Bundled libexpat upgraded to 2.6.4 (XML DoS). (gh-126623)
• gh-122792 — IPv4-mapped ipaddress.IPv6Address now consistently uses the mapped IPv4 address (classification-bypass hardening).

Python 3.11.12 (April 8, 2025)

• CVE-2025-0938 — urllib.parse.urlparse() accepted domain names containing [ or ], diverging from RFC 3986 and enabling SSRF via parser differentials. (gh-105704)
• gh-119511 — Out-of-memory DoS in imaplib when reading a malicious server's response.
• gh-131261, gh-131809 — Bundled libexpat upgraded to 2.7.0/2.7.1 (covers expat CVE-2024-8176, stack-overflow DoS).
• gh-80222, gh-121284 — email quoted-string folding fixes (address-spoofing hardening).
• Python 3.11.13 (June 3, 2025)
• CVE-2024-12718, CVE-2025-4138, CVE-2025-4330, CVE-2025-4435, CVE-2025-4517 — Multiple bypasses of the tarfile extraction filters (filter="data" / filter="tar") using crafted symlinks and hard links. (gh-135034)
• CVE-2025-4516 (gh-133767) — Use-after-free in the unicode-escape decoder with a non-strict error handler.
• gh-128840 — Excessive memory consumption / DoS from overly long IPv6 addresses in ipaddress.

Python 3.11.14 (October 2025)

• CVE-2025-6069 (gh-135462) — Quadratic-complexity DoS in html.parser.HTMLParser, plus several related HTMLParser parsing fixes (gh-135661, gh-102555, gh-118350, gh-86155).
• CVE-2025-8194 (gh-130577) — tarfile now validates that member offsets are non-negative (infinite-loop DoS with crafted archives).
• CVE-2025-59375 — Bundled libexpat upgraded to 2.7.3 (allocation DoS). (gh-139312)
• CVE-2025-47273, CVE-2024-6345 — ensurepip's bundled setuptools updated to 79.0.1 (path traversal / code injection in setuptools). (gh-135374)
• gh-139700 — zipfile zip64 end-of-central-directory consistency checks (crafted-archive hardening).

Python 3.11.15 (March 3, 2026) — latest as of today

• CVE-2026-24515, CVE-2026-25210 — Bundled libexpat upgraded (XML DoS). (gh-144363)
• CVE-2025-59375 follow-up — Expat allocation-tracker APIs for large-allocation DoS mitigation. (gh-90949)
• CVE-2024-6923 follow-up — Additional email comment-folding fix preventing header injection. (gh-144125, gh-143935)
• Hardening batch: control-character rejection in wsgiref.headers, http.cookies.Morsel, and data: URL media types (gh-143916, gh-143919, gh-143925); memory-DoS/complexity fixes (gh-119342, gh-119451, gh-119452, gh-136065); use-after-free and out-of-bounds fixes in core/SSL (gh-144833, gh-120298, gh-120384).

[gaogaotiantian]

Python 3.8 has been end of life for more than 2 years yet we still ran tests with it at least until a few weeks ago. We still use Python 3.9 as the major python version to test spark 3.5 and it has been EOL for about a year too. I can list a bunch of unpatched CVEs for Python 3.9 too because it's not maintained anymore. It's not my imagination that Apache Spark is still using it in their github actions. I won't be surprised at all if there are some users that have a pinned Python version that's old because that happens to be the version when they pinned it.

It's about what we claim to support and what we do support. I'm okay if we say in our documentation that we only support Python 3.11 that's greater than 3.11.4 - that's fine. But if we claim to support 3.11, we should not have something in our install script that won't work in 3.11.0.

More importantly, what do we lose if we check the version? Would it be a worse experience for users if we generate an exception with a better explanation of what's happening and how to deal with it? Will the code be less maintainable with the extra if statement which could be easily removed when we do not support 3.11 anymore? What's the reason not to do the safety check when we know that there are cases where this won't work? Because we are absolutely sure that there is no user out there that's using old Python version? Or we should not care about them because they are not serious users that upgrade their Python version frequently?

[peter-toth]

How about merging the previous version (4435bbd) of the PR to all active branches now and once we declared minimum version >= 3.12 we upgrade the code to its current form?
We can just leave a note that this can be handled using data filter nicely.

[gaogaotiantian]

I'm curious about the original issue. You mentioned that os.path.relpath does not resolve .., but you are using the exact method to resolve it.

I tested it on my machine

>>> os.path.realpath("/usr/../bin")
'/bin'

So what's the issue? How to reproduce this problem?

[peter-toth]

@gaogaotiantian the snippet tests os.path.realpath, but the extract loop uses os.path.relpath (to strip the leading spark-x/ package dir), and relpath emits .. rather than resolving it:

>>> os.path.relpath("spark-x/../../etc/evil", "spark-x/")
'../../etc/evil'

So member.name still contains .. when it reaches tar.extract(member, dest) (no filter), which writes outside dest. Repro:

import io, os, tarfile, tempfile
with tempfile.TemporaryDirectory() as tmp:
    dest = os.path.join(tmp, "dest"); os.makedirs(dest)
    p = os.path.join(tmp, "m.tar")
    with tarfile.open(p, "w") as t:
        info = tarfile.TarInfo("spark-x/../evil.txt"); info.size = 5
        t.addfile(info, io.BytesIO(b"pwned"))
    with tarfile.open(p) as t:
        for m in t.getmembers():
            m.name = os.path.relpath(m.name, "spark-x" + os.path.sep)
            t.extract(m, dest)
    print(os.path.exists(os.path.join(tmp, "evil.txt")))  # True -> escaped dest

realpath does resolve .. - which is exactly why a realpath(os.path.join(dest, name))-stays-under-realpath(dest) check is an effective, version-independent guard.

[gaogaotiantian]

@peter-toth oh it's my mistake. Did not realize that's relpath and realpath. Then yes, I think it's okay to have a universal solution like this for now and switch to the method when we drop 3.11 support.

Stale review opinion.

[dongjoon-hyun]

For this one, it seems that I didn't add a value here. I dropped my previous comment and leave it to you, @peter-toth and @gaogaotiantian . Feel free to proceed in your ways.

[peter-toth]

I've reverted the filter='data' change back to the previous general version and added a comment.

Agreed filter='data' is the nicest fix - it handles all of this natively. The only problem is that technically it might not be available in every Python version we support - it's missing in older patch releases like 3.11.3 and earlier. As @dongjoon-hyun said, these are unlikely to still be in use, but it might happen.

[gaogaotiantian]

I think we can keep it this way for both master and branch-4.x for now. spark 5.0 is supposed to be released next year and it's possible that we still want to support python 3.11 for spark 5.0. The missing candidate pool would be really small (just 3.11.0-3.11.3) then but it's still there. The ideal way is to get rid of it when we drop 3.11.

The current implementation might not be the cleanest, but it's not horrible either. Just a few extra lines of code. We also have a really nice note there for the next step in the future so even if all of us forgot about it there will be someone to pick it up and it would be easy.

My suggestion for now is to keep the master and branch-4.x sync.

[peter-toth]

@gaogaotiantian , @dongjoon-hyun may I get an approval for the current state?

[peter-toth]

Merge Summary:

• merged into master d61f250
• merged into branch-4.x 1593edd
• merged into branch-4.2 fc46f2a
• merged into branch-4.1 b32d80d
• merged into branch-4.0 833224c
• merged into branch-3.5 4596d25

Posted by merge_spark_pr.py

[peter-toth]

Thank you @dongjoon-hyun and @gaogaotiantian for the review.