Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-101283: Try to load the fallback cmd.exe by an absolute path #101286

Merged
merged 25 commits into from
Feb 8, 2023

Conversation

arhadthedev
Copy link
Member

@arhadthedev arhadthedev commented Jan 24, 2023

@arhadthedev
Copy link
Member Author

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\a\cpython\cpython\build\test_python_2036�\test_python_worker_5676�'

Looks like a fluke; restarting.

@arhadthedev arhadthedev marked this pull request as ready for review January 24, 2023 13:46
Lib/subprocess.py Outdated Show resolved Hide resolved
Lib/subprocess.py Outdated Show resolved Hide resolved
Co-authored-by: Eryk Sun <eryksun@gmail.com>
Lib/subprocess.py Outdated Show resolved Hide resolved
@arhadthedev
Copy link
Member Author

@eryksun Thank you for improvements! Since none of my code is left, I've reattributed the news entry to you.

Lib/subprocess.py Outdated Show resolved Hide resolved
Co-authored-by: Eryk Sun <eryksun@gmail.com>
@eryksun eryksun requested a review from a team January 24, 2023 17:09
@arhadthedev arhadthedev marked this pull request as draft January 24, 2023 18:48
@arhadthedev arhadthedev marked this pull request as ready for review January 24, 2023 18:48
Lib/subprocess.py Outdated Show resolved Hide resolved
# cmd.exe is missing, or the system environment
# variables are broken, or they're undefined and the
# system is installed into a non-standard location.
# This is highly unlikely, and we cannot help here.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should a RuntimeWarning be raised or logging.warn call made in this case as an FYI?

Copy link
Contributor

@eryksun eryksun Jan 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea. How about this?

                        comspec = 'cmd.exe'
                        warnings.warn(f'spawning "{comspec}" using a '
                                      'relative file path',
                                      RuntimeWarning)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we keep the fallback at all rather than just make it an error, something like that. probably with a stacklevel= set.

related to my other comment, i wouldn't want to add a new warning in a security backport either.

@gpshead gpshead added type-security A security issue needs backport to 3.7 needs backport to 3.8 only security fixes needs backport to 3.9 only security fixes needs backport to 3.10 only security fixes labels Jan 24, 2023
@AlexWaygood AlexWaygood added needs backport to 3.11 only security fixes and removed needs backport to 3.11 only security fixes labels Feb 8, 2023
@miss-islington
Copy link
Contributor

Thanks @arhadthedev for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

@miss-islington
Copy link
Contributor

Sorry @arhadthedev and @zooba, I had trouble checking out the 3.11 backport branch.
Please retry by removing and re-adding the "needs backport to 3.11" label.
Alternatively, you can backport using cherry_picker on the command line.
cherry_picker 23751ed826ee63fb486e874ec25934ea87dd8519 3.11

@AlexWaygood AlexWaygood added needs backport to 3.11 only security fixes and removed needs backport to 3.11 only security fixes labels Feb 8, 2023
@miss-islington
Copy link
Contributor

Thanks @arhadthedev for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

@bedevere-bot
Copy link

GH-101711 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Feb 8, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Feb 8, 2023
…ue on Windows (pythonGH-101286)

(cherry picked from commit 23751ed)

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
@zooba
Copy link
Member

zooba commented Feb 8, 2023

Bah, just noticed in the backports that now there's a case where comspec may be used without being defined. We should fix and backport that immediately

@zooba
Copy link
Member

zooba commented Feb 8, 2023

Turns out I can commit directly to the backports, so those will be fine. But we need an else: comspec = executable branch in main

@bedevere-bot
Copy link

GH-101713 is a backport of this pull request to the 3.7 branch.

zooba pushed a commit to zooba/cpython that referenced this pull request Feb 8, 2023
@zooba
Copy link
Member

zooba commented Feb 8, 2023

GH-101712 is the comspec=executable fix for main.

@@ -111,6 +111,14 @@ underlying :class:`Popen` interface can be used directly.
Added the *text* parameter, as a more understandable alias of *universal_newlines*.
Added the *capture_output* parameter.

.. versionchanged:: 3.11.2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This version number is wrong. 3.11.2 was released before this change was merged. As this PR is in the main branch, listing it as 3.12 in these docs is more appropriate. Documentation readers should not be expected to know which patch release versions came before a major release.

We lack the ability to list all versions in versionchanged markers in our docs so have done it this way elsewhere as well when a visible API change has been made in patch releases.

@arhadthedev
Copy link
Member Author

Bah, just noticed in the backports that now there's a case where comspec may be used without being defined. We should fix and backport that immediately

This version number is wrong. 3.11.2 was released before this change was merged. As this PR is in the main branch, listing it as 3.12 in these docs is more appropriate. Documentation readers should not be expected to know which patch release versions came before a major release.

Got working on it; the PR will follow in a few hours.

@arhadthedev
Copy link
Member Author

Bah, just noticed in the backports that now there's a case where comspec may be used without being defined. We should fix and backport that immediately

To be fixed by gh-101712. Thank you, @zooba!

This version number is wrong. 3.11.2 was released before this change was merged. As this PR is in the main branch, listing it as 3.12 in these docs is more appropriate. Documentation readers should not be expected to know which patch release versions came before a major release.

Addressed in gh-101728.

ambv pushed a commit that referenced this pull request Feb 9, 2023
…ue on Windows (GH-101286) (#101709)

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
ambv pushed a commit that referenced this pull request Feb 9, 2023
…ue on Windows (GH-101286) (#101710)

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
ambv added a commit that referenced this pull request Feb 9, 2023
…ue on Windows (GH-101286) (#101713)

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Oleg Iarygin <dralife@yandex.ru>
carljm added a commit to carljm/cpython that referenced this pull request Feb 9, 2023
* main: (82 commits)
  pythongh-101670: typo fix in PyImport_ExtendInittab() (python#101723)
  pythonGH-99293: Document that `Py_TPFLAGS_VALID_VERSION_TAG` shouldn't be used. (#pythonGH-101736)
  no-issue: Add Dong-hee Na as the cjkcodecs codeowner (pythongh-101731)
  pythongh-101678: Merge math_1_to_whatever() and math_1() (python#101730)
  pythongh-101678: refactor the math module to use special functions from c11 (pythonGH-101679)
  pythongh-85984: Remove legacy Lib/pty.py code. (python#92365)
  pythongh-98831: Use opcode metadata for stack_effect() (python#101704)
  pythongh-101283: Version was just released, so should be changed in 3.11.3 (pythonGH-101719)
  pythongh-101283: Fix use of unbound variable (pythonGH-101712)
  pythongh-101283: Improved fallback logic for subprocess with shell=True on Windows (pythonGH-101286)
  pythongh-101277: Port more itertools static types to heap types (python#101304)
  pythongh-98831: Modernize CALL and family (python#101508)
  pythonGH-101696: invalidate type version tag in `_PyStaticType_Dealloc` (python#101697)
  pythongh-100221: Fix creating dirs in `make sharedinstall` (pythonGH-100329)
  pythongh-101670: typo fix in PyImport_AppendInittab() (pythonGH-101672)
  pythongh-101196: Make isdir/isfile/exists faster on Windows (pythonGH-101324)
  pythongh-101614: Don't treat python3_d.dll as a Python DLL when checking extension modules for incompatibility (pythonGH-101615)
  pythongh-100933: Improve `check_element` helper in `test_xml_etree` (python#100934)
  pythonGH-101578: Normalize the current exception (pythonGH-101607)
  pythongh-47937: Note that Popen attributes are read-only (python#93070)
  ...
carlosroman added a commit to DataDog/cpython that referenced this pull request Jun 22, 2023
* Post 3.8.16

* [3.8] Update copyright years to 2023. (pythongh-100852)

* [3.8] Update copyright years to 2023. (pythongh-100848).
(cherry picked from commit 11f9932)

Co-authored-by: Benjamin Peterson <benjamin@python.org>

* Update additional copyright years to 2023.

Co-authored-by: Ned Deily <nad@python.org>

* [3.8] Update copyright year in README (pythonGH-100863) (pythonGH-100867)

(cherry picked from commit 30a6cc4)

Co-authored-by: Ned Deily <nad@python.org>
Co-authored-by: HARSHA VARDHAN <75431678+Thunder-007@users.noreply.github.com>

* [3.8] Correct CVE-2020-10735 documentation (pythonGH-100306) (python#100698)

(cherry picked from commit 1cf3d78)
(cherry picked from commit 88fe8d7)

Co-authored-by: Jeremy Paige <ucodery@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>

* [3.8] Bump Azure Pipelines to ubuntu-22.04 (pythonGH-101089) (python#101215)

(cherry picked from commit c22a55c)

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

* [3.8] pythongh-100180: Update Windows installer to OpenSSL 1.1.1s (pythonGH-100903) (python#101258)

* pythongh-101422: (docs) TarFile default errorlevel argument is 1, not 0 (pythonGH-101424)

(cherry picked from commit ea23271)

Co-authored-by: Owain Davies <116417456+OTheDev@users.noreply.github.com>

* [3.8] pythongh-95778: add doc missing in some places (pythonGH-100627) (python#101630)

(cherry picked from commit 4652182)

* [3.8] pythongh-101283: Improved fallback logic for subprocess with shell=True on Windows (pythonGH-101286) (python#101710)

Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>

* [3.8] pythongh-101981: Fix Ubuntu SSL tests with OpenSSL (3.1.0-beta1) CI i… (python#102095)

[3.8] pythongh-101981: Fix Ubuntu SSL tests with OpenSSL (3.1.0-beta1) CI issue (pythongh-102079)

* [3.8] pythonGH-102306 Avoid GHA CI macOS test_posix failure by using the appropriate macOS SDK (pythonGH-102307)

[3.8] Avoid GHA CI macOS test_posix failure by using the appropriate macOS SDK.

* [3.8] pythongh-101726: Update the OpenSSL version to 1.1.1t (pythonGH-101727) (pythonGH-101752)

Fixes CVE-2023-0286 (High) and a couple of Medium security issues.
https://www.openssl.org/news/secadv/20230207.txt

Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Ned Deily <nad@python.org>

* [3.8] pythongh-102627: Replace address pointing toward malicious web page (pythonGH-102630) (pythonGH-102667)

(cherry picked from commit 61479d4)

Co-authored-by: Blind4Basics <32236948+Blind4Basics@users.noreply.github.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

* [3.8] pythongh-101997: Update bundled pip version to 23.0.1 (pythonGH-101998). (python#102244)

(cherry picked from commit 89d9ff0)

* [3.8] pythongh-102950: Implement PEP 706 – Filter for tarfile.extractall (pythonGH-102953) (python#104548)

Backport of c8c3956

* [3.8] pythongh-99889: Fix directory traversal security flaw in uu.decode() (pythonGH-104096) (python#104332)

(cherry picked from commit 0aeda29)

Co-authored-by: Sam Carroll <70000253+samcarroll42@users.noreply.github.com>

* [3.8] pythongh-104049: do not expose on-disk location from SimpleHTTPRequestHandler (pythonGH-104067) (python#104121)

Do not expose the local server's on-disk location from `SimpleHTTPRequestHandler` when generating a directory index. (unnecessary information disclosure)

(cherry picked from commit c7c3a60)

Co-authored-by: Ethan Furman <ethan@stoneleaf.us>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

* [3.8] pythongh-103935: Use `io.open_code()` when executing code in trace and profile modules (pythonGH-103947) (python#103954)

Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>

* [3.8] pythongh-68966: fix versionchanged in docs (pythonGH-105299)

* [3.8] Update GitHub CI workflow for macOS. (pythonGH-105302)

* [3.8] pythongh-105184: document that marshal functions can fail and need to be checked with PyErr_Occurred (pythonGH-105185) (python#105222)

(cherry picked from commit ee26ca1)

Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

* [3.8] pythongh-102153: Start stripping C0 control and space chars in `urlsplit` (pythonGH-102508) (pythonGH-104575) (pythonGH-104592) (python#104593) (python#104895)

`urllib.parse.urlsplit` has already been respecting the WHATWG spec a bit pythonGH-25595.

This adds more sanitizing to respect the "Remove any leading C0 control or space from input" [rule](https://url.spec.whatwg.org/GH-url-parsing:~:text=Remove%20any%20leading%20and%20trailing%20C0%20control%20or%20space%20from%20input.) in response to [CVE-2023-24329](https://nvd.nist.gov/vuln/detail/CVE-2023-24329).

I simplified the docs by eliding the state of the world explanatory
paragraph in this security release only backport.  (people will see
that in the mainline /3/ docs)

(cherry picked from commit d7f8a5f)
(cherry picked from commit 2f630e1)
(cherry picked from commit 610cc0a)
(cherry picked from commit f48a96a)

Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Co-authored-by: Illia Volochii <illia.volochii@gmail.com>
Co-authored-by: Gregory P. Smith [Google] <greg@krypto.org>

* [3.8] pythongh-103142: Upgrade binary builds and CI to OpenSSL 1.1.1u (pythonGH-105174) (pythonGH-105200) (pythonGH-105205) (python#105370)

Upgrade builds to OpenSSL 1.1.1u.

Also updates _ssl_data_111.h from OpenSSL 1.1.1u, _ssl_data_300.h from 3.0.9.

Manual edits to the _ssl_data_300.h file prevent it from removing any
existing definitions in case those exist in some peoples builds and were
important (avoiding regressions during backporting).

(cherry picked from commit ede89af)
(cherry picked from commit e15de14)

Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Ned Deily <nad@python.org>

* Python 3.8.17

* Post 3.8.17

* Updated CI to build 3.8.17

---------

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Benjamin Peterson <benjamin@python.org>
Co-authored-by: Ned Deily <nad@python.org>
Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Co-authored-by: HARSHA VARDHAN <75431678+Thunder-007@users.noreply.github.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Co-authored-by: Jeremy Paige <ucodery@gmail.com>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Co-authored-by: Steve Dower <steve.dower@python.org>
Co-authored-by: Owain Davies <116417456+OTheDev@users.noreply.github.com>
Co-authored-by: Éric <earaujo@caravan.coop>
Co-authored-by: Oleg Iarygin <oleg@arhadthedev.net>
Co-authored-by: Steve Dower <steve.dower@microsoft.com>
Co-authored-by: Dong-hee Na <donghee.na@python.org>
Co-authored-by: Blind4Basics <32236948+Blind4Basics@users.noreply.github.com>
Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Pradyun Gedam <pradyunsg@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Sam Carroll <70000253+samcarroll42@users.noreply.github.com>
Co-authored-by: Ethan Furman <ethan@stoneleaf.us>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>
Co-authored-by: stratakis <cstratak@redhat.com>
Co-authored-by: Illia Volochii <illia.volochii@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-security A security issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants