-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[python3] Update to 3.11.4 #31727
[python3] Update to 3.11.4 #31727
Conversation
ports/python3/portfile.cmake
Outdated
vcpkg_find_acquire_program(PYTHON3) | ||
get_filename_component(PYTHON3_DIR "${PYTHON3}" DIRECTORY) | ||
set(ENV{PythonForBuild} "${PYTHON3_DIR}/python.exe") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why can't we do this unconditionally? vcpkg_find_acquire_program(PYTHON3)
should give us a Python that will run on the host 😕.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is python3 now a bootstrapping requirement for native builds?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Hoikas we can. I'm just little worried whether vcpkg_find_acquire_program
will always give us python of required version (right now it's 3.10 which is ok for bootstrapping)
@dg0yt I think it started with this commit python/cpython@09b4ad1
Theoretically this change can be omitted if we allow find_python.bat
to use system python or download it from nuget (failed in CI for some reason).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If python is needed now, there is only vcpgk_find_acquire_program
(if version requirements are relaxed) or vcpkg_download_distfile
for a specific download.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is python3 now a bootstrapping requirement for native builds?
Python has needed a working python interpreter for its own build since before Python 3.7.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
potential build issues in ports that depend on Python 3 when the interpreter from
vcpkg_find_acquire_program(PYTHON3)
does not match the libraries from the python3 port.
Hmm could you give an example? Or confirm that my understanding is correct:
- Port A depends on
python3
and produces python modules - Port B uses
vcpkg_find_acquire_program(PYTHON3)
and uses this interpreter to import modules from port A.
Shouldn't it use python from tools instead (probably with patches from [python3] Some patches up for discussion #31279)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This case would be that port foo depends on python3 to build Python modules. It also depends on python.exe as part of its build process to perform operation bar at build time. Currently, vcpkg uses the python.exe provided by vcpkg_find_acquire_program(PYTHON3)
for this. If the port's upstream CMake requests the development modules and Python interpreter simultaneously and vcpkg_find_acquire_program(PYTHON3)
and python3 do not match, CMake will reject this. The results of this rejection are difficult to determine but could include failing the port build or finding the system Python.
Option 2 is probably closest to this situation, but we don't have a Python tool port yet. That should be addressed by #31748, if I read it correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm just little worried whether
vcpkg_find_acquire_program
will always give us python of required version (right now it's 3.10 which is ok for bootstrapping)
This is not correct. vcpkg_find_acquire_program will give you a version of python that you can use as an interpreter that satisfies constraints. On most POSIX, for example, it's going to give you /usr/bin/python3
which will basically never have a version that matches what the port wants to build from source. Note that making vcpkg-tool-python3
won't change anything about this situation: we have both scenarios that want to
- load in to python, where the versions of everything need to match and thus we must build from source, and
- just want to use python to run a
.py
, which usually do not want to build python from source, and thus are going to get whatever the system says - (There's another scenario about embedding python with an application that looks closer to the first one, but I don't have a good understanding of what's going on there)
Ports need to make the judgement call about which of these cases they are in, and depend on the python port in the first case and call vcpkg_find_acquire_program in the second.
The guarantee that you appear to be attempting to make here, that the 'tool' copy of python will match the 'built from source' copy of python, is not a guarantee that we are capable of making. The problem is that we don't have a good enough understanding of what you are trying to do with that guarantee to suggest what the right approach going forward is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Port B uses
vcpkg_find_acquire_program(PYTHON3)
and uses this interpreter to import modules from port A.
Port B cannot do this. vcpkg_find_acquire_program(PYTHON3)
does not give a version of python that has any relationship with the modules built by Port A. Port B must also depend on the python3
port and use the interpreter built by it, not what vcpkg_find_acquire_program(PYTHON3)
says.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add'l note onto what @BillyONeal said above: Port B may need to depend on both python3
for the host as well as python3
for the target. This will give you a python.exe
executable on the current system with the same source version as a python.lib
for the target (to use Windows nomenclature).
[
{ "name": "python3", "host": true },
"python3"
]
ports/python3/portfile.cmake
Outdated
vcpkg_find_acquire_program(PYTHON3) | ||
get_filename_component(PYTHON3_DIR "${PYTHON3}" DIRECTORY) | ||
set(ENV{PythonForBuild} "${PYTHON3_DIR}/python.exe") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is python3 now a bootstrapping requirement for native builds?
@@ -1,19 +1,24 @@ | |||
if(CMAKE_HOST_WIN32) | |||
set(program_name python) | |||
set(program_version 3.10.7) | |||
set(program_version 3.11.3) | |||
if(VCPKG_TARGET_ARCHITECTURE STREQUAL "x86") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this check for host architecture instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should, but ...
- it is not a regression ;-),
- there is no
VCPKG_HOST_ARCHITECTURE
, because host triplets are not loaded, - parsing of triplet names is not desired,
CMAKE_HOST_SYSTEM_PROCESSOR
could be uninitialized in script mode.
This leaves ENV{PROCESSOR_ARCHITECTURE}
etc. for WIN32 hosts. Cf. CLANG and GN scripts.
This is also required for gobject-introspection
So that python from vcpkg is picked before system python
@Osyotr Additionally, I think we should drop the Windows 7 backcompat patch. It's been two years since support for Windows 7 ended, and these changes appear to be related to security. |
Please do not drop the Windows 7 patch. A significant plurality of our end users still use this OS, even though it is EOL 😞. My understanding from @BillyONeal and @ras0219-msft is that this patch is acceptable in vcpkg as long as someone is willing to maintain it. |
Is someone signing up to fix the patch? |
This patch does not change observable behavior for users with Windows 10 or later.
I already fixed the patch for 3.11. |
|
@JavierMatosD I'm afraid I don't understand your comment. As I understand it, the position of the maintainers as expressed by @BillyONeal is that as long as contributors are willing to maintain the Windows 7 patch for this port, then it is acceptable. Has the official stance of the vcpkg maintainers changed? From what I can tell, @Osyotr has been very willing to do the work here, and I have previously been involved with maintaining the patch and port. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is someone signing up to fix the patch?
I already fixed the patch for 3.11.
Sorry for my underinformed question about this I posted before; I didn't read all the history before, I just was replying to the specific question and was unaware of the security relevant discussion. I just spoke with @JavierMatosD about this in person and he told me about some of the context.
As I understand it, the position of the maintainers as expressed by @BillyONeal is that as long as contributors are willing to maintain the Windows 7 patch for this port, then it is acceptable. Has the official stance of the vcpkg maintainers changed? From what I can tell, @Osyotr has been very willing to do the work here, and I have previously been involved with maintaining the patch and port.
The policy has never been "we are OK with any patch as long as someone is willing to do the work to maintain the patch". At one time, we looked at the patch that was required to keep XP support, and it looked like it was just adding polyfills around some relatively minor Windows APIs. It was a bigger patch than we would normally have accepted. However, a motivated contributor was present who signed up to do that work, and we were clear that in future version updates, the patch might go away.
The current rules for patches we accept have not changed much over time. Currently they are located in the maintainer guide: https://learn.microsoft.com/en-us/vcpkg/contributing/maintainer-guide#patching . Quoting that:
We want to avoid patches that:
- upstream would disagree with
- cause vulnerabilities or crashes
- we are incapable of maintaining across upstream version updates
- are large enough to cause license entanglement with the vcpkg repository itself
The difference between before and today is that the patch now contains security relevant content. This appears to be explicitly undoing a fix for https://nvd.nist.gov/vuln/detail/CVE-2020-8315 (thanks to @JavierMatosD who found python/cpython#18231 ). That means resolving the third bullet by having someone signed up to do the work of maintaining the patch across versions is no longer sufficient, the first and second bullet are affected. Clearly, if Python is telling their customers 'Python 3.8.2 and later have a fix for CVE-2020-8315', and we are giving them something we claim to be Python 3.11 which does not have a fix for CVE-2020-8315, upstream is going to not like that. It also is explicitly creating a vulnerability, which entangles the second bullet.
Similarly, if a vcpkg customer gets exploited by an attacker using CVE-2020-8315 because we explicitly undid the fix for that, that's a Serious problem. We aren't qualified to review some other fix that claims to fix CVE-2020-8315 while retaining Win7 support. (I also made another comment on why I think the patch is questionable)
As a result I am personally strongly against keeping the patch.
Additionally, I think we should drop the Windows 7 backcompat patch. It's been two years since support for Windows 7 ended, and these changes appear to be related to security.
This patch does not change observable behavior for users with Windows 10 or later. Can we please keep the patch for at least until the next update?
If the other maintainers are willing to keep it one more version as a gesture of goodwill to contributors who have done most of the python port maintenance, I don't like it but at least understand it.
It does not. That combined two commits in one PR: python/cpython#18231 (comment) |
@Hoikas I've checked whether it's feasible to keep the patch for 3.12 and this commit python/cpython@0f17576 uses new Windows API that I don't know how to replace correctly. Otherwise 3.12 builds fine with just minor adjustments to patches. :) |
@Osyotr I'm unfamiliar with that API... I'll try to look into it before Python 3.12 comes out. I'm a little worried that, some day, we may not be able to easily polyfill our way out of the problems :(
@BillyONeal @JavierMatosD This is incorrect. The patch is reverting, only on Windows 7, to the method previously used in Python 3.8.2 and higher. When I initially proposed this patch, I asked @ras0219-msft (in discord) to review it due to the "security content", and he approved of it because it matched the previous behavior of the CVE fix from Python 3.8. The patch is still just polyfilling Windows APIs. The only difference is that, as Python continues to evolve without Windows 7 support, we have to continue replicating the polyfills from Python 3.8.2, which means the patch grows in size. Eventually, it will probably become too much effort to be worthwhile. Until then, I'd appreciate it if we retained the patch and all parties fully informed themselves of the history of the port's development and previous reviews thereof.
The patch adds Windows 7 support, not Windows XP support. |
Okay, it looks like the patch addresses the specific, known CVE. However, the problem is that today, two years after support has been dropped, we have no idea what other CVEs lurk in the codebase and we aren't qualified to review python security fixes. Additionally, we know that this patch will continue to grow over time. After speaking with @ras0219-msft, he agreed to allow the patch to stay until the next update with the condition that we put a big comment in the portfile saying it will be removed on the next update. |
Could someone from vcpkg team do that? Touching python port at this point will result in another 24h half-world rebuild. |
There is definitively a bug here:
so python3 is currently building stuff for the host and not the target. |
I agree. Thankfully, Python is maintaining its code upstream (3.8 is not EOL until October 2024), and the security patching has already been done by Python upstream.
Would it be helpful if all of the polyfills were pulled out into a separate file and set of clearly defined functions instead of replicated verbatim each time? There is nothing different here (except for the length of the patch) than when this patch was initially proposed and the understanding was that the patch was acceptable as long as contributors maintained it. As it stands, the maintainers are making a seemingly invalid/incorrect argument to justify removing a patch that many of our end users are depending on. If vcpkg drops this patch, we will be in a bad position with our users and in a sticky spot with certain stakeholders. I find this seeming reversal of policy quite frustrating, but I would like to find a way forward where we can continue to maintain this functionality for as long as we, as contributors, can, as per the initial understanding. |
Which industry/country is this?
I mean there is always the possibility to provide an port overlay or an version override.
maybe that should have an overlay than? |
@Hoikas, I empathize with your frustration and the importance of this patch. However, we will be dropping the patch in the upcoming update. This decision is based on the evolution of the patch and not on a change of policy. When the patch was first introduced, its primary purpose was to reintroduce the Win7 fix for Python, which was straightforward and concise. Over the past two years, the patch has grown considerably in size and complexity. Its current state is significantly different from its initial version, making it challenging to maintain and at odds with upstream. We genuinely want to ensure that you aren't being left to dry. Before the next update, we recommend two potential solutions:
Please feel free to reach out if neither solution works for you. |
testing...