Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows 10 ARM64 platform support #77306

Closed
StevenNoonan mannequin opened this issue Mar 23, 2018 · 40 comments
Closed

Windows 10 ARM64 platform support #77306

StevenNoonan mannequin opened this issue Mar 23, 2018 · 40 comments
Assignees
Labels
3.11 bug and security fixes OS-windows type-feature A feature request or enhancement

Comments

@StevenNoonan
Copy link
Mannequin

StevenNoonan mannequin commented Mar 23, 2018

BPO 33125
Nosy @pfmoore, @tjguk, @zware, @zooba, @sdownum, @pablogsal, @paulmon, @NiQ1, @rubin55, @nsait-linaro, @tommyvct
PRs
  • bpo-33125: Add support for building and releasing Windows ARM64 packages #16828
  • [3.8] bpo-33125: Add support for building and releasing Windows ARM64 packages #17480
  • bpo-33125: Enables building traditional installer for Windows ARM64 #30885
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2022-02-03.21:57:12.003>
    created_at = <Date 2018-03-23.01:03:31.833>
    labels = ['type-feature', 'OS-windows', '3.11']
    title = 'Windows 10 ARM64 platform support'
    updated_at = <Date 2022-02-03.22:32:06.560>
    user = 'https://bugs.python.org/StevenNoonan'

    bugs.python.org fields:

    activity = <Date 2022-02-03.22:32:06.560>
    actor = 'TommyVCT'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2022-02-03.21:57:12.003>
    closer = 'steve.dower'
    components = ['Windows']
    creation = <Date 2018-03-23.01:03:31.833>
    creator = 'Steven Noonan'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 33125
    keywords = ['patch']
    message_count = 40.0
    messages = ['314295', '314296', '314308', '314309', '314311', '323310', '323368', '323382', '350843', '356617', '356724', '356884', '356999', '357090', '357091', '357094', '357931', '357934', '357935', '388334', '389837', '397443', '397489', '398284', '403203', '403215', '403253', '406725', '410295', '410316', '410346', '410427', '411586', '411589', '411603', '412007', '412010', '412012', '412476', '412478']
    nosy_count = 14.0
    nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'steven.downum', 'pablogsal', 'Steven Noonan', 'Paul Monson', 'dan', 'wayoman', 'rubin', 'jay.swanson', 'nsait-linaro', 'TommyVCT']
    pr_nums = ['16828', '17480', '30885']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue33125'
    versions = ['Python 3.11']

    @StevenNoonan
    Copy link
    Mannequin Author

    StevenNoonan mannequin commented Mar 23, 2018

    The Windows 10 ARM64 release is out along with a bunch of ARM64 devices. This version of Windows has full support for building native Win32 applications (this isn't just some rehash of Windows RT). It also can run x86 (but not x86_64) apps under a transparent emulation layer.

    I would like to see a native build of Python on Windows 10 ARM64.

    I did some very basic work to get it compiling (add 10.0.16299.0 as DefaultWindowsSDKVersion, add WindowsSDKDesktopARM64Support property). But there's still a lot missing: ssl, tk, and ctypes don't build.

    ssl/ctypes have some assembly that needs writing/porting.

    tk has some kind of build failure with the newer Windows SDK: https://core.tcl.tk/tk/tktview?name=3d34589aa0

    @StevenNoonan StevenNoonan mannequin added OS-windows type-feature A feature request or enhancement labels Mar 23, 2018
    @StevenNoonan
    Copy link
    Mannequin Author

    StevenNoonan mannequin commented Mar 23, 2018

    Oh, another change I had to make was remove all the BaseAddress elements in the Link sections. The linker complains if these are used (the lower 4GB of memory are apparently reserved for the x86 emulation). Also, from what I was told by someone over at Microsoft, the BaseAddress options don't do anything useful on modern Windows versions unless you build with -fixed as well, because everything gets relocated anyway.

    @zooba
    Copy link
    Member

    zooba commented Mar 23, 2018

    I'd like to see this as well, but just to set expectations, it can't be any earlier than 3.8 at this stage (for upstream support), and we need access to some real hardware to regularly run tests on.

    That said, most of the changes you describe are on my list of things to do anyway. If you've done them and would like to submit a pull request (against master) then we'll happily take them now and include them in 3.7. But you'll still need to build the ARM versions yourself.

    @zooba zooba added the 3.8 only security fixes label Mar 23, 2018
    @StevenNoonan
    Copy link
    Mannequin Author

    StevenNoonan mannequin commented Mar 23, 2018

    I originally tagged this issue against 3.6 just because that's what I was attempting to build. But I'm not super concerned about what release these changes actually land in, I can always backport it to my own builds (and at my own risk).

    Even though I'm super interested in seeing these changes done, I'm not sure I'm the right person to do the heavy lifting on this. I don't know if the seemingly undocumented (?) Windows ARM64 ABI matches the well-documented Linux AArch64 ABI, for example. If it does match, that would make porting the asm bits pretty straightforward (mostly just translating GNU assembler AT&T syntax to MASM, I suppose). I could do that. But if it doesn't match then I'd probably need to dig into ARM architecture manuals. Someone else could likely do it in their sleep.

    @zware
    Copy link
    Member

    zware commented Mar 23, 2018

    Ideally, all of the changes necessary in OpenSSL, libffi, and Tcl/Tk will happen upstream and we'll just update to new versions of them. We have PR 3806 in progress to extract libffi from our repo and treat it just as any other external dependency on Windows, but neither Steve nor I have had a chance to work on it in quite a long time. That will probably be a prerequisite for properly supporting ARM64 Windows.

    @paulmon
    Copy link
    Mannequin

    paulmon mannequin commented Aug 9, 2018

    I'm interested in getting python working on Windows running on Raspberry Pi (Thumb-2 instruction set). I can also contribute to ARM64 Windows. I've made some progress getting ARM32 working on a fork. The next roadblock for my investigation is libffi support for ARM32 on Windows. Can I help with that?

    @zooba
    Copy link
    Member

    zooba commented Aug 10, 2018

    If libffi already has support and just needs to be updated in CPython, then sure. I thought we'd finished extracting it, but Zach posted aPR link above that apparently still needs finishing.

    However, if libffi doesn't have the support yet then you'll have to approach them.

    @paulmon
    Copy link
    Mannequin

    paulmon mannequin commented Aug 10, 2018

    The PR hasn't changed since September. Is it still active?

    Should I consider updating the libffi_msvc code to support ARM32 in Python? Completing the switch to libffi makes more sense to me.

    I fetched the PR code to my fork and rebased it on the current master branch. When I run python -m test there are 2 tests that fail. Is that what's holding the PR up? or is it something more? I'm new to the Python codebase but I can try to help debugging if the failing tests are the issue.

    @NiQ1
    Copy link
    Mannequin

    NiQ1 mannequin commented Aug 30, 2019

    Hi,
    I've just noticed this but a couple of months ago I managed to compile Python 2.7 on win32 ARM64 including the said ssl, tk, ctypes modules.
    You can download from here and test -
    https://mega.nz/#F!PclhDIRB!-yhBZ6UM7S596ijNU3dx0A
    You will need to install the MSVC 2015 runtime first and if you want ssl to work then also download the OpenSSL package from the same link.

    For ctypes I used the (then most) latest version of libffi. There's an msvc_build directory in the master branch which allows ARM64 builds. With the library working the ctypes module builds just fine (BTW from looking at the libffi code it seems that the code is a straightforward port from the Linux ARM64 code, which is generally fine since Windows on ARM seems to use the standard ARMv8 ABI but who knows what end cases may occur).

    For SSL, I managed to compile OpenSSL with a lot of headache. Also as noted I had to give-up on ASM optimizations so the ssl module may not be as fast as the native i686, however it will probably still be way faster than x86 emulation mode.

    For TCL/TK I compiled the TCL/TK libraries from source as well. There's an issue with compiling that on MSVC 2015 because apparently the source uses some names which are now reserved keywords on the newer compiler so some search & replace is needed but once the module is built the Python modules compile just fine.

    Overall this is a pretty much complete Python 2.7 package as it has all modules usually shipped built-in compiled and seemingly working (as far as I tested). The whole thing took several days but if you want to port it to Python 3.8 it shouldn't be too difficult and I'll gladly share my (very slightly) modified sources with you if you wish.

    @zooba
    Copy link
    Member

    zooba commented Nov 14, 2019

    I just added a PR that will produce ARM64 releases (as embeddable, nuget and Microsoft Store packages).

    The only missing feature is tkinter (and everything that depends on it). For users who require that, I'd suggest grabbing the x86 installer from python.org which works fine (apart from some minor arithmetic and process management bugs in the Windows emulation layer).

    This is not saying we're going to release it just yet. Really it shouldn't be released until 3.9, as it's a new platform and we need to allow the 3rd party package ecosystem to catch up, but at least having the ability to build packages easily will enable those who are already trying to do it themselves.

    Allegedly there's a lot of demand for Python on ARM64, but I haven't actually seen it other than this bug. And if that demand immediately extends to numpy or pywin32 on ARM64 then we're still a very long way from actually satisfying it.

    I'll likely tweak the PR a bit more to prevent releasing ARM64 packages by default, but then I don't see any issue having it in the 3.8 and master branches for those who would be helped. Any thoughts/comments?

    @zooba zooba added the 3.9 only security fixes label Nov 14, 2019
    @zooba zooba self-assigned this Nov 14, 2019
    @NiQ1
    Copy link
    Mannequin

    NiQ1 mannequin commented Nov 15, 2019

    Steve, the 2.7 build I've posted includes native tk, do you need the necessary patches?

    @zooba
    Copy link
    Member

    zooba commented Nov 18, 2019

    If the patches are for Tcl/Tk, then you should submit it to them.

    I'm not a Tk maintainer, and I never want to be one, so frankly I'd be happier to say it's not supported on ARM :)

    But if it builds fine with our PCbuild/prepare_tcltk.bat script and people want it then I'll keep it.

    @NiQ1
    Copy link
    Mannequin

    NiQ1 mannequin commented Nov 19, 2019

    According to this -
    https://wiki.tcl-lang.org/page/Building+with+Visual+Studio+2017
    the issue will be fixed in the next version (8.6.10). TLDR explanation: Tcl/Tk has variables with names that are now reserved keywords in VS2017, meaning that it can't be build with recent compilers and since VS2017 is the first compiler that has official ARM64 support it means that it currently can't be built for that target.
    Anyway when version 8.6.10 is eventually released then it should (hopefully) build fine with the current scripts.
    Maybe we can add it later when that happens?

    @zooba
    Copy link
    Member

    zooba commented Nov 20, 2019

    New changeset de148f2 by Steve Dower in branch 'master':
    bpo-33125: Add support for building and releasing Windows ARM64 packages (GH-16828)
    de148f2

    @zooba
    Copy link
    Member

    zooba commented Nov 20, 2019

    I've merged the change to master, but note that the release is not enabled, so we aren't actually releasing ARM64 builds yet (though they will be compiled [but not tested] in PRs - nothing should be different from other Windows builds though). There's no harm in giving the build scripts a bit of bake time.

    I'm also not concerned about backporting to 3.8 to make it easier to build that version, which I've heard a little bit of demand for. But let's run the scripts in master for a little while first.

    Deliberately leaving the stage as "commit review" rather than "backport needed" for now.

    @zooba
    Copy link
    Member

    zooba commented Nov 20, 2019

    I also filed bpo-38867 specifically for enabling Tkinter on Windows ARM.

    @zooba
    Copy link
    Member

    zooba commented Dec 6, 2019

    New changeset c9f480d by Steve Dower in branch '3.8':
    bpo-33125: Add support for building and releasing Windows ARM64 packages (GH-17480)
    c9f480d

    @zooba
    Copy link
    Member

    zooba commented Dec 6, 2019

    Closing this as the work we need for support is now added. Whether we start releasing binaries from python.org is a discussion to have on python-dev and with the broader community.

    Right now, I don't think the ecosystem is ready to add support (mainly because there is no CI system that can build/test easily). And most of our tooling isn't up to the task either. Until that can be improved, it's not fair to expect third party packagers to provide wheels or support the platform.

    @zooba zooba closed this as completed Dec 6, 2019
    @zooba
    Copy link
    Member

    zooba commented Dec 6, 2019

    This site seems like a pretty good aggregation of news around Windows on ARM64 devices: https://www.windows-arm.com/

    Short story is, it's a real thing, but for (what I consider) obvious reasons the OSS developer experience isn't great yet.

    @rubin55
    Copy link
    Mannequin

    rubin55 mannequin commented Mar 9, 2021

    Requesting that this issue be re-opened or re-evaluated; it would be pretty great if Python could provide Windows on ARM binaries for download. There's been a lot of traction on ARM in general and having Windows on ARM downloads available by default will also give the Python package ecosystem a better opportunity to get ready (although it's already impressively far ahead, take a look at https://cloudbase.it/python-on-windows-arm64/, 23 of 25 of the most popular packages already build out-of-the-box, and the remaining two, numpy and cffi have patches available).

    @zooba
    Copy link
    Member

    zooba commented Mar 30, 2021

    Our builds are fine, so there's no need for an issue on our side. The decision to release is based on the ecosystem catching up.

    If you're able to help projects like setuptools, pip, and the core parts of the data science stack (which will need the most help) be able to support ARM64 builds (despite the current lack of CI/CD support, which I'm working on separately), that will accelerate us being able to make the ARM64 releases available/supported.

    @jayswanson
    Copy link
    Mannequin

    jayswanson mannequin commented Jul 13, 2021

    I know this is closed, but is it possible that the ARM64EC support coming in Windows 11 that allows mixed native/emulated code is something that would help this along? That along with the sunsetting of 32-bit systems.

    @zooba
    Copy link
    Member

    zooba commented Jul 14, 2021

    I'm sure it will help, but official support will continue to be blocked by availability of CI/CD systems for first- and third-party packages/testing.

    @nsait-linaro
    Copy link
    Mannequin

    nsait-linaro mannequin commented Jul 27, 2021

    I know this issue is closed but just wanted to have a discussion around adding CI support for cpython on windows on arm targets.

    We at linaro are working on adding windows/arm support for popular packages such as numpy, cffi etc. Hopefully, very soon we will have many of the popular packages working out of the box on windows on arm targets.

    Testing windows/arm has been an open question. I heard about the potential availablity of windows/arm machines in azure cloud sometimes in the future but not sure about the timescale.

    In Linaro, we have a WoA farm which has couple of surface pro which we are happy to use for CI and for release builds etc. It could be an interim testing system until we have support in azure. This would enable us to start official support for cpython for windows/arm sooner.

    I would like to hear your thoughts on the proposal

    @rubin55
    Copy link
    Mannequin

    rubin55 mannequin commented Oct 5, 2021

    Hi Python peeps,

    It sounds to me like all stars are aligned for a Windows on ARM64 release, it would be rather great if this could happen!

    @nsait-linaro
    Copy link
    Mannequin

    nsait-linaro mannequin commented Oct 5, 2021

    Python buildbot worker for woa is up and running as well

    https://buildbot.python.org/all/#/workers/45

    Would be great to see official support and installer added for woa

    @zooba
    Copy link
    Member

    zooba commented Oct 5, 2021

    Yeah, we're getting close. I'll reopen this issue for tracking.

    We still need that pip release and it'll have to be merged into ensurepip. We'll also need better access to test systems than any of us currently have (the buildbot is great, but we need interactive systems too).

    There are already binary releases at https://www.nuget.org/packages/pythonarm64/ that should be suitable for most testing. I'll probably make the ARM64 packages available through the Windows Store for 3.11's prereleases, and possibly as a side-loadable MSIX from python.org.

    I'm strongly considering not offering the regular executable installer on ARM64, since it seems we don't have the back-compat demand right now. It would be a good opportunity to try and fix the issues with that installer and really put the focus on the MSIX as the user-friendly approach. But that's still future thoughts, no concrete plans yet :)

    @zooba zooba added 3.11 bug and security fixes and removed 3.8 only security fixes 3.9 only security fixes labels Oct 5, 2021
    @zooba zooba reopened this Oct 5, 2021
    @gvanrossum
    Copy link
    Member

    I could really use 3.10 here as well.

    @tommyvct
    Copy link
    Mannequin

    tommyvct mannequin commented Jan 11, 2022

    Hi everyone,

    I'm Tommy from OBS Projects, and I'm currently working on bring OBS Studio to Windows ARM64.

    I still prefer the regular executable installer on ARM64, because it's probably the least intrusive way of making Python scripting work with OBS Studio. Please provide the regular executable installer for ARM64.

    I'm also interested in ARM64EC. I haven't tried it yet, but what's your confidence that it will compile on ARM64EC toolchain?

    @zooba
    Copy link
    Member

    zooba commented Jan 11, 2022

    Hi Tommy! Glad to have you join the discussion (I'm a huge fan of OBS Studio, and if the license wasn't incompatible with my employer, I'd be actively contributing :) )

    Our challenge with the installer is the portability of Wix Toolset. I'm pretty sure our current version doesn't support ARM64, and they haven't released a newer version that does yet. It's possible that emulation will handle the main part of our installer, but AFAIK nobody has tested it to see whether it confuses the underlying installer service. (I can probably do that at some point, as I have access to some test machines now.)

    At the same time, any reason why OBS doesn't just include a copy of Python with it? It should be quite easy to include the embeddable runtime on Windows, which would then mean that you're able to ship the exact version necessary and have the paths preconfigured. (It *might* complicate installing 3rd party packages a bit, but you can always allow search paths to be set/PYTHONPATH to be respected, or the executable path overridden.)

    Possibly this is something you guys have already looked into, so if you know there's an issue I'd love to hear about it. Making it easy/easier to bundle Python with apps like OBS is one of my major focuses.

    @tommyvct
    Copy link
    Mannequin

    tommyvct mannequin commented Jan 11, 2022

    Hi Steve,

    We appreciate your interest in OBS Studio!

    In my opinion, the installer doesn't have or need to be in ARM64, because it's not more complex than copying files and writing registry keys, and as an installer, it should be the most forgotten part of python that usually runs for once or twice.

    For the reason why OBS doesn't include a copy of python, OBS Studio on Windows is already huge (~130MB installer, ~500MB after install), and we don't want to add another disk space eater on already huge installer.

    We are working on a way to find Python installed on the system, but it seems pretty complicated.

    @zooba
    Copy link
    Member

    zooba commented Jan 12, 2022

    PEP-514 is the way to find it on Windows (you should be able to look directly for your supported version, rather than enumerating to search). As far as I know, the best way to find it on macOS and Linux is to search PATH for "python3.X".

    Appreciate the size concerns. The Python runtime comes down to about 10MB with some basic tricks (already done on Windows in the embeddable distro), such as excluding the docs, test suite and zipping the standard library. It's possible with different tricks to do the same on other platforms, so you may even find the best balance for yourselves and your users is to repackage those and do a download-on-demand, rather than looking for a system installed Python.

    as an installer, it should be the most forgotten part of python that usually runs for once or twice.

    Oh I wish this were the case! Installers should be forgotten, but they need to be 100% reliable or people *really* notice :)

    But updating the installer build to target ARM64 while generating an x86 executable should be possible, it just hasn't been done yet.

    @zooba
    Copy link
    Member

    zooba commented Jan 25, 2022

    Our challenge with the installer is the portability of Wix Toolset. I'm pretty sure our current version doesn't support ARM64, and they haven't released a newer version that does yet.

    Turns out they've released some test versions that support it (partially, but enough), just nothing that they've declared stable. I'll push updated tools for 3.11 so we have a chance to discover any breakage in earlier versions, but should be able to do ARM64 regular installer next alpha (Pablo, FYI - this will require updates to your side of things).

    @pablogsal
    Copy link
    Member

    No problem! What do I need to update? The script to add the binaries to the release page?

    @zooba
    Copy link
    Member

    zooba commented Jan 25, 2022

    Yeah, will just be the same files again but with "arm64" in place of "win32"/"amd64".

    I'm also going to stop uploading the old webinstall files with my change here, though that shouldn't matter. The code we added to filter them out should handle them being completely absent, IIRC.

    @zooba
    Copy link
    Member

    zooba commented Jan 28, 2022

    New changeset 45faf15 by Steve Dower in branch 'main':
    bpo-33125: Enables building traditional installer for Windows ARM64 (GH-30885)
    45faf15

    @zooba
    Copy link
    Member

    zooba commented Jan 28, 2022

    The ARM64 installer will be part of the next 3.11 release, minus the py.exe launcher (which needs a chunk of work in bpo-46566) and Tcl/Tk (which needs upstream support for the platform) and hence IDLE.

    I'm hesitant to make a final release without these components. At the same time, I'm concerned about other release blocking issues that may arise that are unrelated, so I do want to get releases out even without them so that we have a chance to discover them.

    @zooba
    Copy link
    Member

    zooba commented Jan 28, 2022

    Tcl/Tk tracking issue is bpo-46567

    @zooba
    Copy link
    Member

    zooba commented Feb 3, 2022

    Closing this issue, as we have others to track individual tasks.

    @zooba zooba closed this as completed Feb 3, 2022
    @tommyvct
    Copy link
    Mannequin

    tommyvct mannequin commented Feb 3, 2022

    Thank you Steve!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 bug and security fixes OS-windows type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants