Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull Windows dependencies from GitHub rather than svn.python.org #74635

Closed
zware opened this issue May 24, 2017 · 52 comments
Closed

Pull Windows dependencies from GitHub rather than svn.python.org #74635

zware opened this issue May 24, 2017 · 52 comments
Assignees
Labels
3.7 build extension-modules OS-windows performance

Comments

@zware
Copy link
Member

@zware zware commented May 24, 2017

BPO 30450
Nosy @pfmoore, @pitrou, @vstinner, @tjguk, @benjaminp, @jkloth, @zware, @zooba, @ammaraskar
PRs
  • #1783
  • #2236
  • #2237
  • #2737
  • #2739
  • #2744
  • #2751
  • #3306
  • #3330
  • #3348
  • #3443
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2017-12-06.21:14:23.747>
    created_at = <Date 2017-05-24.01:33:46.861>
    labels = ['extension-modules', 'build', '3.7', 'OS-windows', 'performance']
    title = 'Pull Windows dependencies from GitHub rather than svn.python.org'
    updated_at = <Date 2017-12-06.22:28:17.743>
    user = 'https://github.com/zware'

    bugs.python.org fields:

    activity = <Date 2017-12-06.22:28:17.743>
    actor = 'vstinner'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2017-12-06.21:14:23.747>
    closer = 'zach.ware'
    components = ['Build', 'Extension Modules', 'Windows']
    creation = <Date 2017-05-24.01:33:46.861>
    creator = 'zach.ware'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 30450
    keywords = ['patch']
    message_count = 52.0
    messages = ['294307', '294359', '294441', '294494', '294533', '294534', '295240', '296150', '296157', '296160', '296351', '296365', '296805', '298469', '298471', '298476', '298497', '298498', '298501', '298504', '298505', '298506', '298507', '298508', '298509', '298512', '298513', '298516', '298517', '298518', '298519', '298520', '298522', '298523', '298526', '298552', '298557', '298559', '298560', '298608', '298609', '298640', '299803', '299809', '301279', '301327', '301328', '301365', '301712', '301966', '307769', '307772']
    nosy_count = 11.0
    nosy_names = ['paul.moore', 'pitrou', 'vstinner', 'tim.golden', 'benjamin.peterson', 'jkloth', 'jeremy.kloth', 'zach.ware', 'steve.dower', 'ammar2', 'schen']
    pr_nums = ['1783', '2236', '2237', '2737', '2739', '2744', '2751', '3306', '3330', '3348', '3443']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'resource usage'
    url = 'https://bugs.python.org/issue30450'
    versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

    @zware
    Copy link
    Member Author

    @zware zware commented May 24, 2017

    Once we've migrated away from svn.python.org for Windows build dependencies, there should be no reason to continue running svn.python.org.

    @zware zware added 3.7 build extension-modules OS-windows performance labels May 24, 2017
    @zooba
    Copy link
    Member

    @zooba zooba commented May 24, 2017

    The reason to keep it running is for all the existing people who are building from source. Unless you're porting all the old versions over and providing instructions for updating old sources? I wouldn't bother - just leave the server up for a few more years.

    Ben probably knows - can we add a custom message to svn requests? Then we could start publicising an end of life date for it.

    @benjaminp
    Copy link
    Contributor

    @benjaminp benjaminp commented May 25, 2017

    The box svn is running on is at risk of falling over any day. Plus with whatever ancient software it's running, it's surely a security risk to keep running an unmaintained svn instance. I certainly don't want to keep it running for years. I don't think we've ever advertised the svn windows deps as a service we provide the community.

    Off the top of my head, I'm not sure how to announce that deprecation to svn client users. If you find something on the internets, I can put it on the server.

    @zooba
    Copy link
    Member

    @zooba zooba commented May 25, 2017

    I don't think we've ever advertised the svn windows deps as a service we provide the community.

    Except in the dev guide :) ([after looking] well, linked from the dev guide https://github.com/python/cpython/blob/master/PCbuild/readme.txt#L230 )

    Maybe we just need to pick a date and publicise it separately. Is this process PEP-worthy? Considering the amount of decisions we had to make regarding putting code in git, it might be worth writing up how it works in a brief PEP and including an EOL for svn.p.o

    @benjaminp
    Copy link
    Contributor

    @benjaminp benjaminp commented May 26, 2017

    How far along are you with removing the svn.python.org dependency. Does Jan 1 2018 sound okay for the end of svn.python.org?

    I think a PEP is up to you. Doesn't seem like something that needs to be approved.

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented May 26, 2017

    I could imagine people wanting to build old releases. But we have Windows binaries up here:
    https://www.python.org/ftp/python/

    Therefore I don't think it's a problem to discontinue svn.python.org once new releases are migrated to github (but only once those new releases are out).

    @benjaminp
    Copy link
    Contributor

    @benjaminp benjaminp commented Jun 6, 2017

    I just disabled viewvc on svn.python.org because of a security problem.

    @zware
    Copy link
    Member Author

    @zware zware commented Jun 16, 2017

    New changeset 51599e2 by Zachary Ware in branch 'master':
    bpo-30450: Pull Windows dependencies from GitHub rather than svn (GH-1783)
    51599e2

    @zware
    Copy link
    Member Author

    @zware zware commented Jun 16, 2017

    New changeset cb8c048 by Zachary Ware in branch 'master':
    bpo-30450: Add NEWS and whatsnew (GH-2236)
    cb8c048

    @zware
    Copy link
    Member Author

    @zware zware commented Jun 16, 2017

    New changeset 04431c9 by Zachary Ware in branch '3.6':
    bpo-30450: Pull Windows dependencies from GitHub rather than svn (GH-1783) (GH-2237)
    04431c9

    @schen
    Copy link
    Mannequin

    @schen schen mannequin commented Jun 19, 2017

    The usage text in build.bat still mentions that svn.exe is required for the '-e' flag. I think it should be updated to reflect the changes made.

    @zooba
    Copy link
    Member

    @zooba zooba commented Jun 19, 2017

    Zach, I just spotted this build log, that appears to have failed to download nuget.exe: http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.6/builds/339/steps/compile/logs/stdio

    Guessing the bot is only succeeding because it currently has all the dependencies. Maybe we should consider just checking in nuget.exe?

    @ammaraskar
    Copy link
    Member

    @ammaraskar ammaraskar commented Jun 25, 2017

    Looks like that error is coming from the fact that the Powershell on that buildbot is outdated. As the documentation notes:

    https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.utility/invoke-webrequest

    This cmdlet was introduced in Windows PowerShell 3.0.

    Before running the web request command maybe do a powershell.exe -Command $PSVersionTable.PSVersion in order to debug this issue on any other machines?

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    The buildbot failures currently being blamed on bpo-30916 are actually due to this issue - we actually need Python 3.6 on these machines in order to download the new externals, and they don't have it and we can't get it via Powershell.

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    My PR adds a small script to use requests.get/urlretrieve with any version of Python to get nuget.exe.

    Theoretically, we *could* use this to download everything, but I'd rather minimize our exposure to insecure downloads for machines that don't already have Python 3.6 available.

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    New changeset 588836d by Steve Dower in branch 'master':
    bpo-30450: Adds alternate download approach for nuget.exe (bpo-2737)
    588836d

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    Compilation still fails, as some of the buildbots don't have py.exe either.

    At this stage, our choices are either to fix the buildbots, most easily by copying py.exe into PATH, or to check nuget.exe (4MB) into the source repository. (I guess we can cover a few more % of cases by checking in py.exe, but that still won't get us 100%)

    Any preferences?

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Jul 17, 2017

    If the dependencies are in git, why don't you use "git clone" instead of trying a HTTP fetch?

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Jul 17, 2017

    For example "git clone --depth 1 https://github.com/python/cpython-bin-deps --branch openssl-bin-1.0.2k myopenssldir"

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    We can't assume git.exe is available either (build from sdist/hggit/.zip), though I guess we can also use it as a fallback. At least on the buildbots it'll be there.

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Jul 17, 2017

    I think it's reasonable to mandate the presence of a git install to fetch CPython externals. After all we used to mandate the presence of a svn install...

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Jul 17, 2017

    We can't assume git.exe is available either

    How do you get CPython source code if git is not available? Also using
    HTTP to get a tarball? I think that it's ok to require git to build
    CPython on Windows. Previously, we required: svn, perl and git :-)

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    we used to mandate the presence of a svn install

    And we regretted that so much that we changed away from it :)

    When discussing this changeover plan, Zach and I decided we needed a fallback requiring only OS dependencies. In this case, the Powershell dependency fails on Windows 7 since it does not have the Invoke-WebRequest command (unless you've been installing all your updates).

    We decided against using git first because then the case that is more reliable (download and extract a .zip file from a URL) would go unused/untested in many cases.

    We also use the potentially installed python.exe in other places in the build, so it's far from a waste to grab it, and we already required *any* Python dependency to do a full build (docs and/or installer), so it isn't really adding anything there. The bit that was overlooked was the PowerShell on old systems limitation, and the absence of py.exe when you don't have Python 3 installed.

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    New changeset efa26bc by Steve Dower in branch 'master':
    bpo-30450: Fall back to git.exe if no Python is found. (bpo-2739)
    efa26bc

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    The buildbots have already successfully built, so I'm declaring this and bpo-30916 resolved.

    @zooba zooba closed this as completed Jul 17, 2017
    @jkloth
    Copy link
    Contributor

    @jkloth jkloth commented Jul 17, 2017

    Or, use the preexisting convention of the HOST_PYTHON envvar that was used prior to the recent merged PRs

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    use the preexisting convention of the HOST_PYTHON envvar that was used prior to the recent merged PRs

    Is that an actual convention? I didn't see any other references, so I figured Zach had made it up.

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Jul 17, 2017

    I think that's used on Unix when cross-compiling, but I don't know about Windows...

    @jkloth
    Copy link
    Contributor

    @jkloth jkloth commented Jul 17, 2017

    In this case, the Powershell dependency fails on Windows 7 since it does not have the Invoke-WebRequest command (unless you've been installing all your updates).

    Just to note, PowerShell must be updated *manually* (at least on Win7). Plus you need to know what to update to get it. In this case, the Windows Management Framework:

    https://www.microsoft.com/en-us/download/details.aspx?id=40855

    I've installed this on my buildbot apparently concurrently with this conversation. Personally, I'd lean toward recommending this update with Windows 7 machines as it stays closer to "naked" OS + Python checkout for building a new Python. (Although the latest merge kinda makes this a mute point).

    @jeremykloth
    Copy link
    Mannequin

    @jeremykloth jeremykloth mannequin commented Jul 17, 2017

    Is that an actual convention? I didn't see any other references, so I figured Zach had made it up.

    It has existed in the Windows build files since 2.5, when x64
    supported was initially added by MvL.

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Jul 17, 2017

    I just got a download failure on an AppVeyor build:
    https://ci.appveyor.com/project/python/cpython/build/3.7.0a0.4571

    Downloading nuget...
    Invoke-WebRequest : The operation has timed out.
    At line:1 char:1
    + Invoke-WebRequest https://aka.ms/nugetclidl -OutFile 'C:\projects\cpy ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:Htt
    pWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShe
    ll.Commands.InvokeWebRequestCommand

    File "<string>", line 1
    C:\projects\cpython\PCbuild\\urlretrieve.py
    ^
    SyntaxError: invalid syntax
    Installing Python via nuget...

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    Hmm... looks like there's another way to fail I didn't account for that goes down a different path in the batch file. I'll try being less clever.

    @zooba zooba reopened this Jul 17, 2017
    @zooba zooba assigned zooba and unassigned zware Jul 17, 2017
    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    Yeouch, it's been causing AppVeyor builds to get stuck, and then they time out after an hour.

    Luckily mine is up next in a minute or so, so I'll try to merge quickly and avoid anyone else getting held up.

    I also restored HOST_PYTHON in the PR, as a fallback for "where do I find a python.exe I can download stuff with" - specific version doesn't matter, as long as it has requests or urlretrieve.

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 17, 2017

    Ah, there's also a nuget.org outage affecting AppVeyor (but not me because I'm in Italy and so connecting to the EU mirror automatically) - https://appveyor.statuspage.io/

    When that recovers it should be fine again. The workaround for those without py.exe is to set PYTHON (or, post-PR, HOST_PYTHON) to a Python 3.6 or later executable.

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 18, 2017

    New changeset 5feda33 by Steve Dower in branch 'master':
    bpo-30450: Fix logic for retrying nuget.exe download (bpo-2744)
    5feda33

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 18, 2017

    Zach - did this get backported to 2.7 or 3.5? It doesn't look like it.

    @zooba
    Copy link
    Member

    @zooba zooba commented Jul 19, 2017

    New changeset e99d3a5 by Steve Dower in branch '3.6':
    [3.6] bpo-30450: Improved logic for obtaining dependencies (bpo-2751)
    e99d3a5

    @pitrou
    Copy link
    Member

    @pitrou pitrou commented Aug 6, 2017

    Has this actually been backported to 3.6? It looks like a lot of Tcl and OpenSSL stuff is being compiled on AppVeyor yet:
    https://ci.appveyor.com/project/python/cpython/build/3.6.1+.5134

    @zooba
    Copy link
    Member

    @zooba zooba commented Aug 6, 2017

    The prebuilt binaries have not, and the OpenSSL change has functional implications and should not be (unless someone wants to make the security justification, but right now I think it's a theoretical rather than an actual benefit). The TclTk build could be, but I haven't gone to the effort to split it up.

    There's also a separate issue for that part - this bug is for the location of sources.

    @zware
    Copy link
    Member Author

    @zware zware commented Sep 4, 2017

    New changeset 986b7ff by Zachary Ware in branch '2.7':
    [2.7] bpo-30450: Pull Windows dependencies from GitHub rather than SVN (GH-1783) (GH-3306)
    986b7ff

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Sep 5, 2017

    New changeset 986b7ff by Zachary Ware in branch '2.7':
    [2.7] bpo-30450: Pull Windows dependencies from GitHub rather than SVN (GH-1783) (GH-3306)

    This change broke Python 2.7 compilation on Windows XP.

    Pending fix: PR 3330.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Sep 5, 2017

    New changeset 986b7ff by Zachary Ware in branch '2.7':
    [2.7] bpo-30450: Pull Windows dependencies from GitHub rather than SVN (GH-1783) (GH-3306)

    This change also broke compilation on AMD64 Windows10 2.7 buildbot:
    http://buildbot.python.org/all/builders/AMD64%20Windows10%202.7/builds/279

    "D:\buildarea\2.7.bolen-windows10\build\PCbuild\pcbuild.proj" (Build target) (1) ->
    "D:\buildarea\2.7.bolen-windows10\build\PCbuild\pythoncore.vcxproj" (Build target) (2) ->
    (Manifest target) ->
    C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppCommon.targets(574,5): error MSB6006: "mt.exe" exited with code -1073741819. [D:\buildarea\2.7.bolen-windows10\build\PCbuild\pythoncore.vcxproj]

    260 Warning(s)
    1 Error(s)
    

    @zware
    Copy link
    Member Author

    @zware zware commented Sep 5, 2017

    New changeset 8905fb8 by Zachary Ware in branch '2.7':
    bpo-30450: Don't use where, XP doesn't have it (GH-3348)
    8905fb8

    @zware
    Copy link
    Member Author

    @zware zware commented Sep 8, 2017

    New changeset 1911cf3 by Zachary Ware in branch '2.7':
    [2.7] bpo-30450: Fall back on the old env.bat (GH-3443)
    1911cf3

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Sep 12, 2017

    This change also broke compilation on AMD64 Windows10 2.7 buildbot:
    http://buildbot.python.org/all/builders/AMD64%20Windows10%202.7/builds/279
    (...) error MSB6006: "mt.exe" exited with code -1073741819. (...)

    I created the bug bpo-31430 to track this regression.

    @zware
    Copy link
    Member Author

    @zware zware commented Dec 6, 2017

    This has been backported and working for nearly 3 months now; closing.

    As for whether to continue operating svn.python.org now that we're officially away from it: that can be determined elsewhere, probably by the box dying and deciding for us :)

    @zware zware closed this as completed Dec 6, 2017
    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Dec 6, 2017

    Noooo! I loved Subversion so much! ... just kidding. Thanks for getting rid of Subversion.

    It became much easier to build CPython on Windows nowadays!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 build extension-modules OS-windows performance
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants