Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'make touch' does not work with git clones of the source repository #67593

Closed
vlee mannequin opened this issue Feb 6, 2015 · 53 comments
Closed

'make touch' does not work with git clones of the source repository #67593

vlee mannequin opened this issue Feb 6, 2015 · 53 comments
Labels
3.7 (EOL) end of life 3.8 (EOL) end of life build The build process and cross-build type-feature A feature request or enhancement

Comments

@vlee
Copy link
Mannequin

vlee mannequin commented Feb 6, 2015

BPO 23404
Nosy @brettcannon, @ncoghlan, @vstinner, @larryhastings, @benjaminp, @ned-deily, @ezio-melotti, @berkerpeksag, @vadmium, @zware, @koobs, @willingc, @MojoVampire, @yan12125
PRs
  • bpo-23404: make touch becomes make regen-all #1405
  • [3.6] bpo-23404: make touch becomes make regen-all (#1405) #1461
  • bpo-30273: Update sysconfig #1464
  • bpo-23404: make touch becomes make regen-all (#1405) (#1461) #1465
  • [2.7] bpo-23404: make touch becomes make regen-all #1466
  • bpo-23404: Document "make regen-all" in What's New in Python 3.6 #2127
  • bpo-23404: make regen-all What's New entry #2128
  • bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 #7620
  • [3.7] bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 (GH-7620) #7621
  • [3.6] bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 #7622
  • [3.5] bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 #7623
  • [2.7] bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 #7624
  • Files
  • 0001-Issue-23404-Do-not-use-Python-2.7-constructs-in-Pars.patch: Do not use Python 2.7 constructs in Parser/asdl.py.
  • iss23404devguide.patch
  • boot-flag.patch
  • boot-flag.patch: Based on Martin's version, with a minor fix
  • boot-flag.py3.5.patch: Needs _freeze_importlib line merged
  • boot-flag.py2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-06-11.09:04:17.337>
    created_at = <Date 2015-02-06.22:46:05.940>
    labels = ['3.8', 'type-feature', '3.7', 'build']
    title = "'make touch' does not work with git clones of the source repository"
    updated_at = <Date 2018-06-11.09:04:17.335>
    user = 'https://bugs.python.org/vlee'

    bugs.python.org fields:

    activity = <Date 2018-06-11.09:04:17.335>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-06-11.09:04:17.337>
    closer = 'ned.deily'
    components = ['Build']
    creation = <Date 2015-02-06.22:46:05.940>
    creator = 'vlee'
    dependencies = []
    files = ['38033', '40044', '44627', '46620', '46772', '46773']
    hgrepos = []
    issue_num = 23404
    keywords = ['patch', 'buildbot']
    message_count = 53.0
    messages = ['235505', '235508', '235509', '235510', '235511', '235513', '235514', '235515', '235516', '247510', '247574', '247575', '275285', '276252', '287493', '287580', '291069', '292067', '292068', '292092', '292126', '292143', '292170', '292194', '292207', '292822', '292823', '292829', '292855', '292858', '292860', '292913', '292916', '293002', '293018', '293026', '293030', '293033', '293034', '293094', '295647', '295764', '295767', '295769', '295770', '299181', '303572', '303624', '319282', '319283', '319285', '319286', '319287']
    nosy_count = 16.0
    nosy_names = ['brett.cannon', 'ncoghlan', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'ezio.melotti', 'python-dev', 'berker.peksag', 'martin.panter', 'zach.ware', 'koobs', 'willingc', 'josh.r', 'vlee', 'yan12125']
    pr_nums = ['1405', '1461', '1464', '1465', '1466', '2127', '2128', '7620', '7621', '7622', '7623', '7624']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue23404'
    versions = ['Python 2.7', 'Python 3.5', 'Python 3.7', 'Python 3.8']

    @vlee
    Copy link
    Mannequin Author

    vlee mannequin commented Feb 6, 2015

    Is there a minimum Python requirement to build Python?

    Python 3.5 does not build with Python 2.6. Python 3.4, Python 3.3, and Python 2.7 build with Python 2.6 so this is recent change in build requirements.

    For example, this build failure occurs on CentOS 6.

    python ./Parser/asdl_c.py -h Include ./Parser/Python.asdl
    Traceback (most recent call last):
      File "./Parser/asdl_c.py", line 6, in <module>
        import asdl
      File "Parser/asdl.py", line 36
        builtin_types = {'identifier', 'string', 'bytes', 'int', 'object', 'singleton'}
                                     ^
    SyntaxError: invalid syntax

    @vlee vlee mannequin added build The build process and cross-build labels Feb 6, 2015
    @MojoVampire
    Copy link
    Mannequin

    MojoVampire mannequin commented Feb 7, 2015

    Looks like this revision, part of bpo-22823, changed it: https://hg.python.org/cpython/rev/4480506137ed

    @MojoVampire
    Copy link
    Mannequin

    MojoVampire mannequin commented Feb 7, 2015

    To answer your question: Since asdl uses set literals (introduced in 2.7 and 3.0), that appears to put an effective minimum version requirement of 2.7 to build 3.5. Whether that was the intent is unclear (they were changing a lot of places from set([a, b, c]) to {a, b, c}; causing problems building Python was probably not the intent, but 2.6 is well out of support at this point).

    @ned-deily
    Copy link
    Member

    Try using "make touch" before "make". Because hg does not preserve precise time stamps when creating working directories, some build steps are run unnecessarily after an initial checkout. 'make touch' updates the file time stamps so that these steps are skipped and a "bootstrap" Python is not needed.

    @ned-deily
    Copy link
    Member

    @vlee
    Copy link
    Mannequin Author

    vlee mannequin commented Feb 7, 2015

    I am building from a git copy of the source repository. "make touch" before "make" does not work for me.

    $ make touch
    cd .; \
    	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
    abort: There is no Mercurial repository here (.hg not found)!

    I've attached a patch that addresses the Python 3.5 build issue for me. The patch contains two syntax construct changes to Parser/asdl.py. Please consider applying this patch upstream.

    @vlee vlee mannequin reopened this Feb 7, 2015
    @ned-deily
    Copy link
    Member

    Unfortunately, those syntax differences aren't the only problems you could run into so making these syntax changes isn't really a general solution, IMO. 'make touch' is the documented and supported way to ensure that the unnecessary build steps are not run. But, as you discovered, 'make touch' only works with an hg copy of the repo; it was not designed for git clones. To fully support git clones, there would need to be a git equivalent of the hg scripts called by 'make touch'. This would be a worthwhile project for someone interested in git support to work on and would be considered for inclusion in the repo.

    @ned-deily ned-deily added docs Documentation in the Doc dir and removed build The build process and cross-build invalid labels Feb 7, 2015
    @ned-deily ned-deily changed the title Python 3.5 does not build with Python 2.6. 'make touch' does not work with git clones of the source repository Feb 7, 2015
    @vlee
    Copy link
    Mannequin Author

    vlee mannequin commented Feb 7, 2015

    With the supplied patch on CentOS 6 , "make" passes and "make test" passes with the exception of test_readline that is bpo-19884.

    372 tests OK.
    1 test failed:
    test_readline
    7 tests altered the execution environment:
    test_calendar test_distutils test_float test_locale test_strptime
    test_types test_warnings
    11 tests skipped:
    test_devpoll test_gdb test_kqueue test_msilib test_ossaudiodev
    test_startfile test_tk test_ttk_guionly test_winreg test_winsound
    test_zipfile64
    Re-running failed tests in verbose mode
    Re-running test 'test_readline' in verbose mode
    testHistoryUpdates (test.test_readline.TestHistoryManipulation) ... ok
    test_write_read_append (test.test_readline.TestHistoryManipulation) ... ok
    test_init (test.test_readline.TestReadline) ... FAIL

    @vlee
    Copy link
    Mannequin Author

    vlee mannequin commented Feb 7, 2015

    I tried a hg copy of the repository. "make touch" worked for me.

    ./configure
    make touch
    make

    @willingc
    Copy link
    Contributor

    Using Ned Deily's feedback, I have added a devguide patch to clarify that 'make touch' only works with a Mercurial clone and was not designed for git clones. This patch, iss23404devguide.patch, resolves the devguide portion of this issue unless anyone has further wording changes.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jul 29, 2015

    New changeset ffbb19026599 by Berker Peksag in branch 'default':
    Issue bpo-23404: Add a note to clarify that "make touch" only works with a hg clone.
    https://hg.python.org/devguide/rev/ffbb19026599

    @berkerpeksag
    Copy link
    Member

    Patch LGTM. Thanks Carol.

    I will left this issue open in case someone wants to work on a git equivalent of "make touch" as described by Ned in msg235514.

    @berkerpeksag berkerpeksag added type-feature A feature request or enhancement and removed docs Documentation in the Doc dir labels Jul 29, 2015
    @vadmium
    Copy link
    Member

    vadmium commented Sep 9, 2016

    Perhaps you can use “make -t <file>” instead of “make touch”. If you know the filename you want to avoid regenerating, that would not need any extra tools, no Mercurial or Git or any other version control. E.g.:

    make -t Include/Python-ast.h

    @vadmium
    Copy link
    Member

    vadmium commented Sep 13, 2016

    Okay so my “make -t” trick has various flaws. You still have to know the filenames to specify, it requires Makefile to be generated by configure in the source tree, and it creates empty files if you use it in a separate build directory.

    Another idea: instead of the “make touch” recipe, we add a simple shell script. Call it say “touch-bootstrap.sh”. Like what “make touch” already does in Python 2, but without embedding it as a makefile rule.

    Or add a flag variable so you can do a build without running any of the regeneration rules or worrying about timestamps: make BOOT="#".

    @yan12125
    Copy link
    Mannequin

    yan12125 mannequin commented Feb 10, 2017

    Just tried boot-flag.patch with 16e0788. Doesn't work out-of-box:

    #./Programs/_freeze_importlib \
    ./Lib/importlib/_bootstrap.py Python/importlib.h
    /bin/sh: line 1: ./Lib/importlib/_bootstrap.py: Permission denied
    make: *** [Makefile:749: Python/importlib.h] Error 126

    My fix is: merge the two lines into a single one. See my attached patch.

    By the way, the BOOT hack is quite similar to GENERATED_COMMENT in introduced in bpo-27641. Maybe they can merge?

    My environment:

    • Arch Linux x86_64
    • /bin/sh symlinks to bash, which is 4.4.12
    • GNU Make 4.2.1

    @zware
    Copy link
    Member

    zware commented Feb 11, 2017

    I've disabled the Touch step on the buildbots until this is fixed.

    @zware zware added the 3.7 (EOL) end of life label Feb 11, 2017
    @vadmium
    Copy link
    Member

    vadmium commented Apr 3, 2017

    I had mainly been using my boot-flag.patch with BSD Make (bmake) rather than Gnu Make. It seems I was relying on a bug in BSD Make that merges escaped newlines in command lines, despite Posix and Gnu Make. Anyway, Chi’s fix seems appropriate.

    I am posting the patches I used for 3.5 and 2.7, in case others are interested. The 3.5 one would also need Chi’s fix merging the “_freeze_importlib” line.

    It may be possible to have the configure script adjust the default for the BOOT flag rather than setting GENERATED_COMMENT. But the side effect would be disabling regeneration of additional things (e.g. that aren’t specifically broken with cross compilation.

    Anyway my preference was to stop automatically regenerating files, and to keep the build system as simple as practical without hacks like the BOOT="#" flag.

    @vstinner
    Copy link
    Member

    FYI 3 FreeBSD buildbots are currently broken because of this issue.

    @vstinner
    Copy link
    Member

    vstinner commented May 3, 2017

    If we go in this direction, maybe we can go further and also run autoconf && autoheader?

    Oh, I just noticed that there is already a "make autoconf" target, I didn't know.

    @vstinner
    Copy link
    Member

    vstinner commented May 3, 2017

    New changeset a5c62a8 by Victor Stinner in branch 'master':
    bpo-23404: make touch becomes make regen-all (bpo-1405)
    a5c62a8

    @vstinner
    Copy link
    Member

    vstinner commented May 3, 2017

    I just created the issue bpo-30259: Test somehow that generated files are up to date: run make regen-all.

    @vstinner
    Copy link
    Member

    vstinner commented May 4, 2017

    I started a thread on python-dev to ask if it's ok to backport the change to 2.7, 3.5 and 3.6.

    I started to backport the change to 3.6:
    #1461

    @vstinner
    Copy link
    Member

    vstinner commented May 4, 2017

    New changeset b109a1d by Victor Stinner in branch 'master':
    bpo-30273: Update sysconfig (bpo-1464)
    b109a1d

    @vstinner
    Copy link
    Member

    vstinner commented May 4, 2017

    New changeset 9d02f56 by Victor Stinner in branch '3.6':
    [3.6] bpo-23404: make touch becomes make regen-all (bpo-1405) (bpo-1461)
    9d02f56

    @vstinner
    Copy link
    Member

    vstinner commented May 5, 2017

    New changeset ab6b962 by Victor Stinner in branch '3.5':
    bpo-23404: make touch becomes make regen-all (bpo-1405) (bpo-1461) (bpo-1465)
    ab6b962

    @vstinner
    Copy link
    Member

    vstinner commented May 5, 2017

    New changeset 8a19eb2 by Victor Stinner in branch '2.7':
    bpo-23404: make touch becomes make regen-all (bpo-1466)
    8a19eb2

    @vstinner
    Copy link
    Member

    vstinner commented May 5, 2017

    I modified Python 2.7, 3.5, 3.6 and master (3.7). I prefer to keep the issue open at least one week in a regression is found, like issue bpo-30273 (this one should already be fixed).

    Maybe the devguide should be updated too?

    @ncoghlan
    Copy link
    Contributor

    ncoghlan commented May 5, 2017

    As noted on python-dev, it would be helpful if this change was mentioned in the 2.7, 3.5, and 3.6 What's New documents under a "Notable changes in maintenance releases" section.

    For 2.7, that would be a retitling of the existing section "New features added to Python 2.7 maintenance releases": https://docs.python.org/dev/whatsnew/2.7.html#new-features-added-to-python-2-7-maintenance-releases

    For 3.5 and 3.6, this would be a new entry at the end of the document, with "Change to handling of pre-generated files" as the sole subsection.

    (As another example of an existing section along these lines, see https://docs.python.org/dev/whatsnew/3.4.html#changed-in-3-4-3)

    @ned-deily
    Copy link
    Member

    As Nick noted, this change needs to be properly documented for 3.6.2 and other maintenance releases.

    @vstinner
    Copy link
    Member

    I wrote #2127 to document "make regen-all". I'm not sure that it's the correct way to document such build change in minor 3.6 releases. Can someone please take a look?

    @ncoghlan
    Copy link
    Contributor

    Thanks for that Victor. I started to review the PR, and then realised my suggestions were going to be extensive enough that it made more sense to post an alternate PR for you to review: #2128

    @ncoghlan
    Copy link
    Contributor

    New changeset e1b6903 by Nick Coghlan in branch '3.6':
    bpo-23404: make regen-all What's New entry (bpo-2128)
    e1b6903

    @ncoghlan
    Copy link
    Contributor

    Moving to deferred blocker and taking 3.6 off the affected versions list, since the 3.6 What's New has now been updated appropriately.

    Items still to be done:

    • make a similar change to the 3.5 What's New on the 3.5 branch
    • update the maintenance release changes section in the 2.7 What's New on the 2.7 branch
    • cherry-pick the 2.7, 3.5, & 3.6 updates in order to make a combined PR for the development branch

    @ned-deily
    Copy link
    Member

    There still might be time to get this into 3.5.4 final, if Larry agrees.

    @ned-deily
    Copy link
    Member

    Nick, were you planning to do the work for the other (non-3.6) branches? If not, can someone else pick this up please?

    @ncoghlan
    Copy link
    Contributor

    ncoghlan commented Oct 3, 2017

    No, I'm not currently working on this (to be completely honest, I entirely forgot we hadn't backported the What's New updates to the various branches).

    They'll need to be independent updates, since they affect a different file for each release (although they can then be cherry-picked forward into the development branch)

    @ned-deily
    Copy link
    Member

    New changeset 9d6171d by Ned Deily in branch 'master':
    bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 (GH-7620)
    9d6171d

    @ned-deily
    Copy link
    Member

    New changeset 144493d by Ned Deily (Miss Islington (bot)) in branch '3.7':
    bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 (GH-7620) (GH-7621)
    144493d

    @ned-deily
    Copy link
    Member

    New changeset 1b5731e by Ned Deily in branch '3.6':
    bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 (GH-7622)
    1b5731e

    @ned-deily
    Copy link
    Member

    New changeset 4fde701 by Ned Deily in branch '2.7':
    bpo-23404: Update/sync What's New files for 3.6/3.5/2.7 (GH-7624)
    4fde701

    @ned-deily
    Copy link
    Member

    I've updated and synced the What's New files across master, 3.7, 3.6, 3.5 (PR pending RM merge), and 2.7. So I think we are finally done here.

    @ned-deily ned-deily added the 3.8 (EOL) end of life label Jun 11, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 (EOL) end of life build The build process and cross-build type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    8 participants