Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

distutils.sysconfig.parse_makefile() regression in Python 3.10 #88517

Closed
hroncok mannequin opened this issue Jun 8, 2021 · 6 comments
Closed

distutils.sysconfig.parse_makefile() regression in Python 3.10 #88517

hroncok mannequin opened this issue Jun 8, 2021 · 6 comments
Labels
3.10 only security fixes 3.11 bug and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@hroncok
Copy link
Mannequin

hroncok mannequin commented Jun 8, 2021

BPO 44351
Nosy @encukou, @hroncok, @frenzymadness, @miss-islington
PRs
  • bpo-44351: Restore back parse_makefile in distutils.sysconfig #26637
  • [3.10] bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637) #26673
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-10-20.14:46:13.099>
    created_at = <Date 2021-06-08.19:00:01.911>
    labels = ['type-bug', 'library', '3.10', '3.11']
    title = 'distutils.sysconfig.parse_makefile() regression in Python 3.10'
    updated_at = <Date 2021-10-20.14:46:13.098>
    user = 'https://github.com/hroncok'

    bugs.python.org fields:

    activity = <Date 2021-10-20.14:46:13.098>
    actor = 'petr.viktorin'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-10-20.14:46:13.099>
    closer = 'petr.viktorin'
    components = ['Library (Lib)']
    creation = <Date 2021-06-08.19:00:01.911>
    creator = 'hroncok'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 44351
    keywords = ['patch']
    message_count = 6.0
    messages = ['395352', '395385', '395387', '395402', '395638', '395873']
    nosy_count = 4.0
    nosy_names = ['petr.viktorin', 'hroncok', 'frenzy', 'miss-islington']
    pr_nums = ['26637', '26673']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue44351'
    versions = ['Python 3.10', 'Python 3.11']

    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Jun 8, 2021

    Hello. I think #23142 changed the behavior of distutils.sysconfig.parse_makefile().

    A downstream Fedora report with an affected petsc package: https://bugzilla.redhat.com/show_bug.cgi?id=1959088

    Reproducers (DeprecationWarnings removed for readability):

    $ cat makefile
    ALL: lib
    DIRS = ssls asls # isls rscs
    LOCDIR = src/tao/complementarity/impls/

    include ${PETSC_DIR}/lib/petsc/conf/variables
    include ${PETSC_DIR}/lib/petsc/conf/rules
    include ${PETSC_DIR}/lib/petsc/conf/test

    $ python3.9
    >>> from distutils.sysconfig import parse_makefile
    >>> makevars = parse_makefile('makefile')
    >>> makevars.get('DIRS','').split()
    ['ssls', 'asls']
    
    $ python3.10
    >>> from distutils.sysconfig import parse_makefile
    >>> makevars = parse_makefile('makefile')
    >>> makevars.get('DIRS','').split()
    ['ssls', 'asls', '#', 'isls', 'rscs']

    And:

    $ cat makefile
    -include ../../../../petscdir.mk
    ALL: lib
    LIBBASE  = libpetscksp
    DIRS     = cr bcgs bcgsl cg cgs gmres cheby rich lsqr preonly tcqmr tfqmr \
               qcg bicg minres symmlq lcd ibcgs python gcr fcg tsirm fetidp hpddm
    LOCDIR   = src/ksp/ksp/impls/

    include ${PETSC_DIR}/lib/petsc/conf/variables
    include ${PETSC_DIR}/lib/petsc/conf/rules
    include ${PETSC_DIR}/lib/petsc/conf/test

    $ python3.9
    >>> from distutils.sysconfig import parse_makefile
    >>> makevars = parse_makefile('makefile')
    >>> makevars.get('DIRS','').split()
    ['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', 'qcg', 'bicg', 'minres', 'symmlq', 'lcd', 'ibcgs', 'python', 'gcr', 'fcg', 'tsirm', 'fetidp', 'hpddm']
    
    $ python3.10
    >>> from distutils.sysconfig import parse_makefile
    >>> makevars = parse_makefile('makefile')
    >>> makevars.get('DIRS','').split()
    ['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', '\\']

    And:

    $ cat makefile
    -include ../../../../petscdir.mk
    ALL: lib
    LIBBASE  = libpetscksp
    DIRS     = jacobi none sor shell bjacobi mg eisens asm ksp composite redundant spai is pbjacobi vpbjacobi ml\
               mat hypre tfs fieldsplit factor galerkin cp wb python \
               chowiluviennacl chowiluviennaclcuda rowscalingviennacl rowscalingviennaclcuda saviennacl saviennaclcuda\
               lsc redistribute gasm svd gamg parms bddc kaczmarz telescope patch lmvm hmg deflation hpddm hara
    LOCDIR   = src/ksp/pc/impls/

    include ${PETSC_DIR}/lib/petsc/conf/variables
    include ${PETSC_DIR}/lib/petsc/conf/rules
    include ${PETSC_DIR}/lib/petsc/conf/test

    $ python3.9
    >>> from distutils.sysconfig import parse_makefile
    >>> makevars = parse_makefile('makefile')
    >>> makevars.get('DIRS','').split()
    ['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml', 'mat', 'hypre', 'tfs', 'fieldsplit', 'factor', 'galerkin', 'cp', 'wb', 'python', 'chowiluviennacl', 'chowiluviennaclcuda', 'rowscalingviennacl', 'rowscalingviennaclcuda', 'saviennacl', 'saviennaclcuda', 'lsc', 'redistribute', 'gasm', 'svd', 'gamg', 'parms', 'bddc', 'kaczmarz', 'telescope', 'patch', 'lmvm', 'hmg', 'deflation', 'hpddm', 'hara']
    
    $ python3.10
    >>> from distutils.sysconfig import parse_makefile
    >>> makevars = parse_makefile('makefile')
    >>> makevars.get('DIRS','').split()
    ['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml\\']

    @hroncok hroncok mannequin added 3.10 only security fixes 3.11 bug and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jun 8, 2021
    @frenzymadness
    Copy link
    Mannequin

    frenzymadness mannequin commented Jun 9, 2021

    Thanks for the report.

    The regression is caused by the fact that the old implementation of parse_makefile in distutils.sysconfig was using feature-rich class TextFile which handles all the functionalities you reported as broken - stripping of comments, joining lines, and more.

    The new implementation of parse_makefile in sysconfig just reads all the lines without any processing.

    The TextFile class is deprecated together with the whole distutils. The class is not used outside of distutils modules, only in these functions:

    distutils.extension:read_setup_file
    distutils.sdist:read_template
    + its tests in distutils/tests/test_text_file.py

    If the functionality is something we want to preserve, we should find a new home for the class and then we can use it in the sysconfig module.

    @encukou
    Copy link
    Member

    encukou commented Jun 9, 2021

    IMO, the functionality should only be preserved until distutils is removed. So:

    • distutils.sysconfig.parse_makefile should use TextFile as before, so projects that use it aren't broken *yet*
    • nothing else should call distutils.sysconfig.parse_makefile, so it doesn't block removing distutils

    Would that make sense?

    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Jun 9, 2021

    Yes, for the purposes of this bug, bringing TextFile-powered parse_makefile() back to distutils (and distutils only) is the right thing to do.

    Whether or not Python needs a public standard library function to parse makefiles and whether that function in sysconfig is imperfect and needs improvements, that is an entirely different discussion.

    @encukou
    Copy link
    Member

    encukou commented Jun 11, 2021

    New changeset fc98266 by Lumír 'Frenzy' Balhar in branch 'main':
    bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637)
    fc98266

    @encukou
    Copy link
    Member

    encukou commented Jun 15, 2021

    New changeset 2f2ea96 by Miss Islington (bot) in branch '3.10':
    bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637) (GH-26673)
    2f2ea96

    @encukou encukou closed this as completed Oct 20, 2021
    @encukou encukou closed this as completed Oct 20, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes 3.11 bug and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant