distutils.sysconfig.parse_makefile() regression in Python 3.10

[hroncok]

BPO	44351
Nosy	@encukou, @hroncok, @frenzymadness, @miss-islington
PRs	bpo-44351: Restore back parse_makefile in distutils.sysconfig #26637 [3.10] bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637) #26673

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-20.14:46:13.099>
created_at = <Date 2021-06-08.19:00:01.911>
labels = ['type-bug', 'library', '3.10', '3.11']
title = 'distutils.sysconfig.parse_makefile() regression in Python 3.10'
updated_at = <Date 2021-10-20.14:46:13.098>
user = 'https://github.com/hroncok'

bugs.python.org fields:

activity = <Date 2021-10-20.14:46:13.098>
actor = 'petr.viktorin'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-20.14:46:13.099>
closer = 'petr.viktorin'
components = ['Library (Lib)']
creation = <Date 2021-06-08.19:00:01.911>
creator = 'hroncok'
dependencies = []
files = []
hgrepos = []
issue_num = 44351
keywords = ['patch']
message_count = 6.0
messages = ['395352', '395385', '395387', '395402', '395638', '395873']
nosy_count = 4.0
nosy_names = ['petr.viktorin', 'hroncok', 'frenzy', 'miss-islington']
pr_nums = ['26637', '26673']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44351'
versions = ['Python 3.10', 'Python 3.11']

[hroncok]

Hello. I think #23142 changed the behavior of distutils.sysconfig.parse_makefile().

A downstream Fedora report with an affected petsc package: https://bugzilla.redhat.com/show_bug.cgi?id=1959088

Reproducers (DeprecationWarnings removed for readability):

$ cat makefile
ALL: lib

DIRS = ssls asls # isls rscs
LOCDIR = src/tao/complementarity/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules
include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['ssls', 'asls']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['ssls', 'asls', '#', 'isls', 'rscs']

And:

$ cat makefile
-include ../../../../petscdir.mk
ALL: lib

LIBBASE  = libpetscksp
DIRS     = cr bcgs bcgsl cg cgs gmres cheby rich lsqr preonly tcqmr tfqmr \
           qcg bicg minres symmlq lcd ibcgs python gcr fcg tsirm fetidp hpddm
LOCDIR   = src/ksp/ksp/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules
include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', 'qcg', 'bicg', 'minres', 'symmlq', 'lcd', 'ibcgs', 'python', 'gcr', 'fcg', 'tsirm', 'fetidp', 'hpddm']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['cr', 'bcgs', 'bcgsl', 'cg', 'cgs', 'gmres', 'cheby', 'rich', 'lsqr', 'preonly', 'tcqmr', 'tfqmr', '\\']

And:

$ cat makefile
-include ../../../../petscdir.mk
ALL: lib

LIBBASE  = libpetscksp
DIRS     = jacobi none sor shell bjacobi mg eisens asm ksp composite redundant spai is pbjacobi vpbjacobi ml\
           mat hypre tfs fieldsplit factor galerkin cp wb python \
           chowiluviennacl chowiluviennaclcuda rowscalingviennacl rowscalingviennaclcuda saviennacl saviennaclcuda\
           lsc redistribute gasm svd gamg parms bddc kaczmarz telescope patch lmvm hmg deflation hpddm hara
LOCDIR   = src/ksp/pc/impls/

include ${PETSC_DIR}/lib/petsc/conf/variables
include ${PETSC_DIR}/lib/petsc/conf/rules
include ${PETSC_DIR}/lib/petsc/conf/test

$ python3.9
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml', 'mat', 'hypre', 'tfs', 'fieldsplit', 'factor', 'galerkin', 'cp', 'wb', 'python', 'chowiluviennacl', 'chowiluviennaclcuda', 'rowscalingviennacl', 'rowscalingviennaclcuda', 'saviennacl', 'saviennaclcuda', 'lsc', 'redistribute', 'gasm', 'svd', 'gamg', 'parms', 'bddc', 'kaczmarz', 'telescope', 'patch', 'lmvm', 'hmg', 'deflation', 'hpddm', 'hara']

$ python3.10
>>> from distutils.sysconfig import parse_makefile
>>> makevars = parse_makefile('makefile')
>>> makevars.get('DIRS','').split()
['jacobi', 'none', 'sor', 'shell', 'bjacobi', 'mg', 'eisens', 'asm', 'ksp', 'composite', 'redundant', 'spai', 'is', 'pbjacobi', 'vpbjacobi', 'ml\\']

[frenzymadness]

Thanks for the report.

The regression is caused by the fact that the old implementation of parse_makefile in distutils.sysconfig was using feature-rich class TextFile which handles all the functionalities you reported as broken - stripping of comments, joining lines, and more.

The new implementation of parse_makefile in sysconfig just reads all the lines without any processing.

The TextFile class is deprecated together with the whole distutils. The class is not used outside of distutils modules, only in these functions:

distutils.extension:read_setup_file
distutils.sdist:read_template
+ its tests in distutils/tests/test_text_file.py

If the functionality is something we want to preserve, we should find a new home for the class and then we can use it in the sysconfig module.

[encukou]

IMO, the functionality should only be preserved until distutils is removed. So:

• distutils.sysconfig.parse_makefile should use TextFile as before, so projects that use it aren't broken *yet*
• nothing else should call distutils.sysconfig.parse_makefile, so it doesn't block removing distutils

Would that make sense?

[hroncok]

Yes, for the purposes of this bug, bringing TextFile-powered parse_makefile() back to distutils (and distutils only) is the right thing to do.

Whether or not Python needs a public standard library function to parse makefiles and whether that function in sysconfig is imperfect and needs improvements, that is an entirely different discussion.

[encukou]

New changeset fc98266 by Lumír 'Frenzy' Balhar in branch 'main':
bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637)
fc98266

[encukou]

New changeset 2f2ea96 by Miss Islington (bot) in branch '3.10':
bpo-44351: Restore back parse_makefile in distutils.sysconfig (GH-26637) (GH-26673)
2f2ea96