Makefile is not correctly generated when compiling pyextat with DXML_POOR_ENTROPY=1

[mgmacias95]

BPO	35184
Nosy	@benjaminp, @ned-deily, @miss-islington, @mgmacias95, @aaronpaulhurst
PRs	bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup.dist #12900 bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing #13064 [3.8] closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064) #13960 [3.7] closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064) #13961

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-11.01:54:27.058>
created_at = <Date 2018-11-07.16:47:20.059>
labels = ['3.8', 'build', '3.7']
title = 'Makefile is not correctly generated when compiling pyextat with DXML_POOR_ENTROPY=1'
updated_at = <Date 2019-06-11.02:15:27.539>
user = 'https://github.com/mgmacias95'

bugs.python.org fields:

activity = <Date 2019-06-11.02:15:27.539>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-11.01:54:27.058>
closer = 'benjamin.peterson'
components = ['Build']
creation = <Date 2018-11-07.16:47:20.059>
creator = 'mgmacias95'
dependencies = []
files = []
hgrepos = []
issue_num = 35184
keywords = ['patch']
message_count = 10.0
messages = ['329424', '340630', '341279', '341292', '341293', '341297', '341304', '345174', '345175', '345176']
nosy_count = 5.0
nosy_names = ['benjamin.peterson', 'ned.deily', 'miss-islington', 'mgmacias95', 'ahurst']
pr_nums = ['12900', '13064', '13960', '13961']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue35184'
versions = ['Python 3.7', 'Python 3.8']

[mgmacias95]

Hello,

I was getting this same error (https://bugs.python.org/issue35139) when compiling pyextat so I added these changes (https://github.com/python/cpython/pull/10291/files) to my Setup.dist.

After running ./configure, the generated Makefile throws the following error:

$ make
Makefile:264: *** missing separator.  Stop.

I have tried this in both Python 3.7 and 3.6 and I get the error in both (the line numbers are different but the error is the same). I have tested this in both Ubuntu 16/18 and CentOS 7.

Any hints in how could I fix this?

Best regards,
Marta

[aaronpaulhurst]

I can reproduce this issue by uncommenting the pyexpat line in Setup.dist and compiling.

The issue is with -DXML_POOR_ENTROPY=1. The equals character causes the line to be incorrectly interpreted as a macro definition by makesetup. This results in an invalid Makefile output.

I've submitted a PR, but a quick work-around is to remove the "=1". It is not necessary.

[ned-deily]

Thanks for the report and thanks for the PR. I am confused here as I cannot reproduce the failure. Can someone show what is invalid in the Makefile? Also, if the change really does need to be made to Setup.dist (3.7) or Setup (master -> 3.8), then setup.py should likely be changed to reduce future confusion (although it presumably doesn't make a difference to the compile). Further, please submit PRs like this against the master branch (unless the problem only exists in a specific released branches); fixes are normally applied to the master branch and the core developer handling the merge will elect to backport to appropriate branches as needed.

[aaronpaulhurst]

Hi Ned,

From a fresh checkout of master on Ubuntu 18.04, I uncomment the pyexpat line in Modules/Setup and run:

cpython$ ./configure
...
cpython$ make
Makefile:273: *** missing separator. Stop.

Here is the offending section of the resulting Makefile:

269 # === Definitions added by makesetup ===
270
271 LOCALMODLIBS=
272 BASEMODLIBS=
273 pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DXML_$
274 PYTHONPATH=$(COREPYTHONPATH)
275 COREPYTHONPATH=$(DESTPATH)$(SITEPATH)$(TESTPATH)

[aaronpaulhurst]

Sorry for my misunderstanding of the process, and thanks for explaining. I resubmitted the PR against the master branch.

[ned-deily]

Thanks for the updates, Aaron. The plot thickens! If I perform the steps on a current macOS system, the result is:

273 pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI

which works just fine. So it seems the underlying problem is actually a difference in the behavior of the makesetup shell script, most likely some difference in one of the utilities it uses, like sed, perhaps a BSD vs GNU difference.

I won't have more time to look into this further today but feel free to do so if you like. In any case, we should fix the root problem in makesetup rather than trying to work around it elsewhere.

[aaronpaulhurst]

Hi Ned,

Thanks for testing this. I also observe that macOS compiles "without error"... but it's still broken... and silently.

This is because the pyexpat line isn't being turned into the expected set of source compilation rules, but it is instead being dumped into the variable definition section. Why is it being interpreted by makesetup as a variable definition? With the equals character, it matches this pattern:

# Lines can also have the form

# <name> = <value>

# which defines a Make variable definition inserted into Makefile.in

But it is intended to match this pattern:

# Lines have the following structure:

# <module> ... [<sourcefile> ...] [<cpparg> ...] [<library> ...]

For reference, here is the corresponding rule in makesetup:

*=*) DEFS="$line$NL$DEFS"; continue;;

I fully support tweaking this pattern to better differentiate when "=" means a variable definition and when "=" is part of a compilation flag, but given that pyexpat is the only such case, my one-line fix makes things consistent again. For now.

[benjaminp]

New changeset 408a2ef by Benjamin Peterson (aaronpaulhurst) in branch 'master':
closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
408a2ef

[miss-islington]

New changeset 5b94b85 by Miss Islington (bot) in branch '3.8':
closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
5b94b85

[miss-islington]

New changeset 30fd7a4 by Miss Islington (bot) in branch '3.7':
closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
30fd7a4