LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found

[neyuru]

BPO	35299
Nosy	@pfmoore, @tjguk, @merwok, @zware, @zooba, @dstufft, @miss-islington, @neyuru
PRs	bpo-35299: Fixes sysconfig and distutils during PGO profiling #11744 bpo-35299: Fixes sysconfig and distutils during PGO profiling #11744 bpo-35299: Fixes sysconfig and distutils during PGO profiling #11744 [3.7] bpo-35299: Fixed sysconfig and distutils during PGO profiling (GH-11744) #11758 [3.7] bpo-35299: Fixed sysconfig and distutils during PGO profiling (GH-11744) #11758 [3.7] bpo-35299: Fixed sysconfig and distutils during PGO profiling (GH-11744) #11758
Files	LIBenRepogiT.txt: Verbose of the compilation with the steps shown

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2019-02-05.01:55:19.153>
created_at = <Date 2018-11-23.06:31:25.303>
labels = ['3.7', '3.8', 'library', 'build', 'tests', 'OS-windows']
title = "LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found"
updated_at = <Date 2019-02-05.01:55:17.147>
user = 'https://github.com/neyuru'

bugs.python.org fields:

activity = <Date 2019-02-05.01:55:17.147>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2019-02-05.01:55:19.153>
closer = 'steve.dower'
components = ['Distutils', 'Tests', 'Windows']
creation = <Date 2018-11-23.06:31:25.303>
creator = 'neyuru'
dependencies = []
files = ['47941']
hgrepos = []
issue_num = 35299
keywords = ['patch', 'patch', 'patch']
message_count = 6.0
messages = ['330299', '330347', '334040', '334780', '334853', '334855']
nosy_count = 8.0
nosy_names = ['paul.moore', 'tim.golden', 'eric.araujo', 'zach.ware', 'steve.dower', 'dstufft', 'miss-islington', 'neyuru']
pr_nums = ['11744', '11744', '11744', '11758', '11758', '11758']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue35299'
versions = ['Python 3.7', 'Python 3.8']

[neyuru]

Build of Python 3.6.7 fails with error
"LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found"
I am using Windows 10 64bit.

Steps to reproduce this error:

1. Install Visual Studio 2017 (With the python development and native dev tools) (I have also installed NVIDIA CUDA Toolkit but I don't think this matters)
2. Download source code from Python to folder E:\RepoGit\3.6 (this is the download folder)
3. Open a command prompt, navigate to "E:\RepoGit" and run "PCBuild\build.bat -p x64"
4. Close the command prompt
5. Copy file pyconfig.h file from E:\RepoGit\3.6\PC and paste it to E:\RepoGit\3.6\include. Also, create an environment variable called "LIB" that contains the folder "E:\RepoGiT\3.6\PCbuild\amd64" (this folder was created at step 3)
6. Open a command prompt and run "Tools\msi\buildrelease.bat -x64"

NOTE1: if step 5 is skipped, the overall build does not fail, but the PGInstrument x64 Interpreter fails in distutils with error "E:\RepoGiT\3.6\include\Python.h(8): fatal error C1083: Cannot open include file: 'pyconfig.h': No such file or directory", as shown in message https://bugs.python.org/msg329247 of issue https://bugs.python.org/issue35157

NOTE2: if pyconfig.h in step 5 is copied but the environment variable is not created, the PGInstrument x64 Interpreter fails in distutils with error "LINK : fatal error LNK1104: cannot open file 'python36.lib'", and the overall build fails with the same error shown in the title of this Issue.

NOTE3: If all the steps are run, the PGInstrument x64 Interpreter succeeds in its tests but the overall build fails with the error shown in the title of this issue. All the verbose of this compilation is included in the file attached.

[zooba]

I'm going to declare that this is a bug in distutils not correctly setting INCLUDE and LIB values when building in a source directory. When this is fixed, there will be no need to copy pyconfig.h anywhere, and the installer build will be fine.

(And it should be noted that we don't rely on 100% pass rate of the test suite when training for PGO - the point is to exercise as many code paths as possible. The test suite has shown to be as good as any other generic workload, and is easily available in our repo.)

[neyuru]

I want to document a workaround if anyone benefits from it, this would be step 7:

7. This error goes away if you remove the pyconfig.h from the include directory before the tests end (and of course, after the test distutils).

This is only a workaround and requires the user placing attention to the build process (maybe for the first 5-10 minutes). Having the option "-q" removed from the buildrelease.bat file (as discussed in issue https://bugs.python.org/issue35739 ) helps with determining when to safely remove the file so that the distutils test pass.

[zooba]

My PR fixes both issues properly, but adds a minor "feature" to distutils by making distutils.sysconfig.get_python_inc() potentially return a set of include paths (with os.path.pathsep separators).

This is fully internal, so right now we know it'll only occur when building in the source tree on Windows, so I'm not concerned about it. It'd also work on any platform through an environment variable, which it is, but it looks like most people simply attach "-I" at the start and pass it to gcc (which again, since they're not on Windows, they're not going to hit the one case we know about).

I'd rather do it properly than have a one-time hack just for this build scenario, and I certainly don't want to be copying files within the source tree. So I'm intending to go ahead with this design, but wanted to allow any interested parties to push back if they know of some reason that I don't.

[zooba]

New changeset 85e102a by Steve Dower in branch 'master':
bpo-35299: Fixed sysconfig and distutils during PGO profiling (GH-11744)
85e102a

[miss-islington]

New changeset 4f6854a by Miss Islington (bot) in branch '3.7':
bpo-35299: Fixed sysconfig and distutils during PGO profiling (GH-11744)
4f6854a