regen-importlib is causing build races against other regen-all targets in Makefile.pre.in

[kanavin]

BPO	33080
Nosy	@vstinner, @tiran, @kanavin

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-07.11:44:05.055>
created_at = <Date 2018-03-15.16:46:10.490>
labels = ['3.8', 'build', '3.7']
title = 'regen-importlib is causing build races against other regen-all targets in Makefile.pre.in'
updated_at = <Date 2021-12-07.11:56:49.294>
user = 'https://github.com/kanavin'

bugs.python.org fields:

activity = <Date 2021-12-07.11:56:49.294>
actor = 'christian.heimes'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-07.11:44:05.055>
closer = 'Alexander Kanavin'
components = ['Build']
creation = <Date 2018-03-15.16:46:10.490>
creator = 'Alexander Kanavin'
dependencies = []
files = []
hgrepos = []
issue_num = 33080
keywords = []
message_count = 5.0
messages = ['313894', '407917', '407918', '407921', '407925']
nosy_count = 3.0
nosy_names = ['vstinner', 'christian.heimes', 'Alexander Kanavin']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33080'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[kanavin]

You can see here:

https://github.com/python/cpython/blob/master/Makefile.pre.in#L708

that regen-importlib is building a binary from .o files which are built from .c and .h files, which are, at the same time, regenerated by other regen- targets.

This does cause build errors in heavily parallelized builds, we've been seeing it regularly in Yocto Project lately:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12596

I tried to see if I can easily correct target dependencies in the makefile, but couldn't figure it out. So, a workaround, for us, would be to issue 'make regen-importlib' ahead of other things:

make regen-importlib
make regen-all

[tiran]

May I ask why you are running the regen-all target at all? A normal end-user build of CPython does not need make regen-all. It's only used for CI and internal development.

[kanavin]

We have long ago updated to a much newer python and removed the workaround, so the whatever the issue was, it is completely obsolete. Thanks!

[kanavin]

(removed both the workaround, and regen-all itself)

[tiran]

Fantasic!