PR check "Check if generated files are up to date" failing intermittently

[ericsnowcurrently]

BPO	47146
Nosy	@tiran, @markshannon, @ericsnowcurrently, @brandtbucher
PRs	bpo-47146: Eliminate a Race Between make regen-deepfreeze and make regen-global-objects #32162 bpo-47146: Avoid Using make Recursively #32206 bpo-47146: Stop Depending On regen-deepfreeze For regen-global-objects #32218

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2022-04-01.15:37:10.755>
created_at = <Date 2022-03-28.16:50:08.590>
labels = ['build', '3.11']
title = 'PR check "Check if generated files are up to date" failing intermittently'
updated_at = <Date 2022-04-01.15:37:10.755>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2022-04-01.15:37:10.755>
actor = 'eric.snow'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2022-04-01.15:37:10.755>
closer = 'eric.snow'
components = ['Build']
creation = <Date 2022-03-28.16:50:08.590>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 47146
keywords = ['patch']
message_count = 13.0
messages = ['416193', '416201', '416211', '416214', '416379', '416396', '416410', '416443', '416444', '416454', '416456', '416457', '416496']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'Mark.Shannon', 'eric.snow', 'brandtbucher']
pr_nums = ['32162', '32206', '32218']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue47146'
versions = ['Python 3.11']

[ericsnowcurrently]

The "Check if generated files are up to date" GitHub check for PRs has been failing recently. It may also impact local usage of "make regen-all".

Example: https://github.com/python/cpython/runs/5719012664

This may be related to #76242.

[ericsnowcurrently]

There's probably something racy with make.

See: https://github.com/python/cpython/runs/5712538599?check_suite_focus=true#step:10:1147

[ericsnowcurrently]

New changeset 4c116f7 by Eric Snow in branch 'main':
bpo-47146: Eliminate a race between make regen-deepfreeze and make regen-global-objects. (gh-32162)
4c116f7

[ericsnowcurrently]

Looks like that fixed it, per #32134.

[ericsnowcurrently]

Looks like this is still an intermittent problem:

• bpo-47168: Mark files generated by make regen-limited-abi as generated #32195
  + failed: https://github.com/python/cpython/runs/5756616733
  + failed: https://github.com/python/cpython/runs/5753267869
  + failed: https://github.com/python/cpython/runs/5757169625
• bpo-40421: Add missing getters for frame object attributes to C-API. #32114
  + failed: https://github.com/python/cpython/runs/5756616733
  + passed: https://github.com/python/cpython/runs/5757213346
• bpo-47151: Fallback to fork when vfork fails in subprocess. #32186
  + failed: ...
  + passed: https://github.com/python/cpython/runs/5757178754

Per Mark Shannon (on discord):

The "Check if generated files are up to date" is still failing consistently. It looks like the makefile is missing a dependency on the ./Programs/_freeze_module for targets that require /Programs/_freeze_module

[tiran]

The problem is likely a recursive make call "$(MAKE)" in combination with parallel execution. As far as I understand it, GNU make cannot track dependencies properly, which can lead to partial or invalid dependency graphs. One make process can modify a file while another uses the file. The permission error or ETXTBSY error are manifestation of one process creating an executable while another process is attempting to execve() it.

The paper https://accu.org/journals/overload/14/71/miller_2004/ explains the issues with recursion in great detail.

[ericsnowcurrently]

New changeset db4dada by Eric Snow in branch 'main':
bpo-47146: Avoid Using make Recursively (gh-32206)
db4dada

[ericsnowcurrently]

I re-ran jobs that had failed before I merged that #76387. Several passed, but the following are still failing:

• [WIP] bpo-23689: re module, fix memory leak when a match is terminated by a signal #32188
  + https://github.com/python/cpython/runs/5773938424
• bpo-47131: Speedup AST comparisons in test_unparse by using node traversal #32132
  + https://github.com/python/cpython/runs/5774054192
• bpo-47152: Convert the re module into a package #32177
  + https://github.com/python/cpython/runs/5773949869

Back to the drawing board...

[ericsnowcurrently]

Brandt pointed out this is consistently reproducible locally:

make clean regen-all -j

I'll get this sorted out today.

[ericsnowcurrently]

New changeset e7bb7c2 by Eric Snow in branch 'main':
bpo-47146: Stop Depending On regen-deepfreeze For regen-global-objects (gh-32218)
e7bb7c2

[ericsnowcurrently]

I'll keep an eye on PRs for the next day or so.

[ericsnowcurrently]

Specifically: https://github.com/python/cpython/actions/workflows/build.yml.

[ericsnowcurrently]

Looks like #76399 worked.