gh-115859: Re-enable T2 optimizer pass by default #116062

gvanrossum · 2024-02-28T19:57:28Z

This undoes the temporary default disabling of the T2 optimizer pass in gh-115860.

Add a new test that reproduces Brandt's example from Type propagation: just because something is const doesn't mean it automatically matches the type #115859; it indeed crashes before gh-115816: Improve internal symbols API in optimizer #116028 with PYTHONUOPSOPTIMIZE=1
Re-enable the optimizer pass in T2, stop checking PYTHONUOPSOPTIMIZE
Rename the env var to disable T2 entirely to PYTHON_UOPS_OPTIMIZE (must be explicitly set to 0 to disable)
Fix skipIf conditions on tests in test_opt.py accordingly
Export sym_is_bottom() (for debugging)
Fix various things in the _BINARY_OP_ specializations in the abstract interpreter:
- DECREF(temp)
- out-of-space check after sym_new_const()
- add sym_matches_type() checks, so even if we somehow reach a binary op with symbolic constants of the wrong type on the stack we won't trigger the type assert

@Fidget-Spinner Please review when you get up. :-)

Fixes gh-115859.

gvanrossum · 2024-02-28T19:58:01Z

Once tests pass I will try to remember to run the leaks buildbot.

Fidget-Spinner

Just a comment on refcounts, everything else looks good to me.

Fidget-Spinner · 2024-02-28T20:41:35Z

Python/optimizer_bytecodes.c

@@ -118,14 +121,17 @@ dummy_func(void) {
            OUT_OF_SPACE_IF_NULL(res = sym_new_const(ctx, temp));


Don't you need to decref it here too on this error branch? Or am I remembering the ownership wrongly? Same for everything below.

Might be better to write it as:

res = sym_new_const(ctx, temp); Py_DECREF(temp); OUT_OF_SPACE_IF_NULL(res);

Good catch!

…ut of space

gvanrossum · 2024-02-28T21:10:41Z

There seems to be wrong with test_external_inspection.py on all the JIT / aarch64-unknown-linux CI runs; from one of the error logs:

Re-running test_external_inspection in verbose mode (matching: test_remote_stack_trace, test_self_trace)
test_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_remote_stack_trace) ... ERROR
test_self_trace (test.test_external_inspection.TestGetStackTrace.test_self_trace) ... ERROR

======================================================================
ERROR: test_remote_stack_trace (test.test_external_inspection.TestGetStackTrace.test_remote_stack_trace)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_external_inspection.py", line 59, in test_remote_stack_trace
    stack_trace = get_stack_trace(p.pid)
                  ~~~~~~~~~~~~~~~^^^^^^^
OSError: [Errno 38] Function not implemented

======================================================================
ERROR: test_self_trace (test.test_external_inspection.TestGetStackTrace.test_self_trace)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_external_inspection.py", line 80, in test_self_trace
    stack_trace = get_stack_trace(os.getpid())
                  ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
OSError: [Errno 38] Function not implemented

----------------------------------------------------------------------
Ran 2 tests in 0.342s

FAILED (errors=2)
test test_external_inspection failed
1 test failed again:
    test_external_inspection

bedevere-bot · 2024-02-28T21:12:25Z

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit bbe8cd9 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

gvanrossum · 2024-02-28T22:37:56Z

Argh, test_type_inconsistency fails when run multiple times. Fix coming up. EDIT: In a separate PR (#116079).

bedevere-bot · 2024-02-28T23:39:47Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu NoGIL Refleaks 3.x has failed when building commit 3409bc2.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1226/builds/1341) and take a look at the build logs.
Check if the failure is related to this commit (3409bc2) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1226/builds/1341

Failed tests:

test_capi

Failed subtests:

test_type_inconsistency - test.test_capi.test_opt.TestUopsOptimization.test_type_inconsistency

Test leaking resources:

test_processes: memory blocks

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/ubuntu/buildarea/3.x.itamaro-ubuntu-aws.refleak.nogil/build/Lib/test/test_capi/test_opt.py", line 903, in test_type_inconsistency
    self.assertIsNone(ex)
    ~~~~~~~~~~~~~~~~~^^^^
AssertionError: <uop_executor object at 0x200006b2d90> is not None


Traceback (most recent call last):
  File "/home/ubuntu/buildarea/3.x.itamaro-ubuntu-aws.refleak.nogil/build/Lib/test/test_capi/test_opt.py", line 903, in test_type_inconsistency
    self.assertIsNone(ex)
    ~~~~~~~~~~~~~~~~~^^^^
AssertionError: <uop_executor object at 0x200006b1dd0> is not None

bedevere-bot · 2024-02-28T23:46:14Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 MacOS M1 Refleaks NoGIL 3.x has failed when building commit 3409bc2.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1368/builds/352) and take a look at the build logs.
Check if the failure is related to this commit (3409bc2) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1368/builds/352

Failed tests:

test_capi

Failed subtests:

test_type_inconsistency - test.test_capi.test_opt.TestUopsOptimization.test_type_inconsistency

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/test_capi/test_opt.py", line 903, in test_type_inconsistency
    self.assertIsNone(ex)
    ~~~~~~~~~~~~~~~~~^^^^
AssertionError: <uop_executor object at 0x200006b2d90> is not None


Traceback (most recent call last):
  File "/Users/ec2-user/buildbot/buildarea/3.x.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/test_capi/test_opt.py", line 903, in test_type_inconsistency
    self.assertIsNone(ex)
    ~~~~~~~~~~~~~~~~~^^^^
AssertionError: <uop_executor object at 0x200006b1190> is not None

…6079) This should fix the refleaks bots. (See #116062 (comment) .)

This undoes the *temporary* default disabling of the T2 optimizer pass in pythongh-115860. - Add a new test that reproduces Brandt's example from pythongh-115859; it indeed crashes before pythongh-116028 with PYTHONUOPSOPTIMIZE=1 - Re-enable the optimizer pass in T2, stop checking PYTHONUOPSOPTIMIZE - Rename the env var to disable T2 entirely to PYTHON_UOPS_OPTIMIZE (must be explicitly set to 0 to disable) - Fix skipIf conditions on tests in test_opt.py accordingly - Export sym_is_bottom() (for debugging) - Fix various things in the `_BINARY_OP_` specializations in the abstract interpreter: - DECREF(temp) - out-of-space check after sym_new_const() - add sym_matches_type() checks, so even if we somehow reach a binary op with symbolic constants of the wrong type on the stack we won't trigger the type assert

python#116079) This should fix the refleaks bots. (See python#116062 (comment) .)

This undoes the *temporary* default disabling of the T2 optimizer pass in pythongh-115860. - Add a new test that reproduces Brandt's example from pythongh-115859; it indeed crashes before pythongh-116028 with PYTHONUOPSOPTIMIZE=1 - Re-enable the optimizer pass in T2, stop checking PYTHONUOPSOPTIMIZE - Rename the env var to disable T2 entirely to PYTHON_UOPS_OPTIMIZE (must be explicitly set to 0 to disable) - Fix skipIf conditions on tests in test_opt.py accordingly - Export sym_is_bottom() (for debugging) - Fix various things in the `_BINARY_OP_` specializations in the abstract interpreter: - DECREF(temp) - out-of-space check after sym_new_const() - add sym_matches_type() checks, so even if we somehow reach a binary op with symbolic constants of the wrong type on the stack we won't trigger the type assert

python#116079) This should fix the refleaks bots. (See python#116062 (comment) .)

This undoes the *temporary* default disabling of the T2 optimizer pass in pythongh-115860. - Add a new test that reproduces Brandt's example from pythongh-115859; it indeed crashes before pythongh-116028 with PYTHONUOPSOPTIMIZE=1 - Re-enable the optimizer pass in T2, stop checking PYTHONUOPSOPTIMIZE - Rename the env var to disable T2 entirely to PYTHON_UOPS_OPTIMIZE (must be explicitly set to 0 to disable) - Fix skipIf conditions on tests in test_opt.py accordingly - Export sym_is_bottom() (for debugging) - Fix various things in the `_BINARY_OP_` specializations in the abstract interpreter: - DECREF(temp) - out-of-space check after sym_new_const() - add sym_matches_type() checks, so even if we somehow reach a binary op with symbolic constants of the wrong type on the stack we won't trigger the type assert

python#116079) This should fix the refleaks bots. (See python#116062 (comment) .)

gvanrossum added 8 commits February 28, 2024 11:09

Export sym_is_bottom

d3a0695

Add a test that crashed before python#116028

b834b29

Tighten test (expect both guard and add)

436a641

Add DECREF(temp) and out-of-space checks

e64d0c1

Re-enable the optimizer

66b76d5

Disable all of Tier 2 only if PYTHON_UOPS_OPTIMIZE=0

af72c76

Fix tests skip for PYTHON_UOPS_OPTIMIZE

d7280d2

Belt and suspenders -- use sym_matches_type before assert

d2078db

gvanrossum added the skip news label Feb 28, 2024

gvanrossum requested a review from Fidget-Spinner February 28, 2024 19:57

gvanrossum requested a review from markshannon as a code owner February 28, 2024 19:57

bedevere-app bot added the awaiting core review label Feb 28, 2024

bedevere-app bot mentioned this pull request Feb 28, 2024

Type propagation: just because something is const doesn't mean it automatically matches the type #115859

Closed

gvanrossum mentioned this pull request Feb 28, 2024

gh-115685: Type/values propagate for TO_BOOL in tier 2 #115686

Merged

gvanrossum changed the title ~~gh-115859: Restore optimizer~~ gh-115859: Re-enable T2 optimizer pass by default Feb 28, 2024

Fidget-Spinner reviewed Feb 28, 2024

View reviewed changes

Arrange things so DECREF(temp) happens even if sym_new_const() runs o…

bbe8cd9

…ut of space

Fidget-Spinner approved these changes Feb 28, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Feb 28, 2024

gvanrossum added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Feb 28, 2024

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Feb 28, 2024

This was referenced Feb 28, 2024

gh-115773: Add tests to exercise the _Py_DebugOffsets structure #115774

Merged

test_external_inspection.py fails on QEMU builds #116075

Closed

gvanrossum enabled auto-merge (squash) February 28, 2024 22:00

gvanrossum mentioned this pull request Feb 28, 2024

Tier 2 optimizations for 3.13 #115419

Open

32 tasks

gvanrossum merged commit 3409bc2 into python:main Feb 28, 2024
48 of 59 checks passed

bedevere-app bot removed the awaiting merge label Feb 28, 2024

gvanrossum deleted the restore-optimizer branch February 28, 2024 22:38

gvanrossum mentioned this pull request Feb 28, 2024

gh-115859: Fix test_type_inconsistency() when run multiple times #116079

Merged

gvanrossum added a commit that referenced this pull request Feb 28, 2024

gh-115859: Fix test_type_inconsistency() when run multiple times (#11…

479ac5c

…6079) This should fix the refleaks bots. (See #116062 (comment) .)

gvanrossum mentioned this pull request Mar 1, 2024

gh-115480: Type propagate _BINARY_OP_ADD_UNICODE #115710

Merged

woodruffw pushed a commit to woodruffw-forks/cpython that referenced this pull request Mar 4, 2024

pythongh-115859: Fix test_type_inconsistency() when run multiple times (

a8631c8

python#116079) This should fix the refleaks bots. (See python#116062 (comment) .)

adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024

pythongh-115859: Fix test_type_inconsistency() when run multiple times (

25b33c4

python#116079) This should fix the refleaks bots. (See python#116062 (comment) .)

diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024

pythongh-115859: Fix test_type_inconsistency() when run multiple times (

8823a53

python#116079) This should fix the refleaks bots. (See python#116062 (comment) .)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-115859: Re-enable T2 optimizer pass by default #116062

gh-115859: Re-enable T2 optimizer pass by default #116062

gvanrossum commented Feb 28, 2024 •

edited

gvanrossum commented Feb 28, 2024

Fidget-Spinner left a comment

Fidget-Spinner Feb 28, 2024

gvanrossum Feb 28, 2024

gvanrossum commented Feb 28, 2024 •

edited

bedevere-bot commented Feb 28, 2024

gvanrossum commented Feb 28, 2024 •

edited

bedevere-bot commented Feb 28, 2024

bedevere-bot commented Feb 28, 2024

		@@ -118,14 +121,17 @@ dummy_func(void) {
		OUT_OF_SPACE_IF_NULL(res = sym_new_const(ctx, temp));

gh-115859: Re-enable T2 optimizer pass by default #116062

gh-115859: Re-enable T2 optimizer pass by default #116062

Conversation

gvanrossum commented Feb 28, 2024 • edited

gvanrossum commented Feb 28, 2024

Fidget-Spinner left a comment

Choose a reason for hiding this comment

Fidget-Spinner Feb 28, 2024

Choose a reason for hiding this comment

gvanrossum Feb 28, 2024

Choose a reason for hiding this comment

gvanrossum commented Feb 28, 2024 • edited

bedevere-bot commented Feb 28, 2024

gvanrossum commented Feb 28, 2024 • edited

bedevere-bot commented Feb 28, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Feb 28, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gvanrossum commented Feb 28, 2024 •

edited

gvanrossum commented Feb 28, 2024 •

edited

gvanrossum commented Feb 28, 2024 •

edited