SyntaxError: unmatched ')' with whitespace in lambda? #105042

thesamesam · 2023-05-28T15:07:59Z

Bug report

Testing on tip of 3.12 which includes the fix for #105013 (thanks!), I get the following difference in behaviour with Python 3.11 vs tip of Python 3.12:

import inspect

from hypothesis.internal.reflection import extract_lambda_source
from hypothesis.strategies import just

#from hypothesis.strategies import just, one_of

# This variant doesn't trigger a TypeError mid-callback, but both variants get the same final inspect error
#one_of_nested_strategy_with_filter = one_of(
#    just(0),
#    just(1),
#    one_of(just(2), just(3), one_of(just(4), just(5), one_of(just(6), just(7)))),
#).filter(lambda x: x % 2 == 0)
#x = get_pretty_function_description(one_of_nested_strategy_with_filter)
#print(inspect.getsource(x))


one_of_nested_strategy_with_filter = (
    just(0)
).filter(lambda x: x % 2 == 0)

x = extract_lambda_source(one_of_nested_strategy_with_filter)

With Python 3.12, I get:

Traceback (most recent call last):
  File "/usr/lib/python3.12/inspect.py", line 1241, in getblock
    for _token in tokens:
  File "/usr/lib/python3.12/tokenize.py", line 450, in _tokenize
    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):
  File "/usr/lib/python3.12/tokenize.py", line 537, in _generate_tokens_from_c_tokenizer
    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
  File "<string>", line 1
    ).filter(lambda x: x % 2 == 0)
    ^
SyntaxError: unmatched ')'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/foo2.py", line 10, in <module>
    x = extract_lambda_source(one_of_nested_strategy_with_filter)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py", line 305, in extract_lambda_source
    sig = inspect.signature(f)
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 3326, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 3070, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 2484, in _signature_from_callable
    raise TypeError('{!r} is not a callable object'.format(obj))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py", line 39, in __repr__
    suffix = "".join(
             ^^^^^^^^
  File "/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py", line 40, in <genexpr>
    f".{name}({get_pretty_function_description(f)})"
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py", line 432, in get_pretty_function_description
    return extract_lambda_source(f)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py", line 312, in extract_lambda_source
    source = inspect.getsource(f)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 1282, in getsource
    lines, lnum = getsourcelines(object)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 1274, in getsourcelines
    return getblock(lines[lnum:]), lnum + 1
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 1248, in getblock
    _, *_token_info = _token
                      ^^^^^^
UnboundLocalError: cannot access local variable '_token' where it is not associated with a value

But with Python 3.11, I get:

Traceback (most recent call last):
  File "/tmp/foo2.py", line 10, in <module>
    x = extract_lambda_source(one_of_nested_strategy_with_filter)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/hypothesis/internal/reflection.py", line 305, in extract_lambda_source
    sig = inspect.signature(f)
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/inspect.py", line 3279, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/inspect.py", line 3027, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/inspect.py", line 2447, in _signature_from_callable
    raise TypeError('{!r} is not a callable object'.format(obj))
TypeError: just(0).filter(lambda x: <unknown>) is not a callable object

If I change the program to drop the whitespace, it works in 3.12 too:

import inspect

from hypothesis.internal.reflection import extract_lambda_source
from hypothesis.strategies import just

one_of_nested_strategy_with_filter = (just(0)).filter(lambda x: x % 2 == 0)

x = extract_lambda_source(one_of_nested_strategy_with_filter)

I noticed this w/ a test failure in priority (the output is huge, so just a snippet here)

ERROR collecting test/test_priority.py ____________________________________________________________________________________
/usr/lib/python3.12/inspect.py:1241: in getblock
    for _token in tokens:
[...]
/usr/lib/python3.12/tokenize.py:537: in _generate_tokens_from_c_tokenizer
    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
E     File "<string>", line 1
E       ).map(lambda blocked: (blocked, active_readme_streams_from_filter(blocked)))
E       ^
E   SyntaxError: unmatched ')'
        c_tokenizer = <module '_tokenize' (built-in)>
        extra_tokens = True
        source     = (').map(lambda blocked: (blocked, '
 'active_readme_streams_from_filter(blocked)))\n'
[...]
ERROR test/test_priority.py - UnboundLocalError: cannot access local variable '_token' where it is not associated with a value

and a perhaps more useful test failure in hypothesis, which priority uses:

test_one_of_flattens_filter_branches_2 ____________________________________________________________________________________
[gw14] linux -- Python 3.12.0 /var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/bin/python3.12
Traceback (most recent call last):
  File "/usr/lib/python3.12/inspect.py", line 1241, in getblock
    for _token in tokens:
  File "/usr/lib/python3.12/tokenize.py", line 450, in _tokenize
    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):
  File "/usr/lib/python3.12/tokenize.py", line 537, in _generate_tokens_from_c_tokenizer
    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
  File "<string>", line 1
    ).filter(lambda x: x % 2 == 0)
    ^
SyntaxError: unmatched ')'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python/tests/quality/test_discovery_ability.py", line 101, in run_test
    runner.run()
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py", line 474, in run
    self._run()
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py", line 880, in _run
    self.generate_new_examples()
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py", line 684, in generate_new_examples
    minimal_example = self.cached_test_function(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py", line 1065, in cached_test_function
    self.test_function(data)
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py", line 209, in test_function
    self.__stoppable_test_function(data)
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py", line 185, in __stoppable_test_function
    self._test_function(data)
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python/tests/quality/test_discovery_ability.py", line 79, in test_function
    value = data.draw(specifier)
            ^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 956, in draw
    return strategy.do_draw(self)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py", line 942, in do_draw
    result = self.do_filtered_draw(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py", line 956, in do_filtered_draw
    value = data.draw(self.filtered_strategy)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 951, in draw
    return strategy.do_draw(self)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py", line 666, in do_draw
    return data.draw(strategy)
           ^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 951, in draw
    return strategy.do_draw(self)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py", line 532, in do_draw
    data.mark_invalid(f"Aborted test because unable to satisfy {self!r}")
                                                               ^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py", line 39, in __repr__
    suffix = "".join(
             ^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py", line 40, in <genexpr>
    f".{name}({get_pretty_function_description(f)})"
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py", line 432, in get_pretty_function_description
    return extract_lambda_source(f)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py", line 312, in extract_lambda_source
    source = inspect.getsource(f)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 1282, in getsource
    lines, lnum = getsourcelines(object)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 1274, in getsourcelines
    return getblock(lines[lnum:]), lnum + 1
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/inspect.py", line 1248, in getblock
    _, *_token_info = _token
                      ^^^^^^
UnboundLocalError: cannot access local variable '_token' where it is not associated with a value

Your environment

CPython versions tested on: 3.11.3, tip of 3.12
Operating system and architecture: Gentoo Linux, amd64

Linked PRs

The text was updated successfully, but these errors were encountered:

thesamesam · 2023-05-28T15:08:18Z

I'm trying to reduce it more to be standalone (not dependent on hypothesis) but not sure if I'll get there.

samuelcolvin · 2023-05-28T17:36:35Z

This is also happening on pydantic-core, see https://github.com/pydantic/pydantic-core/actions/runs/5105586096/jobs/9177258340?pr=629

AlexWaygood · 2023-05-28T17:46:23Z

Cc. @pablogsal @lysnikolaou

Bug: python/cpython#105042 Signed-off-by: Sam James <sam@gentoo.org>

pablogsal · 2023-05-28T19:24:01Z

Will try to fix this this next week

pablogsal · 2023-05-28T19:26:37Z

Meanwhile can someone provide a self contained reproducer that doesn't rely on hypothesis?

pablogsal · 2023-05-28T20:07:01Z

@lysnikolaou if you have some spare cycles can you look at this? The problem is that the tokeniser now raises with unbalanced parens when before it didn't and this now doesn't allow to analyse code that is not syntactically correct by its own. We may need to emit ERRORTOKEN tokens for these or something like that.

lysnikolaou · 2023-05-29T11:30:48Z

This all comes from inspect, which uses tokenize. Here's a more self-contained reproducer for now:

import inspect

(
); x = lambda: 1

inspect.getsource(x)

As @pablogsal said, it's because tokenize now raises for unbalanced parens. Maybe we can deactivate that in the C tokenizer level when it's called from the Python tokenize module?

pablogsal · 2023-05-29T11:32:12Z

Yeah but we need to ensure we don't leave the tokenizer in a bad state by ignoring the errors

…nize

…105061)

…nize (pythonGH-105061) (cherry picked from commit 70f315c) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

pablogsal · 2023-05-30T21:54:37Z

I'm resolving this, please feel free to reopen if something is missing after the fix

…enize (GH-105061) (#105120) gh-105042: Disable unmatched parens syntax error in python tokenize (GH-105061) (cherry picked from commit 70f315c) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

thesamesam added the type-bug An unexpected behavior, bug, or error label May 28, 2023

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue May 28, 2023

dev-python/priority: enable py3.12 (xfail)

494aa75

Bug: python/cpython#105042 Signed-off-by: Sam James <sam@gentoo.org>

samuelcolvin mentioned this issue May 29, 2023

try testing and building for 3.12 pydantic/pydantic-core#629

Merged

lysnikolaou added a commit to lysnikolaou/cpython that referenced this issue May 29, 2023

pythongh-105042: Disable unmatched parens syntax error in python toke…

1c38078

…nize

bedevere-bot mentioned this issue May 29, 2023

gh-105042: Disable unmatched parens syntax error in python tokenize #105061

Merged

pablogsal pushed a commit that referenced this issue May 30, 2023

gh-105042: Disable unmatched parens syntax error in python tokenize (#…

70f315c

…105061)

bedevere-bot mentioned this issue May 30, 2023

[3.12] gh-105042: Disable unmatched parens syntax error in python tokenize (GH-105061) #105120

Merged

pablogsal closed this as completed May 30, 2023

erlend-aasland mentioned this issue Jun 6, 2023

3.12 backport gh 105236 #105358

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SyntaxError: unmatched ')' with whitespace in lambda? #105042

SyntaxError: unmatched ')' with whitespace in lambda? #105042

thesamesam commented May 28, 2023 •

edited by bedevere-bot

thesamesam commented May 28, 2023

samuelcolvin commented May 28, 2023

AlexWaygood commented May 28, 2023

pablogsal commented May 28, 2023

pablogsal commented May 28, 2023

pablogsal commented May 28, 2023

lysnikolaou commented May 29, 2023

pablogsal commented May 29, 2023

pablogsal commented May 30, 2023

SyntaxError: unmatched ')' with whitespace in lambda? #105042

SyntaxError: unmatched ')' with whitespace in lambda? #105042

Comments

thesamesam commented May 28, 2023 • edited by bedevere-bot

Bug report

Your environment

Linked PRs

thesamesam commented May 28, 2023

samuelcolvin commented May 28, 2023

AlexWaygood commented May 28, 2023

pablogsal commented May 28, 2023

pablogsal commented May 28, 2023

pablogsal commented May 28, 2023

lysnikolaou commented May 29, 2023

pablogsal commented May 29, 2023

pablogsal commented May 30, 2023

thesamesam commented May 28, 2023 •

edited by bedevere-bot