KeyError('context_class') with library click

[Jonas1312]

Things to check first

• I have searched the existing issues and didn't find my bug already reported there

• I have checked that my bug is still present in the latest release

Typeguard version

3.0.1

Python version

3.7

What happened?

Hi,

I found another bug.

The code was working well on typeguard==2.13.3, but breaks since release v3.0.x.

The print(result, result.exception, result.exit_code) gives: <Result KeyError('context_class')> 'context_class' 1

When I remove the typechecked decorator, it gives: <Result okay> None 0

Thanks!

How can we reproduce the bug?

click==8.1.3

from typing import Iterable, Optional

import click
from click.testing import CliRunner
from typeguard import typechecked


class Arguments:  
    """Common arguments for the CLI"""

    files = click.argument("files", type=click.Path(), required=False, nargs=-1)

    project_id = click.argument("project_id", type=str, required=True)


class Options:  
    """Common options for the CLI"""

    project_id = click.option("--project-id", type=str, required=True, help="Id of the project")


@click.command(name="import")
@Arguments.files
@Options.project_id
@typechecked
def import_assets(files: Optional[Iterable[str]], project_id: str):
    assert len(files) == 2
    assert project_id == "image_project"


def main():
    arguments = [
        "test_tree/image1.png",
        "test_tree/leaf/image3.png",
        "--project-id",
        "image_project",
    ]

    runner = CliRunner()
    with runner.isolated_filesystem():
        result = runner.invoke(import_assets, arguments)
    print(result, result.exception, result.exit_code)


if __name__ == "__main__":
    main()

[agronholm]

Not much to go on here, as there isn't even a traceback of any kind. Any clue as to where this KeyError comes from? That key isn't present anywhere in the snippet.

[Jonas1312]

There is no traceback since the code doesn't really crash.

The runner.invoke runs the click command as if it was ran from the command line.

I have no idea where the KeyError comes from. Only thing I know is that it was working well with typeguard==2.13.3. And when I remove the @typechecked, the code works well.

I will try to investigate, but I don't know the typeguard codebase well.

[agronholm]

What makes this even harder to debug is that click spawns a subprocess, so I can't even run a debugger to figure out where it comes from. Does this not occur without CliRunner?

[Jonas1312]

Yes sorry the CliRunner doesn't help...

I'll try to get rid of it

[Jonas1312]

bug_typeguard.py:

from typing import Iterable, Optional

import click
from typeguard import typechecked


class Arguments:  # pylint: disable=too-few-public-methods
    """Common arguments for the CLI"""

    files = click.argument("files", type=click.Path(), required=False, nargs=-1)

    project_id = click.argument("project_id", type=str, required=True)


class Options:  # pylint: disable=too-few-public-methods
    """Common options for the CLI"""

    project_id = click.option("--project-id", type=str, required=True, help="Id of the project")


@click.command(name="import")
@Arguments.files
@Options.project_id
@typechecked
def import_assets(files: Optional[Iterable[str]], project_id: str):
    print("import_assets func")
    assert len(files) == 1
    assert project_id == "image_project"


@click.group()
def main():
    # arguments = [
    #     "test_tree/image1.png",
    #     "--project-id",
    #     "image_project",
    # ]
    print("main func")


if __name__ == "__main__":
    main.add_command(import_assets)
    main()

ran with:

python bug_typeguard.py import img.jpg --project-id "image_project"

gives:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/python37/lib/python3.7/site-packages/typeguard/_memo.py", line 70, in __init__
    self.type_hints = _type_hints_map[func]
  File "/opt/anaconda3/envs/python37/lib/python3.7/weakref.py", line 396, in __getitem__
    return self.data[ref(key)]
KeyError: <weakref at 0x7f92600662f0; to 'Command' at 0x7f9260046f10>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bug_typeguard.py", line 43, in <module>
    main()
  File "/opt/anaconda3/envs/python37/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/anaconda3/envs/python37/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/anaconda3/envs/python37/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/envs/python37/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/anaconda3/envs/python37/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "bug_typeguard.py", line 21, in import_assets
    @click.command(name="import")
  File "/opt/anaconda3/envs/python37/lib/python3.7/site-packages/typeguard/_memo.py", line 130, in __init__
    param = signature.parameters[key]
KeyError: 'context_class'

[agronholm]

Interesting. This is actually helpful. I'll delve deeper into this later.

[Jonas1312]

Great! I'll try to investigate too when I have time.

Should be easier to attach a debugger now

Thanks a lot

[agronholm]

The problem seems to be that since Typeguard looks up the function from the global namespace, it then finds the one that click replaced with its own, which then confuses Typeguard. The fix for this is nontrivial, but in the long term probably a necessary one. What I need to do is to copy the annotations to the function body so that they're not lost.

[agronholm]

I should mention that click wraps functions in such a way that it doesn't even provide a standard __wrapped__ attribute which would probably have avoided this problem.

[agronholm]

The good news: I managed to fix this locally.
The bad news: It requires potentially breaking changes to the API.

[agronholm]

Would you mind trying your app with the 4.0-dev branch?

[Jonas1312]

Seems like it has fixed the issue, yes! Thanks a lot

[alexander-held]

Hi, I ran into an issue that produced a very similar error and can also confirm that 4.0-dev resolves it. Thanks for tracking this down and fixing it!

You mentioned that this might be a breaking change, do you have an estimate for when this may make it into a release? Using that branch in CI works fine, but I wonder if this branch will receive potentially other breaking updates with further development towards 4.0 (which may then make it harder to use in CI). Sticking with typeguard<3 works too, though it would be great to be able to take advantage of the additional typing issues that 3.0 seems to be able to catch.

[agronholm]

Once I finish #317, I should be able to release 4.0 quickly. The hard part there is done already, I just need some time to polish the code and update the documentation.

[agronholm]

4.0.0rc1 is out. Please clear any *-typeguard.pyc files if you're upgrading if you've been using the import hook. I will update the suffix in the final release to avoid problems.