3.14.0b2 yield gives TypeError: _pystart_callback expected 2 arguments, got 3

[ThomasWaldmann]

Bug report

Bug description:

I thought I add 3.14-dev to our CI on github actions and after fixing some obvious issues, this one was left (the code is tested successfully on 3.10 .. 3.13):

  File "/home/runner/work/borg/borg/src/borg/archiver/create_cmd.py", line 472, in _rec_walk
    with OsOpen(
         ~~~~~~^
        path=path, parent_fd=parent_fd, name=name, flags=flags_dir, noatime=True, op="dir_open"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as child_fd:
    ^
  File "/opt/hostedtoolcache/Python/3.14.0-beta.2/x64/lib/python3.14/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/home/runner/work/borg/borg/src/borg/archive.py", line 254, in OsOpen
    yield fd
TypeError: _pystart_callback expected 2 arguments, got 3

borgbackup/borg#8919

We use tox and pytest as test runner, in case that matters. Tried updating these (plus their plugins) to the latest versions, but it did not help.

CPython versions tested on:

3.14.0b2

Operating systems tested on:

Linux

Linked PRs

• gh-135429: Fix the argument mismatch in lsprof throw event #135442
• [3.14] gh-135429: Fix the argument mismatch in lsprof throw event (GH-135442) #135446

[ThomasWaldmann]

Another one, this time just a bare yield:

    @contextmanager
    def _stats_updater(self, key, msg):
        """update call counters and overall times, also emulate latency and bandwidth"""
        # do not use this in generators!
        volume_before = self._stats_get_volume(key)
        start = time.perf_counter_ns()
>       yield
E       TypeError: _pystart_callback expected 2 arguments, got 3

[ZeroIntensity]

Hi! There is very little we can do without a repro. Would you mind trying to shrink this down to something using only the stdlib?

That said, _pystart_callback is a function used for profiling. Are you tests using some sort of low-level profiler?

[ThomasWaldmann]

Most unit tests are just normal pytest tests, a few are using unittest. There are some using pytest-benchmark though, could that be it?

[ZeroIntensity]

That could definitely be it. If they're using internal profiling APIs that changed in 3.14, that would explain it. Does it pass without pytest-benchmark?

[ThomasWaldmann]

I just found that the 2 failing tests test our --debug-profile option, which enables the cProfile.Profile profiler from stdlib. .enable, .disable, .snapshot_stats is all we use from that.

            with open(args.debug_profile, "wb") as fd:
                profiler = cProfile.Profile()
                variables = dict(locals())
                profiler.enable()
                try:
                    return get_ec(func(args))
                finally:
                    profiler.disable()
                    profiler.snapshot_stats()
                    if args.debug_profile.endswith(".pyprof"):
                        marshal.dump(profiler.stats, fd)
                    else:
                        # We use msgpack here instead of the marshal module used by cProfile itself,
                        # because the latter is insecure. Since these files may be shared over the
                        # internet we don't want a format that is impossible to interpret outside
                        # an insecure implementation.
                        # See scripts/msgpack2marshal.py for a small script that turns a msgpack file
                        # into a marshal file that can be read by e.g. pyprof2calltree.
                        # For local use it's unnecessary hassle, though, that's why .pyprof makes
                        # it compatible (see above).
                        msgpack.pack(profiler.stats, fd, use_bin_type=True)

[gaogaotiantian]

I know what happened. It was due to the clinical refactoring for _pystart_callback. I will investigate why we did not find it in our test suite. Anyway I'll take care of this, thanks for reporting.

[gaogaotiantian]

Okay so the reason why our test suite did not catch this is because in #111069 we optimized how generator works and the throw event is not there anymore.