Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assertion failed: !absl::exchange(g_registered, true) #30640

Closed
edvgui opened this issue Aug 19, 2022 · 2 comments · Fixed by #30642
Closed

assertion failed: !absl::exchange(g_registered, true) #30640

edvgui opened this issue Aug 19, 2022 · 2 comments · Fixed by #30642

Comments

@edvgui
Copy link

edvgui commented Aug 19, 2022

What version of gRPC and what language are you using?

Here are the installed python packages, no idea which version of grpc runs underneath.

grpcio-1.49.0rc1
grpcio-tools-1.49.0rc1

What operating system (Linux, Windows,...) and version?

guillaume@latitude-5411:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.9.7

What did you do?

guillaume@latitude-5411:~$ cd /tmp/
guillaume@latitude-5411:/tmp$ git clone https://github.com/inmanta/terraform.git > /dev/null
guillaume@latitude-5411:/tmp$ cd terraform/
guillaume@latitude-5411:/tmp/terraform$ mktmpenv -p 3.9 > /dev/null
(tmp-730f4738cbc9bf7) guillaume@latitude-5411:/tmp/terraform$ make install > /dev/null
(tmp-730f4738cbc9bf7) guillaume@latitude-5411:/tmp/terraform$ pip list | grep grpcio
grpcio                    1.49.0rc1
grpcio-tools              1.49.0rc1
(tmp-730f4738cbc9bf7) guillaume@latitude-5411:/tmp/terraform$ pytest tests/ -xvv
/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
====================================================================================================================================================================================================== test session starts ======================================================================================================================================================================================================
platform linux -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0 -- /home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/bin/python
cachedir: .pytest_cache
rootdir: /tmp/terraform, configfile: pytest.ini
plugins: asyncio-0.19.0, inmanta-2.4.0.dev20220819141146, postgresql-4.1.1, inmanta-extensions-7.0.1.dev20220819140631, inmanta-tfplugin-5.2.0, env-0.6.2
asyncio: mode=auto
collected 21 items                                                                                                                                                                                                                                                                                                                                                                                                              

tests/features/test_config_tree.py::test_config_serialization PASSED                                                                                                                                                                                                                                                                                                                                                      [  4%]
tests/features/test_config_tree.py::test_deprecated_config PASSED                                                                                                                                                                                                                                                                                                                                                         [  9%]
tests/features/test_config_tree.py::test_block_config Fatal Python error: Aborted

Thread 0x00007f0bd67fc700 (most recent call first):
  File "/usr/lib/python3.9/logging/__init__.py", line 1175 in _open
  File "/usr/lib/python3.9/logging/__init__.py", line 1146 in __init__
  File "/tmp/tmp5thwsb35/libs/terraform/plugins/tf/terraform_provider.py", line 97 in _io_logger
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bd6ffd700 (most recent call first):
  File "/tmp/tmp5thwsb35/libs/terraform/plugins/tf/terraform_provider.py", line 102 in _io_logger
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Current thread 0x00007f0bd77fe700 (most recent call first):
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/grpc/_channel.py", line 1478 in __init__
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/grpc/__init__.py", line 1978 in insecure_channel
  File "/tmp/tmp5thwsb35/libs/terraform/plugins/tf/terraform_provider.py", line 220 in open
  File "/tmp/tmp5thwsb35/libs/terraform/plugins/terraform_resource.py", line 259 in pre
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/inmanta/agent/handler.py", line 748 in check_facts
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 83 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bd7fff700 (most recent call first):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bec9cb700 (most recent call first):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bed1cc700 (most recent call first):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bedc0d700 (most recent call first):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bee40e700 (most recent call first):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0beec0f700 (most recent call first):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bef410700 (most recent call first):
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.9/threading.py", line 917 in run
  File "/usr/lib/python3.9/threading.py", line 980 in _bootstrap_inner
  File "/usr/lib/python3.9/threading.py", line 937 in _bootstrap

Thread 0x00007f0bf66b2740 (most recent call first):
  File "/usr/lib/python3.9/selectors.py", line 469 in select
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1869 in _run_once
  File "/usr/lib/python3.9/asyncio/base_events.py", line 601 in run_forever
  File "/usr/lib/python3.9/asyncio/base_events.py", line 634 in run_until_complete
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pytest_asyncio/plugin.py", line 454 in inner
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/python.py", line 1761 in runtest
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/runner.py", line 166 in pytest_runtest_call
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/runner.py", line 259 in <lambda>
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/runner.py", line 338 in from_call
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/runner.py", line 258 in call_runtest_hook
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/runner.py", line 219 in call_and_report
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/runner.py", line 130 in runtestprotocol
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/main.py", line 347 in pytest_runtestloop
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/main.py", line 322 in _main
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/main.py", line 268 in wrap_session
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/config/__init__.py", line 164 in main
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/lib/python3.9/site-packages/_pytest/config/__init__.py", line 187 in console_main
  File "/home/guillaume/.virtualenvs/tmp-730f4738cbc9bf7/bin/pytest", line 8 in <module>
Aborted (core dumped)

This is not very minimal, but I couldn't find any other way to reproduce it so far.

What did you expect to see?

The test suite should work.

What did you see instead?

E0819 04:34:59.521602551 3104795 forkable.cc:42]             assertion failed: !absl::exchange(g_registered, true)
Fatal Python error: Aborted

Thread 0x00007f48ebfff700 (most recent call first):
  File "/usr/lib64/python3.9/logging/__init__.py", line 1175 in _open
  File "/usr/lib64/python3.9/logging/__init__.py", line 1146 in __init__
  File "/tmp/tmpvh_l8_8y/libs/terraform/plugins/tf/terraform_provider.py", line 97 in _io_logger
  File "/usr/lib64/python3.9/threading.py", line 910 in run
  File "/usr/lib64/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/usr/lib64/python3.9/threading.py", line 930 in _bootstrap

Thread 0x00007f48eb7fe700 (most recent call first):
  File "/tmp/tmpvh_l8_8y/libs/terraform/plugins/tf/terraform_provider.py", line 102 in _io_logger
  File "/usr/lib64/python3.9/threading.py", line 910 in run
  File "/usr/lib64/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/usr/lib64/python3.9/threading.py", line 930 in _bootstrap

Current thread 0x00007f4908f79700 (most recent call first):
  File "/home/jenkins/workspace/modules_terraform_master/env/lib64/python3.9/site-packages/grpc/_channel.py", line 1478 in __init__
  File "/home/jenkins/workspace/modules_terraform_master/env/lib64/python3.9/site-packages/grpc/__init__.py", line 1978 in insecure_channel
  File "/tmp/tmpvh_l8_8y/libs/terraform/plugins/tf/terraform_provider.py", line 220 in open
  File "/tmp/tmpvh_l8_8y/libs/terraform/plugins/terraform_resource.py", line 259 in pre
  File "/home/jenkins/workspace/modules_terraform_master/env/lib64/python3.9/site-packages/****/agent/handler.py", line 748 in check_facts
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 52 in run
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 77 in _worker
  File "/usr/lib64/python3.9/threading.py", line 910 in run
  File "/usr/lib64/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/usr/lib64/python3.9/threading.py", line 930 in _bootstrap

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See TROUBLESHOOTING.md for how to diagnose problems better.

Anything else we should know about your project / environment?

The testing environment uses some multi-threading and asyncio.

@gnossen
Copy link
Contributor

gnossen commented Aug 19, 2022

Thanks for reporting! I'm glad to see our prereleases are catching issues!

@drfloob
Copy link
Member

drfloob commented Aug 19, 2022

Thanks for pointing this out! I'll have a fix for this shortly, and we'll put out an rc2 release afterwards.

drfloob added a commit that referenced this issue Aug 19, 2022
This fixes a bug that could occur on repeated grpc initialization (after a complete shutdown)

Fixes #30640
drfloob added a commit to drfloob/grpc that referenced this issue Aug 19, 2022
This fixes a bug that could occur on repeated grpc initialization (after a complete shutdown)

Fixes grpc#30640
drfloob added a commit that referenced this issue Aug 19, 2022
* Fix forkable globals (#30608)

* Fix forkable repeated registration (#30642)

This fixes a bug that could occur on repeated grpc initialization (after a complete shutdown)

Fixes #30640
gcf-merge-on-green bot pushed a commit to googleapis/python-storage that referenced this issue Aug 23, 2022
Unblock ci/cd by excluding pre-release version in question `1.49.0rc1`

Update: grpc/grpc#30642 fixes issue grpc/grpc#30651 and grpc/grpc#30640
Restore noxfile once the merged fix is released
veblush added a commit that referenced this issue Sep 20, 2022
* Bump 1.49.x branch to 1.49.0.pre1 (#30615)

* bump version to 1.49.0-pre1

* regenerate projects

* [backport][v1.49.x] forkable fixes (#30646)

* Fix forkable globals (#30608)

* Fix forkable repeated registration (#30642)

This fixes a bug that could occur on repeated grpc initialization (after a complete shutdown)

Fixes #30640

* Drop support for ruby 2.5 (#30699) (#30762)

* Drop ruby 2.5 support

* Backport: "stabilize the C2P resolver URI scheme" to v1.49.x (#30654)

* stabilize the C2P resolver URI scheme

* Bump 1.49 branch to 1.49.0.pre2 (#30786)

* bump version to 1.49.0-pre2

* regenerate projects

* Update protobuf on ancillary packages (#30795) (#30805)

* Bump release version on 1.49 to 1.49.0.pre3 (#30814)

* bump version to 1.49.0-pre3

* regenerate projects

* xDS interop: enable pod log collection in the buildscripts (#30735) (#30856)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

* xDS interop: buildscripts: fix run_test return status (#30768) (#30875)

To capture the return status of the test in run_test the last command must be the call to the test itself.
This removes `set +x`, which makes the run_test always return success, and not propagate the test status.

I can't find it, but this exact error bit us before. Looks like it leaked to other scripts.
The good thing is if the test was executed, it's failure would still be picked up from the result xml.

However, if the test framework didn't start in the first place, the result will be false positive.
Example: https://source.cloud.google.com/results/invocations/98d3e679-ec8a-40bd-9f36-88179747b0d6/targets

```
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3: Error while finding module specification for 'tests.authz_test' (ModuleNotFoundError: No module named 'tests')
+ set +x
Failed test suites: 0

[ID: 3548168] Command finished after 625 secs, exit value: 0
```

* client_channel: fix crash when cancelling a watch after SHUTDOWN (#30885) (#30928)

* Support Python 3.11 (#30818) (#30944)

* Support Python 3.11

* Update build images for 3.11

* Whoopsie

* The architecture of this thing is garbage

* Silence ownership warning

* Account for change in git behavior

* Fix directory

* I am in great pain

* Update Windows and arm linux

* Agh

* Clean up

* Bump 1.49 branch to 1.49.0 (#30974)

* bump version to 1.49.0

* regenerate projects

* Update protobuf to v21.6 on 1.49.x (#31028)

* Update third_party/protobuf

* run tools/distrib/python/make_grpcio_tools.py

* update build_handwritten.yaml

* regenerate projects

* Bump v1.49.x to v1.49.1 (#31037)

* bump version to 1.49.1

* regenerate projects

* Automated change: Fix sanity tests

Co-authored-by: gnossen <gnossen@users.noreply.github.com>

* Fix ruby windows ucrt build (#31053)

Co-authored-by: apolcyn <apolcyn@google.com>
Co-authored-by: AJ Heller <hork@google.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Mark D. Roth <roth@google.com>
Co-authored-by: Richard Belleville <gnossen@gmail.com>
Co-authored-by: gnossen <gnossen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants