Adding `--level=error` to xargs-piped pants command causes internal exception

[engnatha]

Describe the bug
I can run the following command and the pants logs look great and everything happens fast.

./pants dependees list --changed-since=master | xargs ./pants dependees list | xargs ./pants --filter-target-type=python_test list

If I add --level=error to the last command to change the log level, I get exceptions in the pants logs.

Unfortunately, no amount of RUST_BACKTRACE=1 setting in my environment was able to show the full details (maybe an issue with xargs?).

Both variants of the command produce the same result, but the latter causes some performance issues since it kills something for pants (maybe the daemon?).

Pants version
2.15.0

OS
Ubuntu 20.04

Additional info
Here are the logs when the command throws the error.

11:19:28.90 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list --changed-since=master`
11:19:43.80 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list --changed-since=master`
11:19:44.49 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list astranis-python/astranis/ground/service/angle_collector:sources astranis-python/astranis/ground/service/angle_collector/client_server_test.py:tests astranis-python/astranis/service/handler.py astranis-python/astranis/ground/service/angle_collector:client astranis-python/astranis/ground/service/angle_collector/client.py:sources`
11:19:44.98 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.15.0 dependees list astranis-python/astranis/ground/service/angle_collector:sources astranis-python/astranis/ground/service/angle_collector/client_server_test.py:tests astranis-python/astranis/service/handler.py astranis-python/astranis/ground/service/angle_collector:client astranis-python/astranis/ground/service/angle_collector/client.py:sources`
.sleep(self.JOIN_TIMEOUT_SECONDS)
  File "/home/nathanael/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 109, in handle_sigterm
    raise self.SignalHandledNonLocalExit(signum, "SIGTERM")
  File "/home/nathanael/.cache/pants/setup/bootstrap-Linux-x86_64/2.15.0_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 94, in __init__
    self.traceback_lines = traceback.format_stack()

11:19:45.89 [WARN] Executor was not shut down explicitly.
11:19:45.89 [ERROR] panic at 'Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.', /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/blocking/shutdown.rs:51
11:19:45.89 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
11:19:45.89 [ERROR] panic at 'A Tokio 1.x context was found, but it is being shutdown.', /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/time/driver/entry.rs:558
11:19:45.89 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.
11:19:45.89 [ERROR] panic at 'panic in a function that cannot unwind', library/core/src/panicking.rs:106
11:19:45.89 [ERROR] Please set RUST_BACKTRACE=1, re-run, and then file a bug at https://github.com/pantsbuild/pants/issues.

[jsirois]

@engnatha can you try with 2.16.0.dev7 or higher? (Preferably 2.16.0rc1). This sounds like a bug @stuhood fixed.

[fengwei-tian]

We still found this issue in the 2.17.0rc1_py39
Here are the logs:
timestamp: 2023-07-08T05:13:43.183791
process title: pantsd [/var/lib/buildkite-agent/builds/buildkite-agent-rippling-main-tests-batches-green-stack-i-01ab64de5614ee8b9-1/rippling/rippling-main-pr-tests]
sys.argv: ['/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/bin/pants', '--pants-bin-name=./pants', '--pants-version=2.17.0rc1', '--spec-files=changed-files.txt', 'test-selection-method', '--output-file=selection-method.txt']
pid: 32055
Signal 15 (SIGTERM) was raised. Exiting with failure.
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/bin/pants", line 8, in
sys.exit(main())
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main
PantsLoader.main()
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 106, in main
cls.run_alternate_entrypoint(entrypoint)
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 84, in run_alternate_entrypoint
entrypoint_fn()
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/pantsd/pants_daemon.py", line 238, in launch_new_pantsd_instance
daemon.run_sync()
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/pantsd/pants_daemon.py", line 211, in run_sync
time.sleep(self.JOIN_TIMEOUT_SECONDS)
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 109, in handle_sigterm
raise self.SignalHandledNonLocalExit(signum, "SIGTERM")
File "/var/lib/buildkite-agent/.cache/pants/setup/bootstrap-Linux-x86_64/2.17.0rc1_py39/lib/python3.9/site-packages/pants/base/exception_sink.py", line 94, in init
self.traceback_lines = traceback.format_stack()

Fatal Python error: Aborted

Thread 0x00007fbc3b77f680 (most recent call first):

[cburroughs]

@engnatha @fengwei-tian Sorry for this hanging out for so long. I'm not able to reproduce this in my repo on 2.31.0. Is this still something you are experiencing?

[engnatha]

I don't use the pantsbuild tooling anymore. Feel free to close out if it seems fixed!

[cburroughs]

Thanks for responding! I hope Pants served you well.