Run tracker failed to create directory for logs

[junyuanZhang-cb]

Hi, I am getting this error when running pants. Looks like it failed to create the folder by calling self.run_logs_file.parent.mkdir(exist_ok=True, parents=True)
but parents=True should work no matter the parents exist or not. I am confused.

docker exec -ti udas_webserver sh -c "./pants --print-stacktrace -ldebug run udas/src/python:launcher -- manage runserver 0.0.0.0:8888 --noreload"
20:50:13.38 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f791fcee970>
20:50:13.41 [DEBUG] terminating pantsd
20:50:13.41 [DEBUG] sending signal 15 to pid 151
20:50:13.62 [DEBUG] successfully terminated pid 151
20:50:13.62 [DEBUG] purging metadata directory: /opt/pynest/.pids/7fe3dfe26c48/pantsd
20:50:13.64 [DEBUG] Launching pantsd
20:50:13.64 [DEBUG] purging metadata directory: /opt/pynest/.pids/7fe3dfe26c48/pantsd
20:50:13.64 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin:/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/lib-dynload:/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/bin/python /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/bin/pants --pants-bin-name=./pants --pants-version=2.13.0 --print-stacktrace -ldebug run udas/src/python:launcher -- manage runserver 0.0.0.0:8888 --noreload
20:50:14.96 [DEBUG] pantsd is running at pid 612, pailgun port is 33789
20:50:14.96 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f791fcee970>
20:50:14.96 [DEBUG] Connecting to pantsd on port 33789
20:50:14.96 [DEBUG] Connecting to pantsd on port 33789 attempt 1/3
20:50:14.97 [DEBUG] Connected to pantsd
20:50:14.99 [DEBUG] Launching 1 roots (poll=false).
20:50:15.03 [DEBUG] computed 1 nodes in 0.047116 seconds. there are 7 total nodes.
20:50:15.20 [INFO] Initializing scheduler...
20:50:15.21 [DEBUG] File handle limit is: 1048576
20:50:15.22 [DEBUG] Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: nailgun::CommandRunner { inner: local::CommandRunner { .. }, .. }, .. }, .. }] for process execution.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.t2vWUx/install/bin, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python38.zip, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python3.8, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /usr/lib/python3.8/lib-dynload, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] Changes to /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages, outside of the buildroot, will not be invalidated.
20:50:15.41 [DEBUG] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f5c2af00fd0>
20:50:15.41 [DEBUG] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f5c2af12850>
20:50:15.41 [DEBUG] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f5c2af00fd0>
20:50:15.41 [DEBUG] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x7f5c2af12850>
20:50:15.42 [INFO] Scheduler initialized.
20:50:15.42 [DEBUG] Launching 1 roots (poll=false).
20:50:15.42 [DEBUG] computed 1 nodes in 0.000391 seconds. there are 7 total nodes.
20:50:15.46 [ERROR] [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'
Traceback (most recent call last):
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/bin/daemon_pants_runner.py", line 131, in single_daemonized_run
    runner = LocalPantsRunner.create(
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 130, in create
    run_tracker = RunTracker(options_bootstrapper.args, options)
  File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/lib/python3.8/site-packages/pants/goal/run_tracker.py", line 95, in __init__
    self.run_logs_file.parent.mkdir(exist_ok=True, parents=True)
  File "/usr/lib/python3.8/pathlib.py", line 1293, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/pynest/.pants.d/run-tracker/pants_run_2023_06_02_20_50_15_463_684ee4e61f3c465db30f03ff385e5b42'


See https://www.pantsbuild.org/v2.13/docs/troubleshooting for common issues.
Consider reaching out for help: https://www.pantsbuild.org/v2.13/docs/getting-help

[benjyw]

Turned into an issue at #19244