pex binary hangs on startup at atomic_directory

[mbakhoff]

My pex binary is hanging on startup. Using python3.7, pex 2.1.21, ubuntu 18.04. Here's the stack:

Traceback (most recent call first):
  <built-in method lockf of module object at remote 0x7fdf05a08f50>
  File "/var/zt/zt_consumers/current/.bootstrap/pex/common.py", line 385, in atomic_directory
  <built-in method next of module object at remote 0x7fdf06709d10>
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/zt/zt_consumers/current/.bootstrap/pex/util.py", line 182, in cache_distribution
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 173, in _write_zipped_internal_cache
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 197, in _load_internal_cache
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 227, in _update_candidate_distributions
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 416, in _activate
  File "/var/zt/zt_consumers/current/.bootstrap/pex/environment.py", line 260, in activate
  File "/var/zt/zt_consumers/current/.bootstrap/pex/pex.py", line 103, in _activate
  File "/var/zt/zt_consumers/current/.bootstrap/pex/pex.py", line 444, in execute
  File "/var/zt/zt_consumers/current/.bootstrap/pex/pex_bootstrapper.py", line 360, in bootstrap_pex
  File "/var/zt/zt_consumers/current/__main__.py", line 68, in <module>
  <built-in method exec of module object at remote 0x7fdf06709d10>
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)

It's a bit tricky to reproduce because it's timing sensitive. The basic steps should be: build a pex binary containing some wheels, launch multiple copies of the binary in parallel. All processes must use the same pex_root.

The hang happens in pex/common.py atomic_directory. What I think happens is that the first process takes a file lock for the atomic dir, finalizes it and releases the lock. If the timing is right, then another process reaches https://github.com/pantsbuild/pex/blob/v2.1.21/pex/common.py#L390 so it also grabs the lock but never releases it.

The issue was introduced in #1062

[jsirois]

Thanks @mbakhoff. That's a facepalm bug. Thanks for identifying, I'll get out a fix for this tomorrow.

[mbakhoff]

@jsirois Thank you for the quick fix! 👍