Race condition when importing Menagerie descriptions concurrently from multiple processes

[hartikainen]

Bug description

When import import MuJoCo Menagerie descriptions concurrently from more than one process, the git clone fails. I was expecting (or at least hoping) for this to work.

Reproduction steps

Steps to reproduce the bug:

1. Install GNU parallel and robot_descriptions
2. Run:

rm -rf ~/.cache/robot_descriptions/ && \
parallel --verbose -j 4 "
    python -c 'from robot_descriptions import fr3_mj_description; print({1}, fr3_mj_description.MJCF_PATH)'
" ::: {1..4}

3. See error

Code

See above

Logs

Full error:

$ rm -rf ~/.cache/robot_descriptions/ && \
$ parallel --verbose -j 4 "
    python -c 'from robot_descriptions import fr3_mj_description; print({1}, fr3_mj_description.MJCF_PATH)'
" ::: {1..4}

    python -c 'from robot_descriptions import fr3_mj_description; print(1, fr3_mj_description.MJCF_PATH)'


    python -c 'from robot_descriptions import fr3_mj_description; print(2, fr3_mj_description.MJCF_PATH)'


    python -c 'from robot_descriptions import fr3_mj_description; print(3, fr3_mj_description.MJCF_PATH)'


    python -c 'from robot_descriptions import fr3_mj_description; print(4, fr3_mj_description.MJCF_PATH)'

Cloning https://github.com/deepmind/mujoco_menagerie.git...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/fr3_mj_description.py", line 14, in <module>
    REPOSITORY_PATH: str = _clone_to_cache(
                           ^^^^^^^^^^^^^^^^
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 138, in clone_to_cache
    clone = clone_to_directory(
            ^^^^^^^^^^^^^^^^^^^
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 80, in clone_to_directory
    os.makedirs(target_dir)
  File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/Users/user/.cache/robot_descriptions/mujoco_menagerie'
Cloning https://github.com/deepmind/mujoco_menagerie.git...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/fr3_mj_description.py", line 14, in <module>
    REPOSITORY_PATH: str = _clone_to_cache(
                           ^^^^^^^^^^^^^^^^
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 138, in clone_to_cache
    clone = clone_to_directory(
            ^^^^^^^^^^^^^^^^^^^
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 80, in clone_to_directory
    os.makedirs(target_dir)
  File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/Users/user/.cache/robot_descriptions/mujoco_menagerie'
Cloning https://github.com/deepmind/mujoco_menagerie.git...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/fr3_mj_description.py", line 14, in <module>
    REPOSITORY_PATH: str = _clone_to_cache(
                           ^^^^^^^^^^^^^^^^
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 138, in clone_to_cache
    clone = clone_to_directory(
            ^^^^^^^^^^^^^^^^^^^
  File "/Users/user/github/Code/code-worktree-3/.venv/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 80, in clone_to_directory
    os.makedirs(target_dir)
  File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/Users/user/.cache/robot_descriptions/mujoco_menagerie'
Cloning https://github.com/deepmind/mujoco_menagerie.git...
4 /Users/user/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
100%|██████████| 1058.0/1058.0 [00:24<00:00, 43.20it/s]

System

• OS: macOS Sequoia 15.0.1
• robot_descriptions version: 1.13.0

Additional context

I spent a bit of time trying to get some sort of locking mechanism to work for clone_to_cache. I never got to finishing it but below's what I had so far. Just putting it out here in case someone finds useful and wants to pick it up.

`clone_to_cache_with_lock`

def clone_to_cache_with_lock(description_name: str, commit: Optional[str] = None) -> str:

    lock_file_path = pathlib.Path(
        os.environ.get(
            "ROBOT_DESCRIPTIONS_CACHE",
            "~/.cache/robot_descriptions/CLONE_LOCK",
        )
    ).expanduser()
    lock_file_path.parent.mkdir(parents=True, exist_ok=True)

    print(f"{lock_file_path=}")

    timeout = 300
    poll_every = 5
    start_time = time.time()
    while True:
        try:
            # Attempt to create the lock file atomically
            lock_fd = os.open(str(lock_file_path), os.O_CREAT | os.O_EXCL | os.O_RDWR)

            try:
                print(f"Process {os.getpid()} is cloning the repository...")
                clone_to_cache(description_name, commit)
                print(f"Process {os.getpid()} completed cloning.")
            except subprocess.CalledProcessError as e:
                print(f"Process {os.getpid()} failed to clone the repository.")
                raise e
            finally:
                os.close(lock_fd)
                lock_file_path.unlink()
            break
        except FileExistsError as e:
            # Lock file exists, wait for cloning to complete
            print(f"Process {os.getpid()} detected cloning in progress. Waiting...")
            time.sleep(poll_every)
            elapsed_time = time.time() - start_time
            if elapsed_time > timeout:
                print(f"Process {os.getpid()} timeout exceeded. Removing stale lock file.")
                try:
                    lock_file_path.unlink()
                except FileNotFoundError:
                    pass
                continue
        except Exception as e:
            print(f"Process {os.getpid()} encountered an error: {e}")
            raise e

[stephane-caron]

Thank you for this detailed report and proposing a first idea 👍 From the logs I'm guessing processes are trying to clone concurrently as Menagerie is not in cache yet when they start.

Actually, even if it the description is already in cache, we will run into a similar error. I've adapted your parallel command to run on my machine but it does the same thing:

parallel -i python -c 'from robot_descriptions import fr3_mj_description; print({}, fr3_mj_description.MJCF_PATH)' -- 1 2 3 4

Error: Another git process seems to be running in this repository, ...

3 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
Traceback (most recent call last):
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 90, in clone_to_directory
    clone.git.checkout(commit)
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 800, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1386, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1183, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git checkout 3d593a327ff8d5efd10f4926ccc477820e9252c1
  stderr: 'fatal: Unable to create '~/.cache/robot_descriptions/mujoco_menagerie/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/fr3_mj_description.py", line 14, in <module>
    REPOSITORY_PATH: str = _clone_to_cache(
                           ^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 139, in clone_to_cache
    clone = clone_to_directory(
            ^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/robot_descriptions/_cache.py", line 98, in clone_to_directory
    clone.git.checkout(commit)
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 800, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1386, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.micromamba/envs/robot_descriptions/lib/python3.12/site-packages/git/cmd.py", line 1183, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git checkout 3d593a327ff8d5efd10f4926ccc477820e9252c1
  stderr: 'fatal: Unable to create '~/.cache/robot_descriptions/mujoco_menagerie/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.'
Commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 not found, let's fetch origin and try again...
Commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 not found, let's fetch origin and try again...
Found commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 successfully!
4 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
Commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 not found, let's fetch origin and try again...
Found commit 3d593a327ff8d5efd10f4926ccc477820e9252c1 successfully!
1 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml

At first read I had a hunch this error is related to the quick-and-dirty way we pass arguments via environment variables, which won't work well when we run in parallel. But in this instance the exception is rather raised from:

robot_descriptions.py/robot_descriptions/_cache.py

Lines 89 to 95 in 380aa5a

	try:
	clone.git.checkout(commit)
	except GitCommandError:
	print(
	f"Commit {commit} not found, "
	"let's fetch origin and try again..."
	)

And the (currently uncaught) exception message reads:

Unable to create '~/.cache/robot_descriptions/mujoco_menagerie/.git/index.lock': File exists.

We can probably make a quick workaround for this one. Spinning it to #115 so that we don't loose track of the initial point here.

[stephane-caron]

OK, after #116 we can at least do:

$ parallel -i -j 4 python -c 'from robot_descriptions import fr3_mj_description; print({}, fr3_mj_description.MJCF_PATH)' -- 1 2 3 4
3 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
4 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
2 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml
1 ~/.cache/robot_descriptions/mujoco_menagerie/franka_fr3/fr3.xml

Thus, if all the descriptions we import are from the same commit of the same repository, one thing that works now is:

1. Import the description once from a single process, waiting for it to be cloned to cache, then
2. Run in as many parallel imports as needed.

I hope this can help with your current use case?

The initial case where we clear the cache right before running in parallel is not handled yet, as the rest of the initialization/recovery logic in clone_to_directory still assumes exclusive access to the cache.

[hartikainen]

This is indeed enough for my current use case because I can easily pre-load the cache. Thanks for such a fast resolution here, Stephane! I'll close this issue for now, but perhaps another one needs to be opened for the "true" concurrency support :)