Serialise publishes to avoid concurrent-rsync failures

[StanFromIreland]

This fixes https://python-software-foundation.sentry.io/issues/7209728782/?query=is%3Aunresolved&referrer=issue-stream.

[hugovk]

For reference, the error from Sentry:

Run: 'rsync -a --delete-delay --filter 'P archives/' /srv/docsbuild/cpython-only-html/Doc/build/html/ /srv/docs.python.org/3.15' KO:

Issue ID: 7209728782
Project: python-docs
Date: 29/06/2026, 09:45:00

Message

Run: 'rsync -a --delete-delay --filter 'P archives/' /srv/docsbuild/cpython-only-html/Doc/build/html/ /srv/docs.python.org/3.15' KO:
file has vanished: "/srv/docs.python.org/3.15/.genindex-Q.html.1Um04T"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1356) [sender=3.2.7]

Tags

• environment: production
• interface_type: contexts
• language: en
• level: error
• logger: root
• runtime: CPython 3.13.2
• runtime.name: CPython
• server_name: docs
• version: 3.15

Breadcrumbs

• subprocess subprocess [info]
  /srv/docsbuild/venv-3.15/bin/python3.13 -m ensurepip --upgrade --default-pip
  {"subprocess.cwd":"/srv/docsbuild/venv-3.15","thread.id":"135275563532416","thread.name":"MainThread"}
• subprocess subprocess [info]
  /srv/docsbuild/venv-3.15/bin/python -m pip install --upgrade --upgrade-strategy=eager python-docs-theme -rrequirements.txt jieba PyStemmer~=2.2.0 matplotlib>=3
  {"subprocess.cwd":"PosixPath('/srv/docsbuild/cpython-only-html/Doc')","thread.id":"135275563532416","thread.name":"MainThread"}
• subprocess subprocess [info]
  /srv/docsbuild/venv-3.15/bin/python -m pip freeze --all
  {"thread.id":"135275563532416","thread.name":"MainThread"}
• log root [info]
  Build start.
  {"asctime":"2026-06-29 06:41:05,325"}
• log root [info]
  Running make autobuild-dev-html
  {"asctime":"2026-06-29 06:41:05,325"}
• subprocess subprocess [info]
  make -C /srv/docsbuild/cpython-only-html/Doc PYTHON=/srv/docsbuild/venv-3.15/bin/python SPHINXBUILD=/srv/docsbuild/venv-3.15/bin/sphinx-build BLURB=/srv/docsbuild/venv-3.15/bin/blurb VENVDIR=/srv/docsbuild/venv-3.15 SPHINXOPTS=-D latex_engine=xelatex -D latex_elements.inputenc= -D latex_elements.fontenc= -t create-social-cards -D ogp_site_url=https://docs.python.org/3.15/ SPHINXERRORHANDLING= autobuild-dev-html
  {"thread.id":"135275563532416","thread.name":"MainThread"}
• log root [info]
  Build done (3m 51s).
  {"asctime":"2026-06-29 06:44:56,242"}
• log root [info]
  Publishing start.
  {"asctime":"2026-06-29 06:44:56,242"}
• log root [info]
  Copying HTML files to /srv/docs.python.org/3.15
  {"asctime":"2026-06-29 06:44:59,304"}
• subprocess subprocess [info]
  rsync -a --delete-delay --filter P archives/ /srv/docsbuild/cpython-only-html/Doc/build/html/ /srv/docs.python.org/3.15
  {"thread.id":"135275563532416","thread.name":"MainThread"}

[hugovk]

Claude review:

---

Nice fix — the lock key maps 1:1 onto the target directory, so this correctly serialises the only-html / only-html-en collision from the Sentry report.

One suggestion on the timeout path: the bare raise re-raises zc.lockfile's Couldn't lock '...publish-en-3.15.lock', which is indistinguishable from a single failed attempt — it doesn't say we waited 10 minutes. It then lands in run()'s blanket except Exception as a generic "Badly handled exception, human, please help" Sentry event, so under sustained contention we'd trade the old rsync noise for new, less diagnosable noise.

Suggest converting the timeout into a self-describing TimeoutError:

        except zc.lockfile.LockError as err:
            if perf_counter() >= deadline:
                raise TimeoutError(
                    f"Gave up waiting for lock {path.name} "
                    f"after {timeout} seconds"
                ) from err
            logging.info("Waiting for lock %s...", path.name)
            sleep(poll_interval)

and giving it a dedicated handler in DocBuilder.run, ahead of the catch-all:

        except TimeoutError as err:
            # Another builder held the publish lock for too long; the docs
            # were built fine and will be published on the next run.
            logging.error("%s", err)
            if sentry_sdk:
                sentry_sdk.capture_exception(err)
            return False

That way a timeout still reaches Sentry (a publish exceeding 10 minutes is worth knowing about) but as a distinct, self-explanatory event that groups separately. If we'd rather stay quiet like build_docs_with_lock's "Another builder is running... dying...", drop the capture_exception and keep the log line.

Relatedly, the timeout would fire far less often if the critical section were slimmer: chgrp(..., recursive=True) and chmod_make_readable(...) on self.checkout / "Doc" / "build" / "html" (and on dist_dir) only touch this builder's private checkout, so they could be hoisted above the with wait_for_lock(...) block — that leaves only the target mkdir/chmod, changed_files(), the rsync, and the archives copy holding the lock.

[StanFromIreland]

One suggestion on the timeout path: the bare raise re-raises zc.lockfile's Couldn't lock '...publish-en-3.15.lock', which is indistinguishable from a single failed attempt — it doesn't say we waited 10 minutes.

I applied the suggestion.

Relatedly, the timeout would fire far less often if the critical section were slimmer

Realistically hitting the ten minute timeout seems very unlikely, and saving a few seconds won't be of much help. I'd rather avoid refactors for theoretical cases.

[hugovk]

Thanks! The main thing is not sending new Sentry reports when we didn't before. I think we hit the timeout more in the past when we still did PDFs and some translations took a long time, especially before Sphinx made some big performance improvements.