Ensure 'pip wheel' can create .so artifacts deterministically

[thundergolfer]

What's the problem this feature will solve?

The Bazel build system has the major selling point of supporting both local and remote-caching.

In order for that caching to work though, Bazel targets must be built deterministically so that the same target always has the same content-addressable hash.

Currently pip wheel is non-deterministic, so our Python Bazel targets will cache miss if they depend on something built with pip wheel.

Describe the solution you'd like

Note: The following is the output of a Bazel execution log. A bit unrelated to the pip wheel command but shows the relevant information.

inputs {
  path: "external/pypi__PyYAML_5_1/PyYAML-5.1.dist-info/LICENSE"
  digest {
    hash: "a2adb9c959b797494a0ef80bdf60e22db2749ee3e0c0908556e3eb548f967c56"
    size_bytes: 1101
    hash_function_name: "SHA-256"
  }
}
inputs {
  path: "external/pypi__PyYAML_5_1/PyYAML-5.1.dist-info/METADATA"
  digest {
    hash: "df7bc0c7cbd2ce350c5c61ceda3a74bbcb6f82446a7c01f7f8e1034a98df231f"
    size_bytes: 1704
    hash_function_name: "SHA-256"
  }
}
inputs {
  path: "external/pypi__PyYAML_5_1/PyYAML-5.1.dist-info/RECORD"
  digest {
    hash: "6fe803b74ab4fcab1f23e96060cf062d12779598af7e72692c492c2dd7cad0ed"
    size_bytes: 1701
    hash_function_name: "SHA-256"
  }
}
inputs {
  path: "external/pypi__PyYAML_5_1/PyYAML-5.1.dist-info/WHEEL"
  digest {
    hash: "cdf2c8f141bc498ae490a88870d655dd174abe3db8c1f57562224b168930c624"
    size_bytes: 104
    hash_function_name: "SHA-256"
  }
}
inputs {
  path: "external/pypi__PyYAML_5_1/PyYAML-5.1.dist-info/top_level.txt"
  digest {
    hash: "ae98f42153138ac02387fd6f1b709c7fdbf98e9090c00cfa703d48554e597614"
    size_bytes: 11
    hash_function_name: "SHA-256"
  }
}
inputs {
  path: "external/pypi__PyYAML_5_1/_yaml.cpython-36m-x86_64-linux-gnu.so"
  digest {
    hash: "a7f3774015f839ccee5e2281bbfdf22a42e0e1dafaac33ef4c91db83a07210d9"
    size_bytes: 1133288
    hash_function_name: "SHA-256"
  }
}
inputs {
  path: "external/pypi__PyYAML_5_1/yaml/__init__.py"
  digest {
    hash: "2af8b6dbcb1df5c63597f215421cad02f2317e291061b181b0f7bbf4f71ac0dd"
    size_bytes: 12012
    hash_function_name: "SHA-256"
  }
}

The following is a subset of the build outputs of the PyYAML package. Of the build outputs, it is the RECORD files and the _yaml.cpython-36m-x86_64-linux-gnu.so shared object file that have non-deterministic hashes build to build. I have inspected the RECORD file and found that it contains the hash of the .so file, so it is non-deterministic because of the .so file, and I think only because of that.

So the problem is the .so file.

I ran the strings program on the .so file and found this printable string: /tmp/pip-wheel-_bd8v3f2/pyyaml. That is coming from here:

pip/src/pip/_internal/wheel.py

Line 649 in 6af9de9

with TempDirectory(kind="wheel") as temp_dir:

So while I found other differences between different _yaml.cpython-36m-x86_64-linux-gnu.so, this tmp directory usage leaking in itself is sufficient to break determinism.

Additional context

rules_python issue discussing this problem: bazel-contrib/rules_python#154
rules_python repo: https://github.com/bazelbuild/rules_python

[chrahunt]

This string is embedded in the debug information:

$ objdump -Wi _yaml.cpython-37m-x86_64-linux-gnu.so  | grep /tmp
    <15>   DW_AT_comp_dir    : (indirect string, offset: 0x76b): /tmp/user/1000/pip-install-94g5rob6/pyyaml

It looks like this is a common issue:

• cmd/go: support reproducible builds regardless of build path? golang/go#16860
• https://tests.reproducible-builds.org/debian/issues/unstable/gcc_captures_build_path_issue.html

It may also be related to pypa/wheel#248.

Using a deterministic build directory name would leave us open to denial-of-service and/or concurrency-related problems.

Using e.g. the GCC flag -fdebug-prefix-map=$SRC_ROOT=. is another option, but is this something that should be in pip, wheel, or the individual build backends?

[thundergolfer]

Thanks for your input Christopher.

would leave us open to denial-of-service and/or concurrency-related problems.

The latter makes sense to me, but how exactly is DOS at play if you use a consistent build dir?

should be in pip, wheel, or the individual build backends?

For my use case, the input of the Bazel / rules_python team would be helpful in deciding. They haven't responded to my issue in rules_python yet. Might be time for a nudge.

[chrahunt]

The latter makes sense to me, but who exactly is DOS at play if you use a consistent build dir?

An unprivileged user on the same host can create the build directory and set its permissions to 700, which prevents you from building.

[manthey]

In a simple test, I was able to get consistent builds by exporting CFLAGS=-g0 before building the wheel. This prevents adding any of the debug information to the generated libraries which is where the TempDirectory was being pulled in. I also have SOURCE_DATE_EPOCH set. I don't know how universal this is (and, of course, you lose debugging symbols).

[ngie-eign]

This is an issue not just with pip wheel, but pip install in general when not run in editable mode :(.

CC: @bdrewery

[ngie-eign]

Interesting... this is a common pattern that can't be overridden:

$ grep -rI TempDirectory  
pip/_internal/build_env.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/build_env.py:        self._temp_dir = TempDirectory(kind="build-env")
pip/_internal/cache.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/cache.py:        self._temp_dir = TempDirectory(kind="ephem-wheel-cache")
pip/_internal/download.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/download.py:    with TempDirectory(kind="unpack") as temp_dir:
pip/_internal/wheel.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/wheel.py:        with TempDirectory(kind="wheel") as temp_dir:
pip/_internal/commands/download.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/commands/download.py:            with RequirementTracker() as req_tracker, TempDirectory(
pip/_internal/commands/install.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/commands/install.py:        target_temp_dir = TempDirectory(kind="target")
pip/_internal/commands/install.py:            with RequirementTracker() as req_tracker, TempDirectory(
pip/_internal/commands/wheel.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/commands/wheel.py:            with RequirementTracker() as req_tracker, TempDirectory(
pip/_internal/req/req_install.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/req/req_install.py:        self._temp_build_dir = TempDirectory(kind="req-build")
pip/_internal/req/req_install.py:        with TempDirectory(kind="record") as temp_dir:
pip/_internal/req/req_tracker.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/req/req_tracker.py:            self._temp_dir = TempDirectory(delete=False, kind='req-tracker')
pip/_internal/req/req_uninstall.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/req/req_uninstall.py:        self.save_dir = TempDirectory(kind="uninstall")
pip/_internal/utils/temp_dir.py:class TempDirectory(object):
pip/_internal/utils/temp_dir.py:        super(TempDirectory, self).__init__()
pip/_internal/vcs/bazaar.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/vcs/bazaar.py:        with TempDirectory(kind="export") as temp_dir:
pip/_internal/vcs/git.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/vcs/git.py:        with TempDirectory(kind="export") as temp_dir:
pip/_internal/vcs/mercurial.py:from pip._internal.utils.temp_dir import TempDirectory
pip/_internal/vcs/mercurial.py:        with TempDirectory(kind="export") as temp_dir:

From pip.util.TempDirectory:

     60     def create(self):
     61         """Create a temporary directory and store it's path in self.path
     62         """
     63         if self.path is not None:
     64             logger.debug(
     65                 "Skipped creation of temporary directory: {}".format(self.path)
     66             )
     67             return
     68         # We realpath here because some systems have their default tmpdir
     69         # symlinked to another directory.  This tends to confuse build
     70         # scripts, so we canonicalize the path by traversing potential
     71         # symlinks here.
     72         self.path = os.path.realpath(
     73             tempfile.mkdtemp(prefix="pip-{}-".format(self.kind))
     74         )
     75         logger.debug("Created temporary directory: {}".format(self.path))

It's unfortunate, but it looks like the mkdtemp wrapper lacks the ability to override the template, which would allow us to set a deterministic build path:

$ man 3 mktemp
...
     char *
     mkdtemp(char *template);

Sidenote: hmmm... the notion of tempfiles in cpython as of 3.9 seems insecure; it bypasses better security practices by reimplementing mkstemp(3), etc, with a deterministic algorithm/template consisting of only 8 "random" characters seeded by the PID :(.

[ngie-eign]

I've fallen back to invoking setup.py directly for building binaries and am using pip only when installing the binaries.

It's really unfortunate that there isn't a better story around this with pip, given that pip is sort of the defacto core installation utility for python. Hopefully the Chan-Zuckerberg work will improve this support/usability, especially since Facebook used buck (alternative to bazel) by default, which does similar tricks in terms of caching artifacts on remote shares to speed up their build process.

[sbidoul]

Is this another argument in favor of in-tree builds (#7555)?