Add support for PYTHON_BUILD_MIRROR_URL when checksums do not exist

[jamescurtin]

Make sure you have checked all steps below.

Prerequisite

• [N/A] Please consider implementing the feature as a hook script or plugin as a first step.
  • pyenv has some powerful support for plugins and hook scripts. Please refer to Authoring plugins for details and try to implement it as a plugin if possible.
• [N/A] Please consider contributing the patch upstream to rbenv, since we have borrowed most of the code from that project.
  • We occasionally import the changes from rbenv. In general, you can expect changes made in rbenv will be imported to pyenv too, eventually.
  • Generally speaking, we prefer not to make changes in the core in order to keep compatibility with rbenv.
• My PR addresses the following pyenv issue (if any)
  • Closes PYTHON_BUILD_MIRROR_URL desn't take effect. #1473
  • Closes Support for pull-through cache-mirror of python packages #1377
  • Specify install path to python tar  #1575
  • It seems PYTHON_BUILD_MIRROR_URL environment variable doesn't work #803

Description

Fixes an issue where PYTHON_BUILD_MIRROR_URL is not used if the mirror does not have the same checksum that is expected. In some cases, the mirror is not expected to have a matching checksum (as in the case of a corporate mirror behind a firewall), which causes errors.

By setting PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM, the PYTHON_BUILD_MIRROR_URL will be used but will not attempt to match a checksum.

Tests

• My PR adds the following unit tests (if any)
  • package is fetched from mirror when checksum is invalid if SKIP_CHECKSUM set

[jamescurtin]

👋 Checking in on this PR: would someone please be able to review? Thank you!

[jamescurtin]

Hi: wanted to check in again--would a maintainer be able to review this PR please? Thank you! (CC @joshfriend sorry for the ping 😄 )

[joshfriend]

why would the download from a mirror have a different checksum?

[jamescurtin]

The downloaded file will have the same checksum--the issue with the current implementation is the assumption that the checksum is always included in the filepath of the mirrored distribution. For example:

3.8.5 can be downloaded from https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz.

My PYTHON_BUILD_MIRROR_URL is https://mirror.example.com/python/, and the source distribution is available at https://mirror.example.com/python/3.8.5/Python-3.8.5.tar.xz.

Based on this logic in python-build, there is an assumption that the path to the mirrored distribution is https://mirror.example.com/python/e3003.../Python-3.8.5.tar.xz, where e3003... is the checksum of the file. This may be the case for some mirrors, but is not the case when configuring a remote repository mirror using Artifactory (as is my situation).

Therefore, the behavior introduced with PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM is to skip templating the checksum into the URL of the download. The verify_checksum function is still called when downloading the tarball from the mirror, so you remain protected against corrupted or poisoned distributions.

[joshfriend]

Ah I understand now :) thank you ❤️

[uzxmx]

Just want to add a comment:

Because of this code snippet, it actually tries downloading the tarball twice. The first time it tries downloading with the checksum appended, but fails. The second time without checksum, hence succeeds.

So I think the name PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM may not be a good name, because it doesn't skip, but actually substitutes the original url with PYTHON_BUILD_MIRROR_URL.

  if [ -n "$PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM" ]; then
      local package_url="$(echo "$1" | sed -e "s|.*//${URL_BASE:-$official_source}|$PYTHON_BUILD_MIRROR_URL|g")"
  else
      local package_url="$1"
  fi

In order to use the name PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM, maybe here is a good place to do some edits.

[lushi516]

The Alibaba Cloud image path does not comply with the official agreement