ValueError: invalid literal for int() with base 10 during pex build

[Github-dm-CDE]

Since the update to apache-airflow 2.9.1 methodtools 0.4.7 is required as a transitive dependency.
Currently there seems to be a problem in the interaction between the mentioned methodtools version and pex 2.3.1 which we cannot quite explain.
Our pex build fails every time since the update.

This is our requirements.txt:

--index-url [https://gitlab-ci-token:[MASKED]@gitlab.[MASKED].com/api/v4/projects/[MASKED]/packages/pypi/simple](https://gitlab-ci-token:%5BMASKED%5D@gitlab.[MASKED].com/api/v4/projects/[MASKED]/packages/pypi/simple)
aiofiles==23.2.1 ; python_version >= "3.9" and python_version < "3.10"
aiohttp==3.9.5 ; python_version >= "3.9" and python_version < "3.10"
aiosignal==1.3.1 ; python_version >= "3.9" and python_version < "3.10"
alembic==1.13.1 ; python_version >= "3.9" and python_version < "3.10"
annotated-types==0.6.0 ; python_version >= "3.9" and python_version < "3.10"
anyio==4.3.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-common-io==1.3.1 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-common-sql==1.13.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-fab==1.1.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-ftp==3.9.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-http==4.11.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-imap==3.6.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-postgres==5.11.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-smtp==1.7.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-sqlite==3.8.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow-providers-ssh==3.11.0 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow==2.9.1 ; python_version >= "3.9" and python_version < "3.10"
apache-airflow[ldap,postgres,ssh]==2.9.1 ; python_version >= "3.9" and python_version < "3.10"
apispec[yaml]==6.6.1 ; python_version >= "3.9" and python_version < "3.10"
argcomplete==3.3.0 ; python_version >= "3.9" and python_version < "3.10"
asgiref==3.8.1 ; python_version >= "3.9" and python_version < "3.10"
astronomer-providers==1.19.0 ; python_version >= "3.9" and python_version < "3.10"
async-timeout==4.0.3 ; python_version >= "3.9" and python_version < "3.10"
attrs==23.2.0 ; python_version >= "3.9" and python_version < "3.10"
babel==2.15.0 ; python_version >= "3.9" and python_version < "3.10"
backoff==2.2.1 ; python_version >= "3.9" and python_version < "3.10"
bcrypt==4.1.3 ; python_version >= "3.9" and python_version < "3.10"
blinker==1.8.2 ; python_version >= "3.9" and python_version < "3.10"
cachelib==0.13.0 ; python_version >= "3.9" and python_version < "3.10"
certifi==2024.2.2 ; python_version >= "3.9" and python_version < "3.10"
cffi==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
charset-normalizer==3.3.2 ; python_version >= "3.9" and python_version < "3.10"
click==8.1.7 ; python_version >= "3.9" and python_version < "3.10"
clickclick==20.10.2 ; python_version >= "3.9" and python_version < "3.10"
colorama==0.4.6 ; python_version >= "3.9" and python_version < "3.10"
colorlog==4.8.0 ; python_version >= "3.9" and python_version < "3.10"
configupdater==3.2 ; python_version >= "3.9" and python_version < "3.10"
connexion[flask]==2.14.1 ; python_version >= "3.9" and python_version < "3.10"
cron-descriptor==1.4.3 ; python_version >= "3.9" and python_version < "3.10"
croniter==2.0.5 ; python_version >= "3.9" and python_version < "3.10"
cryptography==42.0.7 ; python_version >= "3.9" and python_version < "3.10"
deprecated==1.2.14 ; python_version >= "3.9" and python_version < "3.10"
dill==0.3.8 ; python_version >= "3.9" and python_version < "3.10"
dm-knox-token==2.0.16 ; python_version >= "3.9" and python_version < "3.10"
dm-webhdfs==9.1.16 ; python_version >= "3.9" and python_version < "3.10"
dnspython==2.6.1 ; python_version >= "3.9" and python_version < "3.10"
docutils==0.21.2 ; python_version >= "3.9" and python_version < "3.10"
email-validator==2.1.1 ; python_version >= "3.9" and python_version < "3.10"
exceptiongroup==1.2.1 ; python_version >= "3.9" and python_version < "3.10"
flask-appbuilder==4.4.1 ; python_version >= "3.9" and python_version < "3.10"
flask-babel==2.0.0 ; python_version >= "3.9" and python_version < "3.10"
flask-caching==2.0.1 ; python_version >= "3.9" and python_version < "3.10"
flask-jwt-extended==4.6.0 ; python_version >= "3.9" and python_version < "3.10"
flask-limiter==3.6.0 ; python_version >= "3.9" and python_version < "3.10"
flask-login==0.6.3 ; python_version >= "3.9" and python_version < "3.10"
flask-session==0.5.0 ; python_version >= "3.9" and python_version < "3.10"
flask-sqlalchemy==2.5.1 ; python_version >= "3.9" and python_version < "3.10"
flask-wtf==1.2.1 ; python_version >= "3.9" and python_version < "3.10"
flask==2.2.5 ; python_version >= "3.9" and python_version < "3.10"
frozenlist==1.4.1 ; python_version >= "3.9" and python_version < "3.10"
fsspec==2024.3.1 ; python_version >= "3.9" and python_version < "3.10"
google-re2==1.1.202[40](https://gitlab.[MASKED].com/cxa/cxa-ck/pex-envs/pex-airflow/-/jobs/35382959#L40)501 ; python_version >= "3.9" and python_version < "3.10"
googleapis-common-protos==1.63.0 ; python_version >= "3.9" and python_version < "3.10"
greenlet==3.0.3 ; python_version >= "3.9" and (platform_machine == "aarch64" or platform_machine == "ppc64le" or platform_machine == "x86_64" or platform_machine == "amd64" or platform_machine == "AMD64" or platform_machine == "win32" or platform_machine == "WIN32") and python_version < "3.10"
grpcio==1.63.0 ; python_version >= "3.9" and python_version < "3.10"
gunicorn==22.0.0 ; python_version >= "3.9" and python_version < "3.10"
h11==0.14.0 ; python_version >= "3.9" and python_version < "3.10"
httpcore==1.0.5 ; python_version >= "3.9" and python_version < "3.10"
httpx==0.27.0 ; python_version >= "3.9" and python_version < "3.10"
idna==3.7 ; python_version >= "3.9" and python_version < "3.10"
importlib-metadata==7.1.0 ; python_version >= "3.9" and python_version < "3.10"
importlib-resources==6.4.0 ; python_version >= "3.9" and python_version < "3.10"
inflection==0.5.1 ; python_version >= "3.9" and python_version < "3.10"
itsdangerous==2.2.0 ; python_version >= "3.9" and python_version < "3.10"
jinja2==3.1.4 ; python_version >= "3.9" and python_version < "3.10"
jsonschema-specifications==2023.12.1 ; python_version >= "3.9" and python_version < "3.10"
jsonschema==4.22.0 ; python_version >= "3.9" and python_version < "3.10"
lazy-object-proxy==1.10.0 ; python_version >= "3.9" and python_version < "3.10"
ldap3==2.9.1 ; python_version >= "3.9" and python_version < "3.10"
limits==3.11.0 ; python_version >= "3.9" and python_version < "3.10"
linkify-it-py==2.0.3 ; python_version >= "3.9" and python_version < "3.10"
lockfile==0.12.2 ; python_version >= "3.9" and python_version < "3.10"
mako==1.3.3 ; python_version >= "3.9" and python_version < "3.10"
markdown-it-py==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
markupsafe==2.1.5 ; python_version >= "3.9" and python_version < "3.10"
marshmallow-oneofschema==3.1.1 ; python_version >= "3.9" and python_version < "3.10"
marshmallow-sqlalchemy==0.28.2 ; python_version >= "3.9" and python_version < "3.10"
marshmallow==3.21.2 ; python_version >= "3.9" and python_version < "3.10"
mdit-py-plugins==0.4.0 ; python_version >= "3.9" and python_version < "3.10"
mdurl==0.1.2 ; python_version >= "3.9" and python_version < "3.10"
methodtools==0.4.7 ; python_version >= "3.9" and python_version < "3.10"
more-itertools==10.2.0 ; python_version >= "3.9" and python_version < "3.10"
multidict==6.0.5 ; python_version >= "3.9" and python_version < "3.10"
numpy==1.26.4 ; python_version >= "3.9" and python_version < "3.10"
opentelemetry-api==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
opentelemetry-exporter-otlp-proto-grpc==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
opentelemetry-exporter-otlp-proto-http==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
opentelemetry-exporter-otlp==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
opentelemetry-proto==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
opentelemetry-sdk==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
opentelemetry-semantic-conventions==0.37b0 ; python_version >= "3.9" and python_version < "3.10"
ordered-set==4.1.0 ; python_version >= "3.9" and python_version < "3.10"
packaging==24.0 ; python_version >= "3.9" and python_version < "3.10"
paramiko==3.4.0 ; python_version >= "3.9" and python_version < "3.10"
pathspec==0.12.1 ; python_version >= "3.9" and python_version < "3.10"
pendulum==3.0.0 ; python_version >= "3.9" and python_version < "3.10"
pluggy==1.5.0 ; python_version >= "3.9" and python_version < "3.10"
prison==0.2.1 ; python_version >= "3.9" and python_version < "3.10"
prompt-toolkit==3.0.43 ; python_version >= "3.9" and python_version < "3.10"
protobuf==4.25.3 ; python_version >= "3.9" and python_version < "3.10"
psutil==5.9.8 ; python_version >= "3.9" and python_version < "3.10"
psycopg2-binary==2.9.9 ; python_version >= "3.9" and python_version < "3.10"
pyarrow==16.0.0 ; python_version >= "3.9" and python_version < "3.10"
pyasn1-modules==0.4.0 ; python_version >= "3.9" and python_version < "3.10"
pyasn1==0.6.0 ; python_version >= "3.9" and python_version < "3.10"
pycparser==2.22 ; python_version >= "3.9" and python_version < "3.10"
pydantic-core==2.18.2 ; python_version >= "3.9" and python_version < "3.10"
pydantic==2.7.1 ; python_version >= "3.9" and python_version < "3.10"
pygments==2.18.0 ; python_version >= "3.9" and python_version < "3.10"
pyjwt==2.8.0 ; python_version >= "3.9" and python_version < "3.10"
pynacl==1.5.0 ; python_version >= "3.9" and python_version < "3.10"
python-daemon==3.0.1 ; python_version >= "3.9" and python_version < "3.10"
python-dateutil==2.9.0.post0 ; python_version >= "3.9" and python_version < "3.10"
python-ldap==3.4.4 ; python_version >= "3.9" and python_version < "3.10"
python-nvd3==0.16.0 ; python_version >= "3.9" and python_version < "3.10"
python-slugify==8.0.4 ; python_version >= "3.9" and python_version < "3.10"
pytz==2024.1 ; python_version >= "3.9" and python_version < "3.10"
pyxattr==0.8.1 ; python_version >= "3.9" and python_version < "3.10"
pyyaml==6.0.1 ; python_version >= "3.9" and python_version < "3.10"
referencing==0.35.1 ; python_version >= "3.9" and python_version < "3.10"
requests-toolbelt==1.0.0 ; python_version >= "3.9" and python_version < "3.10"
requests==2.31.0 ; python_version >= "3.9" and python_version < "3.10"
rfc3339-validator==0.1.4 ; python_version >= "3.9" and python_version < "3.10"
rich-argparse==1.4.0 ; python_version >= "3.9" and python_version < "3.10"
rich==13.7.1 ; python_version >= "3.9" and python_version < "3.10"
rpds-py==0.18.1 ; python_version >= "3.9" and python_version < "3.10"
setproctitle==1.3.3 ; python_version >= "3.9" and python_version < "3.10"
setuptools==69.5.1 ; python_version >= "3.9" and python_version < "3.10"
six==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
sniffio==1.3.1 ; python_version >= "3.9" and python_version < "3.10"
sqlalchemy-jsonfield==1.0.2 ; python_version >= "3.9" and python_version < "3.10"
sqlalchemy-utils==0.[41](https://gitlab.[MASKED].com/cxa/cxa-ck/pex-envs/pex-airflow/-/jobs/35382959#L41).2 ; python_version >= "3.9" and python_version < "3.10"
sqlalchemy==1.4.52 ; python_version >= "3.9" and python_version < "3.10"
sqlparse==0.5.0 ; python_version >= "3.9" and python_version < "3.10"
sshtunnel==0.4.0 ; python_version >= "3.9" and python_version < "3.10"
tabulate==0.9.0 ; python_version >= "3.9" and python_version < "3.10"
tenacity==8.3.0 ; python_version >= "3.9" and python_version < "3.10"
termcolor==2.4.0 ; python_version >= "3.9" and python_version < "3.10"
text-unidecode==1.3 ; python_version >= "3.9" and python_version < "3.10"
time-machine==2.14.1 ; python_version >= "3.9" and python_version < "3.10" and implementation_name != "pypy"
typing-extensions==4.11.0 ; python_version >= "3.9" and python_version < "3.10"
tzdata==2024.1 ; python_version >= "3.9" and python_version < "3.10"
uc-micro-py==1.0.3 ; python_version >= "3.9" and python_version < "3.10"
unicodecsv==0.14.1 ; python_version >= "3.9" and python_version < "3.10"
universal-pathlib==0.2.2 ; python_version >= "3.9" and python_version < "3.10"
urllib3==2.2.1 ; python_version >= "3.9" and python_version < "3.10"
wcwidth==0.2.13 ; python_version >= "3.9" and python_version < "3.10"
werkzeug==2.3.8 ; python_version >= "3.9" and python_version < "3.10"
wirerope==0.4.7 ; python_version >= "3.9" and python_version < "3.10"
wrapt==1.16.0 ; python_version >= "3.9" and python_version < "3.10"
wtforms==3.1.2 ; python_version >= "3.9" and python_version < "3.10"
yarl==1.9.4 ; python_version >= "3.9" and python_version < "3.10"
zipp==3.18.1 ; python_version >= "3.9" and python_version < "3.10"

This is our build script:
(We are currently using pex version 2.3.1)

#!/bin/sh
set -e

rm -rf dist && mkdir dist
pip install --upgrade pip

poetry install --only build
poetry export -f requirements.txt --output dist/requirements.txt --without-hashes --with-credentials
cat dist/requirements.txt
poetry run pex -i https://rtf.[MASKED].com/artifactory/api/pypi/pypi-remote/simple \
    -r dist/requirements.txt . \
    -c airflow \
    -o dist/airflow.pex \
    --venv prepend

And this is the resulting error message:

pid 75 -> /root/.pex/venvs/7fd29932d86a744f8a544214470ad8d101a53805/108a3ddc84230ab282ea6312e06cb68f51008ce5/bin/python -sE /root/.pex/venvs/7fd29932d86a744f8a544214470ad8d101a53805/108a3ddc84230ab282ea6312e06cb68f51008ce5/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /root/.pex/pip/20.3.4-patched/pip_cache download --dest /root/.pex/downloads/resolver_download.a1d87g0r/builds.cxa.cxa-ck.pex-envs.pex-airflow.pex_build.bin.python3.9 --requirement dist/requirements.txt . --index-url https://pypi.org/simple --extra-index-url https://rtf.[MASKED].com/artifactory/api/pypi/pypi-remote/simple --retries 5 --timeout 15 exited with 1 and STDERR:
    ERROR: Command errored out with exit status 1:
     command: /root/.pex/venvs/7fd29932d86a744f8a544214470ad8d101a53805/108a3ddc84230ab282ea6312e06cb68f51008ce5/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/root/.pex/pip/20.3.4-patched/pip_cache/.tmp/pip-download-jb56u_x2/methodtools/setup.py'"'"'; __file__='"'"'/root/.pex/pip/20.3.4-patched/pip_cache/.tmp/pip-download-jb56u_x2/methodtools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base /root/.pex/pip/20.3.4-patched/pip_cache/.tmp/pip-pip-egg-info-9jym9h3c
         cwd: /root/.pex/pip/20.3.4-patched/pip_cache/.tmp/pip-download-jb56u_x2/methodtools/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/root/.pex/pip/20.3.4-patched/pip_cache/.tmp/pip-download-jb56u_x2/methodtools/setup.py", line 3, in <module>
        assert tuple(map(int, setuptools.__version__.split('.')[:3])) >= (39, 2, 0), \
    ValueError: invalid literal for int() with base 10: '0+3acb925dd708430aeaf197ea53ac8a752f7c1863'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The problem seems to result from this section of the methodtools package: https://github.com/youknowone/methodtools/blob/0.4.7/setup.py
But we cannot explain why it is called during the pex build process.

Does anyone have a clue as to why this could be or can they give us a hint?

[jsirois]

Please try adding --pip-version latest to your Pex command line. I can explain what's going on later.

[Github-dm-CDE]

Hi @jsirois, thank you for the quick feedback and the suggested solution!
In fact, adding the --pip-version latest argument fixed the problem. 😊

We were also wondering whether there is a way to use the pip version already available in the venv.

We are very much looking forward to your explanation 😉

[jsirois]

But we cannot explain why it is called during the pex build process.

It's called during any Python build or resolve process because methodtools 0.4.7 is a source distribution: https://pypi.org/project/methodtools/0.4.7/#files. In order for Pip to resolve methodtools dependencies or Pex to build a wheel from it (Pex requires wheels), the sdist must be ~built. Building the sdist requires evaluating its setup.py.

So that's the why. The reason this is a problem in Pex (with default settings) and not Pip is Pex's default settings are backwards compatible to Pex 2.0.0 which used a patched version of Pip 20.3.4, and - more to the point, a patched version of setuptools. That patched version of setuptools is here: https://github.com/pex-tool/pex/tree/main/pex/vendor/_vendored/setuptools/setuptools-44.0.0+3acb925dd708430aeaf197ea53ac8a752f7c1863.dist-info and the version is 44.0.0+3acb925dd708430aeaf197ea53ac8a752f7c1863. The methodtools setup.py code assume a version of the form X.Y.Z with X, Y and Z all being integers. Here, the Pex vendored patched version of setuptools has a +3ac... local version identifier suffix that defeats the methodtools setup.py assumption.

The fix --pip-version latest could in fact be --pip-version <any version except the default>, see pex --help for all the supported values of --pip-version. All versions of Pip except the default use a production release of setuptools with a version in the standard X.Y.Z form; thus the fix.

You probably want to read more of the Pex CLI --help since features and options are added frequently to improve performance and compatibility, but they are never defaulted to since Pex never wants to break existing users.

[jsirois]

@Github-dm-CDE I'm going to close this as an answered question, but please speak up if this is still confusing.

[Github-dm-CDE]

Hi @jsirois , thank you very much for your explanation!