Installation failures within a Docker container, problem with ctypesgen?

[Ying322]

Package origin

• I confirm to be using an official package of pypdfium2 from PyPI or GitHub/pypdfium2-team.

Description

Since a few hours ago, I have been experiencing installation failures within a Docker container. The error message is as follows:

Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python /tmp/tmp432vfd7g get_requires_for_build_wheel /tmp/tmp8qd4q16r
       cwd: /tmp/pip-install-w8jdg0h2/pypdfium2
  Complete output (57 lines):
  usage: ctypesgen [-h] [--version] -i HEADERS [HEADERS ...] [-l LIBRARY] -o
                   FILE [--other-headers HEADER [HEADER ...]]
                   [-m MODULE [MODULE ...]] [-I INCLUDEDIR [INCLUDEDIR ...]]
                   [-L LIBDIR [LIBDIR ...]]
                   [--compile-libdirs LIBDIR [LIBDIR ...]]
                   [--runtime-libdirs LIBDIR [LIBDIR ...]]
                   [--no-system-libsearch] [--no-embed-preamble] [--cpp CPP]
                   [--allow-gnu-c] [-D MACRO [MACRO ...]] [-U NAME [NAME ...]]
                   [--save-preprocessed-headers FILENAME] [--optimize-lexer]
                   [-a] [--builtin-symbols] [--no-macros] [--no-undefs]
                   [--symbol-rules SYMBOL_RULES [SYMBOL_RULES ...]]
                   [--no-stddef-types] [--no-gnu-types] [--no-python-types]
                   [--no-load-library] [--no-missing-symbols]
                   [--insert-files FILENAME [FILENAME ...]]
                   [--output-language LANGUAGE]
                   [--dllclass {CDLL,WinDLL,OleDLL,PyDLL}] [--no-srcinfo]
                   [--no-symbol-guards] [--no-macro-guards] [--all-errors]
                   [--show-long-errors] [--no-macro-warnings]
                   [--debug-level DEBUG_LEVEL]
  ctypesgen: error: unrecognized arguments: --strip-build-path=/tmp/pip-install-w8jdg0h2/pypdfium2/data/bindings/headers
  ['git', 'ls-remote', 'https://github.com/bblanchon/pdfium-binaries.git'] (cwd=None)
  Downloading binary 6164 ...
  Fetching chromium refs ...
  ['git', 'ls-remote', '--sort', '-version:refname', '--tags', 'https://chromium.googlesource.com/chromium/src', '*.*.*.0'] (cwd=None)
  Resolved 6164 -> PdfiumVer(major=121, minor=0, build=6164, patch=0)
  Downloading headers...
  Building bindings ...
  ['ctypesgen', '--strip-build-path=/tmp/pip-install-w8jdg0h2/pypdfium2/data/bindings/headers', '--library', 'pdfium', '--no-srcinfo', '--no-macro-guards', '--runtime-libdirs', '.', '--no-system-libsearch', '--compile-libdirs', '/tmp/pip-install-w8jdg0h2/pypdfium2/data/linux_x64', '--no-symbol-guards', '--symbol-rules', 'if_needed=\\w+_$|\\w+_t$|_\\w+', '--headers', 'fpdf_annot.h', 'fpdf_attachment.h', 'fpdf_catalog.h', 'fpdf_dataavail.h', 'fpdf_doc.h', 'fpdf_edit.h', 'fpdf_ext.h', 'fpdf_flatten.h', 'fpdf_formfill.h', 'fpdf_fwlevent.h', 'fpdf_javascript.h', 'fpdf_ppo.h', 'fpdf_progressive.h', 'fpdf_save.h', 'fpdf_searchex.h', 'fpdf_signature.h', 'fpdf_structtree.h', 'fpdf_sysfontinfo.h', 'fpdf_text.h', 'fpdf_thumbnail.h', 'fpdf_transformpage.h', 'fpdfview.h', '-o', '/tmp/pip-install-w8jdg0h2/pypdfium2/data/bindings/bindings.py'] (cwd=PosixPath('/tmp/pip-install-w8jdg0h2/pypdfium2/data/bindings/headers'))
  'https://github.com/bblanchon/pdfium-binaries/releases/download/chromium%2F6164/pdfium-linux-x64.tgz' -> '/tmp/pip-install-w8jdg0h2/pypdfium2/data/linux_x64/pdfium-linux-x64.tgz'
  Traceback (most recent call last):
    File "/tmp/tmp432vfd7g", line 280, in <module>
      main()
    File "/tmp/tmp432vfd7g", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/tmp432vfd7g", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-_u0_w0of/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-_u0_w0of/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-_u0_w0of/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 154, in <module>
    File "<string>", line 149, in main
    File "/tmp/pip-install-w8jdg0h2/pypdfium2/setupsrc/pypdfium2_setup/emplace.py", line 70, in prepare_setup
      _get_pdfium_with_cache(pl_name, pdfium_ver, flags, use_v8)
    File "/tmp/pip-install-w8jdg0h2/pypdfium2/setupsrc/pypdfium2_setup/emplace.py", line 45, in _get_pdfium_with_cache
      build_pdfium_bindings(req_ver, flags=req_flags, compile_lds=compile_lds)
    File "/tmp/pip-install-w8jdg0h2/pypdfium2/setupsrc/pypdfium2_setup/packaging_base.py", line 517, in build_pdfium_bindings
      run_ctypesgen(DataDir_Bindings, headers_dir, **kwargs)
    File "/tmp/pip-install-w8jdg0h2/pypdfium2/setupsrc/pypdfium2_setup/packaging_base.py", line 452, in run_ctypesgen
      run_cmd(args, cwd=headers_dir)
    File "/tmp/pip-install-w8jdg0h2/pypdfium2/setupsrc/pypdfium2_setup/packaging_base.py", line 399, in run_cmd
      comp_process = subprocess.run(command, cwd=cwd, check=check, **kwargs)
    File "/usr/lib/python3.8/subprocess.py", line 516, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['ctypesgen', '--strip-build-path=/tmp/pip-install-w8jdg0h2/pypdfium2/data/bindings/headers', '--library', 'pdfium', '--no-srcinfo', '--no-macro-guards', '--runtime-libdirs', '.', '--no-system-libsearch', '--compile-libdirs', '/tmp/pip-install-w8jdg0h2/pypdfium2/data/linux_x64', '--no-symbol-guards', '--symbol-rules', 'if_needed=\\w+_$|\\w+_t$|_\\w+', '--headers', 'fpdf_annot.h', 'fpdf_attachment.h', 'fpdf_catalog.h', 'fpdf_dataavail.h', 'fpdf_doc.h', 'fpdf_edit.h', 'fpdf_ext.h', 'fpdf_flatten.h', 'fpdf_formfill.h', 'fpdf_fwlevent.h', 'fpdf_javascript.h', 'fpdf_ppo.h', 'fpdf_progressive.h', 'fpdf_save.h', 'fpdf_searchex.h', 'fpdf_signature.h', 'fpdf_structtree.h', 'fpdf_sysfontinfo.h', 'fpdf_text.h', 'fpdf_thumbnail.h', 'fpdf_transformpage.h', 'fpdfview.h', '-o', '/tmp/pip-install-w8jdg0h2/pypdfium2/data/bindings/bindings.py']' returned non-zero exit status 2.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python /tmp/tmp432vfd7g get_requires_for_build_wheel /tmp/tmp8qd4q16r Check the logs for full command output.

Install Info

My dockerfile

FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get upgrade -y && \
    apt-get install -y python3 python3-pip && \
    apt-get install -y gawk git && \
    apt-get install -y poppler-utils 
RUN ln -s /usr/bin/python3.8 /usr/bin/python
WORKDIR /app

Run the command python -m pip install -U pypdfium2 inside the container

[mara004]

AFAICS, this will be a reoccurrence of #264 (comment):

Setup requires our fork of ctypesgen, which has recently received API-breaking changes. The latest release is still authored for the old codebase, however the main branch is updated. As a workaround, I'd suggest to just install from main using pip3 install git+https://github.com/pypdfium2-team/pypdfium2.

I'm currently making efforts to overhaul the ctypesgen codebase, which inevitably involves API-breaking changes.
Unfortunately this means the current latest release is not installable from source ATM.

Again, as in the linked issue, I don't understand why the wheel wasn't used and a source install happened, though...

Update: Previously this post complained about the edited issue template and missing debug info. However, taking a step back I acknowledge it was not overly applicable here. Two of the commands would fail, and the platform we can deduce from the Dockerfile. Apologies.

[mara004]

I've implemented something to pin ctypesgen to an exact commit when building the sdist, which should prevent future occurrences of this type of issue.

pyproject.toml doesn't support env var substitution, which makes the matter slightly inconvenient.
Also we have to use the github API, because git itself to my knowledge doesn't support getting a branch head sha of a remote repo.

[Ying322]

Thank you for your reply. I used 'pip3 install git+https://github.com/pypdfium2-team/pypdfium2'.