Package issue: hugo

[agriyakhetarpal]

Package name

hugo

Package version

all

PyPI URL

https://pypi.org/project/hugo

piwheels URL

https://www.piwheels.org/project/hugo/

Python version

• Python 3.7
• Python 3.9
• Python 3.11

I am aware this is the issue tracker for a Python package index specifically for Raspberry Pi

• Yes

I have checked for duplicate issues

• Yes

I am the maintainer

• Yes

More information

Hi, there! I currently maintain https://github.com/agriyakhetarpal/hugo-python-distributions, which aims to provide pip-installable binaries for Python 3.8 and later (which are community-enabled and unofficial at this time) for the extended version of the Hugo static site generator for several platforms and architectures, and wanted to expand support for armv7l via piwheels (xref: agriyakhetarpal/hugo-python-distributions#91). The issue with the current broken builds from the source distribution on piwheels is that the package requires some external, non-PyPI dependencies to compile the binary:

1. Golang – most likely armv6l should work from https://go.dev/dl/, since they don't publish armv7l binaries anymore, I'm not sure.
2. Git, for Golang to be able to download and compile the dependencies for the upstream project from GitHub and other code repositories
3. A C compiler, because CGO_ENABLED is set to 1 to build the extended version of Hugo. Either Clang or a fairly recent GCC will work. Zig can be used as well, to enable cross-compilation.

Without these, the wheel builds will fail, as they have been doing so currently. Native compilation should be trivial after having access to these, but there might be a few bugs that I would be happy to fix with the packaging infrastructure that I have currently implemented. I don't have access to a CI server for building armv7l wheels, but I came here because I can receive help on getting the builds successful plus I'm willing to cooperate by providing a recipe or pre-installation scripts to deal with the above three points. I don't have a Raspberry Pi either right now, so if there is a manylinux equivalent of a Docker image available where one can build with emulation that I can be directed to, that would be great – I'm sure that the package build time won't pass the 3 hour CI time limit in any way; in general, it should not take more than an hour to build via emulation, considering that wheels for aarch64x, s390x, and ppc64le; all pass under 45 minutes. Native compilation takes 3 to 5 minutes without caches being preserved.

[bennuttall]

We do have golang and a C compiler available. I think there might be an issue with golang dependencies as we block git access to HTTP(S) for security reasons.

The latest build is failing with:

KeyError: 'armv7l'

Though I've not looked too closely to what this is.

See https://www.piwheels.org/logs/0000/1179/5137.txt

If there's a way we can build these separately, outside the automated piwheels build procedure, perhaps we can import these wheels.

[bennuttall]

piwheels@piwheels50:~$ pip3 wheel hugo==0.125.4 --no-deps --extra-index-url https://www.piwheels.org/simple
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting hugo==0.125.4
  Using cached hugo-0.125.4.tar.gz (21 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Traceback (most recent call last):
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-7450rfeb/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-7450rfeb/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-7450rfeb/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 34, in <module>
      KeyError: 'armv7l'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[agriyakhetarpal]

Thanks for pointing this out, @bennuttall – I think the error is coming from this dictionary in the code:

import platform

HUGO_ARCH = {
    "x86_64": "amd64",
    "arm64": "arm64",
    "AMD64": "amd64",
    "aarch64": "arm64",
    "x86": "386",
    "s390x": "s390x",
    "ppc64le": "ppc64le",
}[platform.machine()]

that I use, where I need to add an additional target to build against armhf first to enable building against it, corresponding to the linux/arm duo with GOARM set to 7. I can push that and other related changes, and hope for another rebuild! Edit: just did!

[bennuttall]

Raspberry Pi 1 and Zero are armv6; the others are armv7. If you're on a 32-bit OS (i.e. not aarch64, which piwheels does not support atm), it's armv6 / armhf userland. Generally speaking, our armv6 and armv7 wheels are the same and will work on all Pi models. Otherwise someone could prep an SD card on a Pi 3/4/5 and insert it into a Pi 1/Zero and it wouldn't work.

[agriyakhetarpal]

Makes sense to me. I think armv7 would be the better choice to add, based on the upstream HugoReleaser configuration. In my case, this is the extended version instead of being the regular one, so I'm not sure how the compilation will go, but it is only through the logs that we will be able to debug.

[agriyakhetarpal]

Hi again, @bennuttall! I see that with my recent release (v0.125.5) from a few hours ago, this failed with a quite strange error log: https://www.piwheels.org/logs/0000/1183/9026.txt – at the stage where it was installing the build-time dependencies with OSError: [Errno 28] No space left on device? I can't find any related issue(s) about this in this repository's tracker or that for the main one, so I guess it would be best to tag you here?

Edit: never mind, found a few issues. I guess triggering another build would be great :) If there is a Docker image or anything I can use to debug builds locally, that would be quite helpful so that I can not rely on doing releases and fix things beforehand.

[agriyakhetarpal]

I think there might be an issue with golang dependencies as we block git access to HTTP(S) for security reasons.

On another run, I think this note came up with these logs: https://www.piwheels.org/logs/0000/1183/9631.txt fatal: transport 'https' not allowed when doing the git clone. Is there a mitigation or a feature flag available to disable this behaviour, since I am not sure if there is a way to not go with the git clone route? ☹️

[MichaIng]

Still the same two errors, OSError: [Errno 28] No space left on device or failing the Git clone.

@agriyakhetarpal as Git access to HTTPS is strictly blocked, would adding https://github.com/gohugoio/hugo as submodule to https://github.com/agriyakhetarpal/hugo-python-distributions work, a way that it is part of the tarball on PyPI?

[agriyakhetarpal]

I think that could work, @MichaIng. I would have two concerns:

• the lack of a Git repository means that the commit information for what appears in hugo version cannot be inferred. Homebrew's hugo version returns hugo v0.151.0+extended+withdeploy darwin/arm64, and uvx hugo version returns hugo v0.151.0-c70ab27ceb841fc9404eab5d2c985ff7595034b7+extended+withdeploy darwin/arm64; i.e., Homebrew does not contain the commit associated with the build.
• It would bump up the size of the sdist by a bit

I could work with the second point, but will need to find a way to work with the first one.

Thanks!

[MichaIng]

@agriyakhetarpal
Sorry for the late reply. Submodules are loaded like regular Git repositories within a repository, so you can define/freeze the commit, and you can access commit information within just as if you would have cloned it separately.

For the sdist: You can define in pyproject.toml or whatever which files/dirs to include or exclude. You could also clone the subdir only after generating the sdist. But I mean the whole problem here is that cloning the repo at build-time fails (at piwheels), so adding it to the sdist would solve it, and not imply any more download bandwidth overall for users.

[agriyakhetarpal]

No worries, thanks for the response! I think I can experiment with submodules; yes. I am also okay with pre-including the submodule in the sdist. I'll get back here if I face any problems.