Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.4: Themes are not installed, leading to failing tests #500

Closed
dvzrv opened this issue Sep 30, 2023 · 5 comments · Fixed by #503
Closed

2.4: Themes are not installed, leading to failing tests #500

dvzrv opened this issue Sep 30, 2023 · 5 comments · Fixed by #503

Comments

@dvzrv
Copy link
Contributor

dvzrv commented Sep 30, 2023

Hi! I'm packaging this project for Arch Linux. This is the build script we are using for 2.3 can be found here: https://gitlab.archlinux.org/archlinux/packaging/packages/sigal/-/blob/1f8a26e7b63f488b70085c6e23e73933cb9162a3/PKGBUILD
The current one does not apply the now irrelevant patch any longer and builds from a github provided source tarball.
Additionally, for the tests I am now installing sigal to a temporary location to test against (more below).

When trying to build 2.4 I noticed that themes directories are no longer installed, which has the tests fail:

E               Exception: Impossible to find the theme /build/sigal/src/sigal-2.4/test_dir/usr/lib/python3.11/site-packages/sigal/themes/colorbox

Full build and test logs:
sigal-2.4-1-x86_64-build.log
sigal-2.4-1-x86_64-check.log

The directory structure created by installing sigal is:

/var/lib/archbuild/extra-testing-x86_64/dave-5/build/sigal/src/sigal-2.4/test_dir
└── usr
    ├── bin
    │   └── sigal
    └── lib
        └── python3.11
            └── site-packages
                ├── sigal
                │   ├── __init__.py
                │   ├── __main__.py
                │   ├── __pycache__
                │   │   ├── __init__.cpython-311.opt-1.pyc
                │   │   ├── __init__.cpython-311.pyc
                │   │   ├── __main__.cpython-311.opt-1.pyc
                │   │   ├── __main__.cpython-311.pyc
                │   │   ├── gallery.cpython-311.opt-1.pyc
                │   │   ├── gallery.cpython-311.pyc
                │   │   ├── image.cpython-311.opt-1.pyc
                │   │   ├── image.cpython-311.pyc
                │   │   ├── log.cpython-311.opt-1.pyc
                │   │   ├── log.cpython-311.pyc
                │   │   ├── settings.cpython-311.opt-1.pyc
                │   │   ├── settings.cpython-311.pyc
                │   │   ├── signals.cpython-311.opt-1.pyc
                │   │   ├── signals.cpython-311.pyc
                │   │   ├── utils.cpython-311.opt-1.pyc
                │   │   ├── utils.cpython-311.pyc
                │   │   ├── version.cpython-311.opt-1.pyc
                │   │   ├── version.cpython-311.pyc
                │   │   ├── video.cpython-311.opt-1.pyc
                │   │   ├── video.cpython-311.pyc
                │   │   ├── writer.cpython-311.opt-1.pyc
                │   │   └── writer.cpython-311.pyc
                │   ├── gallery.py
                │   ├── image.py
                │   ├── log.py
                │   ├── plugins
                │   │   ├── __init__.py
                │   │   ├── __pycache__
                │   │   │   ├── __init__.cpython-311.opt-1.pyc
                │   │   │   ├── __init__.cpython-311.pyc
                │   │   │   ├── adjust.cpython-311.opt-1.pyc
                │   │   │   ├── adjust.cpython-311.pyc
                │   │   │   ├── compress_assets.cpython-311.opt-1.pyc
                │   │   │   ├── compress_assets.cpython-311.pyc
                │   │   │   ├── copyright.cpython-311.opt-1.pyc
                │   │   │   ├── copyright.cpython-311.pyc
                │   │   │   ├── extended_caching.cpython-311.opt-1.pyc
                │   │   │   ├── extended_caching.cpython-311.pyc
                │   │   │   ├── feeds.cpython-311.opt-1.pyc
                │   │   │   ├── feeds.cpython-311.pyc
                │   │   │   ├── media_page.cpython-311.opt-1.pyc
                │   │   │   ├── media_page.cpython-311.pyc
                │   │   │   ├── nomedia.cpython-311.opt-1.pyc
                │   │   │   ├── nomedia.cpython-311.pyc
                │   │   │   ├── nonmedia_files.cpython-311.opt-1.pyc
                │   │   │   ├── nonmedia_files.cpython-311.pyc
                │   │   │   ├── titleregexp.cpython-311.opt-1.pyc
                │   │   │   ├── titleregexp.cpython-311.pyc
                │   │   │   ├── upload_s3.cpython-311.opt-1.pyc
                │   │   │   ├── upload_s3.cpython-311.pyc
                │   │   │   ├── watermark.cpython-311.opt-1.pyc
                │   │   │   ├── watermark.cpython-311.pyc
                │   │   │   ├── zip_gallery.cpython-311.opt-1.pyc
                │   │   │   └── zip_gallery.cpython-311.pyc
                │   │   ├── adjust.py
                │   │   ├── compress_assets.py
                │   │   ├── copyright.py
                │   │   ├── encrypt
                │   │   │   ├── __init__.py
                │   │   │   ├── __pycache__
                │   │   │   │   ├── __init__.cpython-311.opt-1.pyc
                │   │   │   │   ├── __init__.cpython-311.pyc
                │   │   │   │   ├── encrypt.cpython-311.opt-1.pyc
                │   │   │   │   ├── encrypt.cpython-311.pyc
                │   │   │   │   ├── endec.cpython-311.opt-1.pyc
                │   │   │   │   └── endec.cpython-311.pyc
                │   │   │   ├── encrypt.py
                │   │   │   └── endec.py
                │   │   ├── extended_caching.py
                │   │   ├── feeds.py
                │   │   ├── media_page.py
                │   │   ├── nomedia.py
                │   │   ├── nonmedia_files.py
                │   │   ├── titleregexp.py
                │   │   ├── upload_s3.py
                │   │   ├── watermark.py
                │   │   └── zip_gallery.py
                │   ├── settings.py
                │   ├── signals.py
                │   ├── templates
                │   │   ├── __pycache__
                │   │   │   ├── sigal.conf.cpython-311.opt-1.pyc
                │   │   │   └── sigal.conf.cpython-311.pyc
                │   │   └── sigal.conf.py
                │   ├── utils.py
                │   ├── version.py
                │   ├── video.py
                │   └── writer.py
                └── sigal-2.4.dist-info
                    ├── LICENSE
                    ├── METADATA
                    ├── RECORD
                    ├── WHEEL
                    ├── entry_points.txt
                    └── top_level.txt

15 directories, 91 files
@saimn
Copy link
Owner

saimn commented Sep 30, 2023

Hi,
I don't know what's happening. The theme files are present in the wheel and sdist on pypi.org, and it works for me locally when building the wheel from the .tar.gz. It also works when using the PKGBUILD

==> Making package: sigal 2.4-1 (Sat 30 Sep 2023 10:42:42 PM CEST)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found sigal-2.4.tar.gz
==> Validating source files with sha512sums...
    sigal-2.4.tar.gz ... Passed
==> Extracting sources...
  -> Extracting sigal-2.4.tar.gz with bsdtar
==> Starting build()...
* Getting build dependencies for wheel...
/usr/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:72: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
validate_pyproject.api.load_builtin_plugin defines `tool.distutils` schema
validate_pyproject.api.load_builtin_plugin defines `tool.setuptools` schema
running egg_info
writing src/sigal.egg-info/PKG-INFO
writing dependency_links to src/sigal.egg-info/dependency_links.txt
writing entry points to src/sigal.egg-info/entry_points.txt
writing requirements to src/sigal.egg-info/requires.txt
writing top-level names to src/sigal.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'src/sigal.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'src/sigal.egg-info/SOURCES.txt'
...

One thing I changed since 2.3 is I removed the MANIFEST, since it's not needed when using setuptools-scm. However when building from the .tar.gz there is not git information, and I get a message in the build stage (above): listing git files failed - pretending there aren't any, message that I cannot see in your log.

@dvzrv
Copy link
Contributor Author

dvzrv commented Oct 27, 2023

I'm building with SETUPTOOLS_SCM_PRETEND_VERSION=$pkgver python -m build --wheel --no-isolation.
We are no longer relying on pypi sdist tarballs because they are not clearly defined and broken more often than not.

I think the issue is with the setuptools configuration (specifically https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-simple-packages), which only finds simple packages. All the themes are not packages or namespaced packages though (they lack __init__.py) and looking at https://setuptools.pypa.io/en/latest/userguide/datafiles.html#include-package-data I believe that the MANIFEST.in would still be required for that (otherwise the auto-generated source tarball can not be used).

It seems the more straight-forward way would be to use https://setuptools.pypa.io/en/latest/userguide/datafiles.html#package-data though (as it doesn't require MANIFEST.in).

dvzrv added a commit to dvzrv/sigal that referenced this issue Oct 27, 2023
@dvzrv
Unconditionally add all files below the themes dir to wheel
fc22803 
Use setuptools' `package-data` [1] to unconditionally add all files
below sigal's `themes` dir to a wheel.
This works without any SCM integration and without `MANIFEST.in`.

Fixes saimn#500

[1] https://setuptools.pypa.io/en/latest/userguide/datafiles.html#package-data

Signed-off-by: David Runge <dave@sleepmap.de>
@dvzrv dvzrv mentioned this issue Oct 27, 2023
Merged
@dvzrv
Copy link
Contributor Author

dvzrv commented Oct 27, 2023

I believe #503 to fix this issue

@dvzrv
Copy link
Contributor Author

dvzrv commented Oct 27, 2023

Hm, that appears to not yet fix it entirely.

There also seems to be at least one .js file with sigal/plugins/encrypt/static/decrypt.js that needs packaging as well. Will update the PR.

dvzrv added a commit to dvzrv/sigal that referenced this issue Oct 27, 2023
@dvzrv
Add data files for themes and plugins using setuptools package-data
ebedd9f 
Use setuptools' `package-data` [1] to unconditionally add all files
below sigal's `themes/` dir and all data files below the
`plugins/encrypt/static/` dir to a wheel.
This works without any SCM integration and without `MANIFEST.in`.

Fixes saimn#500

[1] https://setuptools.pypa.io/en/latest/userguide/datafiles.html#package-data

Signed-off-by: David Runge <dave@sleepmap.de>
@dvzrv
Copy link
Contributor Author

dvzrv commented Oct 27, 2023

Okay, with #503 applied I can now build from the auto-generated tarball and all tests pass!

@saimn saimn closed this as completed in #503 Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add data files for themes and plugins using setuptools package-data dvzrv/sigal
2 participants
@saimn @dvzrv