srpm fails when spec file is not in the project root dir and there is a custom `create-archive:` action #1621

martinpitt · 2022-06-14T12:00:04Z

I've been scratching my head in cockpit-project/cockpit-podman#1016 and cockpit-project/cockpit#17450 about some packit config cleanups which we want to do. Part of it is that they stop dynamically creating a spec file in the project root directory, and move to a valid spec file in packaging/projectname.spec. We really don't want to keep the .spec in the root dir, that's just clutter -- it should go together with debian, arch, etc. packaging.

What I am seeing can be reproduced with https://github.com/martinpitt/python-dbusmock/ , which is a fairly simple and standard (wrt. packit config). With a pristine checkout, packit srpm works fine. But when I add a custom create-archive:

--- packit.yaml
+++ packit.yaml
@@ -12,6 +12,10 @@ files_to_sync:
   - src: packaging/python-dbusmock.spec
     dest: python-dbusmock.spec
 
+actions:
+  create-archive:
+    - sh -exc 'git archive HEAD | xz > python-dbusmock-$(git describe).tar.xz; ls python-dbusmock-*.tar.xz'
+
 jobs:
   - job: copr_build
     trigger: pull_request

Then packit srpm fails the same way as I saw in these (much more complex) PRs:

2022-06-14 13:53:13.818 schema.py         WARNING The 'metadata' key in jobs is deprecated and can be removed.Nest config options from 'metadata' directly under the job object.
2022-06-14 13:53:13.818 schema.py         WARNING The 'metadata' key in jobs is deprecated and can be removed.Nest config options from 'metadata' directly under the job object.
2022-06-14 13:53:13.818 schema.py         WARNING The 'metadata' key in jobs is deprecated and can be removed.Nest config options from 'metadata' directly under the job object.
2022-06-14 13:53:13.818 schema.py         WARNING The 'metadata' key in jobs is deprecated and can be removed.Nest config options from 'metadata' directly under the job object.
2022-06-14 13:53:13.827 base_git.py       INFO   Using user-defined script for ActionName.create_archive: [['sh', '-exc', 'git archive HEAD > python-dbusmock-$(git describe).tar.xz; ls python-dbusmock-*.tar.xz']]
2022-06-14 13:53:13.829 logging.py        INFO   + git archive HEAD
2022-06-14 13:53:13.829 logging.py        INFO   ++ git describe
2022-06-14 13:53:13.836 logging.py        INFO   + ls python-dbusmock-0.27.5-8-g203cfe6.tar.xz
2022-06-14 13:53:13.836 logging.py        INFO   python-dbusmock-0.27.5-8-g203cfe6.tar.xz
2022-06-14 13:53:13.837 upstream.py       INFO   Created archive:
2022-06-14 13:53:13.837 upstream.py       INFO   	parsed   path: /var/home/martin/upstream/python-dbusmock/python-dbusmock-0.27.5-8-g203cfe6.tar.xz
2022-06-14 13:53:13.837 upstream.py       INFO   	absolute path: /var/home/martin/upstream/python-dbusmock/python-dbusmock-0.27.5-8-g203cfe6.tar.xz
2022-06-14 13:53:13.837 command_handler.py INFO   Nothing to clean.
2022-06-14 13:53:13.837 utils.py          ERROR  Preparation of the repository for creation of an SRPM failed: '/var/home/martin/upstream/python-dbusmock/python-dbusmock-0.27.5-8-g203cfe6.tar.xz' is not in the subpath of '/var/home/martin/upstream/python-dbusmock/packaging' OR one path is relative and the other is absolute.

This doesn't make sense to me -- why would it require the spec file to be in the root dir? The docs explicitly say it can be somewhere else, and without create-archive that works fine.

The text was updated successfully, but these errors were encountered:

martinpitt · 2022-06-14T12:06:11Z

One workaround which is arguably a bit weird is to build the tarball in packaging/ subdirectory:

--- packit.yaml
+++ packit.yaml
@@ -12,6 +12,10 @@ files_to_sync:
   - src: packaging/python-dbusmock.spec
     dest: python-dbusmock.spec
 
+actions:
+  create-archive:
+    - sh -exc 'git archive HEAD | xz > packaging/python-dbusmock-$(git describe).tar.xz; ls packaging/python-dbusmock-*.tar.xz'
+
 jobs:
   - job: copr_build
     trigger: pull_request

However, while that is easy in this reproducer, it's much harder in real projects where you call make dist -- one would have to explicitly mv it.

Keep `Version: 0` in the spec file. This works fine with `autogen rpm`, `dnf builddep, and Packit. It was the last macro, so now it's a real working .spec. This avoids having to invoke this script all the time. This simplifies our packit config a bit and stops swimming against the stream. The only place where we want a real version is `make dist`, so that built RPMs have a correct version. Rework the rule to modify the file in-place in distdir. Unfortunately this uncovers another packit bug: If the .spec file is not in the same directory as the tarball, `packit srpm` gets upset: <packit/packit#1621>. Work around that by copying the spec to the root directory, and pointing to it.

TomasTomecek · 2022-06-15T13:40:58Z

Found the problem @_@

it's one-liner

the function we are using for computing relative paths cannot work with subdirs; python has another that can - seriously...

`Path.relative_to` can raise `ValueError` when `specfile_dir` is in a subdir and archive path is in the project's root. This commit uses `os.path.relpath` instead that supports having the source path in a subdir. It's a different implementation than `.relative_to`. Fixes: packit#1621 Signed-off-by: Tomas Tomecek <ttomecek@redhat.com>

Keep `Version: 0` in the spec file. This works fine with `autogen rpm`, `dnf builddep, and Packit. It was the last macro, so now it's a real working .spec. This avoids having to invoke this script all the time. This simplifies our packit config a bit and stops swimming against the stream. The only place where we want a real version is `make dist`, so that built RPMs have a correct version. Rework the rule to modify the file in-place in distdir. Unfortunately this uncovers another packit bug: If the .spec file is not in the same directory as the tarball, `packit srpm` gets upset: <packit/packit#1621>. Work around that by copying the spec to the root directory, and pointing to it.

[PurePath.relative_to()](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.relative_to) requires self to be the subpath of the argument, but [os.path.relpath()](https://docs.python.org/3/library/os.path.html#os.path.relpath) does not. Fixes: packit#1621 Signed-off-by: Tomas Tomecek <ttomecek@redhat.com> Co-Authored-by: Hunor Csomortáni <csomh@redhat.com>

allow having custom archives in root while specfile is in a subdirectory TODO: Write new tests or update the old ones to cover new functionality. Path.relative_to can raise ValueError when specfile_dir is in a subdir and archive path is in the project's root. This commit uses os.path.relpath instead that supports having the source path in a subdir. It's a different implementation than .relative_to. Fixes: #1621 RELEASE NOTES BEGIN Packit now correctly handles creation of custom archives in root while a specfile is in a subdirectory. RELEASE NOTES END Reviewed-by: Jiri Popelka <None> Reviewed-by: Hunor Csomortáni <csomh@redhat.com> Reviewed-by: Tomas Tomecek <tomas@tomecek.net>

martinpitt · 2022-06-21T10:31:00Z

Thanks! I retried it in cockpit-project/cockpit-podman#1016 and it fails differently now:

2022-06-21 10:28:25.695 upstream.py       INFO   Created archive:
2022-06-21 10:28:25.695 upstream.py       INFO   	parsed   path: /tmp/tmpueaiwen2/cockpit-podman-49.1.10.gd95b5ed.tar.xz
2022-06-21 10:28:25.696 upstream.py       INFO   	absolute path: /tmp/tmpueaiwen2/cockpit-podman-49.1.10.gd95b5ed.tar.xz
2022-06-21 10:28:25.696 upstream.py       INFO   Linking to the specfile directory: /tmp/tmpueaiwen2/packaging/cockpit-podman-49.1.10.gd95b5ed.tar.xz -> ../cockpit-podman-49.1.10.gd95b5ed.tar.xz (given path to archive: /tmp/tmpueaiwen2/cockpit-podman-49.1.10.gd95b5ed.tar.xz)
[...]

cmd: ['rpmbuild', '-bs', '--define', '_sourcedir /var/lib/copr-rpmbuild/workspace/workdir-nfw3ragk/srcdir', '--define', '_rpmdir /var/lib/copr-rpmbuild/workspace/workdir-nfw3ragk/srcdir', '--define', '_builddir /var/lib/copr-rpmbuild/workspace/workdir-nfw3ragk/srcdir', '--define', '_specdir /var/lib/copr-rpmbuild/workspace/workdir-nfw3ragk/srcdir', '--define', '_srcrpmdir /var/lib/copr-rpmbuild/results', '/var/lib/copr-rpmbuild/workspace/workdir-nfw3ragk/srcdir/cockpit-podman.spec']
cwd: .
rc: 1
stdout: 
stderr: error: File /var/lib/copr-rpmbuild/workspace/workdir-nfw3ragk/srcdir/cockpit-podman-49.1.10.gd95b5ed.tar.xz: No such file or directory

New issue?

TomasTomecek · 2022-06-21T15:49:10Z

I'll have a look tomorrow.

TomasTomecek · 2022-06-22T12:23:19Z

I'm unable to reproduce locally, so let's track it as a new issue.

martinpitt mentioned this issue Jun 14, 2022

Keep .spec in git cockpit-project/cockpit-podman#1016

Closed

3 tasks

martinpitt mentioned this issue Jun 14, 2022

Eliminate tools/create-spec, keep real .spec file in git, simplify packit cockpit-project/cockpit#17450

Merged

TomasTomecek added the kind/bug Something isn't working. label Jun 15, 2022

TomasTomecek self-assigned this Jun 15, 2022

TomasTomecek mentioned this issue Jun 15, 2022

allow having custom archives in root while specfile is in a subdirectory #1622

Merged

1 task

softwarefactory-project-zuul bot closed this as completed in #1622 Jun 17, 2022

TomasTomecek mentioned this issue Jun 22, 2022

SRPM cannot be created in Copr when packaging files are in a subdir #1637

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

srpm fails when spec file is not in the project root dir and there is a custom `create-archive:` action #1621

srpm fails when spec file is not in the project root dir and there is a custom `create-archive:` action #1621

martinpitt commented Jun 14, 2022 •

edited

martinpitt commented Jun 14, 2022 •

edited

TomasTomecek commented Jun 15, 2022

martinpitt commented Jun 21, 2022

TomasTomecek commented Jun 21, 2022

TomasTomecek commented Jun 22, 2022

srpm fails when spec file is not in the project root dir and there is a custom create-archive: action #1621

srpm fails when spec file is not in the project root dir and there is a custom create-archive: action #1621

Comments

martinpitt commented Jun 14, 2022 • edited

martinpitt commented Jun 14, 2022 • edited

TomasTomecek commented Jun 15, 2022

martinpitt commented Jun 21, 2022

TomasTomecek commented Jun 21, 2022

TomasTomecek commented Jun 22, 2022

srpm fails when spec file is not in the project root dir and there is a custom `create-archive:` action #1621

srpm fails when spec file is not in the project root dir and there is a custom `create-archive:` action #1621

martinpitt commented Jun 14, 2022 •

edited

martinpitt commented Jun 14, 2022 •

edited