RFE: set builsubdir to the *first* extracted archive not the last one

[nim-nim]

rpm convention is to put the main archive in SOURCE0 and ancilliary sources in SOURCEX
(this way SOURCE0 is sure to exist and be stable, and can be aliased to SOURCE, while the number of addons and their index can vary over a package time)

However rpm does not respect its own convention, sets buildsubdir to the last processed archive, and cds to this last archive at the start of build

This has the following deleterious side effects:

1. the packager ends up working in an addon subdirectory, which is utterly confusing, because this is not the subdirectory he cares about most, and will change over time depending on the number of addon archives he adds to his spec. So his first addition to the spec will be to cd back to the correct directory, or change %setup invocation order to process SOURCE0 last (let's all count up towards zero, yay, hard to be less natural)

2. you can not do any cleanup of the extracted archives in %prep, that involves things that should end up in the packager main work directory, because this directory does not exist before all archives have been extracted. Instead of the natural:

%prep
for each SOURCEX
  %setup
  cd to extracted archive
  archive prep work
done

you need to code two loops

%prep
for each SOURCEX
  %setup
  remember the extracted dir
done ← at this point only builsubdir exists but the variable is not set !
for each extracted dir
  cd extractdir
  do prep work that involves the final workdir
done

This is needlessly byzantine, just set builsubdir to the first archive subdirectory, so it is garanteed to exist after the first %setup invocation, and packagers get warped to their main project files at the start of %build (or better, to the subdir that matches SOURCE0, so %setup order does not matter past %prep)

[ascherer]

I believe that's what the -n option of %setup is for. You can (or have to) set/repeat the desired %buildsubdir on every invocation of %setup. (IIRC, multiple %setups are deemed esoteric/byzantine anyway.)

[nim-nim]

The problem is not setting it. The problem is that rpm uses the last extracted directory as default workdir, preventing any preparation that involves this directory before every archive has been extracted

[Conan-Kudo]

You're supposed to use %setup once, and tell it how many archives it needs to extract before it it should change into the workdir.

For example: %setup -b 3 tells it to unpack Source0, Source1, Source2, and Source3 before changing into the %name-%version workdir. If you want another directory, you need to override with -n.

[nim-nim]

I don't doubt that's what the setup authors intended, but really, it's overly clever for human beings, that like their commands simple and easy to remember, it's overly clever for scripts, that like to loop over the same command not compute complex lists of arguments, it's overly clever for whoever is tasked to explain it so anyone else, and besides, it does not handle patching of intermediate sources well, packagers want to operate disabling or removal of aditional Sources by commenting out or removing related spec lines, not by argument surgery, and so on.

Which is why from time immemorial, starting with the heroic period when maximum rpm was written¹, the documented and actual %setup use was not what the setup authors intended, but one setup call per archive. And it's getting a tad late to change it.

So could we make the way people actually use %setup work better please?

¹ http://ftp.rpm.org/max-rpm/s1-rpm-inside-macros.html#S3-RPM-INSIDE-SETUP-MULTI-SOURCE

[pmatilai]

Part of the problem is the lengths people go to in order to use %setup for unpacking sources even when it ends up being counter-beneficial. Just call it once to setup the main directory and do the rest with tar/unzip/whatever. It's not like the magic %setup incantation required to unpack dozen sources in just the right combo is going to be somehow easier to understand than doing the same manually as a script.

Another point is that indeed, if it's not supposed to be used more than once then rpm should prevent that, or at least warn about it. Telling people "it's not supposed to be used that way" doesn't scale.

[nim-nim]

Panu, one huge reason %setup is still a mess after all this time it that it is linked to the SourceX black magic. And yes rpmbuild will complain loudly if you deviate from the way it thinks things should be done. And yes it will refuse to process the resulting spec. And the documentation is piss-poor.

There are layers over layers of workarounds in SourceX and %setup just to avoid admitting parts of the original rpm design didn't work out in practice (lots of things did work out, it's natural to make some mistakes). This design got tweaked just enough to avoid a rewrite, without a lot of effort to hide that the way it is used today, is not what the author envisioned.

So you have to shoehorn archive names behind an artificial # in SourceX (because admitting the file name is not always part of the URL and a separate tag is needed is out of the question), you have the way distros have been forced to invent %dist (making Release, in effect, two-tags-in-one), you have the way rpm makes no provision for archive or topdir being different from %{name}-%{version} (forcing pretty much every packager to invent his own metadata convention for the same info), you have the way %setup still tries to chainbuild, when maximum rpm, written in 2000, already documents the way people actually use it and expect it to work is multiple %setup calls and so on I probably forget half of it.

So it's not surprising people just use %setup by rote, just to get past this ordeal as fast as possible, and get to the parts of the spec where they can do things without rpm complaining and trying to impose unintuitive overcomplicated misdesigns on them.

Just provide in rpm a %setup alternative that just does what you propose (unpack one archive, without esotheric chaining rules to avoid writing one more extraction command, and with simple ways to query or set buildsubdir that rpm won't stomp on later), and you'll see %setup use and misuse drop quickly.

Or, alternatively, fix %setup to work how people expect it to work, without the magic they don't expect and don't need.

But having sour grapes about people using %setup the way maximum rpm documented 18 years ago? And trying to teach them a lesson in the correct way to do things via warnings? That will end badly for everyone involved.

[ffesti]

@n3npq It looks like you are deleting most of your comments after posting them here. This disrupts the discussion and renders the tickets unusable. Also note that they are still available in the mailing-list archive and everyone's mail folder - if you don't want your words to be archived and available for future review please post them else where.

[pmatilai]

Just provide in rpm a %setup alternative that just does what you propose (unpack one archive, without esotheric chaining rules to avoid writing one more extraction command, and with simple ways to query or set buildsubdir that rpm won't stomp on later), and you'll see %setup use and misuse drop quickly.

@nim-nim, but isn't that just about exactly what %setup by default does? Calling it once with no fancy arguments will unpack the first source and define %_buildsubdir, and from there on you can do whatever you want with the rest. Or call it just to create a directory of your choosing and do the rest manually.

Like said rpm could do better there, and having different semantics for %_buildsubdir is trivial, but if calling %setup multiple times hurts and makes things unnecessarily complicated, then don't!

[nim-nim]

@nim-nim, but isn't that just about exactly what %setup by default does?

Not really, since the chaining interferes %setup reuse with the next archive

People just want to align

%somesetup -z 1
%somesetup -z 2
%somesetup -z 3

Like they align path or source declarations

And they want to have some simple way to set the primary root where %build will chdir and various other macros like autodeps will look for generated files.

They do not want to use one macro for Source0 and switch to something else for other sources.
They do not want this simple stupid natural pattern to have weird side effects because the expansion macro was really designed for a chaining that almost not one actually uses or need.

Calling it once with no fancy arguments will unpack the first source and define %_buildsubdir,

No it won't, the variable is not really set before %build and rpm will overwrite any attempt to set it manually. This variable, which is key to many other rpm macros, is set as a %setup side-effect that depends on the chaining design.

[pmatilai]

No it won't, the variable is not really set before %build

NOW we're getting somewhere. %setup parsing certainly defines it, and this happens before any shell script is executed, but I see it gets lost somewhere in the cracks of %prep parsing. I can take a look at it, all I remember off-hand is that %prep parsing is entirely different from the rest of the build-sections.

However I still don't see the big problem here. %buildsubdir is not some magic random value calculated by rpm, it's always %{name}-%{version}, or literally whatever you pass to %setup with -n argument. It's also the directory you'll find yourself in the shell, so derivable from "basename ${PWD}" inside %prep.

I have no idea what you mean by chainbuild in this context, nor do I know what %somesetup -zN is supposed to do. Please give us concrete examples and issues so we don't need to guess and assume because I'm sure we all know how well that works. What I can see here is that you be in charge of %buildsubdir. Well, here's an artificial example that will unpack all your sources in a pre-determined randomly named directory and prove that it's what ends up as %buildsubdir.

%global mysub %{basename:%(mktemp -d)}

%prep
%setup -c -T -n %{mysub}
expr %{mysub} = $(basename $(pwd)) || exit
for s in %{sources}; do
    tar xf ${s};
done

%build
expr %{mysub} = %{buildsubdir} || exit

You could hide the detail of calling %setup exactly once in the %somesetup macro I'm sure.
And if all that has nothing to do with what you want to do, then that just goes to show the need for concrete issues and examples.

Note that none of this is average "something people want to do", I can only assume this is you wanting to do something with the %forgesetup stuff. Which is all fine and cool, but please speak about the needs of your macros instead of some vague "they" that doesn't exist.

[pmatilai]

Oh and BTW, if the reason for wanting to use %setup "at any cost" is that it knows how to extract tar, zip, ruby gems and whatnot automatically, then that might be another concrete something we could look into, ie add a simpler lower level %extract or such helper macro/tool that can do what %setup can, but only for one source, and without the side-effects.

[nim-nim]

No it won't, the variable is not really set before %build

NOW we're getting somewhere. %setup parsing certainly defines it, and this happens before any shell script is executed, but I see it gets lost somewhere in the cracks of %prep parsing. I can take a look at it, all I remember off-hand is that %prep parsing is entirely different from the rest of the build-sections.

However I still don't see the big problem here. %buildsubdir is not some magic random value calculated by rpm, it's always %{name}-%{version}, or literally whatever you pass to %setup with -n argument. It's also the directory you'll find yourself in the shell, so derivable from "basename ${PWD}" inside %prep.

The packaging pattern is

Source0: main-archive
Source1: addon1
Source2: addon2
Source3: addon3

[...]
%prep
%setupsomething -z 0 (main archive)
%setupsomething -z 1 (addon 1)
%setupsomething -z 2 (addon 2)
%setupsomething -z 3 (addon 3

It has the nice properties that:

1. it is simple and easy to understand
2. it is simple and easy to explain
3. it is patch and preparation friendly (just put the %patch or any other archive fixup directive after the corresponding %setupsomething line)
4. it is spec-maintenance and branching friendly (want to remove a particular addon ? just remove the corresponding lines, you end with a simple diff with added and removed lines, without side-effects on other lines – or changed lines if you prefer to call it that way)

ASIDE
I used -z instead of the horrible -a -b -c -T -n mix that no one understands at first and is only there to permit chaining and to avoid standardizing:

1. a variable/tag to specify the upstream archive name (without tar.foo extension)

• let's call it %{archivenameX} with X matching the corresponding %{sourceX})
• that's workarounded by the # hack,
• the drawback is that you can not rely on %{archivenameX} existing and being called this way in other macros

2. a variable/tag to specify the topdir upstream uses with its archives

• let's call it %{topdirX} with X matching the corresponding %{sourceX}
• can trivially use %{archivenameX} as default value

3. the extraction directory the packager wishes to generate

• can trivially be set to %{topdirX}, except when %{topdirX} is nil or empty, in which case %{archivenameX} is the default anyone sensible expects

ASIDE END

That's the packaging pattern Maximum rpm already documented in the last century, and having spent some months writing macros which are intended to be used in %prep, and tried to document them to others, I understand 100% why the Maximum rpm authors chose to document it that way (and I agree with them).

Maximum rpm is the main vulgarisation document that attemps to explain how to use %setup with multiple archives. I'd say it succeeds poorly, in large part due to %setup quirks, but at least it tries.

Now, as Neal explains, those quirks are largely due to the historical %setup chaining design. But almost nobody uses %setup that way. First, it is not documented in any specfile writing vulgarization document. And second, even it it was documented, it does not have the nice packager-friendly properties of the no-chaining approach which is documented in Maximum rpm.

However, even even if you use %setup without chaining, you still get bitten by the chaining design.

Because %setup assumes you will use it in one call, it makes no provision to set %{buildsubdir} cleanly. %{buildsubdir} is whatever you passed %setup with -n, why you use several %setup calls, how weird, why are you doing that? (I've already explained why).

So packagers mostly accept this is dark rpm vodoo and that rpm will chdir to the last addon directory in %build (which is definitively not their main work directory when processing multiple archives).

This RFE is about setting %{buildsubdir} by default to something sensible: the extraction directory of the %{source0} archive, which:

• will almost always be the main archive processed by the spec, due to historical conventions
• will almost always be the first extraction directory created, because people count from zero.
• will almost always exist in all the variations of the specfile, regardless of the number or archive addons a particular branch uses, because people add addons to an initial %{source}, and not this initial %{source} to a (variable) pile of addons

Therefore this is tthe directory the packager can use to centralize files that need centralizing in %prep, without worrying it may have not been created yet, or does not exist in one of the spec declinations.

Alternatively, just making it possible to override in %prep the mess %setup makes of this all, works for me too.

I'm definitely not interested in anything that would be equivalent to

%prep
%myprepmacro

%build
%fixprep ← workaround %{buildsubdir} selection in %prep

[nim-nim]

For example, the common case of a single well formed compresset tarball with a patch in a spec file is even simpler than your design:

%prep
%setup
%patch

And that can be trivialy declined in

%prep
%setup -z 0
%patch0
%setup -z 1
%patch1
%setup -z 2
%patch2

(of course one could genereate %setupX names too, that's another can of worms created by some of historical rpm choices).

But anyway I'm not here to discuss why single archive is simpler than multiple archives. That's self-evident. However some people do want to process multiple archives. Being able to handle multiple archive was the first request of Fedora's rpm maintainers when I posted my first macro iterations months ago. So bzzzt people wanted me to look at it, so I looked at it. And it was a major PITA due to many rpm quirks that ended up as RFEs here.

My personal use cases do not necessarily match the use cases of everyone in the distribution or its derivatives.

of spec files, not how easy it is to "maintain" spec files. While it is certainly easy to delete lines Han it is to, say, change a macro value by editing, the drudgery of changing hundreds or thousands of files in order to conform to some Newer! Better! Bestest! Syntax or policy.

I'm not in the business of setting policies, this is free software, people will use the result if they find it useful and convenient, which is why I make damn sure it is useful and convenient, because I have no interest in wasting my efforts on things almost no one uses in reality, or in explaining to the users of my macros they are idiots that should just do unnatural and unintuitive xxx due to some rpm historical implementation quirk they have no interest in (and just want to be fixed).

Also, I do not rely on others to vulgarize and document the usage of my macros, which is a wonderful way to detect an implementation detail is stupid from the user point of view, because documenting it suddenly takes pages.

[pmatilai]

This could potentially be solved by #860 (ie just allow macros to do what they want with it)

[nim-nim]

Thanks, I need to investigate if #860 solves the problem for me

[pmatilai]

FWIW, #1976 moves all the %setup extraction logic into an external rpmuncompress command, so you can just call that on any archive format rpm knows about and never mind the bollocks with %setup. And since %buildsubdir is now (since 6caca84) a regular macro, you really don't need to muck with %setup at all if it only gets in the way.

Changing it as per requested here is still being considered, but that has the chance to break stuff that has been carefully arranged for the current, strange scheme.

[ffesti]

With the latest changes to the %setup macro this may now be easier and should be looked at again.