RFE: BuildRequires generator

[ignatenkobrain]

In many of languages (rust, python, golang) BuildRequires are normal Requires + Test deps. It's always (in many of cases) possible to get names of packages in advance, before building.

Since Requires are dynamic, BuildRequires also, but nowadays you still have to specify it manually and check those after each update of package. Would be nice to implement support for generating BuildRequires.

Usually such generator will require some additional package to be installed (cargo, python), so we will need to introduce someting like BuildRequires(pre) or BuildRequires(source) which should be installed for building SRPM. Having this is also beneficial for normal process since you don't need to modify buildroot all the time in order to have some macro.

[pmatilai]

For the record, there are a number of (other) build-requires that could be relatively easily collected automatically:

• (de)compression utilities needed for processing sources + patches
• utilities needed for patching (if any), for example %autosetup -S git would optimally pull in git as a build dependency
• bash --rpm-requiresis likely to work fairly well with %prep / %build / %install / %check because unlike normal shell scripts, there are few conditionals to throw it off

[ignatenkobrain]

utilities needed for patching (if any), for example %autosetup -S git would optimally pull in git as a build dependency

I submitted bug about this some time ago and it was closed as WONTFIX ;)

[mlschroe]

Build require generators will not work in the open build service...

[ffesti]

There clearly is some chicken - egg problem here as one will likely need some tools to determine the dependencies for the build. So there needs to be a multi step process:

Getting the BuildRequires that are needed to determine the dynamic Buildrequires
Generating the SRPM with all the BuildRequires
Get all BuildRequires.
Building the Binary RPMs.

This is clearly not possible with most of the existing build systems. So changing this will require changes to basically all of them. We would probably need to add something like BuildRequires(srpm) and add a new section in the spec file that can be used to create the dynamic BuildRequires.

Another question is on what files this BuildDeps generator/section should operate. Obviously executing %prep before hand would be very helpful for many use cases.

The next question is whether the Dependencies should be generated automatically - by scripts provided by rpm and run in the background or if packagers would need to write their own section generating the dependencies - may be by executing some rpm provided scripts.

[cgwalters]

I think the solution is to not try to improve RPM here, just do what everyone in the world that hit this has done: move to generating specs via a better build system.

Examples:

• Yocto/OpenEmbeded
• Clear Linux autospec

And there's the vast existing set of dynamic language -> spec tools, like the venerable cpan2rpm
and this gofed one for golang in fedora.

There are obviously challenges to this route too, like if you support multiple generators then they need to learn how to interact for dependencies. However, I think a lot of those issues can be solved by having everything use provides rather than package names for example. (And this would require generating provides for everything)

[nim-nim]

@ffesti You're complicating things unecessary, rpm does not distinguish between manual and dynamic provides, there's no need to distinguish between manual and dynamic BuildRequires either

In a dynamic BuildRequires world, the spec still contains static BuildRequires (sufficient to pull in the build root whatever is necessary to compute the dynamic BuildRequires) and the packager executes at the end of %prep whatever command or commands are appropriate to compute those BuildRequires

That allows the packager to "fix" the project state that serves as a base to the computation, to massage the command output if needed, etc

All it needs is an rpm entry point that pipes a string or a string list into the BuildRequires list during %prep, for mock or whatever to read the final BuildRequires state at the end of %prep and complete the build root as needed

If you want maximum flexibility you can even forget about %prep or not %prep, and do it with two spec verbs:

1. one verb that accepts piping new BuildRequires (one per line, without the BuildRequires prefix) or alternatively as arguments
2. another verb that basically kicks the build system and tells it "add to the build root all the BuildRequires not already present"

and let packagers define the best dynamic BR strategy over time. Adding BRs just in time would simplify a lot of ifdefing in %build and %check for example: just request the BRs in the optional section instead of trying to sync several optional sections in different parts of the spec

[praiskup]

If we had %build_requires -f <script> section (to be handled post %prep time),
-- piping out the additional build-requires line-by-line -- I don't see any reason
why RPM should feel blocked to implement this. Likely the SRPM won't be
parseable by older RPMs, likely old build-system won't know how to handle
them, but that's non problem.

So one thing is definition of this feature (done on RPM side), and the second thing
is implementing this feature in build-systems. From the build-system POV, there are
IMO two ways how to handle this (since the new action needs to be done
right after %prep section):

• either split the build into two parts, which is not easily doable in RPM nowadays AFAIK
• execute the %prep twice, first time solely for the purpose of BuildRequiresAction...

I don't think the former is needed at least at the beginning.

To me, I think we only need a volunteer to implement this in RPM (PoC pull request).

[nim-nim]

@n3npq That's why not trying to force a specific strategy when there is so little previous art in rpm land to draw on, and providing a verb that the packager can use to tell the buildsys "I am finished computing new BRs, take those into account then continue from the next spec line" may be best

[praiskup]

On Saturday, February 17, 2018 12:08:53 AM CET Jeff Johnson wrote: @nim-nim: there are classes of BuilRequires: that are not known until after a build

This sounds interesting, don't you have specific example? It rather sounds like bootstrapping issue which the BuildRequires generator isn't supposed to solve. Maybe I finally see what you meant by chicken-egg problem, but this is not what the RFE is about..

Start designing a collection/query system that can be added to existing build systems before fiddling around with pipe/file mechanisms and spec file syntax (the answer will be obvious if you succeed in capturing/querying BuilRequires).

I think I don't really understand what what you propose, how this querying should work?

[praiskup]

On Saturday, February 17, 2018 7:18:47 AM CET Jeff Johnson wrote: Contrarian examples are trivial to devise. Consider an autoconf based generated file that builds if (and only if) certain files are detected. None of those BuildRequires can be automated and generated during a spec file parse with a pipe/file redirection.

Understood, but (at least I hope) we aren't trying to find ultimate answer for all packages/languages out there. Some languages try to solve this problem by having list of build requires set upstream, e.g. python: https://packaging.python.org/specifications/declaring-build-dependencies/ We could also think of some heuristic which goes through extracted upstream release tarball, and calculating "what is needed". Packagers would still have %prep for doing `rm` for particular parts of sources which they don't want to build (which is good safety measure anyway, see java packaging practices).

The only solution (I can see) is an attempted build on a system with "everything" installed that then repeats with the BuildRequires: detected from the "everything" build.

Theoretically yes, but I don't think we have to go that far. Btw. the set of build-requires wouldn't be "constant" for each build of particular package and that would be big -1 from me (at least if you consider bootstrapping scenario where everything isn't completed yet). Whatever the calculated build-requires set would be, I wish it was constant for each build, regardless of what's in buildroot at that time. Maybe I'm missing the point of the issue (OP to decide), but I think this has real and pretty trivial engineering solution for some languages: - distribution provides heuristic for language Foo in foo-build package - package Baz puts 'foo-build' into build requires - package Baz adds "%build_requires --script %foo_analyzer" to declare that the buildsystem should attempt to generate build requires after %prep with %foo_analyzer script - this brings new file/value in generated SRPM to let build-system know - mock finds %build_requires section in SRPM, so it - does installroot + and installs BuildRequires as usually - runs %prep - runs %foo_analyzer from %build_requires - runs the rest of the build In the stack it means: - build system needs an easy way to split build process to two parts (there's --short-circuit, but packages created with this basically aren't installable) - incompatible chnage, so build-systems need to be adjusted

[praiskup]

On Saturday, February 17, 2018 7:40:53 AM CET Jeff Johnson wrote: If -- as this RFE seems to assume -- you are going to limit the implementation to "... (Rust, Python, golang) ..." that have alternative non-specfile means to specify BuildRequires, then all known rpm build systems will require a change to augment the installed files with newly discovered "dynamic" BuildRequires.

It is up to the build system whether it is going to implement this; the worst case scenario would be to ignore the new format so people won't be allowed to use the feature "because guidelines".

Repopulating a buildroot with additional "dynamic" BuildRequires and restarting an rpm build either needs to teach rpm how to install additional packages as a side effect of parsing, or needs to be handled by depsolvers that populate the buildroot (entirely out of scope for the current rpmbuild implementation) before rpmbuild is invoked.

Agreed. IMO rpm should just speficy "script" which prints build-requires to standard output. That can/could be: %build_requires # the script content /bin/awk ... do something or %build_requires -f <script-available-in-sources> or %build_requires -s <script-available-in-buildroot> . this way you don't have to adjust depsolver, only the build-system.

[nim-nim]

Contrarian examples are trivial to devise. Consider an autoconf based generated file that builds if (and only if) certain files are detected. None of those BuildRequires can be automated and generated during a spec file parse with a pipe/file redirection.

That's more or less the Go system right now, computing deps is just a special compiler mode where it dumps what it will need instead of building the project (it is slowly evolving as a separate command, but still, it will continue to call Go compiler innards behind). So that depends what you call "build" if "building" means calling the compiler yes that's building, my definition of building is producing files.

I don't think it needs more than one phase at present though.

Repopulating a buildroot with additional "dynamic" BuildRequires and restarting an rpm build either needs to teach rpm how to install additional packages as a side effect of parsing,

It only needs to teach rpm to:

• pass dynamic BRs requests to the buildsystem,
• bomb if the BuildSystem is unable to provide them (and in case of no build system, bomb if the dynamic BRs are not already present).
• continue otherwise

There is not need to add BR provisioning logic to rpm itself, that has not been its role for quite a long time.

or needs to be handled by dep solvers that populate the build root (entirely out of scope for the current rpmbuild implementation) before rpmbuild is invoked.

I think everyone would prefer the current responsibility separation where rpm does the parsing and building and the build system does the provisioning

There is currently no known way other than "works" to verify that the BuildRequires passed to the depsolvers that populates the build system used.

You're over-engineering things. Builds can fail with static BR too, a requires system has never promised builds will succeed, only that the material they should need is present at build time.

Only looping to test that the BuildRequires are sufficient.

That's why I proposed a two-command compute-BR/populate-BR system, that makes incremental dynamic BR composition possible, and places the packager/language integrator in command of when dynamic BR population is needed, and whether it is needed once or several times

(everyone will understand that the populate-BR is expensive in build time, so the packager's interest will be to batch as much compute-BRs as possible to limit populate-BRs calls)

Of course you can make the looping implicit, force the packager to declare all the compute-BR engines
(a project can include code in several languages) before a specific step in the spec, and then at this step loop

• execution of all compute-BR
• populate-BR

till first step produces no dynamic BR not computed before.

That's a more brute-force approach. It will possibly be simpler to use by packagers, at the expense of increased build times (because the compute-BR logic will necessarily be simpler and more brutal too) and making some use-cases like declaring the BRs needed by unit tests in %check impossible.

Hence any attempt to automate BuilRequires MUST have a persistent incremental store from which the automagically generated BuildRequires can be retrieved on the final build.

"final" build is a murky concept when a project can include code in several languages, and %checks can include many tests which are all small build units. I'd rather have a system that either declares the end of %prep or %prep-br the final limit after which no BR resolving occurs, or no rpm-enforced limit with an explicit populate-BR command that can occur in all spec sections that need new BRs

Now as you astutely noted:

1. adding BRs is necessarily incremental (dynamic BR removal is too horrid to contemplate and all the systems I know are additive, not substractive, at worst the build commands know how to ignore stuff it doesn't need or want)
2. rpm needs to remember the result of the last compute-BR call to

• abort the loop when no new BRs are produced in case of implicit looping
• return a "you already asked this" error code from populate-BR if you go the explicit call route