RFE: BuildRequires generator #104
Comments
|
For the record, there are a number of (other) build-requires that could be relatively easily collected automatically:
|
I submitted bug about this some time ago and it was closed as WONTFIX ;) |
|
Build require generators will not work in the open build service... |
|
There clearly is some chicken - egg problem here as one will likely need some tools to determine the dependencies for the build. So there needs to be a multi step process: Getting the BuildRequires that are needed to determine the dynamic Buildrequires This is clearly not possible with most of the existing build systems. So changing this will require changes to basically all of them. We would probably need to add something like BuildRequires(srpm) and add a new section in the spec file that can be used to create the dynamic BuildRequires. Another question is on what files this BuildDeps generator/section should operate. Obviously executing %prep before hand would be very helpful for many use cases. The next question is whether the Dependencies should be generated automatically - by scripts provided by rpm and run in the background or if packagers would need to write their own section generating the dependencies - may be by executing some rpm provided scripts. |
|
I think the solution is to not try to improve RPM here, just do what everyone in the world that hit this has done: move to generating specs via a better build system. Examples: And there's the vast existing set of dynamic language -> spec tools, like the venerable cpan2rpm There are obviously challenges to this route too, like if you support multiple generators then they need to learn how to interact for dependencies. However, I think a lot of those issues can be solved by having everything use provides rather than package names for example. (And this would require generating provides for everything) |
|
@ffesti You're complicating things unecessary, rpm does not distinguish between manual and dynamic provides, there's no need to distinguish between manual and dynamic BuildRequires either In a dynamic BuildRequires world, the spec still contains static BuildRequires (sufficient to pull in the build root whatever is necessary to compute the dynamic BuildRequires) and the packager executes at the end of %prep whatever command or commands are appropriate to compute those BuildRequires That allows the packager to "fix" the project state that serves as a base to the computation, to massage the command output if needed, etc All it needs is an rpm entry point that pipes a string or a string list into the BuildRequires list during %prep, for mock or whatever to read the final BuildRequires state at the end of %prep and complete the build root as needed If you want maximum flexibility you can even forget about %prep or not %prep, and do it with two spec verbs:
and let packagers define the best dynamic BR strategy over time. Adding BRs just in time would simplify a lot of ifdefing in %build and %check for example: just request the BRs in the optional section instead of trying to sync several optional sections in different parts of the spec |
|
If we had So one thing is definition of this feature (done on RPM side), and the second thing
I don't think the former is needed at least at the beginning. To me, I think we only need a volunteer to implement this in RPM (PoC pull request). |
|
@n3npq That's why not trying to force a specific strategy when there is so little previous art in rpm land to draw on, and providing a verb that the packager can use to tell the buildsys "I am finished computing new BRs, take those into account then continue from the next spec line" may be best |
|
On Saturday, February 17, 2018 12:08:53 AM CET Jeff Johnson wrote:
@nim-nim: there are classes of BuilRequires: that are not known until after a
build
This sounds interesting, don't you have specific example? It rather
sounds like bootstrapping issue which the BuildRequires generator isn't
supposed to solve. Maybe I finally see what you meant by chicken-egg
problem, but this is not what the RFE is about..
Start designing a collection/query system that can be added to existing
build systems before fiddling around with pipe/file mechanisms and spec
file syntax (the answer will be obvious if you succeed in
capturing/querying BuilRequires).
I think I don't really understand what what you propose, how this querying
should work?
|
|
On Saturday, February 17, 2018 7:18:47 AM CET Jeff Johnson wrote:
Contrarian examples are trivial to devise. Consider an autoconf based
generated file that builds if (and only if) certain files are detected.
None of those BuildRequires can be automated and generated during a spec
file parse with a pipe/file redirection.
Understood, but (at least I hope) we aren't trying to find ultimate answer
for all packages/languages out there. Some languages try to solve this
problem by having list of build requires set upstream, e.g. python:
https://packaging.python.org/specifications/declaring-build-dependencies/
We could also think of some heuristic which goes through extracted
upstream release tarball, and calculating "what is needed". Packagers
would still have %prep for doing `rm` for particular parts of sources
which they don't want to build (which is good safety measure anyway, see
java packaging practices).
The only solution (I can see) is an attempted build on a system with
"everything" installed that then repeats with the BuildRequires:
detected from the "everything" build.
Theoretically yes, but I don't think we have to go that far. Btw. the set
of build-requires wouldn't be "constant" for each build of particular
package and that would be big -1 from me (at least if you consider
bootstrapping scenario where everything isn't completed yet). Whatever
the calculated build-requires set would be, I wish it was constant for
each build, regardless of what's in buildroot at that time.
Maybe I'm missing the point of the issue (OP to decide), but I think this
has real and pretty trivial engineering solution for some languages:
- distribution provides heuristic for language Foo in foo-build package
- package Baz puts 'foo-build' into build requires
- package Baz adds "%build_requires --script %foo_analyzer" to declare
that the buildsystem should attempt to generate build requires after
%prep with %foo_analyzer script
- this brings new file/value in generated SRPM to let build-system know
- mock finds %build_requires section in SRPM, so it
- does installroot + and installs BuildRequires as usually
- runs %prep
- runs %foo_analyzer from %build_requires
- runs the rest of the build
In the stack it means:
- build system needs an easy way to split build process to two parts
(there's --short-circuit, but packages created with this basically
aren't installable)
- incompatible chnage, so build-systems need to be adjusted
|
|
On Saturday, February 17, 2018 7:40:53 AM CET Jeff Johnson wrote:
If -- as this RFE seems to assume -- you are going to limit the implementation
to "... (Rust, Python, golang) ..." that have alternative non-specfile means
to specify BuildRequires, then all known rpm build systems will require a
change to augment the installed files with newly discovered "dynamic"
BuildRequires.
It is up to the build system whether it is going to implement this; the worst
case scenario would be to ignore the new format so people won't be allowed to
use the feature "because guidelines".
Repopulating a buildroot with additional "dynamic" BuildRequires and
restarting an rpm build either needs to teach rpm how to install additional
packages as a side effect of parsing, or needs to be handled by depsolvers
that populate the buildroot (entirely out of scope for the current rpmbuild
implementation) before rpmbuild is invoked.
Agreed. IMO rpm should just speficy "script" which prints build-requires to
standard output. That can/could be:
%build_requires
# the script content
/bin/awk ... do something
or
%build_requires -f <script-available-in-sources>
or
%build_requires -s <script-available-in-buildroot>
. this way you don't have to adjust depsolver, only the build-system.
|
That's more or less the Go system right now, computing deps is just a special compiler mode where it dumps what it will need instead of building the project (it is slowly evolving as a separate command, but still, it will continue to call Go compiler innards behind). So that depends what you call "build" if "building" means calling the compiler yes that's building, my definition of building is producing files. I don't think it needs more than one phase at present though.
It only needs to teach rpm to:
There is not need to add BR provisioning logic to rpm itself, that has not been its role for quite a long time.
I think everyone would prefer the current responsibility separation where rpm does the parsing and building and the build system does the provisioning
You're over-engineering things. Builds can fail with static BR too, a requires system has never promised builds will succeed, only that the material they should need is present at build time.
That's why I proposed a two-command compute-BR/populate-BR system, that makes incremental dynamic BR composition possible, and places the packager/language integrator in command of when dynamic BR population is needed, and whether it is needed once or several times (everyone will understand that the populate-BR is expensive in build time, so the packager's interest will be to batch as much compute-BRs as possible to limit populate-BRs calls) Of course you can make the looping implicit, force the packager to declare all the compute-BR engines
till first step produces no dynamic BR not computed before. That's a more brute-force approach. It will possibly be simpler to use by packagers, at the expense of increased build times (because the compute-BR logic will necessarily be simpler and more brutal too) and making some use-cases like declaring the BRs needed by unit tests in %check impossible.
"final" build is a murky concept when a project can include code in several languages, and %checks can include many tests which are all small build units. I'd rather have a system that either declares the end of %prep or %prep-br the final limit after which no BR resolving occurs, or no rpm-enforced limit with an explicit populate-BR command that can occur in all spec sections that need new BRs Now as you astutely noted:
|
I'm pretty sure it will be constant, if only because once you go the dynamic BR route, the initial build root is likely to be sparse with few things that can influence the build. But even if it weren't, is that's such a big deal? The set of BRs is already not constant – the only constant part is the first level explicitly declared BRs, but those can pull in different second level BRs depending on the repos state or depending on their own version. And a lot of the weird effects in Go stem from their attempt to remove second-level BRs from the picture (in a lala-lala dev world no rpm-style dependencies — no rpm dependency hell, when if fact it only moves the tricky effects to another point) |
That would work too, as long as you take into account a package may declare several %build_requires s and you probably need to loop their execution with BR population till there are no new results. That's an analog of my "implicit looping". Replace script with --command to make the people that reimplement everything as binaries happy And make sure command can take arguments :). So maybe just this ? Simpler is better
%buildrequires <command1 with arguments>
%buildrequires <command2 with arguments>
…(commandX can be a macro, a script in the package archive, a system command, maybe even a %{sourceX}? Our should %{sourceX}s be deployed in %prep?)
do
DynamicBRs=nil
for each BR_command do
DynamicBRs += BR_command_output
done
until install DynamicBRs returns "nothing to do"Step one and 2 will probably be automated in language-specific macros such as %gometa for Go in Fedora |
The main drawback being that if rpm is completely unaware of this, it can not abort local builds with a sensible error if the local system is missing one of the dynamic BRs |
|
On Saturday, February 17, 2018 9:57:13 AM CET nim-nim wrote:
> [snip, mock could ... ]
> - does installroot and installs BuildRequires as usually
> - runs %prep
> - runs %foo_analyzer from %build_requires
> - runs the rest of the build
[snip]
That would work too, as long as you take into account a package may declare
several %foo_analyzer
Well we could stick with
%build_requires
%foo_analyzer
%bar_analyzer
if we baked the script into SRPM _unexpanded_, and build system was
responsible (through in-chroot rpm) for expanding ... user could pick how
many analysers he wanted.
Well, we wouldn't have to bake this into SRPM at all in the end -- but
just add some "flag" mentioning that "this is SRPM which needs multi-step
buildrequires resolution", so build system knows how to switch to the
appropriate multistep build mode. The actual run of %build_requires would
be performed from extracted specfile.
and you probably need to loop their execution with BR population till there
is no new results to handle complex cases transparently. That's an analog of
my "implicit looping".
I'm not sure calculating "fixed point" for build requires dependency graph
is required in the first place. Having BuildRequires and then single-step
dynamic build requires would be powerful enough I think, and easier to
implement from buildsystem pov. Slight benefit would be that packagers
would be much less motivated to do **very complicated magic** in
build-depsolving.
> this way you don't have to adjust depsolver, only the build-system.
The main drawback being that if rpm is completely unaware of this, it
can not abort local builds with a sensible error if the local system is
missing one of the dynamic BRs
If I understand you correctly, the dynamic build requires would be
slightly weaker compared to the standard BuildRequires; is that a real
problem? IOW, the semantics would be like `rpmbuild --nodeps` was enabled
for them.
In theory, rpm could reexec the %build_requires script before continuing
with %build just to check that nothing is missed... but then we would have
to expect that build system does the "fixpoint" calculation as mentioned
above. I wouldn't be sure then that we are not stepping to far to academic
field..
|
To be honest, I don't think I need looping for Go, but I'm not sure I won't need it either (the Go dep situation is evolving fast those days as things have reached a "can't continue like before" point). The "correct" future-proof way to do it is looping (for IT definitions of future-proof, that is). It is so hard to implement ? You know you've reached the end of the loop when the last dynamic BR install command does nothing. do
DynamicBRs=nil
for each BR_command do
DynamicBRs += BR_command_output
done
until install DynamicBRs returns "nothing to do"
I'm not fearing packagers as much as upstream language ecosystems. Sure, some packagers will write their own custom BR logic but most major languages will just call upstream language tools in ( With Asking them for a mode where the tooling does not download anything during BR computation and delegates to the package buildsystem is doable. And actually, just blocking downloads will usually result in a partial BR answer since the tooling can not compute on things not available yet. So it gets things done without annoying language people with rpm limitations. Asking language people to rethink their BR logic so it's single phase is a lot more difficult. They are usually convinced their native tooling is better than Linux packages I'd rather not have a brand-new dynamic BR system that lasts a month before they obsolete it. |
Seeing an existing example would really help to justify the additional complexity. Such problem smells like equivalent to bootstrapping distro from scratch problem. And what |
As I said, I don't have such an example (and hope won't have for some time). It's just the kind of things I see upstreams do.
Since most Go projects do not use dynamic libraries, setting up the deps of a non-trivial project is similar to bootstraping, yes. They tried to ignore lots of things and it just moved the complexity at the project level. That's why the soonest it is all plugged in rpm where there is know-how to deal with software dep complexity, the better
As long as you're ready to add it later, I don't care much (only that I'm quite sure that if it's required we'll manage to get EL stuck on the feature-free, and pay the price in "can not be packaged for EL" later). As I've shown you don't even need to maintain a dynamic BR state to loop, just to test in the install does something |
|
Anyway, to put things back into perspective. Since I spent a one more year trying to progress on the subject. When you remove all the fluff and optional feel-good additions, the basic raw need is very simple. The constraints of a BuildRequires generator are the following:
So concretely you can not run a generator before the end of some processing that does
Notice something? That's pretty much the definition of So to run any BuildRequires generator you need a All the solutions that pretend computing BuildRequires without executing So, with a year of hindsight, I've simplified the requirements to
So, it mainly needs to define how the BuildRequires compute engine must present the computed deps to the rest of the tooling. The simplest way is probably just to generate a file that contains standard The moment when the rest of the tooling must parse the file is just before And optionally, as a further enhancement, because some people like to run
Does not matter as long as you can run You can even avoid the need of a define a separate But, the second part is an enhancement. Something some people feel strongly about, but is not necessary to produce working packages. The basic need one can not avoid, only hide behind abstraction layers is: 1. run classical So the most minimal changes required to integrate it all cleanly in |
|
I agree that %prep is needed to do this. But there is another thing to think about:
|
You have static |
|
Of course, the SRPM format needs to be updated first; so we can store the dynamic build requires "unexpanded" there. |
|
Well, the issue here is what is the procedure to build the package. There are two options: Overall the static Buildrequires are not quite the same as the Requires needed to calculate the dynamic BuildRequires and they should be kept separate. Especially for the cases where there is no script for dynamic build requires. |
|
I thought the same thing as you at first but you don't absolutely need the separate BuildRequires syntax. You can perfectly limit static BuildRequires to the part needed to compute additional BuildRequires before (technically that's just putting those BuildRequires in an expand, and cat-ing >> the result to the dynamic BuildRequires file. Ugly for those that have to do it manually, if not hidden within a macro). So, from a technical POW, sure the separate syntax would be nice to have, but it's not absolutely required. |
That was me and after playing with the concept for a year I agree you can live without it in the real world. A single install transaction before |
|
One thing that concerns me is that now when srpms require certain packages, this information is visible from the source repo. I can run repoquery to check that nothing requires what I intent to retire etc. If we generate those, we should make sure the srpms we put in the source repo have the info in them available. |
|
That's interesting thing from the policy POV, thanks. That would certainly be an issue for FESCO before allowing us to use that in Fedora. But I don't think it is necessarily a blocker for the actual implementation in mock/rpm. |
That's pretty much what I wrote in if you want to use repoquery this way, you need either to redefine srpms as "current srpm + result of And then just output this file in mock/copr/koji, store it in repos the usual way, query it with repoquery, etc. Since the solution proposed by @ffesti does not require any special new rpm header, as long as you output this file, all the tooling that will index or query it need not be aware some of the BuildRequires in rpm headers have been generated dynamically. Or, if I misunderstood @ffesti, and he intends to go the whole way with a new header, it does need slight adaptations in createrepo and dnf to index and query But, as @praiskup wrote, that's all enhancement land, the repoquery part is not required to start creating clean packages that use BuildDependencies. |
I actually like this idea and as maintainer of mock, I can promise implementation of this in Mock. |
I can imagine
I can make support in mock for that and we can make this default in Koji, so all builds from Koji will have full set of buildrequires and you can still repoquery them. I am not afraid about Koji/Mock, but what about OBS @mlschroe ? |
|
Well, the result of %buildrequires could be added if the srpm is build side by side with the binary packages. We could add some marker if it is build stand alone. So you could know in advance that there are dynamic buildrequirements still missing. |
|
Ok, I tried to sketch a POC patch and it turns out executing arbitrary scripts and breaking builds is something not too foreign to rpm. We still need a good name for the section. Unfortunately %buildrequires already is taken by the macro with the (static) buildrequires. It would be nice to have a naming scheme that would work for other types of dependencies, too. Just in case we'd like to add something like that in the future. Suggestions? |
|
How about a Warning: I suck at naming |
|
I proposed somewhere above a %build_requires (with some options, too), but that would probably be too huge overlap, right? Can we operate with brackets or options? Like |
|
something not too long to type would be nice however |
|
Oh, sorry I should have been more specific what I mean with "other dependencies". I don't think we need more different kind of build requires. But it may be interesting to add scripts for Provides, "normal" Requires, Conflicts, all the weak deps, well probably not obsoletes. This is currently totally speculative but I could imagine a future where we have build time scripts for those. Basically dependency generators on spec level. |
|
What a wonderful idea. You’re right. BuildRequires are the current pain point, but there's no reason the rest won’t want to be done after this problem is fixed. However I think the way rpm syntax tries to pretend rpm and srpm are symetrical makes you miss the obvious. rpm and srpm are not symetrical. srpm-things are package-wide and need to be evaluated at specific points of the build process. rpm-things OTOH can all be evaluated at the end of the build process when rpm packages are assembled, but they are (sub-)package specific. Therefore I don't feel the script model is the right one. Computing deps is a script, but declaring them is just declarative. Something like Of course you can also add the script sections to avoid overloading So how about something like this? Provide containers for all the lists of dep things to allow moving them out of headers, where they can be generated, and where their scope is clear. During years of transition period, read both traditional headers and the new dep containers. # Declarative section. For syntax symmetry
# Anything needed in %%prep or by the macros used before %%prep
# Existing BuildRequires could be migrated here over time, if they’re needed before %%build
# Alternatively, don’t implement it and make existing declarations an implicit %%sourcedeps
# Eventually make it accept a -f argument taken from sources
%sourcedeps
BuildRequires: x
BuildRequires: y
# Script section
%prep
# Another declarative section. With an optional computed list and static declarations
%builddeps -f <computed list>
BuildRequires: z
# Script section
%build
[…]
# Declarative sections for the built packages
%files -f <computed list>
%doc xxx
%deps -f <computed list>
Requires: moo
Obsoletes: foo < %{version}-%{release}
%files devel -f <computed list>
%doc yyy
%deps devel -f <computed list>
Recommends: a_compiler
In fact I just realized it could simplify some of the packaging macros I'm writing now. Having to inject dep things precisely after And that opens the way to more syntax cleanups in the future, like straightening up of the headers that apply to the srpm and those who apply to the rpm (once upon a time Build* applied to the srpm and the rest to the rpm, but that has changed since, and the limit between those is not obvious nowadays as long as they are not in separate sections). Or merging Short-term, we only need |
|
But if that’s too hard to do please just do the Let’s not start chasing rainbows instead of doing the fixing we know we need and we know how to do now. |
|
The main problem I see is that current rpm does not like more than one Alternatively, (and probably simpler and more versatile), just accept an arbitrary number of The script approach, while it allows calling several commands easily, does not lend itself to splitting output by (sub-)package, which is what you want for non-srpm things. Many generators won’t be (sub-)package aware (the (sub-)package layout is a packager decision). So in all cases you will need to capture their output in a file or variable, and massage this file or variable to separate the (sub-)package components. |
|
|
|
|
Prepare for the infrastructure changes described in – rpm-software-management/rpm#104 and – rpm-software-management/rpm#593
|
Haven't read this all, but it seems to fit the bill wrt. use case rpm-software-management/mock#11 Nice to see it's becoming a popular request, finally :-) |
|
Will it be possible to condition BR generator to output only main deps without tests deps? In some case it could be useful to break a cyclic deps to not have the tests deps included. |
|
@eclipseo My understanding of things is that you just need to have a BR generator that supports this distinction, no need for special rpm support That's the nice thing of getting buildrequires generation integrated within rpmbuild, you get all the existing spec framework for free. |
|
Implemented as of commit 58dcfdd |
Prepare for the infrastructure changes described in – rpm-software-management/rpm#104 and – rpm-software-management/rpm#593
In many of languages (rust, python, golang) BuildRequires are normal Requires + Test deps. It's always (in many of cases) possible to get names of packages in advance, before building.
Since Requires are dynamic, BuildRequires also, but nowadays you still have to specify it manually and check those after each update of package. Would be nice to implement support for generating BuildRequires.
Usually such generator will require some additional package to be installed (cargo, python), so we will need to introduce someting like
BuildRequires(pre)orBuildRequires(source)which should be installed for building SRPM. Having this is also beneficial for normal process since you don't need to modify buildroot all the time in order to have some macro.The text was updated successfully, but these errors were encountered: