Dynamic Specs File Generation

[ffesti]

Dynamic Specs File Generation

This gives a more detailed background to #1485 and which tries to address #329

Issues with static specs

Rpmbuild parses the spec file completely before starting the building process. This is of course necessary to know what to actually do as many important details are in there like the list of sources and patches and the build scripts. This has the disadvantage that information only discovered during the build can't be part of the spec file. So while there is a %include macro it can't be used on build artifacts as it also needs to be expanded at parse time.

To get around this limitations several pieces are expanded later on - after the build scripts have been run. But all of these are special mechanisms with limited scope. These include:

• Dynamic Build Dependencies
• Globs, sub directories and manifests for files sections
• Dependency generators

Everything not covered by these mechanisms needs to be statically declared in the spec file itself and though cannot be derived from the sources or the build process. Main issue here are sub packages which cannot be created after the initial parsing of the spec file. There are a few other issues like generating dependencies that are not bound to installed files but e.g. to manifest files in the sources for interpreted languages.

Other approaches

There are different possible ways to work around this limitation. A more powerful implementation may allow accessing and altering previously declared sub packages. There may be different rounds of declaring sub packages, process their file lists and then alter the packages - triggering pre-processing of the files and generated dependencies.

While this amount of power - that would allow reformatting unaware packages - is tempting it requires a lot of new APIs and forces the
code doing all this dynamic processing to be very intimately linked to these RPM APIs.

In the real world it is expected that most use cases are more simple and just generating sub package definitions will be
sufficient. Instead of altering previously declared packages these can be removed from the static spec file and also generated
dynamically. This shrinks the required feature set to be able to generate sub packages after the build scripts and feed them into the
regular file processing and package assembling mechanism as normal packages. This can even be achieved by parsing spec file sections using the normal spec parser.

Minimal viable Product

Between %install and %check read and parse *.specpart files generated by the build scripts and put in the buildSubdir. The files are read in alphabetical for consistent results.

This has multiple benefits:

• Simple and easy to implement
• Spec syntax is simple and well known to packagers
• Spec syntax can be generated with any tool and language
• Mechanism does not care how and when the file was created
  • Explicitly executing a script or tool
  • Implicitly being added to %__spec_install_post macro or similar
  • Executed from brp script
• Has access to buildSubdir and installRoot
• Can get passed macro contents via cli parameters
• Can make use of macros when parsed
• Previously create files can be read and altered (although tooling is currently lacking)

Without additional feature this approach comes with a few limitations, though:

• No direct access to the parsed spec file
  • Tools could parse the spec themselves but may come to different results than rpmbuild
  • Rpmbuild could pass the package data to the build scripts
• No way to alter previously define sub packages
  • Allowing to overwrite sub package definitions seems easy but will likely get messy soon.
  • Simple way to "steal" files from packages may help for many use cases
• No access to the expanded file lists as they are only calculated later
• No good fit for replacing current debuginfo implementation due to the limitations above
• No good integration with the lua API
  • May be %lua can be used in the build scripts and expanded right before the execution. May be we can add lua based build scripts that are executed at the right time in addition to the traditional build scripts

But it seems like these can be addressed as a second or third step. Some use cases - like language sub packages - can already be
implemented with this simple version.

Use cases

• Using manifest files from languages like rust to create the sub packages from upstream layout
• Language sub packages
• Putting generated files like pre-compiled byte code or caches into separate packages
• Generate optional doc packages
• Generating dependencies unrelated to files

Post scriptum

We are not looking at the build scripts themselves here. As they are executable they are less in need of dynamic features. There is a line of injury on how to automate the build process. But this is not it.

[pmatilai]

May be %lua can be used in the build scripts and expanded right before the execution. May be we can add lua based build scripts that are executed at the right time in addition to the traditional build scripts

Related to this: we now have rpmlua which can be used to run the rpm embedded Lua externally. It obviously wont have access to macros defined in the spec, but that could be considered a feature. But indeed, we could have a Lua hook where you can do stuff that external scripts cannot. We even have code for such hooks since early millenium, classic case of a solution looking for a problem 😆

My concerns (and related) wrt packager/distro control over packaging still stand, that sort of interaction and plan for compatibility needs to be in the design I think.

[ffesti]

My concerns (and related) wrt packager/distro control over packaging still stand, that sort of interaction ... needs to be in the design I think.

Im my view there are actually three cases:

1. Spec executes some tool
2. Distro enables some brp script or %__spec_install_post macro
3. Some upstream Makefile/build script produces the specpart

So:

1. Is just fine
2. has established ways of controlling them. They could be better but that's another topic.
3. Seems very odd to me. Why would they do that unless explicitly commanded to do so? Do we really need to defend against that? Build scripts can do all kind of weird things. From a design POV they just shouldn't and even if they did e.g. just ship a file the spec can just delete or alter it after the upstream build script has run.

[ffesti]

[A] plan for compatibility needs to be in the design I think.

I agree that the compatibility issue is not sufficiently addressed.

One problem right now is that if a spec making use of this feature is build on an older rpm version the spec looks just fine and might even build. More likely it is going to fail with unpackaged files. But there is no good indication on what is the issue.
This is actually a larger theme. Specs don't have anything that says what rpm version and environment is needed. And we don't really want to force packagers to explicitly state the compatibility level. Similar issues arise when packages rely on e.g. dependency generators or file triggers in other packages.
For many other new feature things are easier as they require lines in the spec file that will trigger somewhat useful error messages. We might be able to mimic that by creating a sub directory where the spec pieces go into. On older rpm versions the directory wouldn't be created and that could at least fail builds at the right time. May be ditching the sub dir was a bit premature...

[Vogtinator]

One use-case is also to add new Requires/Provides/... to subpackages or the main package during build, sometimes that's not (easily) possible with dependency generators.

[ffesti]

That was kinda hinted at in the issues section but I added it to the use cases so it is more obvious.

[DemiMarie]

I recommend not actually generating spec file fragments, but rather using a Lua API. Generating text is convenient, but it is also very fragile and tends to break at unexpected times. Instead, a better approach would be to use Lua to modify the in-memory representation of the spec file, and provide an option to RPM to dump the spec file for debugging.

[pmatilai]

better approach would be to use Lua to modify the in-memory representation of the spec file

You clearly haven't much looked at the build code 😆

[ffesti]

Well, as hinted at in the top post I think it is just not practical to implement most use cases in lua - even if we add a nice API for that. Most languages already have tools to deal with their meta data and will want to use them instead of re-implementing them in lua.
I don't see much of an issue of them producing spec syntax. What we really should avoid is people writing their own spec parsers. We already have rpmspec. But it is not as useful as it probably needs to be. We should offer a way to get the parsed content of the spec (and spec parts) as some sane universal format like json - and a way to turn it back into spec. This is not as trivial as it sounds as there are %files sections that are different from actual files and the question of macro expansion. But this is one of the next steps. I think this feature will already be very useful with write only spec parts.

[DemiMarie]

Yes please!

I don't see much of an issue of them producing spec syntax.

I’d prefer to avoid that, mostly because of escaping issues.

[ffesti]

I think one issue which is kinda related to the compatibility concerns but are actually a different thing is discoverability. This feature is very implicit. Yes, it prints the files it reads and error reporting on parsing should point to the right place but these are all build time things. Right now the generated files are deleted together with the rest of the build. Yes, one can inspect them on error but the mainly remain invisible. The code producing them will often be centralized and also be invisible or at least out of sight. This is not by accident but the main benefit of the feature.
Still we can make things a bit more obvious - at least for the build and built:

• Print the spec parts content during build
• Add the spec parts to the srpm (although there's not a great place for them now)
• Make the build srpm that it uses this feature

Not sure if I am really a fan of adding something to the srpm - it seems kinda odd to me. Otoh it is in line with what we do with dynamic build requires. May be someone has an even better idea. Mark the binary packages with the spec part they come from?

[dcermak]

• Print the spec parts content during build

I'd be for including this one at least, it's really not intrusive at all and could be potentially also not printed to stdout via a flag passed to rpmbuild.

• Add the spec parts to the srpm (although there's not a great place for them now)

I kind of like this idea. Or how about creating a second spec with the additional parts included in there as well, while keeping the original intact?

[pmatilai]

The implicitness is indeed a problem, but it's by no means limited to this feature. The more things get done by magic macros from distros and rpm itself alike, the less clue you have what a build will actually do, or how a particular package was actually built. Incidentally we already have a ticket for that: #1241. The spec parts would end up in that parsed spec, and we could/should annotate the parts somehow to make it clear what is part of the original spec and what's something else.

[dcermak]

So how about putting a $pkg_name.spec file into the source rpm, alongside a $pkg_name-post-processed.spec where all the dynamic parts have been filled in?

[pmatilai]

Putting additional files into the payload quite significantly changes what an src.rpm is. See #2047.

[ffesti]

Ok, with #2047 we now have a place where the (parsed) content of the spec including these dynamic parts get stored.

Guess printing the unparsed content out during build should make this visible enough for packagers. I'll add a patch that does that.

[pmatilai]

Considering we don't print out macro expansions either, I think printing out the entire snippets would be excessive (I mean, consider autogenerated texlive.spec equivalent...). Just print out a message with the file name for each dynamically generated spec part we parse, I think that's quite enough to signal that there's something going on.

It wouldn't hurt to automatically add a comment including the dynamic parts file name to the parsed spec, maybe with explicit begin-end markers, eg add something like this in the parsed content.

# begin 01-foo.specpart
[...]
# end  01-foo.specpart

[ffesti]

Well we can easily only print the content on verbose builds. That way packagers have an easy way to take a look - especially on failing builds which won't have an SRPM created.

The current code already prints a message for each spec part being read.

These markers are a great idea to easier allow making sense of the expanded spec. Add them as soon as I figure out how to inject them into the parsing machinery.

[pmatilai]

Problem is, builds are verbose by default. But it's a fair point, debuggability of remote buildsystem failures is important. So there needs to be some way to invoke printing the whole thing. One possibility would be adding an option to dump the parsed spec at any failing build, this would help debugging macro issues too.

Adding to the parsed content should be a simple matter of appendStringBuf(spec->parsed, <msg>).

[Conan-Kudo]

Wait, I thought the RPM header had space to store macro definitions. I know we don't use that for anything (like we don't use the RPM header that stores compiler build flags), but couldn't that be part of the solution?

[pmatilai]

I don't know what gives you that idea. There are handful of special tags which can be set through macros, including %optflags, but nothing generic.

[ffesti]

OK, #1485 is finally merged. This is the minimally useful step. I'll resubmit the patch to find-lang.sh that uses this to create language sub-packages as a use/show case.

Still the current state is limited to complete sub packages and can't really interact with the content of the spec file. So we need to look at other - more complicated - use cases and come up with the next features needed.

[pmatilai]

FWIW, I tend to think the "limited to complete sub-packages" a rather good and sensible limitation to start with. There may be cases where loosening that limit might make sense (%files comes to mind), we'll see.

[ffesti]

The first version of the find-lang.sh PR is now online: #2300

Feedback welcome!