New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
%exclude should not permit files to bypass check-files and be omitted from all packages built from spec #994
Comments
(as for why I'm filing this now... well, I forgot about this in the shuffle two years ago, and I was just reminded of this again today...) |
Sounds like a bug indeed. Patches welcome ;) |
I am predicting this will break multiple packages in Fedora, but I think this would be good behavior. |
It will definitely break something, probably a lot of things. OTOH, it would probably make the things more consistent, e.g. it would avoid issues such as [1], because one would need to delete the file instead of excluding it. |
This will break a lot of packages in Fedora indeed. For some of my packages I even rely on the behavior, such as here: https://src.fedoraproject.org/rpms/python-xmlschema/blob/a11ca456b41794af7aec847af66f1e7974a698ff/f/python-xmlschema.spec#_59 |
The current behavior of |
The spirit of %exclude always was merely to support sub-packaging, not for leaving random junk in buildroot. But where even the tiniest crack exists, just like water will find its way, so will packagers 😄 |
My current guess on how this slips through is that files that are marked as excluded are still being passed to check-files, so it passes the check-files check. But my attempts at trying to make it not do that seem to be in vain... 😕 I at least have a test case to see if I fixed it, but so far, no dice. 😢 |
…nt#994) The intent of %exclude was always to merely support sub-packaging with wildcards in %files sections, not to permit leaving junk in the buildroot. Enforce this by checking against the actually packaged contents rather than everything we encountered during collection. This also gets rid of the ugly static check_fileList buffer - besides being ugly, such things are bad for parallelism. This has been widely abused so the change is likely to break quite a few packages in the wild. As a side-effect this also cures a long-standing bug where unpackaged excluded files leak their debuginfo into packaged contents, as such a package will now fail to build at (RhBug:878863) Fixes: rpm-software-management#994
…nt#994) The intent of %exclude was always to merely support sub-packaging with wildcards in %files sections, not to permit leaving junk in the buildroot. Enforce this by checking against the actually packaged contents rather than everything we encountered during collection, and document the behavior. This has been widely abused so the change is likely to break quite a few packages in the wild. As a side-effect this also cures a long-standing bug where unpackaged excluded files leak their debuginfo into packaged contents, as such a package will now fail to build at (RhBug:878863) As a nice side-bonus, this also gets rid of the ugly static check_fileList buffer - besides being ugly, such things are bad for parallelism. Fixes: rpm-software-management#994
The intent of %exclude was always to merely support sub-packaging with wildcards in %files sections, not to permit leaving junk in the buildroot. Enforce this by checking against the actually packaged contents rather than everything we encountered during collection, and document the behavior. This has been widely abused so the change is likely to break quite a few packages in the wild. As a side-effect this also cures a long-standing bug where unpackaged excluded files leak their debuginfo into packaged contents, as such a package will now fail to build at (RhBug:878863) As a nice side-bonus, this also gets rid of the ugly static check_fileList buffer - besides being ugly, such things are bad for parallelism. Fixes: #994
Again: how is this an improvement? I have seen many specfiles deliberately using |
@kkofler Just because @hroncok is doing that does not mean it was a good idea to do it that way to begin with. Additionally, that would have been broken anyway if you tried to |
Yes, I too have seen an endless stream of specfiles deliberately doing all manner of strange things and abusing loopholes in rpmbuild, and we've been systematically closing those loopholes as we come across them and time permits, for (more) consistent and defined behavior. Just like compilers do. This is just another in a long string of such changes - buildroot always was only for packaged material. |
I am also complaining just the same way about compilers doing this. Your "closing loopholes" is your users' incompatible changes that unnecessarily break their builds. |
If it was unnecessary, I'd agree... |
I think your definition of "necessary" differs significantly from mine. RPM will not break down if this incompatible change is not made (or reverted, now that you pushed it), so I do not see why it is necessary. And to give some context: as a maintainer of TIGCC, I was consistently reverting that kind of changes to our patched GCC (C only though, because we did not support C++ (whose GCC frontend is much worse in this respect) for other, unrelated reasons). I did all I could to maintain backwards source compatibility all the way to the initial alpha release of TIGCC and I strongly believe that this should be the standard to which to hold all compilers. E.g., the TIGCC patch reenables multiline C string literals, C casts as lvalues, etc. So I am not just a passive complainer. |
RPM may not immediately break down if one such change is not done, but maintaining bug compatibility for bug compatibility's sake is a colossal waste of time, and worse, they sooner or later end up preventing new developments from taking place. That's why maintaining those undocumented dark corner behaviors that somebody found is bad for everybody. |
I think there is no point in arguing. I understand both sides. Let's try measure the impact of this? Maybe it's not horrible, only pretty bad :) |
Previously we only checked for unpackaged files and symlinks, completely ignoring eg extra directories that might be there. Just check for everything instead. Related to rpm-software-management#994. Directories are a little tricky as some of them are almost always unowned so we need to allow all path components leading to packaged files.
Previously we only checked for unpackaged files and symlinks, completely ignoring eg extra directories that might be there. Just check for everything instead. Related to rpm-software-management#994. Directories are a little tricky as some of them are almost always unowned so we need to allow all path components leading to packaged files. This would be a *lot* of relatively expensive lookups, but we only need to do the dance when the directory index changes, and we can stop when no new paths get added to the pool.
It is horrible. See #1442 (comment) |
Previously we only checked for unpackaged files and symlinks, completely ignoring eg extra directories that might be there. Just check for everything instead. Related to #994. Directories are a little tricky as some of them are almost always unowned so we need to allow all path components leading to packaged files. This would be a *lot* of relatively expensive lookups, but we only need to do the dance when the directory index changes, and we can stop when no new paths get added to the pool.
The change got reverted for now, reopening. Sigh. |
😭 |
@pmatilai Could we have an (extra) knob for this behavior and have Fedora switch it off by default? I think of the main distributions using/contributing to RPM, only Fedora does not expect to enforce package file list consistency because it runs no package build verification tools as automatic post-build checks that can fail the build. |
Ideas for progress:
Note that those ideas are not dependent on each other and can happen at different timelines. |
The reason for reverting is that there's this unexpected link to %check usage (and ambiguity) and I think those changes are better handled together. I have zero more cycles and/or will to spare on this topic right now. |
As part of some of the work I've done in OpenMandriva in transitioning the RPM stack from rpm5.org to rpm.org RPM, I've discovered that there was an interesting behavioral difference with
%exclude
.In rpm5.org RPM,
%exclude
does not give you a "get out of jail free" card to bypass the file list check. A file that is marked by%exclude
in one subpackage but isn't included in any other subpackage triggers the unpackaged files error. This does not happen in rpm.org RPM.I would argue that the rpm.org behavior is a bug, as there's not a particularly obvious reason for why it works this way. Moreover, it leads to accidental packaging bugs.
Can we change this behavior for the upcoming RPM 4.16?
The text was updated successfully, but these errors were encountered: