Remove underused targets (mainly building upstream RPMs) #1520
Conversation
It complicates our build system and we rarely use it. The workflow is to build a tarball and a zip file and then build RPMs or other packages from that in downstream. Upstream git repo is not the right place to be building RPMs.
|
I see your point, but also disagree with it. IMO, I find it extremely useful for testing, exploring new features/bugfixs, etc. when upstream incorporates packaging in the build system, and find it a little annoying when you have to go find a spec file in some downstream system somewhere. IMO, it would be better to have the spec files upstream (except %changelog) rather than have multiple spec files that have to be updated. But, I am sure that there are reasons here that I am not seeing. |
|
What's the reason to make an RPM and then install it when you can just run We can definitely do some sort of CI/CD flow in Jenkins that will build nightly RPMs. We don't have to carry a spec file and the machinery to build RPMs in the upstream git to do that. |
|
I see your point. I would think it would be easier to setup automated testing for QE tasks if we can ever get to that point. |
|
You do however remove the |
|
@redhatrises Yeah, if we decide to rip out the RPM building the git-tag target won't do much more than |
|
Ack. |
|
Dear SSG users, you have now found why the RPM target is missing in the SSG release you are using. Please be mad at @mpreisler and @redhatrises equally, they both deserve it. |
|
Not sure why my reply didn't get picked up by GitHub...
So, if RPM building is now removed from SSG, where is the most appropriate place to put RPM build code? Should I make a new project (e.g. OpenSCAP/rpm-build) to put all this back into? |
|
@shawndwells must have happened during the DOS attacks that were happening. |
|
@shawndwells In Fedora git, RHEL git, COPR git, etc... http://pkgs.fedoraproject.org/cgit/rpms/scap-security-guide.git/tree/scap-security-guide.spec |
|
On 10/24/16 9:54 AM, Martin Preisler wrote:
Not sure I follow. How will that help me make on-demand RPMs? Place that Shawn Wells |
|
Why would you want to upload an unreleased SSG to NIST.gov? Why would anyone want to build an RPM from an unstable git version of SSG? Both of those sound like massive support nightmares. EDIT: We don't have |
|
Besides, uploading unsigned RPM content built from unofficial git spec to NIST.gov sounds like a bad idea. If it were me I would upload an official signed RPM or a plain ZIP file. |
|
Definitely need rpm creation to stay. It's a staple in almost all demos, downstream creates RPMs (e.g. NSA SIMP), and has become incredibly invaluable when customers want to create their own rpm based off newer versions than what ships in RHEL. Not to mention, whenever my NIST.gov account gets recreated I'll be uploading RPMs per SSG release. Rom building definitely has to stay. If it doesn't, we'll have to build an entirely different project to create rpm releases.... Shawn Wells
|
|
Two related references:
I will reiterate that creating RPM on demand from unreleased git bits is not how packages are meant to work. If there is no repository and no signing involved I don't understand what the value is over zip or tarball files. We could run |
|
@shawndwells I think @mpreisler brings up a good point about having a signed RPM rather than unsigned. I does seem weird that a security content that advocates enabling and using signed RPMs is not signed itself. I wonder as an alternative if RH could internally build the SSG RPM nightly as well sign the rpm? Or if OpenSCAP can sign an rpm it generates if a key is imported? However, either of those may not be a possibility. |
|
So, I would like to request some easy way of building RPMs from the SSG repositories. Given that I have to fork everything to make custom modifications, and I have my own signing keys, I'm happy to do the signing myself as long as I have an easy way to do so. I definitely need the ability to generate a valid tarball and spec file from the latest git release. I do not need the ability to build a full RPM or SRPM, I already have build scaffolding via mock to do that. The SSG is a highly forked project and I like to have a nice consistent environment with RPMs so I would like to request that at least the ability to generate a valid spec file is restored. |
Could you please elaborate on what kind of custom modifications you need to make? Very likely we (or yourself) could make upstream changes that would make this easier if we knew what you were doing.
I am not sure what your release process is but if you make customizations and then release your own version of SSG you should rename it and mark it as your own. Using upstream spec that lists one of upstream developers as the maintainer will confuse users. I suggest you take the Fedora spec, alter it with your copyright, changelog, website and whatnot and use the tarball from upstream to build the RPM. |
|
The custom mods that I need are:
We certainly will be pushing material back upstream once it has been proven to function as expected. In the mean time, as with all other forks of software that we use, we will be installing the software via RPM in our environment. Usually, what we do is we fork, maintain the existing spec file, then add our versioning to the top of the stack. This is 100% accurate and clearly denotes our custom changes in the local RPM changelog. Without a local spec file that maintains all history...well.. frankly we'll do what we usually do which is to scavenge the spec file out of a Fedora SRPM and go from there. While the absence of a spec file ins't a disaster, it does make it way easier for us to fork since forking is a requirement by design at this point. We will be making commits, but you can't expect most of your Government users to do this. As such, the project should make it as easy as possible to distribute forks of the content into the field in a reusable, consistent, and auditable manner. AKA, a RPM. |
|
@trevor-vaughan Correct me if I am wrong but it sounds like your use-case could be met with Have you looked into
I recommend pushing material back as soon as possible. Otherwise you may be writing a bunch of checks in a way that we can never accept. Testing is definitely encouraged but proving that it works everywhere as expected is not necessary or practical. People often postpone contributing back and then drop a huge pile of code on us. That is almost impossible for us to review and accept. |
|
@trevor-vaughan : I second @mpreisler 's comment about pushing things upstream ASAP. Waiting not only does creates an unreasonable load on the maintainers when you do push it, but it also tends to get your stuff out of sync, so that before you can get it accepted, you'll need to make major changes to reflect new upstream reorganization. I also understand the sometimes political nature of pushing upstream. There are times when uploading things immediately can't be done for political reasons. But I recommend that you push back on those political decision makers as hard as you reasonably can, to keep your version and the upstream version in sync as closely as possible. You won't always succeed (or at least, I haven't), but the effort is better spent that way than reworking things in the code later on. |
|
@radzy and @mpreisler I 100% get the need to push early and push often. However, this circles back around to my argument that I should be able to have a very small overlay that I can maintain on top of your materials instead of needing to push back upstream because it may simply be irrelevant to most of the world. scap-as-rpm looks OK but it's missing the Changelog. I feel that this is one of the more useful parts of the RPM to auditors and admins. Also, this now has me relying on a tool that may, or may not, keep up with the structure of the SSG codebase. At this point, it sounds like you don't want to maintain a spec file in the codebase. This is fine but, once again, adding to the complexity of using the project thereby reducing the chance that you'll have additional outside committers moving forward. (Yes, I say this as the maintainer of a project that has a stupidly complex build process and keep trying to figure out how to fix it. So far, the answer has not been to make people look for more parts.) |
You are 100% able to do that and always will be. This is an open source project. This PR is about making a divide between upstream and downstream that I believe is important going forward. It is definitely not about shutting down forks or anything of the sort.
It looks like you want a well maintained RPM without maintaining it. The thing is, we really don't maintain a changelog in the upstream. It's a joke. Look at the last 10 entries:
SSG is very much like that project. We are doing all this cleaning up and fat trimming to make it manageable going forward. SSG has grown organically into a copy pasted monster with a ton of weird "features" that nobody is using. With a ton of second-class products that somebody code dropped on us, that wouldn't build or validate. I think making SSG easy to start with and easy to patch is a much more important factor than being able to build arguably broken upstream RPMs using |
|
@mpreisler Yeah, I had noticed the Changelog entries and that was a bit saddening. Well, I've made my point and certainly have the ability to roll my own RPMs in my own special way. I guess I would suggest putting a link to scap-as-rpm in the README.md file for people that need to roll their own and you can judge if it should go back in based on the number of hits to that repo (maybe). |
|
For me, the complexity is needless. I cannot use the built-in RPM creation functionality during my So, again for me, the value is very very low, and what little The fact that it is RPM-centric is also a concern. Although my (Aside: the same holds true of the XCCDF and OVAL content related So I'm certainly not opposed to deprecating RPM creation, and for |
|
On 10/31/16 6:22 PM, radzy wrote:
I give about 3-5 SCAP conversations and workshops per week. Out of the For some reason I'm failing to express how disruptive abruptly dropping |
|
And at this point, I'm literally going to have to maintain a fork of SSG, and point customers to contribute there, to get RPMs. Them I'm going to have to merge my fork, which will include US Gov contributions, periodically back to SSG. Total fucking disaster. |
|
In addition to @shawndwells comment, the FreeIPA project maintains rpm .spec files as well as build targets for making rpms. IMO, I think that we should revert as I think @shawndwells provides the justification to reverting and maintaining rpm builds. |
|
Alright, fair enough. I am still not convinced that building upstream RPMs has value but other people seem to think so and that's a good enough reason to keep it. I will clean up this mess and add it back, hopefully tomorrow. In retrospect we should have kept this PR unmerged for a while before people expressed their opinion. |
Agreed. My fault on that one. |
|
+1 @shawndwells : great justification for having it. With that, I agree that we should restore RPM creation. |
|
On 11/3/16 3:39 PM, Martin Preisler wrote:
Even on the Red Hat side: For USGCB content, we can upload releases So - what is the best way to re-add the RPM build? As the one kicking |
|
Adding RPM building back in #1540 Marked that as a blocker for the release. |
Let's get rid of underused weird complexities in the build system. In the upstream repo it makes sense to build a tarball or a zip file but IMO it makes no sense to be building RPMs. The place to build RPMs is downstream distribution like Fedora or RHEL.