Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get representatives for NixOS assigned to the OSS-Security "distros" mailing list #14819

Closed
peti opened this issue Apr 19, 2016 · 38 comments

Comments

@peti
Copy link
Member

commented Apr 19, 2016

NixOS should have representatives subscribed to the "distros" mailing list at http://oss-security.openwall.org/wiki/mailing-lists/distros so that we are informed about embargoed security issues and have a chance to provide our users with fast updates on the coordinated release date.

@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 19, 2016

Great idea! I'd love to do at least part of it. I have a strong background in security and would set aside some time to do the NixOS work needed for this. How many representatives can we get involved?

@domenkozar

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

I have little background in security (coursera Crypto), but I'd also like to help on this front. I think we should start by having a team/herd that cares about security and make a roadmap for each NixOS release what to focus on.

@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

I asked for more details on IRC (#oss-security on freenode):

<copumpkin> http://oss-security.openwall.org/wiki/mailing-lists/distros mentions that discussion of new distro membership should happen on the list. Is there a standard way to propose it, or does one just send an email to the list? I couldn't find any recent such discussions
<sarnold>   copumpkin: just send an email; I think the last successful "please add me to this list" came from amazon probably a year or two ago
<sarnold>   copumpkin: be sure to demonstrate that you've got users, that you've been publishing security updates for those users for a while
<copumpkin> cool, thanks. We were considering applying over in NixOS. Not sure what the threshold for users should be, and we don't have an easy way of gauging user count, but it's a reasonably notable project with lots of IRC users and github participation. Do you think that would be eligible?
<copumpkin> (we've definitely been pushing security updates, etc.)
<sarnold>   nixos seems pretty plausible to me, I think most people have a good opinion of your work anyway.
<copumpkin> sweet! How many representatives do distros typically propose? I'd like for it not to rely on one person for a few (probably obvious) reasons :)
<sarnold>   one or two, probably best if both are known in their own rights somehow..
<copumpkin> makes sense
@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

/participate

@peti

This comment has been minimized.

Copy link
Member Author

commented Apr 20, 2016

The purpose of that mailing list is to coordinate important security updates behind-the-scenes, i.e. serious vulnerabilities are disclosed to the members of that list before the general public learns about them. This gives distributors a head start of, say, 2 weeks until the publication of the CVE so that they have a chance to prepare package updates ahead of time and to release them to their users at the same time as the CVE is published, thereby minimizing the window of exposure of their users.

Since the members of that list have access to extremely sensitive material, representatives we suggest should be reliable, long-term contributors of the project who are well connected within the NixOS community.

Furthermore, these representatives must have the ability to prepare package updates secretly. For example, suppose a serious vulnerability is discovered in glibc. Then the best possible solution would be for our representative to patch glibc, re-build most of the Nixpkgs package set for Linux/i686 and Linux/x86_64, and to upload all those builds to cache.nixos.org. At the time of the coordinated release date -- when the vulnerability becomes public --, the appropriate patch to the glibc Nix expression is then checked into our Git repository so that everyone can see it. The important bit it is that at the time that patch hits the Nixpkgs repository all the binary packages for the new state are already available for download.

Obviously, that procedure can be re-fined a lot, still, but the point I am trying to make is that our representatives need the ability to mess with hydra.nixos.org and cache.nixos.org in a way that's invisible to the rest of the world, i.e. without going through the public Git repository.

This means, IMHO, that our representative pretty much have to be @edolstra and @rbvermaa, because no-one else has the necessary privileges and technical skills to pull off the necessary updates.

@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

@peti I buy your point today, but I'd like to move away from centralizing further on @edolstra and @rbvermaa. With @nbp's upcoming changes, security fixes don't necessarily need to rebuild the world, so the special tie-ins with Hydra become less necessary. Furthermore, this project is already buckling under the weight of our dependency on Eelco and Rob, and adding more seems like the wrong direction. If anything, I'd like to figure out ways to make access to Hydra less magical while still retaining the nice trust properties we have today.

@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

Here's more that could simplify our decisions a bit:

<copumpkin> what's the expected level of secrecy here? are the two representatives expected to single-handedly prepare/coordinate the patch releases for their distros, or can they confer with others as needed on the distro security team
<copumpkin> (need-to-know, trusted, etc.)
<sarnold>   copumpkin: probably confer as needed, assuming they'd instruct the others in the appropriate protocols
<copumpkin> cool
<copumpkin> yeah, that sounds a lot more manageable than lumping all the responsibility onto two people
@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

ability to mess with hydra.nixos.org and cache.nixos.org in a way that's invisible to the rest of the world

It would be best if hydra supported jobs invisible to most users. A simpler option would be to create another parallel instance, running on the same HW, but only accessible to the selected individual(s) (and in effect also to anyone with SSH access to those machines).

@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

I suppose there will always be that information leak when build farm puts unusually little resources to the visible jobs, but I think that's perfectly acceptable, as it's even common to pre-announce security updates.

@nbp

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

I totally agree, some well acknowledged, trust-able, and available community member should apply there.

On the other hand, I think this might be a bit premature to do that if we have no way to ship these security updates in a timely manner. (#10851 and the long tail of static analysis reports to fix)

@nbp

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

@peti, do we have any idea of the volume of CVE that these persons would have to handle?

@grahamc

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

According to sarnold,there is very little traffic on the mailing list.

@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

If one is given a week of heads up, even a full rebuild is feasible on our Hydra...

What is very little traffic? About a pair of threads with several e-mails in a month?

@grahamc

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

Not sure. I asked, but haven't received a reply yet. I would suspect no more than what you said.

I believe it might also be helpful to have a team for following oss-security in general and make sure we handle those in a timely manner as well.

@grahamc

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

Update: "it's kind of hard to quantify; there's 208 mails in the last year, over what feels like less than a dozen "actionable" things, a handful of CVE assignments."

@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

OK, I believe I could handle the distros list and the related agenda without any money for it, especially if paired with someone else. I currently can't claim that about the oss-security list. (But note that I'm rather cheap ;-)

@peti

This comment has been minimized.

Copy link
Member Author

commented Apr 20, 2016

@nbp, the volume on distros is small and the vulnerability reports usually come with ready-to-apply fixes attached. If we'd care only about master, then handling those issue would be almost no effort: you'd simply apply the patches (or update the package to the new version) and set off some kind of at job to push the update to Nixpkgs at the coordinated release date. Things become more interesting (and more effort) only when it comes to fixing older versions of the package which might still be in use in previous release branches since upstream may not provide a patch for a package version other than the latest one. In that case, you'll have to back-port the patch to the older code base -- or update the version in the release branch. That might be tricky. On such occasions, it's probably a good idea to involve someone else who has the necessary expertise. Anyhow, NixOS has no LTS branches where 3+ year old core packages are still in active use, so in our case dealing with such issues will be very straightfoward almost all of the time.

IMHO, the biggest problem is how to prepare binary packages to release on the CRD without revealing any details about the update beforehand.

@nbp

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

Note, this mailing list does not change the fact that we would have to handle 0-days, without any rebuild.

So I do not see why we should enforce rebuilds when we have the option to not do that, as this also involve bandwidth that not all users can spare on a daily bases.

@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

@nbp I'm treating it as a given that we'll merge your PR. I see this ticket more about organizational issues.

@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 20, 2016

Yes, I consider that rebuild-reducing work orthogonal to this discussion.

@grahamc

This comment has been minimized.

Copy link
Member

commented Apr 22, 2016

What is the next step in resolving this issue? I'd hate to see this fall by the way side.

@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 23, 2016

I think now we "only" need to find who would be on that list; two seems the best number. As I wrote, I can be one if there's lack of interest from others.

@grahamc

This comment has been minimized.

Copy link
Member

commented Apr 23, 2016

OK. I deleted my comment suggesting myself. Assuming they agree, perhaps @edolstra and @rbvermaa should pick a couple people.

@edolstra

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

@nbp We can push out -small updates requiring a full rebuild in < 3 hours. And in any case Hydra would have to rebuild something (e.g. for a Glibc patch it will need to rebuild Glibc). So regardless of #10851, we need either a private Hydra instance, or some access control in Hydra. There was some interest at work in having private jobsets, so I may be able to spend some time implementing that.

Regarding our representatives, how about @copumpkin and @domenkozar? (It wasn't clear to me if @peti was volunteering himself.)

@peti

This comment has been minimized.

Copy link
Member Author

commented Apr 28, 2016

@edolstra, I didn't mean to nominate myself. I just felt like pointing out work that I'd like other people to do. :-)

@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

For more information on the last addition to the linux-distros list, I dug up this old thread from when Amazon got a representative added. It might help clarify the sorts of things they look for in a distro and representative:

http://www.openwall.com/lists/oss-security/2014/04/10/5

@copumpkin

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

In short, what they appear to be looking for:

  • Representatives with past participation in the public oss-security list (probably not good at this)
  • Evidence that the distro backs the representatives (this thread can be evidence)
  • Evidence that the distro puts out timely security releases (I think we're generally good about this and will only get better)
  • Evidence that the distro includes information about which CVEs are fixed (we're patchier on this)
  • Evidence that the distro actively notifies users about available security updates (I don't think we do this today)

These all seem like reasonable things to look for. Perhaps we should put together a NixOS security mailing list for those of us who are interested, so we can further discuss the various finer points around this stuff? Or I guess we could do it all on GitHub, but it might get unwieldy to notify all interested parties.

@vcunat

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

Heh, that list's not easy to fulfill.

@domenkozar

This comment has been minimized.

@domenkozar

This comment has been minimized.

Copy link
Member

commented May 20, 2016

Also relevant: #13515

@peti

This comment has been minimized.

Copy link
Member Author

commented Sep 10, 2016

In my opinion, the community members who volunteer to represent NixOS on the distros mailing list should take matters into their own hands and initiate some kind of "poll for support" via nix-dev to legitimize their claim. Once a significant number of Nix'er has expressed their support for the candidates, our representatives should get in touch with distros and try to make things happen. As it is now, nobody feels responsible for achieving this goal, which means that we won't. That would be a shame.

@davidak

This comment has been minimized.

Copy link
Member

commented Sep 28, 2016

I like how Ubuntu notifies it's users. They have a mailing list and webpages with the same informations, for example http://www.ubuntu.com/usn/usn-3089-1/

That is great to give your team leader a link to inform him about a serious update. A RSS feed on the NixOS homepage would also be great.

Thanks for taking the initiative. I really appreciate it as a user!

@grahamc

This comment has been minimized.

Copy link
Member

commented Oct 21, 2016

CoreOS is trying to get on the distro list. Here are some requirements they've noted on the ML:

It looks like CoreOS is shipping Linux and respecting the various licenses
in a volume sufficient to make sense for them being given access to the
Linux distros list, and shipping security updates (I would say they could
benefit from shipping advisories, but they put the CVE's in the ChangeLog
so I really can't complain). Assuming they can handle embargoed issues (do
you have private bug tracking/code repos/CI/whatever else you need to ship
an update?) I would have no objections to them joining the Linux distros
list. Can you confirm you have infrastructure to handle embargoed issues?
If yes I guess it's up to Solar to add you.
@Mic92

This comment has been minimized.

Copy link
Contributor

commented Oct 22, 2016

So a private hydra is required?

@grahamc

This comment has been minimized.

Copy link
Member

commented Oct 22, 2016

Not sure. I could imagine this being The Most Enterprise:

private

I also think there is talk about private jobsets on the public hydra.

Note, this also requires a private nixpkgs repository and bug tracker.

@Moredread

This comment has been minimized.

Copy link
Contributor

commented Oct 4, 2017

Has there been made progress on that front?

@ckauhaus

This comment has been minimized.

Copy link
Contributor

commented Oct 30, 2018

FTR: We discussed this issue during the Hackday @ NixCon 2018 and decided that it is not about time to apply for oss-security now. We are currently not in a position to produce evidence in all required aspects.

I'd personally recommend to work hard towards meeting the criteria and reconsider the issue then. Perhaps this issue can be closed and we will open a new when it is really actionable.

@peti

This comment has been minimized.

Copy link
Member Author

commented Oct 30, 2018

Yeah, let's close this issue as it's not actionable right now. We need to take other, smaller steps first before we can tackle this one.

@peti peti closed this Oct 30, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
You can’t perform that action at this time.