Please use OVAL export for Ubuntu vulnerabilities source

[cjwatson]

Description of Problem / Feature Request

Hi. I'm one of the engineers responsible for running git.launchpad.net.

At the moment, the Ubuntu vulnerabilities data source is implemented by cloning/pulling https://git.launchpad.net/ubuntu-cve-tracker. Unfortunately this is pretty expensive for us to service: it's a large repository, and if it's imperfectly packed then cloning it causes git pack-objects to use a substantial amount of memory trying to serve a suitable pack to clients. If there are more than a trivial number of parallel clones at once, then this effectively DDoSes our git service (as happened today). Obviously we'd like to improve our service's scalability, and we do have work in progress for that, but this won't happen overnight and in any case it can't scale indefinitely.

It's hard for me to tell from our logs how many of the clients are related to Clair; all I can see is a large number of git processes from widely varying IP addresses trying to clone the same large repository and resulting in a large spike in our memory use. It does look like a likely candidate, though.

Could you please consider using the OVAL exports that our security team provide instead of cloning this git repository? I'm not in the Ubuntu security team, but I'm told they're here. The Oracle, RHEL, and SUSE vulnerability sources all use OVAL, so I would hope this is just a matter of plugging different URLs into much the same framework.

Let me know if there are any problems with the data that make this difficult, and I can pass that on to the Ubuntu security team.

Expected Outcome

Clair uses OVAL exports for Ubuntu, resulting in (I hope) the same effective vulnerabilities feed but with a lighter footprint on our infrastructure.

Actual Outcome

Clair sends many clients to clone the same large repository from git.launchpad.net, which creaks under the load.

Environment

N/A

[alexmurray]

Another easier short-term option would be mirroring https://git.launchpad.net/ubuntu-cve-tracker to github and updating clair to use that mirror instead of launchpad?

[jzelinskie]

I didn't realize that Ubuntu produces OVAL reports, too!

@cjwatson Can you get someone from the Ubuntu security team to chime in? It would be a shame to make the effort to port the code to use this service only to find out that the Clair clients simply overload a different service ran by Canonical and the owner of people.canonical.com creates a new issue that their website is overloaded.

[cjwatson]

@jzelinskie, @alexmurray is on our security team and chimed in on this thread earlier today, so hopefully he can point out if my suggestion is inappropriate. (But I did already discuss this with our security team internally before filing this issue.)

[alexmurray]

@jzelinskie - @cjwatson would be best able to advise on the infrastructure side but my understanding is that the current load problem is caused by interactions with git and the launchpad git backend - when using OVAL, this is just served over HTTP and there are no complex interactions to fetch this etc - so the impact should be much less. I would be happy to try and submit a PR for this - can you point me in the right direction where I should be looking to modify clair in this regard?

[tahriabd]

@alexmurray Oval datas doesn't seems to provide CVE informations for ESM (Ubuntu Extended Security Maintenance) and it's less updated unlike to ubuntu-cve-tracker

[alexmurray]

@tahriabd the OVAL data is generated multiple times a day from the contents of the ubuntu-cve-tracker, however this is only for the standard support releases. We are currently looking at how to generate this for ESM as well.

[ldelossa]

We recently merged your oval changes into ClairV3. Can you please backport those changes into ClairV2. ClairV2 development occurs on the release-2.0 branch.

Also to note, we are currently working on ClairV4 - a rewrite. In ClairV4 all data sources are Oval - so this issue no longer exists.

[alexmurray]

@ldelossa I assume you meant me not @alexlarsson in the ping above? Sure I can give that a go, I am not able to easily test the result however (as noted in PR #853) so once I get something that appears to be working I will be relying on someone else in the clair project to actually verify it is still functioning as expected - as noted in #853 (comment) I think a quick guide which explains - here's how to go from a standard server install of Ubuntu/Debian/Fedora or something to a running clair instance, and then here's how to submit an existing container from dockerhub (or similar) to your clair instance and how to interpret the results would be very beneficial.

[ldelossa]

@alexmurray noted.

[hdonnay]

We’re declaring bug bankruptcy as part of the release process for a new major version of Clair. Please open a ticket in our issue tracker if you feel this still needs to be addressed, and we'll triage as part of our v4 development process. Thanks!

[setharnold]

I've filed https://issues.redhat.com/browse/PROJQUAY-1099 for this issue. Thanks.

[Tejuvmware]

Did we get a fix for this in any of the versions of Clair ? I'm facing this issue https://bugs.launchpad.net/ubuntu-cve-tracker/+bug/1943825 . Using OVAL is going to solve my problem. May I know in which versions of clair image this has been implemented ?