TAC: publishing repositories for "CVE benchmarking" initiative in Security Tooling WG

[sj]

Hi TAC 👋!

Within the Security Tooling working group, we've been working on an initiative called "CVE benchmarking" (working title, might change). The idea is quite simple:

• Construct a large data set of CVEs, including metadata on fix commits and exact source code location of the vulnerability
• Tooling that allows the following workflow
  1. Select a bunch of CVEs (e.g. anything related to the OWASP top-10)
  2. Select a bunch of tools to benchmark (e.g. eslint, jslint, jshint)
  3. For every CVE, feed the code base to each of the tools that needs to be benchmarked.
  4. Determine whether the tool (1) detects the vulnerability and (2) recognises the fix
  5. Generate a report that tells you which tool is best for your purposes

All of the above lives in a collection of repositories:

• A single repo containing the metadata and tooling
• For every CVE: a repository (a fork of sorts) with the source code of the affected project. We expect there to be ~300 CVEs (so: ~300 repositories) at launch time.

We've almost finished creating the first version of the tooling and the data set (specifically for JavaScript CVEs). In fact, our talk proposal for Black Hat Europe has been accepted, so we'll be announcing it there on 7-10 December.

We'd love to release this tooling and data under the OpenSSF flag. In particular, we'd love for the repositories to be published in an OpenSSF GitHub org. For example, we could consider:

• https://github.com/ossf-cve-benchmark/cve-2019-123456 (one repo for every CVE)
• https://github.com/ossf-cve-benchmark/ossf-cve-benchmark (repo for tooling and metadata)

(It goes without saying that I'm very happy to talk about details like repository structure — the above is just an example)

It'd also be great if we could bundle our efforts on the announcement side of things.

What's the best way to proceed? To give an idea of timelines: the BlackHat talk needs pre-recording, which we're hoping to do in the week of Monday 9 November.

People

• PM for this initiative: @sj (GitHub)
• OpenSSF Security Tooling Working group lead: @psiinon (OWASP)
• Engineering: @esbena (IC), @calumgrant (EM)
• Team working on announcement on GitHub side: @secure-dev, @samanthachau, @margheritadicerbo
• Presenters for BlackHat Europe: @sj and @kevinbackhouse
• GitHub Security Lab: @xcorail and @kevinbackhouse

cc @greysteil, @mayakacz

[jenniferfernick]

Hi @sj!!

I am merely one member of the TAC, so I wouldn't pretend to speak for us all, but my personal view is that this looks very nice and would be a great thing to release under the OpenSSF flag, especially if it has been worked on within the tooling WG. I really like the idea of generating comparative benchmarks for these tools' efficacy and I can see easily how the extensibility of this can make it more powerful and general-purpose over time as well.

@rhaning -- what would we need to do from a TAC perspective to discuss this together and decide on next steps?? Bring to next TAC meeting?

[kaywilliams]

+1

[MarcinHoppe]

@sj I love this idea!

I am wondering if the vulnerability metadata schema that we are working on within the Vulnerability Disclosures WG could be helpful.

One of our goals with the schema is to facilitate capturing information about remediation, including fix commits, etc.

[sj]

Thanks for the support @jenniferfernick, @kaywilliams, and @MarcinHoppe!

@rhaning: when will the next TAC meeting be? I'd of course be very happy to join the call to explain this further and ask any questions if/when necessary.

@MarcinHoppe: it definitely looks like the work you're doing in the disclosures WG is very relevant to us! Apologies for having missed it! can think of a variety of ways in which we could incorporate this work in the future. Might you be available for a quick call with myself (and possibly @esbena) today or tomorrow so we can discuss the two initiatives and how we can work together pre and post BlackHat? I'll ping you an email.

[rhaning]

@sj This is really great. There could be some collaboration with the Security Metrics project happening in the Identifying Security Threats WG as well. cc: @scovetta

The next TAC meeting is scheduled for Tuesday November 3rd - 8:00AM PT. Details are in the invite in the OpenSSF community calendar. If you can put together an overview to present, we can discuss it at the next meeting and decide on the next steps and how to best coordinate with the other working groups.

[xcorail]

@rhaning @sj I think the Nov 3rd meeting is about to be moved (US Election day).

[mayakacz]

Would love to discuss at next TAC. Very cool stuff.

This is really great. There could be some collaboration with the Security Metrics project happening in the Identifying Security Threats WG as well.

It sounds like no - this is about measuring tools' efficacity (what that WG focuses on) vs projects' security... but it certainly could help us decide what tools we may want to run on projects, or recommend!

[dlorenc]

To help unblock this, do you think you could get the repos staged somewhere? It sounds like you'll probably want your own organization (rather than sticking everything into the ossf one) because you need to organize a potentially large set of repos.

Getting a few of these created somewhere should make it easier for TAC members to look at, then later we can get a "real" one created under the ossf accounts and transfer everything into there.

[sj]

It sounds like you'll probably want your own organization (rather than sticking everything into the ossf one) because you need to organize a potentially large set of repos.

That's indeed what we were thinking as well.

To help unblock this, do you think you could get the repos staged somewhere?

Sure thing! I'll get this together ASAP in private repositories in a separate org: https://github.com/ossf-cve-benchmarking (name can be changed if people prefer). I'll make sure all TAC members in this issue have access.

[david-a-wheeler]

@sj @psiinon @esbena @calumgrant @kevinbackhouse - this "CVE Benchmarking" effort sounds very interesting!

I'll try to connect you with Paul Black at US NIST. NIST runs the Software Assurance Metrics And Tool Evaluation (SAMATE) effort for evaluating software assurance tools. They have a set of test suites for evaluating tools. Most of their test suite is artificial, but they do use some live code. They might be interested in adding this to their suite and/or have information that'd be useful to you.

I tried to do this years ago & the CVE databases didn't provide enough data to really locate the vulnerabilities without LOTS of help; I hope that's changed.

[sj]

Thanks @david-a-wheeler, but we've already spoken to Paul and his team back in June 🙂

[david-a-wheeler]

@sj - excellent, glad to hear it! Never mind then :-).

[estesp]

+1 this looks really interesting

[sj]

@estesp, @david-a-wheeler, @dlorenc, @mayakacz, @xcorail, @rhaning, @MarcinHoppe, @kaywilliams, @jenniferfernick: I've given you all access to our tooling and metadata repo: https://github.com/ossf-cve-benchmark/ossf-cve-benchmark

Most of the development happened on internal GitHub repositories in the past, which we've combined into a single commit. Currently, development is happening in feature branches of that repository. Very much work in progress!

We'll add some repositories with vulnerable code (eventually there'll be one such repo for every CVE in the dataset) later today or tomorrow. As soon as those are live, I'll give you access to those as well.

When will the TAC meeting be, @mayakacz / @kaywilliams? I'd be very happy to present our plans there and tell you more about our BlackHat talk. It might also be useful to discuss whether the OpenSSF would like to make any announcements on this, and if so: how. We'll be posting a blog on the GitHub website as well.

Any questions: give me a shout!

[rhaning]

Thanks @sj! The next TAC meeting will be on November 17, 2020 at 08:00am PT. I'll put you on the agenda to present, we're all excited to learn more. Thanks!

[sj]

Thanks @rhaning; I'll see you on the 17th!

The deadline for us to hand in the recording of the BlackHat talk is 16 November, though. It'll be broadcast 7-10 December. I think it's fine for us to talk through some details at the TAC meeting (after the recording), but it'd be great if we could discuss a few things asynchronously here so the recording of the BlackHat talk is not blocked. Hope that's OK!

If possible, I would love to get a thumbs-up / thumbs-down on the following:

1. @kevinbackhouse (GitHub Security Lab researcher) and I will present (= record) the BlackHat talk. Our affiliation for that talk is listed as GitHub, but we will emphasise and acknowledge that this initiative was developed with support from OpenSSF (in particular, the security tooling working group).
2. During the broadcast, we will publish our work in public GitHub repositories in the newly-created ossf-cve-benchmark organisation.
   • The CVE metadata and tooling will live in https://github.com/ossf-cve-benchmark/ossf-cve-benchmark (license: MIT)
   • There'll be an additional repository for every CVE in the data set. Every such repository hosts the (open source) code of the code base affected by the CVE. It's basically a fork of the code base in question, frozen in time at the fix commit.
   • I've chosen the org name ossf-cve-benchmark to be in line with the https://github.com/ossf org name, but if people prefer openssf-cve-benchmark as an org name, that's absolutely fine. Just leave a comment.
   • While the repositories will remain private prior to the broadcast of the talk at BlackHat, we would of course like to refer to the repository/org name during the pre-recorded talk.

Very happy to discuss this further in comments below. If people are completely happy with the above, it'd be great if you could leave a 👍. Many thanks!

Everything else we can discuss in virtual person at the TAC meeting on the 17th!

[jenniferfernick]

FWIW: Love this - am fully in support of it as written. Looking forward to seeing the talk!

Curious also to learn more about future directions for the project, and to learn more about the current and future activities within the tooling group more generally. I suppose we'll learn some of this at the Town Hall! :)

[sj]

Thanks for the support everybody! Considering the many 👍 on my comment (including from members of TAC and Governing Board), we'll go full steam ahead.

I've just created a Team early-access in the ossf-cve-benchmark repo. You'll all receive invites to that, which should give you preview access to the repositories in that org.

[dlorenc]

Excellent!

@LindsayLF - do you know what logistics would have to happen to "officially" transfer this org over to LF/OSSF management? I think if the org is free we just need to add "thelinuxfoundation" as an OWNER on the org. Not sure what happens if we need billing.

[LindsayLF]

@dlorenc Yes, please add me as an owner @LindsayLF and I will add "thelinuxfoundation" in. Do you anticipate any fees in setting up this org? / If so, what would the cost be, and what billing info is needed?

[david-a-wheeler]

I haven't had time to delve into everything, but the basic idea looks very sound! Thanks for working on this!

[david-a-wheeler]

👍

[sj]

@LindsayLF: would you mind if we waited with adding extra owners and any formalities until we've safely recorded our Black Hat talk (deadline: Mon 16 Nov)? We can talk about the details at the TAC meeting on 17 November.

At this stage I don't anticipate this involving costs anytime soon.

[LindsayLF]

@sj Sure thing! Add me the week of 11/16, thanks!

[dlorenc]

@sj can @LindsayLF be added now?

Anything else needed to close this out?

[sj]

At the TAC meeting a week or two ago, we decided that we'd hold off with the transfer of ownership until after Black Hat has actually taken place (Wednesday next week). Hope that's OK with you too, @dlorenc!

[dlorenc]

Ah sure! Makes sense.

[LindsayLF]

That works for me! @sj

[mayakacz]

@sj Is this now done? Seems to be published here: https://github.com/ossf-cve-benchmark/ossf-cve-benchmark

[dlorenc]

Closing, this is done now.