revise security program for higher signal #789

Closed
chadwhitacre opened this Issue Aug 23, 2016 · 3 comments

Comments

Projects
None yet
1 participant
@chadwhitacre
Contributor

chadwhitacre commented Aug 23, 2016

Originally posted by @nashe at #773 (comment) ...


We should get the 300th report this week, time to make statistics and suggestions!

Since the bounty started in June 2015, we got 12.4% of "Resolved" reports (reports leading to a commit or more in our codebase), 15.5% of "N/A" (unrelated to our infrastructure or code, third-party products... we don't use this category anymore), 34.6% of "Informative" (not a vulnerability issue, risk to low to be a really risk…) and… 37.5% of "Duplicates" (some of the informative ones are even in fact duplicates). The reports categorized as "low-quality" ("Informative" + "N/A", even if we should count the duplicates of already public reports) increased since your (very good) post about the creation of Gratipay's bounty program.

First point: I find this amount of duplicates incredibly high. It's definitely related to our high delay to properly resolve issues and to the lack of information given in response of the reports we refuse and make publicly available. We need to address this by trying to fix it as soon the report is triaged. Handling reports takes time: don't close any hazardously formulated but valid reports, try to do the same steps, validate the find or not, reply personally to the researcher… enough reasons to avoid getting duplicates. We'll win time, make quicker fixes, lead to less duplicates, etc.

We get a lot of copied and pasted reports related to "Best practices" that have been reported (and not always awarded…) elsewhere. This kind of "Informative" reports belong more to our Github queue than HackerOne.

We don't attract the right researchers: it's rare to get a report originating from somebody with a positive Signal value. This mechanism is already a good indicator of the serious of the researcher, in addition the followings points:

  • the presence (or not) of copied/pasted part of the report from other public reports,
  • presence of steps to reproduce/demonstrate the vulnerability,
  • possible consequences of the report (exploitation scenario, even if it requires another vulnerability to happen).

Some researchers are often trying to make "easy" money by sending a lot of (often low-quality) reports, "just in case, because it was awarded elsewhere", even if it's not applicable. It's not what we want to receive, but that's up to us give them the opportunity to improve their reports and help them expand their skill set by giving detailed answers and not sending generic messages.

My suggestions are the following:

  • I will now try to systematically write summaries on publicly disclosed reports, in order to make the information more easily available. Regarding our CSRF mitigation method, often misunderstood, I went ahead and tried to explain the full reasoning behind this choice and offered one additional bounty as proof of good faith (only for this one).
  • Regarding the categories:
    • Start using again "N/A" for irrelevant reports (third-party, not a risk nor a bug). Since, everybody can make mistakes, it would be good to apply it only when the researcher already sent at least one irrelevant report in the last weeks or signs of "bad will" (not sure if this expression really exists in English…).
    • Keep "Informative" for best practices and things not seen as a risk, create a Github ticket, link the GH ticket in the report and close the HackerOne report.
  • Award bounty very quickly, but take time to coordinate a fix with the researcher. Involving him in the process will be beneficent for everybody. At least to ensure that the fix is correct and no bypass subsists. Taking time ≠ forgetting the report.
  • Offer to tweet a "thank you" message from @gratipay for important issues / good reports,
  • Offer to co-write a post on https://gratipay.news/ regarding the vulnerability finding (him) and fix (us) for most severe ones,
  • Offer a variable bonus in addition of the bounty to great reports (with PoC, active participation of the researcher in the resolution…),
  • Offer a bounty per vulnerability type rather than per "risk" (it's too generic). Twitter's policy is a good example (even if we can't afford the same amount ;-D). I'll make a PR for this.
  • Add all the domains (being discussed in #511).
  • Find a way to include software components managed by Gratipay (postgres.py?) that doesn't already have a separate bug bounty (aspen should get one soon?). Create a specific reward range for these reports.
  • Continue to focus the "0 New / 0 Triaged" on HackerOne. I still think it's possible!

Thoughts on theses (draft) ideas?

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Aug 23, 2016

Contributor

And from #782 (comment) ...

After taking a look at ownCloud's policy, I found:

Q: Where should I report bugs without security implication or hardening / best practise guidance?
A: Please report all non-security bugs as well as general hardening advice at https://github.com/owncloud/core. Rule of thumb: if it on it's own is not directly exploitable it is likely to be hardening.

I guess it's worth to tell the users to do it for our case too. In addition of my last week's suggestion,

Keep "Informative" for best practices and things not seen as a risk, create a Github ticket, link the GH ticket in the report and close the HackerOne report.

Contributor

chadwhitacre commented Aug 23, 2016

And from #782 (comment) ...

After taking a look at ownCloud's policy, I found:

Q: Where should I report bugs without security implication or hardening / best practise guidance?
A: Please report all non-security bugs as well as general hardening advice at https://github.com/owncloud/core. Rule of thumb: if it on it's own is not directly exploitable it is likely to be hardening.

I guess it's worth to tell the users to do it for our case too. In addition of my last week's suggestion,

Keep "Informative" for best practices and things not seen as a risk, create a Github ticket, link the GH ticket in the report and close the HackerOne report.

@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Sep 21, 2016

Contributor

I've discovered Google's list of non-qualifying reports. A quick skim suggests some overlap with our own backlog. One way to focus our security program would be to from google import nonvuln.

Contributor

chadwhitacre commented Sep 21, 2016

I've discovered Google's list of non-qualifying reports. A quick skim suggests some overlap with our own backlog. One way to focus our security program would be to from google import nonvuln.

@chadwhitacre chadwhitacre referenced this issue in gratipay/gratipay.com Dec 27, 2016

Closed

Full security report #4263

2 of 8 tasks complete
@chadwhitacre

This comment has been minimized.

Show comment
Hide comment
@chadwhitacre

chadwhitacre Aug 9, 2017

Contributor

@EdOverflow What's the status of this ticket relative to recent updates to our program? Anything we want to take from here or can we close?

Contributor

chadwhitacre commented Aug 9, 2017

@EdOverflow What's the status of this ticket relative to recent updates to our program? Anything we want to take from here or can we close?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment