First cut at security audit guidelines #125
Conversation
|
closing this PR, content included in #140 |
security audit => assessment
|
figured out how to submit PR to @JustinCappos original PR, so re-opened this and closed #140 which is now redundant |
simplified readme with high level goals and link to guide
remove SAFE references, add outline
adjust README content for improved navigation
| * Vulnerability Response Process. Who is responsible for responding to a report. What is the reporting process? How would you respond? | ||
| * Ecosystem. How does your software fit into the cloud native ecosystem? (e.g. Flibber is integrated with both Flocker and Noodles which covers virtualization for 80% of cloud users. So, our small number of "users" actually represents very | ||
| wide usage across the ecosytem since every virtual instance uses Flibber | ||
| encryption by default.) |
There was a problem hiding this comment.
Within the ecosystem, is there a concern of delegation of security responsibilities from one project to another and vice versa? Perhaps a project is relying on security mechanisms of another CNCF technology which has not been reviewed yet? How would this affect the security posture of the project? I suppose that this would be a rare case that can be handled whenever it comes up.
Side comment: Along the lines of a point mentioned in the earlier section about AES/sha256, I like the idea of a list of technologies and algorithms/libraries/projects with their security assumptions accepted. This could help in getting project-leads to reduce the assumptions they are making and/or default to more well established security libraries/components.
There was a problem hiding this comment.
+1 on explicitly listing the technologies and assumptions. This would also inform mapping which domains and subject matter experts are needed. even more important it might highlight areas where stuff should NOT have been invented (we rolled our own symmetric encryption algorithm instead of using X cuz we thought X was too slow...oops we never considered side channel attacks)
There was a problem hiding this comment.
This is covered by the assessment, but could benefit from more detailed documentation of effective techniques. I've opened an issue on that, if anyone wants to elaborate further on the need: #155
| of security audits for a software project they manage. Note that it is | ||
| encouraged to have participation (shadowing) from participants that are not | ||
| yet qualified to help them gain the necessary skills to be a reviewer | ||
| in the future. |
There was a problem hiding this comment.
Should we add something about managing conflict of interest of the reviewer/project?
|
|
||
| ## Time and effort | ||
|
|
||
| The level of effort for the reviewers is expected to be 10 hours. |
There was a problem hiding this comment.
Should we require (encourage) folks to log their time so we can actually benchmark this and track and normalize (ie # hours per SLOC or feature count, etc?)
There was a problem hiding this comment.
I think this is mostly to set expectations about the level of commitment. Some reviewers will want to spend more time, based on their interest in the project or level of experience. If reviewers find that the time estimate is way off, then I think running an experiment where people track time could be helpful. For now, I think we need a bit more experience with the process first.
| organizations. The ideal reviewer should also have been the recipient | ||
| of security audits for a software project they manage. Note that it is | ||
| encouraged to have participation (shadowing) from participants that are not | ||
| yet qualified to help them gain the necessary skills to be a reviewer |
There was a problem hiding this comment.
Perhaps we should explicitly catalog what those skills are? Not only is that useful for those hoping to participate, but it lends itself to segregating the work and parallel processing. And not all experts will be experts in everything. for example I have deep crypto review expertise, but very little linux kernel review expertise.
There was a problem hiding this comment.
There's not a clear consensus on what this is, so we decided to do a number of assessments before adding more detail -- which i think we set as 6-8 assessments, but could revisit earlier if it causes friction in the process or there are specific recommendations that help the process
@rficcaglia -- thanks for chiming in on #142 -- feel free to add more thoughts / ideas there on this topic, and we'll revisit at a future checkpoint
| there are hidden assumptions, underestimated risk, or design issues that | ||
| harm the security of the project. It may be useful to reach out to | ||
| community members to understand the answers to some questions, especially | ||
| involving deployment scenarios and the impact of attacks. |
There was a problem hiding this comment.
Perhaps there should be a formal "request for comment" process where domain experts are solicited and there is a waiting period to allow those experts to weigh in? eg 10 days? perhaps we should put the burden on the project under review to nominate 2-3 non-affiliated subject matter experts to participate and advise the reviewer team?
There was a problem hiding this comment.
This makes sense but I think might be a little too heavyweight for the initial iterations of this process. We will grow and add to this over time in the most helpful ways possible and this certainly could be one of those directions.
There was a problem hiding this comment.
that is specified in outline.md "Project posts their document to the group mailing list, allowing at least
one week for review before presenting to the WG"
| @@ -0,0 +1,31 @@ | |||
| # Security Reviewers | |||
|
|
|||
| Reviewers will try to understand the | |||
There was a problem hiding this comment.
I recommend we have at least N (N>=2, 3?) reviewers to ensure diverse and possibly even conflicting opinions, lest there is too much dependence on one or two reviewers? it also encourages expansion of the network of reviewers so we cultivate a large, diverse experience base.
There was a problem hiding this comment.
specified in outline.md " assigned to lead security reviewer who
will recruit a second reviewer"
| * Vulnerability Response Process. Who is responsible for responding to a report. What is the reporting process? How would you respond? | ||
| * Ecosystem. How does your software fit into the cloud native ecosystem? (e.g. Flibber is integrated with both Flocker and Noodles which covers virtualization for 80% of cloud users. So, our small number of "users" actually represents very | ||
| wide usage across the ecosytem since every virtual instance uses Flibber | ||
| encryption by default.) |
There was a problem hiding this comment.
+1 on explicitly listing the technologies and assumptions. This would also inform mapping which domains and subject matter experts are needed. even more important it might highlight areas where stuff should NOT have been invented (we rolled our own symmetric encryption algorithm instead of using X cuz we thought X was too slow...oops we never considered side channel attacks)
| * Internal. How do team members communicate with each other? | ||
| * Inbound. How do users or prospective users communicate with the team? | ||
| * Outbound. How do you communicate with your users? (e.g. flibble-announce@ mailing list) | ||
| * Vulnerability Response Process. Who is responsible for responding to a report. What is the reporting process? How would you respond? |
There was a problem hiding this comment.
at a minimum shouldn't this SIG be informed "formally"? i suggest there is a mechanism (GHI, commit to some markdown file, etc) for explicit capture of all known vulns over time.
| how keys are likely to be managed and stored. | ||
|
|
||
| ## Project design | ||
| * Design. A description of the system design that discusses how it works. |
There was a problem hiding this comment.
is there any interest in promoting/recommending formal design specification ,eg TLA+ and such?
There was a problem hiding this comment.
This seems great if a project has it. I don't think we want to require this at the outset.
|
|
||
| ## Additional Process Notes | ||
|
|
||
| Iteration is expected; however, we expect quick turnaround (at most a week). |
There was a problem hiding this comment.
Is there a desire to have an explicit followup process, ie an annual review? I think the notion that a single point in time review ignores the reality that threats change over time. someone using this review process as evidence of "trustworthiness" can be lulled into thinking something is "safe" since it was reviewed N years ago.
There was a problem hiding this comment.
yes! added as open issue: #152 (so we can get this PR merged for initial reviews that are currently blocked, then add more detail on annual review process)
| cloud native ecosystem | ||
|
|
||
| Due to the nature and timeframe for the analysis, *this review is not meant | ||
| to subsume the need for a professional security audit of the code*. Audits |
There was a problem hiding this comment.
I agree with separating the concerns here, but I think there would be huge benefit in providing open source code audits that do not rely on vendors, and encourage transparent and diverse methodologies. Perhaps an open code audit sub-SIG? maybe put the burden on the USERS to volunteer to participate and collectively encourage more active engagement with the user community instead of just passively harvesting value from open source?
I think open source code audit would also have the side benefit of providing real world education to a larger group of engineers on how to code securely (and how to review code for security). this is a skill that should really be part of every developer's forte, and the more we can model and inculcate secure coding skills, the better for all of us.
There was a problem hiding this comment.
There's a different CNCF audit process -- we should link that here (or open an issue if it's not yet documented).
There was a problem hiding this comment.
I believe I have captured all of the actionable comments as issues tagged with assessment
|
I checked in with @JustinCappos who originally authored this PR based on prior security reviews that were presented to and accepted by the CNCF TOC. Based on his feedback (and prior agreement in one of the live meetings a few weeks ago), I'm going to merge this PR and we'll address improvements with additional PRs, as needed or with data from retrospective checkpoint after 6-8 assessments. |
First cut at security audit guidelines
No description provided.