New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Security service #176

Open
terah opened this Issue Jan 26, 2017 · 16 comments

Comments

Projects
None yet
8 participants
@terah

terah commented Jan 26, 2017

Hi,

I just read https://blog.gopheracademy.com/advent-2016/saga-go-dependency-management/ and thought I'd add some ideas.

Sorry if this has been discussed before or is outside the scope of this project.

PHP has this security advisory service for packages: https://github.com/FriendsOfPHP/security-advisories

It would be useful for your dep tool to tell you that certain deps have known security issues.

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Jan 27, 2017

Member

It's a great suggestion, and not out of scope at all! We definitely need something like this.

Up until now, I've been more focused on the mechanics of dealing with insecure versions within the solver itself (I intend to model it after Cargo's yanking system). I haven't done a ton of thinking about what the service would look like that provides the information - obviously, we don't have anything like a crates.io to rely on for it right now.

Frankly, I'd love it if someone volunteered to step up and take some ownership over working out a design and API for such a system. hint hint 😉

Member

sdboyer commented Jan 27, 2017

It's a great suggestion, and not out of scope at all! We definitely need something like this.

Up until now, I've been more focused on the mechanics of dealing with insecure versions within the solver itself (I intend to model it after Cargo's yanking system). I haven't done a ton of thinking about what the service would look like that provides the information - obviously, we don't have anything like a crates.io to rely on for it right now.

Frankly, I'd love it if someone volunteered to step up and take some ownership over working out a design and API for such a system. hint hint 😉

@terah

This comment has been minimized.

Show comment
Hide comment
@terah

terah Jan 29, 2017

Hi Sam,

This is a project I'd enjoy getting my teeth into however I'm over committed for the next couple of months and it'd be irresponsible for me to take on another project right now. I will follow golang/dep but if someone else wants to pick up this component of it then go right ahead.

terah commented Jan 29, 2017

Hi Sam,

This is a project I'd enjoy getting my teeth into however I'm over committed for the next couple of months and it'd be irresponsible for me to take on another project right now. I will follow golang/dep but if someone else wants to pick up this component of it then go right ahead.

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Jan 30, 2017

Member

of course, i totally understand. i'd much rather people be honest about what their commitments are than overextend. the latter just ends up putting me in an awkward position!

hopefully we'll have it well underway by a couple months from now, but if not, well :)

Member

sdboyer commented Jan 30, 2017

of course, i totally understand. i'd much rather people be honest about what their commitments are than overextend. the latter just ends up putting me in an awkward position!

hopefully we'll have it well underway by a couple months from now, but if not, well :)

@Alex-Wauters

This comment has been minimized.

Show comment
Hide comment
@Alex-Wauters

Alex-Wauters Feb 17, 2017

For similar examples, node has https://nodesecurity.io/ and for Java there's Owasp Dependency Checker. The latter is quite cumbersome as it needs to download the entire NIST NVD Data Feeds.

Alex-Wauters commented Feb 17, 2017

For similar examples, node has https://nodesecurity.io/ and for Java there's Owasp Dependency Checker. The latter is quite cumbersome as it needs to download the entire NIST NVD Data Feeds.

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany Feb 17, 2017

Contributor

So what would this be? A Go-specific list of CVEs for version tags that dep caches locally and uses when resolving dependencies? Essentially an exclusion list with a specific purpose and error messages.

Something like that could provide the equivalent behaviour of a yanked package, but without having a central repository for the actual code.

Perhaps not entirely related, there are a few security scanning tools for Go such as:

Contributor

nathany commented Feb 17, 2017

So what would this be? A Go-specific list of CVEs for version tags that dep caches locally and uses when resolving dependencies? Essentially an exclusion list with a specific purpose and error messages.

Something like that could provide the equivalent behaviour of a yanked package, but without having a central repository for the actual code.

Perhaps not entirely related, there are a few security scanning tools for Go such as:

@Alex-Wauters

This comment has been minimized.

Show comment
Hide comment
@Alex-Wauters

Alex-Wauters Feb 18, 2017

Another option which scanners such as GoASTScanner could include is to provide an api which, for a list of dependencies, can provide which versions are insecure and link to the relevant advisories. Someone does have to maintain the highly available list (as builds will depend on it) & ensure it is secure (to avoid DoS by including every package/version).

What the previously mentioned examples don't do and which I think should definitely be included in a Go dep scanner; is to check the version of Go itself (example: CVE-2016-3959). If go dep does not yet provide a standard way on how to define which version of Go should be used during the build (every PaaS will have its own definition on how to specify it, but if there were a standard way they could rely on it instead) it would be a good idea to add it (similar to the engine field in package.json for node).

Alex-Wauters commented Feb 18, 2017

Another option which scanners such as GoASTScanner could include is to provide an api which, for a list of dependencies, can provide which versions are insecure and link to the relevant advisories. Someone does have to maintain the highly available list (as builds will depend on it) & ensure it is secure (to avoid DoS by including every package/version).

What the previously mentioned examples don't do and which I think should definitely be included in a Go dep scanner; is to check the version of Go itself (example: CVE-2016-3959). If go dep does not yet provide a standard way on how to define which version of Go should be used during the build (every PaaS will have its own definition on how to specify it, but if there were a standard way they could rely on it instead) it would be a good idea to add it (similar to the engine field in package.json for node).

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Feb 18, 2017

Member

What the previously mentioned examples don't do and which I think should definitely be included in a Go dep scanner; is to check the version of Go itself (example: CVE-2016-3959)

There's a similar need for this independent of the security considerations - simply, being able to indicate (or have the tool infer) the version of Go required by a project. Ideally, I'd like to see us deal with both of these at once.

If we treat them together, though, then it needs its own issue, as the handling of the toolchain version information itself is orthogonal to the security service.

Member

sdboyer commented Feb 18, 2017

What the previously mentioned examples don't do and which I think should definitely be included in a Go dep scanner; is to check the version of Go itself (example: CVE-2016-3959)

There's a similar need for this independent of the security considerations - simply, being able to indicate (or have the tool infer) the version of Go required by a project. Ideally, I'd like to see us deal with both of these at once.

If we treat them together, though, then it needs its own issue, as the handling of the toolchain version information itself is orthogonal to the security service.

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Feb 18, 2017

Member

So what would this be? A Go-specific list of CVEs for version tags that dep caches locally and uses when resolving dependencies? Essentially an exclusion list with a specific purpose and error messages.

Something like that, yeah. I'm more inclined towards an HA service.

Member

sdboyer commented Feb 18, 2017

So what would this be? A Go-specific list of CVEs for version tags that dep caches locally and uses when resolving dependencies? Essentially an exclusion list with a specific purpose and error messages.

Something like that, yeah. I'm more inclined towards an HA service.

@nathany

This comment has been minimized.

Show comment
Hide comment
@nathany

nathany Mar 3, 2017

Contributor

build systems like Maven can not be relied upon to evict old versions.
https://opensource.googleblog.com/2017/03/operation-rosehub.html?m=1

Contributor

nathany commented Mar 3, 2017

build systems like Maven can not be relied upon to evict old versions.
https://opensource.googleblog.com/2017/03/operation-rosehub.html?m=1

@sdboyer

This comment has been minimized.

Show comment
Hide comment
@sdboyer

sdboyer Mar 3, 2017

Member

From the link she linked to in that post, it seems like we shouldn't have the "lurking in transitive deps" problem if we design this right.

Member

sdboyer commented Mar 3, 2017

From the link she linked to in that post, it seems like we shouldn't have the "lurking in transitive deps" problem if we design this right.

@zacps

This comment has been minimized.

Show comment
Hide comment
@zacps

zacps Mar 21, 2017

I'd be interested in helping with this.

Trying to brake down the steps we need to do:

  1. Resolve package versions
  2. Query a service to see if the resolved versions are secure.
  3. Refuse to perform operations until the resolved packages are secure (bypass through a flag)

The first issue I can think of is when do this check. Obviously this has to be done every-time we resolve/update. Ideally a command like dep status would also check this. However I think this would make it too slow in normal use. Adding another command to check this is just a recipe for it being forgotten until the the dev updates dependencies. Might not be a problem with git hooks or CI though.

If/when we integrate with the go tool we could refuse to go build with insecure deps but I don't think a check at build is ideal either.


In terms of the service there are a few obvious options:

  • Use the CVE list
  • Use something controlled through version control, like git tags
  • Use our own registry that could be updated by package authors or automatically from CVEs, git tags, or both

zacps commented Mar 21, 2017

I'd be interested in helping with this.

Trying to brake down the steps we need to do:

  1. Resolve package versions
  2. Query a service to see if the resolved versions are secure.
  3. Refuse to perform operations until the resolved packages are secure (bypass through a flag)

The first issue I can think of is when do this check. Obviously this has to be done every-time we resolve/update. Ideally a command like dep status would also check this. However I think this would make it too slow in normal use. Adding another command to check this is just a recipe for it being forgotten until the the dev updates dependencies. Might not be a problem with git hooks or CI though.

If/when we integrate with the go tool we could refuse to go build with insecure deps but I don't think a check at build is ideal either.


In terms of the service there are a few obvious options:

  • Use the CVE list
  • Use something controlled through version control, like git tags
  • Use our own registry that could be updated by package authors or automatically from CVEs, git tags, or both
@Alex-Wauters

This comment has been minimized.

Show comment
Hide comment
@Alex-Wauters

Alex-Wauters Mar 21, 2017

I would suggest the own registry than the CVE database (approach made by OWASP Dependency Checker but requires a large download to set-up, not ideal for from-scratch builds).

A separate command is useful when you want to validate a set-up without building. For example, scanning done by centralized security teams on all team repositories.

I have made a preliminary design on https://docs.google.com/document/d/1qJJTBNUEngzG1B-mgD1uL0CC549-gVrEhmmYrHvJW6M/ ; comments and edits are welcome!

Alex-Wauters commented Mar 21, 2017

I would suggest the own registry than the CVE database (approach made by OWASP Dependency Checker but requires a large download to set-up, not ideal for from-scratch builds).

A separate command is useful when you want to validate a set-up without building. For example, scanning done by centralized security teams on all team repositories.

I have made a preliminary design on https://docs.google.com/document/d/1qJJTBNUEngzG1B-mgD1uL0CC549-gVrEhmmYrHvJW6M/ ; comments and edits are welcome!

@zacps

This comment has been minimized.

Show comment
Hide comment
@zacps

zacps Mar 22, 2017

zacps commented Mar 22, 2017

@Alex-Wauters

This comment has been minimized.

Show comment
Hide comment
@Alex-Wauters

Alex-Wauters Mar 23, 2017

Elaborating further on design, including the process steps which are now left empty. I'd also like to gather some feedback to test the assumptions made, specifically regarding up or downgrading: e.g. an update is not available for a vulnerable dependency and the user must be guided to point to the correct revisionId and set the advisory to mitigated.

The steps which the solver should make are also missing, the document only mentions the stand-alone command and how it defines a dependency as vulnerable or mitigated (requiring semver to be used or specifically denoting an advisory as mitigated by the user) for now.

Alex-Wauters commented Mar 23, 2017

Elaborating further on design, including the process steps which are now left empty. I'd also like to gather some feedback to test the assumptions made, specifically regarding up or downgrading: e.g. an update is not available for a vulnerable dependency and the user must be guided to point to the correct revisionId and set the advisory to mitigated.

The steps which the solver should make are also missing, the document only mentions the stand-alone command and how it defines a dependency as vulnerable or mitigated (requiring semver to be used or specifically denoting an advisory as mitigated by the user) for now.

kris-nova pushed a commit to kris-nova/dep that referenced this issue Apr 21, 2017

Don't try to pop dep when it's the root
Project-level cycles are fine, but we can't apply the same logic to
the root project as non-root projects when cleaning up the selection
while backtracking. Fixes golang#176.
@Zanadar

This comment has been minimized.

Show comment
Hide comment
@Zanadar

Zanadar Jan 15, 2018

Contributor

@Alex-Wauters I was browsing through dep issues and this caught my eye. I read through the doc you and made some edits and adding some thoughts and suggestion (sorry I kind of went a little buck-wild). Hopefully I can stir the pot a bit and get some traction on this issue because I think its super important.

Contributor

Zanadar commented Jan 15, 2018

@Alex-Wauters I was browsing through dep issues and this caught my eye. I read through the doc you and made some edits and adding some thoughts and suggestion (sorry I kind of went a little buck-wild). Hopefully I can stir the pot a bit and get some traction on this issue because I think its super important.

@jasinner

This comment has been minimized.

Show comment
Hide comment
@jasinner

jasinner Jun 18, 2018

I think a list of security audit contact people makes sense rather than consuming what NIST produce piecemeal. A Go dependency will rarely have an official CPE which makes identifying Go dependencies in NIST reports error prone.
OWASP dependency checker tries makes the mapping of library to CPE less error prone, so perhaps we could consider adding Go dependency support to OWASP Dependency Check to setup an automated feed into the security registry as described by @Alex-Wauters.
Starting with a 'contact us' form, or mailing list for reporting security issues in Go dependencies, seems like a good way to go. These people can use the security registry to submit reports. This is going to be a lot of work for the group, but starting with this base we can build out a Go Security service which could potentially be a viable commercial entity in it's own right, ala NodeSecurity which was eventually purcahsed by NPM.

jasinner commented Jun 18, 2018

I think a list of security audit contact people makes sense rather than consuming what NIST produce piecemeal. A Go dependency will rarely have an official CPE which makes identifying Go dependencies in NIST reports error prone.
OWASP dependency checker tries makes the mapping of library to CPE less error prone, so perhaps we could consider adding Go dependency support to OWASP Dependency Check to setup an automated feed into the security registry as described by @Alex-Wauters.
Starting with a 'contact us' form, or mailing list for reporting security issues in Go dependencies, seems like a good way to go. These people can use the security registry to submit reports. This is going to be a lot of work for the group, but starting with this base we can build out a Go Security service which could potentially be a viable commercial entity in it's own right, ala NodeSecurity which was eventually purcahsed by NPM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment