Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interactive audit resolver #18

Open
wants to merge 4 commits into
base: latest
from

Conversation

Projects
None yet
6 participants
@naugtur
Copy link

naugtur commented Aug 5, 2018

`npm audit resolve` runs audit and iterates over all actions from the output with a prompt for a decision on each.
If fix is available, it's offered as one of the options to proceed. Other options include: ignore, remind later and delete.

All decisions are stored in `audit-resolv.json` file as key-value, where key is `${dependency path}|${advisory id}` and value is a structure specific to the resolution.

This comment has been minimized.

@ljharb

ljharb Aug 5, 2018

Why resolv instead of resolve?

This comment has been minimized.

@naugtur

naugtur Aug 5, 2018

Author

I don't have a strong opinion about this. Wrote it the way it is after some well known files like resolv.conf and didn't give it a second thought. Didn't have to choose between 'resolve' or 'resolved' or 'resolver' but now that I see you ask, I think audit-resolve.json is probably a better name.

### resolutions
- **fix** runs the fix proposed in audit action and marks the issue as fixed in `audit-resolv.json`. On each subsequent run of `npm audit resolve` if the issue comes up again, the user is also warned that the problem was supposed to be fixed already.
- **ignore** marks the issue as ignored and `npm audit` will no longer fail because of it (but should display a single line warning about the issue being ignored). If a new vulnerability is found in the same dependency, it does not get ignored. If another dependency is installed with the same vulnerability advisory id it is not ignored. If the same package is installed as a dependency to another dependency (different path) it is not ignored.
- **postpone** marks the issue as postponed to a timestamp 24h in the future. Instead of ignoring an issue permanently just to make a build pass, one can postpone it when in rush, but make it show up as a CI faiure on the next working day.

This comment has been minimized.

@ljharb

ljharb Aug 5, 2018

The next working day is often more than 24 hours later; is this just meant as an example?

This comment has been minimized.

@naugtur

naugtur Aug 5, 2018

Author

Yes. I didn't intend it to count in working days.

If noone's at work and the build turns red, does it make a sound?


- `package-lock.json` is not a good option for storing resolutions because even if we overlook the fact that it'd be adding another purpose to a single-purpose file, the community at large is still used to removing and regenerating it often.
- `package.json` is not a good option, because it is already overloaded for so many functionalities. The output of resolutions could be lengthy and clutter the file. User will need the resolution information much less often than they look in the package.json file. ALso, resolution file is not meant to be edited by hand.
- `.npmrc` is not a good option because it doesn't live in the repository and being a dotfile is easy to miss, so a developer could overlook it affecting the audit. Also using it for the purpose of storing resolutions to project specific issues seems counter-intuitive.

This comment has been minimized.

@ljharb

ljharb Aug 5, 2018

npmrc could live in the repo, it just also might not.

This comment has been minimized.

@naugtur

naugtur Aug 5, 2018

Author

I'm aware of that, but we probably wouldn't want to make an exception and accept global resolutions too. That could lead to hard to debug problems.

This comment has been minimized.

@ljharb

ljharb Aug 5, 2018

Totally agree; just hoping to clarify the npmrc explanation.

Why is postpone useful at all? It's designed to build a secure development culture where one didn't yet form. Without it, a developer under time pressure would mark an issue as ignored with intention to un-mark it later.
While shipping with a known vulnerability is a bad practice, NPM's mission with the community should be to empower people to build more secure products and trust their skill and understanding of their project's particular needs. We should also aspire to help teams introduce more secure workflows effortlessly, so letting a build pass without risking compromising security long-term is a win.

Remove is only useful as a convenience. Imagine a developer introducing `npm audit` and having to go through tens of issues. If they notice one of the first issues is caused by a dependency they no longer use, instead of remembering to clean it up later, they can choose this option.

This comment has been minimized.

@ljharb

ljharb Aug 5, 2018

Presumably this could also narrow the list of action items, if some of them are from transitive deps of the removed item?

This comment has been minimized.

@naugtur

naugtur Aug 5, 2018

Author

In my experience audit grouped these together nicely, so when I choose remove it's unlikely the dependency or any of its children would come up again.

But we could easily run audit again after the remove command and thanks to audit-resolv being written after each decision, it'd automatically continue from the same point on the list.

@naugtur naugtur referenced this pull request Aug 5, 2018

Closed

Add npm audit resolve command #10

2 of 3 tasks complete
@CADBOT

This comment has been minimized.

Copy link

CADBOT commented Sep 28, 2018

What's the latest on this? Would be great to have something to replace .nsprc/exceptions from nsp.

@naugtur

This comment has been minimized.

Copy link
Author

naugtur commented Sep 28, 2018

I don't really know what's next other than moving my repo to npm org on GitHub.
There's no disagreement on anything here.

I have more free time than usual for the next few days so I'd be happy to move this forward too.

But you can use it now. Install npm-audit-resolver. I'm using it at work with my team for a while now

### resolutions
- **fix** runs the fix proposed in audit action and marks the issue as fixed in `audit-resolv.json`. On each subsequent run of `npm audit resolve` if the issue comes up again, the user is also warned that the problem was supposed to be fixed already.
- **ignore** marks the issue as ignored and `npm audit` will no longer fail because of it (but should display a single line warning about the issue being ignored). If a new vulnerability is found in the same dependency, it does not get ignored. If another dependency is installed with the same vulnerability advisory id it is not ignored. If the same package is installed as a dependency to another dependency (different path) it is not ignored.
- **postpone** marks the issue as postponed to a timestamp 24h in the future. Instead of ignoring an issue permanently just to make a build pass, one can postpone it when in rush, but make it show up as a CI faiure on the next working day.

This comment has been minimized.

@evilpacket

evilpacket Nov 6, 2018

How do we handle the situation in which somebody keeps hitting snooze (postpone) on a vulnerability. Do we really have to rely on git commit history to see this? Should we track all history not just the current state / decisions? This adds complexity that we maybe don't need but also could be a useful audit trail?

This comment has been minimized.

@naugtur

naugtur Nov 7, 2018

Author

I thought about a few cases of audit trail with similar conclusions. IMHO the benefit of audit-resolv file's simplicity outweighs the inconvenience of using git history. Also, people are quite familiar with git so the less we do on our end, the less learning they have to do.

I'd like to avoid the file growing to overwhelming lengths, but other than that I'm fine with adding more capability to this file format if it's useful for the server side ignores etc.

This comment has been minimized.

@naugtur

naugtur Dec 4, 2018

Author

Leaving only the 24h option it should be annoying enough for most reasonable people to eventually fix. :)
Unless someone gets a job postponing vulnerabilities every morning before the team comes in.

With the audit-resolve format change I just pushed we can also add a count${dependency path}er. I'd consider that a nice touch, not the core functionality though.

`npm audit resolve` runs audit and iterates over all actions from the output with a prompt for a decision on each.
If fix is available, it's offered as one of the options to proceed. Other options include: ignore, remind later and delete.

All decisions are stored in `audit-resolv.json` file as key-value, where key is `${dependency path}|${advisory id}` and value is a structure specific to the resolution.

This comment has been minimized.

@evilpacket

evilpacket Nov 6, 2018

Can you document the format of this file? Also adding in the option to have a reason for each decision.

This comment has been minimized.

@evilpacket

evilpacket Nov 6, 2018

Another though is that the audit-resolve.json file should not be published as part of a module publication. We will want to assert exceptions outside of the package.json and update those exceptions outside of the normal publication lifecycle.

This comment has been minimized.

@naugtur

naugtur Nov 7, 2018

Author

Will update the RFC to contain the file format spec

This comment has been minimized.

@naugtur

naugtur Dec 4, 2018

Author

only the file next to top level package.json should be taken into account at all (and that's how it's implemented in npm-audit-resolver)


Remove is only useful as a convenience. Imagine a developer introducing `npm audit` and having to go through tens of issues. If they notice one of the first issues is caused by a dependency they no longer use, instead of remembering to clean it up later, they can choose this option.

Investigate option has a lot of potential for the future where NPM could do things ranging from linking to resources on mitigation to generating local fixes or providing a marketplace of companies offering services fixing the issue for customers (business potential for NPM).

This comment has been minimized.

@evilpacket

evilpacket Nov 6, 2018

I would make investigate, beyond simply telling a user where to submit a PR if possible, as a future phase of implementation and scratch hypotheticals / future vision from the RFC.

So to be clear, define investigate as you have it now vs the potential other things in the future as that's useful for things that do not have fixes / require manual review.

@evilpacket

This comment has been minimized.

Copy link

evilpacket commented Nov 6, 2018

In general I'm very excited about the potential of the npm audit resolve feature. I think some work needs to be done to really make it a good interactive experience presenting the information in the best way possible for users.

@zkat

This comment has been minimized.

Copy link
Member

zkat commented Feb 7, 2019

(Heads up, we are tracking this internally and it's something we intend to get to, but I can't give you an exact timeline right now. Thanks for your patience.)

@naugtur

This comment has been minimized.

Copy link
Author

naugtur commented Feb 11, 2019

If you want to consider one bigger push to move this forward, my family is out for winter vacation 25-28Feb while I stay at work, so I'll have unusually many hours in the evenings in that period.

@ersel

This comment has been minimized.

Copy link

ersel commented Feb 19, 2019

This would be an awesome addition to npm CLI. We used to run npm audit on CI with a custom script outlined here.

Recently we've switched to @naugtur 's npm-resolve-audit module by installing it as a dev dependency and adding a script to run node node_modules/npm-audit-resolver/check.js

Thanks for building this tool @naugtur !

@naugtur

This comment has been minimized.

Copy link
Author

naugtur commented Feb 19, 2019

@ersel if you're running it inside an npm script, you can use it as a command check-audit and it resolves from dependencies. Otherwise I'd go with npx -p npm-audit-resolver check-audit - feels a little cleaner and you don't need to install resolver as a dependency - npx will download and cache it. (npx is shipped with npm so no need to install it)

Check out my slides on using audit in CI (and other things) https://naugtur.pl/pres3/securedev/#/18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.