Security update needed: mongodb

[abea]

npm audit is throwing high severity warnings for mongodb. It looks like we would need to update to mongodb@3.3.3 to resolve those, and we're currently on "mongodb": "^2.2.36". Clearly possible breaking changes.

We probably need to update apostrophe-db-mongo-3-driver as well, though easier there.

[boutell]

Please see https://github.com/apostrophecms/apostrophe/blob/affce2c5b43b34888bc5bd421d3afabf4f11e793/CHANGELOG.md#2981-2019-10-21

[abea]

Do we know the new warnings are the same thing? If they're whitelisted why would they still be flagged? The issue at the moment is that high severity warnings are being displayed, at the very least impacting comfort with the package.

[abea]

So I understand now we have the whitelist in Github for that audit warning. This was raised by someone in the forums, so I worry it's going to effect faith in the project.

[boutell]

For some reason I can't find it in the forum. I agree it is a problem. There was an npm issue about it but it didn't survive the transition to the new npm community forums. A new rfc ticket should be opened: https://github.com/npm/rfcs/issues It should be possible for module creators to flag issues as irrelevant with a justification ("we don't use feature X").

…

On Mon, Nov 11, 2019 at 12:11 PM Alex Bea ***@***.***> wrote: So I understand now we have the whitelist in Github for that audit warning. This was raised by someone in the forums, so I worry it's going to effect faith in the project. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2079?email_source=notifications&email_token=AAAH27KUTOETUE2237GMAYDQTGG4NA5CNFSM4JLXFDO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDXPJGQ#issuecomment-552531098>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAH27JKBJOBSJFR23PSHG3QTGG4NANCNFSM4JLXFDOQ> .

-- THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his

[boutell]

I opened an RFC on this. It's an upstream issue we can't fix here (short of a bc break).

[bobclewell]

@boutell Am I reading this right? It was whitelisted to hide it which worked for a certain version, but then un-whitelisted again?

[abea]

I think the whitelist is only for Github. So Github has its own security warnings, which are suppressed by whitelisting. There's no such whitelisting for NPM, so that warning remains.

[boutell]

Yes, this is a bad developer relations problem. cc @agilbert @localghost443

There is no whitelist for github and no whitelist for npm audit... there's just a whitelist we created for our own "npm run audit" project level npm task, which generates a JSON report from npm audit, removes anything on our own whitelist, and decides whether there is still a problem. This is great for our own use in regression testing but completely unsatisfying psychologically for developers, who are still being scared off every time they "npm install" for the first time.

These are the possible resolutions:

• Convince npm to implement a safelist feature (as discussed with them by many parties, it may happen)
• Write a 100% 2.x driver compatible wrapper for the 3.x driver, and use that in apostrophe 2.x. There are backwards incompatible differences, we would have to wrap them all, or enough, and use this audit failure as justification for small frustrations resulting from our actions
• Finish apostrophe 3.x, making this an issue only for the much smaller audience that will be doing long term 2.x support and likely be patient with explanations like this one

[boutell]

(We have already written an optional apostrophe-mongo-3-driver module, but it requires changes to the developer's code because it doesn't try to paper over all of the incompatibilities. Also having it present in your project won't stifle the audit error because both versions of the module are then installed.)

[stuartromanek]

@boutell can you ballpark the effort on point 2: 100% 2.x driver compatible wrapper for the 3.x driver ?

[boutell]

So, I had a nice long chat in 2018 about this with Matt Broadstone, the lead engineer of MongoDB. You can find it here: #1474 In a nutshell: I wanted to monkeypatch methods like find() so they would still take a projection as a second argument. But I couldn't because the *internals* of the mongodb 3.x driver call find() and they call it expecting the 3.x behavior. So I gave up on that approach, and we made apostrophe-mongo-3-driver, which requires the developer to be mindful of what version they're using and does NOT fix the npm audit warning (2.x is still a dependency of apostrophe). Now, I do think an alternative might exist: rather than patching the collection object, return an entirely new object (decorator pattern) in which I reimplement the 2.x methods in terms of the 3.x methods. So when they made internal calls they'd see the internal methods, but our code would see the wrapper. That is a viable idea. I would estimate it at three days of full time effort on my part to arrive at something that most/all of our sites would accept and operate acceptably with. ... Regarding how this happened: Apostrophe 2.x is an LTS (Long Term Support) release. It depends on a lot of stuff, and there's no chance you'll ever get a perfect match in the long term support plans of all the things your large system depends upon. We've been able, so far, to pull it off without too much pain because the things that ended support were taken up by other people (Sails took up Lodash 3.x support) or, with minimal effort required, by us (Nunjucks 2.x is supported by us as @apostrophecms/nunjucks)... Uh... OH YEAH ... There's an easier solution, for now. We can fork the mongodb 2.x driver, like we did nunjucks (and like sails did lodash), and fix the reported security issue. And then we should also watch for mongodb 2.x driver bug reports going forward. However, there may come a point — as Matt warned — where a future version of MongoDB server will just refuse to talk to the 2.x driver. And then we would have to do some harder work on it, or wrap the 3.x driver. So, we need to decide which way to go forward: wrap 3.x in a 2.x compatible form (bigger near term effort, lots of long term peace of mind), or fork the 2.x driver and just fix the vulnerability (totally OK for now, and will most likely carry us into Apostrophe 3.x stable release era).

…

On Mon, Nov 25, 2019 at 9:35 PM Stuart Romanek ***@***.***> wrote: @boutell <https://github.com/boutell> can you ballpark the effort on #2 <#2>: 100% 2.x driver compatible wrapper for the 3.x driver ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2079?email_source=notifications&email_token=AAAH27IZDXNEAVXKFJW2VQDQVSDQXA5CNFSM4JLXFDO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFEP5WA#issuecomment-558431960>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAH27LMNYSQQNZE6OHUBRDQVSDQXANCNFSM4JLXFDOQ> .

-- THOMAS BOUTELL | CHIEF TECHNOLOGY OFFICER APOSTROPHECMS | apostrophecms.com | he/him/his

[boutell]

This got done, with emulate-mongo-2-driver.