fix: Pin npm and core

[dawsontoth]

I introduced a new npm run core:sync script to make core sync easier, and to take over some of what npm run package was doing with dependencies.

With that new core sync, we now:

1. Update to the latest commit from core.
2. Grab the lock file from core.
3. Grab the overrides and optionalDependencies from core.
4. npm install after that, which will update the lockfile with our pro particulars, but leave the dependency versions untouched.

That means the package process is stable, with regards to versions:

1. We can npm ci instead of npm install.
2. @datadog/pprof was already a dependency of core, we don't need to npm install @datadog/pprof as a result -- which was updating the version at package time to an untested, unvalidated version.
3. We package.json remains untouched during packages. The lock gets translated to shrinkwrap, and then restored at the conclusion of the build script.

The end result of all this should be:

1. Stable dependencies that we control.
2. Renovate can then keep them up to date. To avoid duplicate dependency maintenance work between harper and harper-pro, I've disabled renovate looking at the package.json in this repo.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	node-forge@​1.3.3
	chai-integer@​0.1.0
	@​types/​gunzip-maybe@​1.4.3
	@​typescript-eslint/​parser@​8.57.0
	undici@​7.24.5
	@​types/​jsonwebtoken@​9.0.10
	typescript-eslint@​8.57.0
	lodash@​4.17.23
	alasql@​4.6.6
	@​types/​micromatch@​4.0.10
	@​harperfast/​rocksdb-js@​0.1.14
	why-is-node-still-running@​1.0.0
	@​types/​fs-extra@​11.0.4
	clone@​2.1.2
	gunzip-maybe@​1.4.2
	human-readable-ids@​1.0.4
	intercept-stdout@​0.1.2
	normalize-path@​3.0.0
	passport-http@​0.3.0
	passport-local@​1.0.0
	recursive-iterator@​3.3.0
	segfault-handler@​1.3.0
	validate.js@​0.13.1
	@​types/​sinon@​17.0.4
	@​harperfast/​extended-iterable@​1.0.3
	@​types/​mocha@​10.0.10
	json-bigint-fixes@​1.1.0
	joi@​17.13.3
	@​types/​tar-fs@​2.0.4
	node-unix-socket@​0.2.7
	@​types/​lodash@​4.17.24
	mkcert@​3.2.0
	ulidx@​0.5.0
See 75 more rows in the dashboard

View full report

[socket-security]

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Warn		High CVE: Prototype Pollution via parse() in NodeJS npm flatted CVE: GHSA-rf6f-7fwh-wjgh Prototype Pollution via parse() in NodeJS flatted (HIGH) Affected versions: < 3.4.2 Patched version: 3.4.2 From: package-lock.json → npm/@typescript-eslint/parser@8.57.0 → npm/typescript-eslint@8.57.0 → npm/rewire@9.0.1 → npm/@harperdb/code-guidelines@0.0.6 → npm/flatted@3.4.1 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/flatted@3.4.1. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: npm lodash vulnerable to Code Injection via `_.template` imports key names CVE: GHSA-r5fr-rjxr-66jc lodash vulnerable to Code Injection via _.template imports key names (HIGH) Affected versions: >= 4.0.0 < 4.18.0 Patched version: 4.18.0 From: package-lock.json → npm/lodash@4.17.23 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/lodash@4.17.23. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: Forge has a basicConstraints bypass in its certificate chain verification (RFC 5280 violation) in npm node-forge CVE: GHSA-2328-f5f3-gj25 Forge has a basicConstraints bypass in its certificate chain verification (RFC 5280 violation) (HIGH) Affected versions: < 1.4.0 Patched version: 1.4.0 From: package-lock.json → npm/node-forge@1.3.3 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-forge@1.3.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: Forge has signature forgery in Ed25519 due to missing S > L check in npm node-forge CVE: GHSA-q67f-28xg-22rw Forge has signature forgery in Ed25519 due to missing S > L check (HIGH) Affected versions: < 1.4.0 Patched version: 1.4.0 From: package-lock.json → npm/node-forge@1.3.3 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-forge@1.3.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: Forge has signature forgery in RSA-PKCS due to ASN.1 extra field in npm node-forge CVE: GHSA-ppp5-5v6c-4jwp Forge has signature forgery in RSA-PKCS due to ASN.1 extra field (HIGH) Affected versions: < 1.4.0 Patched version: 1.4.0 From: package-lock.json → npm/node-forge@1.3.3 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-forge@1.3.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: Forge has Denial of Service via Infinite Loop in BigInteger.modInverse() with Zero Input in npm node-forge CVE: GHSA-5m6q-g25r-mvwx Forge has Denial of Service via Infinite Loop in BigInteger.modInverse() with Zero Input (HIGH) Affected versions: < 1.4.0 Patched version: 1.4.0 From: package-lock.json → npm/node-forge@1.3.3 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-forge@1.3.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: Picomatch has a ReDoS vulnerability via extglob quantifiers CVE: GHSA-c2c7-rcm5-vvqj Picomatch has a ReDoS vulnerability via extglob quantifiers (HIGH) Affected versions: >= 4.0.0 < 4.0.4; >= 3.0.0 < 3.0.2; < 2.3.2 Patched version: 2.3.2 From: package-lock.json → npm/micromatch@4.0.8 → npm/picomatch@2.3.1 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/picomatch@2.3.1. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: Picomatch has a ReDoS vulnerability via extglob quantifiers CVE: GHSA-c2c7-rcm5-vvqj Picomatch has a ReDoS vulnerability via extglob quantifiers (HIGH) Affected versions: >= 4.0.0 < 4.0.4; >= 3.0.0 < 3.0.2; < 2.3.2 Patched version: 4.0.4 From: package-lock.json → npm/@typescript-eslint/parser@8.57.0 → npm/typescript-eslint@8.57.0 → npm/picomatch@4.0.3 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/picomatch@4.0.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		High CVE: Serialize JavaScript is Vulnerable to RCE via RegExp.flags and Date.prototype.toISOString() in npm serialize-javascript CVE: GHSA-5c6j-r48x-rmvq Serialize JavaScript is Vulnerable to RCE via RegExp.flags and Date.prototype.toISOString() (HIGH) Affected versions: < 7.0.3 Patched version: 7.0.3 From: package-lock.json → npm/mocha@11.7.5 → npm/serialize-javascript@6.0.2 ℹ Read more on: This package | This alert | What is a CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Remove or replace dependencies that include known high severity CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/serialize-javascript@6.0.2. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Low adoption: npm @harperfast/extended-iterable Location: Package overview From: package-lock.json → npm/@harperfast/extended-iterable@1.0.3 ℹ Read more on: This package | This alert | What are unpopular packages? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Unpopular packages may have less maintenance and contain other problems. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/@harperfast/extended-iterable@1.0.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Deprecated by its maintainer: npm glob Reason: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me From: package-lock.json → npm/mocha@11.7.5 → npm/glob@10.5.0 ℹ Read more on: This package | This alert | What is a deprecated package? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Research the state of the package and determine if there are non-deprecated versions that can be used, or if it should be replaced with a new, supported solution. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/glob@10.5.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Deprecated by its maintainer: npm glob Reason: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me From: package-lock.json → npm/mqtt@4.3.8 → npm/glob@7.2.3 ℹ Read more on: This package | This alert | What is a deprecated package? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Research the state of the package and determine if there are non-deprecated versions that can be used, or if it should be replaced with a new, supported solution. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/glob@7.2.3. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Deprecated by its maintainer: npm inflight Reason: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful. From: package-lock.json → npm/mqtt@4.3.8 → npm/inflight@1.0.6 ℹ Read more on: This package | This alert | What is a deprecated package? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Research the state of the package and determine if there are non-deprecated versions that can be used, or if it should be replaced with a new, supported solution. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/inflight@1.0.6. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Low adoption: npm node-unix-socket-darwin-arm64 Location: Package overview From: package-lock.json → npm/node-unix-socket@0.2.7 → npm/node-unix-socket-darwin-arm64@0.2.7 ℹ Read more on: This package | This alert | What are unpopular packages? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Unpopular packages may have less maintenance and contain other problems. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-unix-socket-darwin-arm64@0.2.7. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Low adoption: npm node-unix-socket-darwin-x64 Location: Package overview From: package-lock.json → npm/node-unix-socket@0.2.7 → npm/node-unix-socket-darwin-x64@0.2.7 ℹ Read more on: This package | This alert | What are unpopular packages? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Unpopular packages may have less maintenance and contain other problems. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-unix-socket-darwin-x64@0.2.7. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Low adoption: npm node-unix-socket-linux-arm-gnueabihf Location: Package overview From: package-lock.json → npm/node-unix-socket@0.2.7 → npm/node-unix-socket-linux-arm-gnueabihf@0.2.7 ℹ Read more on: This package | This alert | What are unpopular packages? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Unpopular packages may have less maintenance and contain other problems. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-unix-socket-linux-arm-gnueabihf@0.2.7. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Low adoption: npm why-is-node-still-running Location: Package overview From: package-lock.json → npm/why-is-node-still-running@1.0.0 ℹ Read more on: This package | This alert | What are unpopular packages? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Unpopular packages may have less maintenance and contain other problems. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/why-is-node-still-running@1.0.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

[kriszyp]

I don't think this worked (I don't think it installed any dependencies).

[dawsontoth]

Tests are failing due to HarperFast/harper#325

[dawsontoth]

@kriszyp when you have a few spare cycles, I'd love your input on this change! See the PR description for the bullet points of what I propose changing and why.

[dawsontoth]

HarperFast/harper#327 will fix the integration test failures, I believe.

[cb1kenobi]

LGTM.

So what's the general flow for updating our local repo? git pull origin main && npm run core:sync?

[dawsontoth]

It depends on if you want to pull in changes in core or not, the sync script will change the commit it references for the sub module if a newer one is available on main. Make sense?

[dawsontoth]

@cb1kenobi so probably git pull origin main && git submodule update --init --recursive, normally. I use a GUI so I don't remember exactly 😉 -- and for those times when you've made changes in core, or someone else did and you're pulling it in, then yeah npm run core:sync will pull them in so you can commit them to this repo.

[dawsontoth]

Yay, socket is happy now! 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳

[dawsontoth]

... we'll wait to merge this PR until Kris gets back and has a chance to look at it, IMO!

[cb1kenobi]

I have been doing git pull origin main && git submodule update && npm i && npm run build. It's a lot of steps.

[dawsontoth]

I mean, we could add husky to the mix, that's the typical solution to that. But then it'll run whenever you switch branches which can get really annoying if you're a rapid branch switcher during rebases, etc.

[kriszyp]

So does core-sync update harper-pro submodule reference to point to the latest in the current core branch? I thought git submodule update did the reverse, it updated the core branch/commit to the reference currently pointed to by the parent.
And what if I don't want to run npm install when I update the reference to core (or update core to the reference), because I don't want it to remove my npm symlinks? (which is the case 99% of the time for me)

[dawsontoth]

What npm links do you have? The submodule gets pointed to the latest from the main branch of harper - I think I have an extra update in there, confusing things. The remote update is the one that actually does something. I pulled in the fix for the exported port env var with this script, for example. And then the removal of the old node 16 based free space module. So I know it's functional.

I'm envisioning this being run from a workflow that gets invoked whenever core is updated, opening a PR here with the changes applied. That will keep the dependencies in sync, and let the tests in pro run.

[kriszyp]

What npm links do you have?

rocksdb-js, lmdb-js, msgpackr, cbor-extract, ordered-binary, weak-lru-cache.

The submodule gets pointed to the latest from the main branch of harper

What if I want to point to a different branch? This is something I frequently do for coordinated PRs (and it is very intuitive because it is just using git).

[dawsontoth]

I mean, you can still do it manually :) with the workflow we could pass the branch name as an arg

[dawsontoth]

@kriszyp I added a script to set the tracked branch by name, which is a passthrough to the appropriate git command. I also added a flag to skip NPM install, which will place the lockfile into an inconsistent state, but it will maintain your links. An untested workflow was created for synchronizing core, too. The permissions of it might be off.

[kriszyp]

Ok, that sounds good, and yeah, it seems fine and easy to just use standard git commands when I am updating core without any dependency changes.
Also: my intention was that @datadog/pprof be a dependency of harper-pro and not harper (which shouldn't be using). Maybe it never got removed from harper (and should be), but how do we maintain dependency "additions" of harper-pro?

[dawsontoth]

@kriszyp for that, we'll need to do some more engineering, or maintain the dependencies manually. Or actually rely on harper as a sub-dependency. The logic right now (that I fixed up to use a lockfile and respect the optional deps and overrides) is a big copy-paste at the moment.

echo -e "\n📦 Copying lock file from core"
cp core/package-lock.json ./

echo -e "\n📦 Copying dependencies & devDependencies from core"
deps=$(cd core && npm pkg get dependencies)
npm pkg set "dependencies=${deps}" --json
devDeps=$(cd core && npm pkg get devDependencies)
npm pkg set "devDependencies=${devDeps}" --json
overrides=$(cd core && npm pkg get overrides)
npm pkg set "overrides=${overrides}" --json
optionalDependencies=$(cd core && npm pkg get optionalDependencies)
npm pkg set "optionalDependencies=${optionalDependencies}" --json

[dawsontoth]

Created #74 to track that work