fix: re-install lmdb when detecting the absence of @LMDB prebuilt packages

[JGAntunes]

Description

Add the ability for gatsby to self heal on the absence of prebuilt lmdb binaries we can use to bundle in lambdas. This is particularly useful to avoid runtime issues with wrong binaries built from source.

We've seen that multiple project structures can lead to this, from particular outdated lockfiles, to npmrc settings instructing the package manager to skip optional package installs.

Documentation

NA

Tests

I've added a set of integration tests which try to first reproduce the issue and then validate that Gatsby was able to self heal and generate the right set of assets for the bundle.

Related Issues

FRA-11

[pieh]

In some cases this would not resolve even if it's actually installed when we try to require(.resolve) from gatsby package context.

Yarn PnP wouldn't allow that because those prebuilt packages are not dependencies of gatsby and are transitive dependency and you can't require/import transitive deps that you package doesn't declare as deps.

In node_modules cases it also is possible this won't work if package manager would decide to structure things like this:

node_modules
├── gatsby
└── lmdb
    └── @lmdb/lmdb-${process.platform}-${process.arch}

this would also mean it doesn't resolve.

For this check to work without relying on node_modules hierarchy we need to resolve prebuilt package from lmdb package context:

Suggested change

	require.resolve(`@lmdb/lmdb-${process.platform}-${process.arch}`)
	const lmdbRequire = mod.createRequire(
	require.resolve(`lmdb`)
	)
	lmdbRequire.resolve(`@lmdb/lmdb-${process.platform}-${process.arch}`)

The other part is that I'm not sure about is if there are prebuilts for all the platform+arch combinations that exist. If prebuilt for the combo doesn't exist we will always try to install lmdb but that wouldn't help, so maybe before we even attempt to install - we should try to check if @lmdb/lmdb-${process.platform}-${process.arch} is included in list of available prebuilts (gathered from lmdb's optional dependencies list):

[
  '@lmdb/lmdb-darwin-arm64', 
  '@lmdb/lmdb-darwin-x64', 
  '@lmdb/lmdb-linux-arm', 
  '@lmdb/lmdb-linux-arm64', 
  '@lmdb/lmdb-linux-x64', 
  '@lmdb/lmdb-win32-x64'
]

[JGAntunes]

Greate suggestions, I'll add those in 👍

[pieh]

I do wonder if there is something we could do to "sandbox" this to not have potential repo/site level .npmrc (or even user level .npmrc locally) potentially cause problems with installing optional packages. If there is .npmrc with omit=optional (or however the syntax it is) - that would use in this directory as well?

Alternatively, maybe instead of installing whole lmdb (and let that install optional packages with prebuilds) - maybe we try to install those prebuild package directly? Maybe then it would be less prone to inherit same problem that cause those not to be installed normally in the first place?

[JGAntunes]

Alternatively, maybe instead of installing whole lmdb (and let that install optional packages with prebuilds) - maybe we try to install those prebuild package directly?

I guess what I wonder is, in the presence of @lmdb/x package will that always be favoured even if there's a build from source in the lmdb directory? (I can dig through the code to try to understand that but tbh I worry that we might miss something by doing so)

[pieh]

I guess what I wonder is, in the presence of @lmdb/x package will that always be favoured even if there's a build from source in the lmdb directory? (I can dig through the code to try to understand that but tbh I worry that we might miss something by doing so)

It looks like general order of loading is to first try to load from build/Release first before trying optional packages. However this relies on fact that there is install script that before trying to compile from source check if there is binary that can be loaded - and when prebuilts are installed - it finds those and that check prevent from even compiling from source ( https://unpkg.com/browse/node-gyp-build-optional-packages/ is doing all those things ).

However - our lmdb-bundling-patch replace runtime usage of node-gyp-build(-optional-packages) from lmdb and replace that with direct path to .node file, so we are not subject to regular behavior when bundling for lambdas as we get rid of it here:

gatsby/packages/gatsby/src/schema/graphql-engine/lmdb-bundling-patch.ts

Lines 74 to 85 in 22c2412

	.replace(
	`require$1('node-gyp-build-optional-packages')(dirName)`,
	`require(${JSON.stringify(lmdbBinaryLocation)})`
	)
	.replace(
	`require$1('node-gyp-build')(dirName)`,
	`require(${JSON.stringify(lmdbBinaryLocation)})`
	)
	.replace(
	`loadNAPI__default["default"](dirName);`,
	`require(${JSON.stringify(lmdbBinaryLocation)})`
	)

So I think if we can install just one of prebuild packages like @lmdb/lmdb-linux-x64 and get path to .node file and make the lmdb bundling patch use that - I think this should be fine (?)

Also when we install whole lmdb in .cache/internal-packages, we don't actually use lmdb instance from there currently - we could add webpack alias to use that instance of lmdb package for lambda, but my biggest hang up with it is that installing main lmdb package it in nested directory could inherit any configuration that caused those prebuilt packages to not be installed in first place.

PS. after going through node-gyp-build-optional-packages I also found that:

build-from-source = true

in .npmrc would also likely cause the issue (similarly to omit = optional)

[JGAntunes]

@pieh I've updated the logic to only install the specific@lmdb package and use the force flag to override for the loader to override the binary in use. Let me know what you think. I've tested this out locally in a project with a package built from source and it appears to be working.
Wanted to write some tests for this but I'm wondering what would be the best approach for it? An e2e-test?

[pieh]

Wanted to write some tests for this but I'm wondering what would be the best approach for it? An e2e-test?

That surely would provide the most confidence, but I'm not sure what setup for it would be best - easiest/quickest right now would be to have fixture test site with .npmrc with one of known triggers for the problem and deploy to Netlify and try to hit SSR or DSG route and assert it's not 500 status. But if setup would change in the future on Netlify that test might not be testing the problem anymore.

More standalone e2e setup could use different docker images for build and serve (picked in a way that would trigger the problem - so image for build would have newer GLIBC than image for serve - at least if I understand this part correctly), but this seems quite a bit of work to get that done?

Alternatively we don't do full on actual e2e - instead something more like integration - we still have fixture test site that would build lmdb from source, let it run build and we "just" inspect .node binary in produced SSR/DSG lambda - not sure what would be the best way to assert that correct .node is being bundled for lambda - maybe just file comparison against node_modules/lmdb/build/Release/lmdb.node would be enough (at least assuming that one that is built from source is not exactly like one that is in prebuilds)? With setup right now we could also check the name of binary - if .cache/query-engine/assets/build/Release/lmdb.node is in lambda - then that's prebuilt - but this relies on some details of bundling that could potentially change (unrelated to changes done in this PR), so I don't quite like that kind of assertion.

[pieh]

Let me know what you think.

The code part seems great and I don't have any more notes on it 👍

[pieh]

LGTM!