feat(gatsby): Support all caching headers for createRemoteFileNode

[cameronbraid]

When using createRemoteFileNode the origin server is always pinged, even for requests with a cache-control: public, max-age=5000 header.

This can add significant amount of time to a site build when hundreds to thousands of files are requested if each takes hundreds of ms (for a 304 response)

This PR proposes to use got's built in cache handling to avoid such cases.

Would gatsby consider something like this ?

[KyleAMathews]

haha yes actually! I'd just noticed a week or so ago that we were only use etag and that this didn't always work (I happened to be testing grabbing a file from a server that didn't set it) and then realized the same thing, that we'd be able to avoid all the revalidations — #28331 (reply in thread)

I did a bit of research and was thinking that we'd want to implement our own lower-level cache to avoid reading/writing the file each time but perhaps got's caching is smart enough to avoid that (or maybe your implementation avoids this) — I didn't research much.

[cameronbraid]

Looks like got's cache stored the response body as base64, and it doesnt look like theres an easy way to workaround that.

It may be best to look at integrating https://www.npmjs.com/package/http-cache-semantics instead

[KyleAMathews]

Hmm yeah — that's really inefficient for our use case.

[cameronbraid]

Another option could be to switch to using https://www.npmjs.com/package/make-fetch-happen (just for remote file download) as it's cache manager is based on https://developer.mozilla.org/en-US/docs/Web/API/Cache which gives direct access to the request and response objects meaning so it should be fairly straightforward to implement a cache manager that stores headers in gatsby cache and body in files

[cameronbraid]

After some more research it looks like the got based cache does support custom Serializer/Deserializer so I will look into that approach first.

[KyleAMathews]

Been mucking around with Got today with #32012 — basically it doesn't give you good low-level control over headers or the body. I don't see a way for us to make this PR work with Got so make-fetch-happen might be the best way forward.

[LekoArts]

Hi @cameronbraid 👋

I'm closing this PR as stale as we've moved a lot of logic around in https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-core-utils/src/fetch-remote-file.ts and as it stands the current state of the PR won't be merged. It this is still something you need, please open a feature request first: https://github.com/gatsbyjs/gatsby/discussions/categories/ideas-feature-requests

Thanks!