Provide method for defining page level headers & ability to transform headers

[techfg]

Background

Adapters are a great path forward for Gatsby, however the current functionality compared to previous methods (e.g., gatsby-plugin-gatsby-cloud, gatsby-plugin-netlify) is not yet at the same level. For example, these two plugins offer allPageHeaders and transformHeaders properties simplifying the ability to create a header for pages only and then transforming special cases for any header on any path. With the migration to Netlify from Gatsby Cloud, and for those already on Netlify (since Netlify is the only adapter currently), losing the functionality can have significant effects.

Currently, the Gatsby Config headers property is an array of path level headers meaning that every page must have an explicit entry. This is incredibly cumbersome for any site let alone sites with a lot of pages. Route matching doesn't work to identify "pages only" unfortunately (at least as far as I can tell - please correct me if I'm wrong).

Additionally, there are situations where the final header value may not be known until farther down in the build process. For example, I generate a Content Security Policy (CSP) with hashes and the hashes aren't able to be determined until the final html is known.

Proposal

Add ability to specify headers that will be applied to all pages and the ability to transform the headers just before the adapter persists them (in the adapt phase).

Potential Approaches:

1. Unfortunately, headers property is an Array which makes extending it problematic. Changing this to an object would allow for the proposed options to be added (as well as future options). For backwards compat, simply checking if headers is an array and transforming to the new structure would work:

type Headers = {
  /**
   * The current headers property
   */
  paths: Array<Header>
  allPageHeaders: IHeader["headers"]
  transformHeaders: (
    headers: IHeader["headers"],
    path: string,
  ) => IHeader["headers"] // possibly Promise<IHeader["headers"]>
}

export interface GatsbyConfig {
  ...
  headers?: Headers
  ...
}

2. Allow for a header value to be a function (equivalent of transformHeaders) so that its value can be determined at runtime:

type Header = {
  /**
   * The path to match requests against.
   */
  source: string
  /**
   * Your custom response headers.
   */
  headers: Array<{
    key: string
    value: string | ((path: string, key: string) => string) // possibly Promise<((path: string, key: string) => string)>
  }>
}

type Headers = {
  /**
   * The current headers property
   */
  paths: Array<Header>
  allPageHeaders: IHeader["headers"]
}

3. Add new properties to Gatsby config to avoid changing existing headers

export interface GatsbyConfig {
  headers: Array<Header>
  allPageHeaders: IHeader["headers"]
  transformHeaders: (
    headers: IHeader["headers"],
    path: string,
  ) => IHeader["headers"] // possibly Promise<IHeader["headers"]>
}

A general approach to any of the above would be:

1. Anywhere createHeaders (from createHeadersMatcher) is called on a page, add allPageHeaders to the list of defaultHeaders (modifications here and here in both cases we can easily identify a page)
2. Within createHeaders (from createHeadersMatcher) call the transform function (or invoke the value if it's a function) if defined (requires a few adjustments here.

There are other approaches (e.g., adding a special source value in Header such as ::allPages), but the above seem the most straightforward and least invasive. Personally, I prefer combining #1 & #2 (transformHeaders for all headers for a path & value function for individual header values) for maximum flexibility, however #1 matches currently functionality of the existing plugin ecosystem and provides all the necessary functionality (there are some perf & consumer configuration trade-offs between #1 & #2 worth considering though). Would also consider making the transformHeaders (or value function) asynchronous as it may need to obtain resources to derive the value(s) (e.g., read a file of cached CSP hashes 😉).

Workaround

Below is a solution that should address the equivalent of allPageHeaders and transformHeaders for SSG pages and redirects - it will not solve for functions or DSG/SSR pages as adapters are not responsible for their headers (at least as far as I can tell).

By creating a custom adapter and composing the Netlify adapter, we can intercept the adapt call, add in page headers & call transform before passing on the context with the modified routeManfiest to gatsby-adapter-netlify. Getting close to back to the functionality in gatsby-plugin-gatsby-cloud & gatsby-plugin-netlify.

Warning

This code has not been fully tested for functionality, performance, etc., it is just a proof-of-concept - use at your own risk.

Note

As mentioned above, this will not work for functions or DSG/SSR pages.

gatsby-config.ts

import { AdapterInit, GatsbyConfig, IRedirectRoute, IStaticRoute } from "gatsby"
import { IHeader } from "gatsby/dist/redux/types"
import { Route } from "gatsby/dist/utils/adapter/types"
import netlifyAdapter from "gatsby-adapter-netlify"
import path from "path"

interface INetlifyAdapterOptions {
  excludeDatastoreFromEngineFunction?: boolean
}

interface INetlifyAdapterExtendedOptions extends INetlifyAdapterOptions {
  transformHeaders?: (
    headers: IHeader["headers"],
    path: string,
  ) => IHeader["headers"]
  allPageHeaders?: IHeader["headers"]
}

const netlifyAdapterEx: AdapterInit<INetlifyAdapterExtendedOptions> = ({
  allPageHeaders = [],
  transformHeaders,
  ...options
} = {}) => {
  const adapter = netlifyAdapter(options)
  return {
    name: `gatsby-adapter-netlify-ex`,
    config: adapter.config,
    cache: adapter.cache,
    adapt: async ({ routesManifest: originalRoutesManifest, ...context }) => {
      const isSSGPage = (route: Route) => {
        // adapters are only responsible for headers on SSG pages
        // https://github.com/gatsbyjs/gatsby/blob/db248ab1b7dfca7023816f53b4ebf1138468a9e2/packages/gatsby/src/utils/adapter/manager.ts#L332
        if (route.type !== `static`) {
          return false
        }

        // no way to know for certain that its a page (vs. a resource)
        // so we look for .html files that aren't "gatsby" files (e.g., slices)
        const ext = path.extname(route.filePath)
        return !!(ext && ext === `.html` && !/^\/_gatsby\//.test(route.path))
      }

      const primitiveMerge = (
        headers: IHeader["headers"] = [],
        pageHeaders: IHeader["headers"] = [],
      ) => {
        const map = new Map(
          [...headers, ...pageHeaders].map((header) => [header.key, header]),
        )

        return Array.from(map.values())
      }

      const getHeaders = (route: IStaticRoute | IRedirectRoute) => {
        const headers = isSSGPage(route)
          ? primitiveMerge(route.headers, allPageHeaders)
          : route.headers
        return transformHeaders
          ? transformHeaders(headers, route.path)
          : headers
      }

      const routesManifest = originalRoutesManifest.map((route: Route) => {
        // function headers are not responsibility of Adapters
        // https://github.com/gatsbyjs/gatsby/blob/db248ab1b7dfca7023816f53b4ebf1138468a9e2/packages/gatsby/src/utils/adapter/types.ts#L30
        // and seem to be taken care of within
        // https://github.com/gatsbyjs/gatsby/blob/db248ab1b7dfca7023816f53b4ebf1138468a9e2/packages/gatsby/src/utils/page-ssr-module/entry.ts#L74
        if (route.type !== `static` && route.type !== `redirect`) {
          return route
        }

        return {
          ...route,
          headers: getHeaders(route),
        }
      })

      return adapter.adapt({
        routesManifest,
        ...context,
      })
    },
  }
}

const config: GatsbyConfig = {
  adapter: netlifyAdapterEx({
    excludeDatastoreFromEngineFunction: false,
    allPageHeaders: [
      { key: `X-Header-Source-Type`, value: `page` },
      { key: `X-PagePath`, value: `` },
    ],
    transformHeaders: (headers, routePath) =>
      (routePath !== `/contact`
        ? headers
        : [...headers, { key: `X-Transformed-Header`, value: `Added!!` }]
      ).map(({ key, value }) => ({
        key,
        value: key === `X-PagePath` ? routePath : value,
      })),
  }),
  headers: [
    {
      source: `/*`,
      headers: [
        {
          key: `X-Header-Source-Type`,
          value: `/*`,
        },
      ],
    },
    {
      source: `/:page`,
      headers: [
        {
          key: `X-Test`,
          value: `Hello`,
        },
      ],
    },
    {
      source: `/contact`,
      headers: [
        {
          key: `X-Test`,
          value: `Goodbye`,
        },
      ],
    },
  ],
}

Thoughts/input welcome!

[techfg]

Tagging #38542 for cross-reference