RFC: Payload Admin Components Abstraction

[jmikrut]

Hi everyone, first up, YOU ARE ALL GREAT. You have been so extremely helpful while we build toward 3.0 - and we are VERY close to stable. There are only a few things left. One item on our radar to solve for yet is something I'd like to discuss here.

Many of you likely already know that we have a pull request open to Next.js in regards to being able to use Payload's Local API in your own frontends / React Server Components.

Click the issue to read more about the challenge. It's a tough one, and technically it's not really something that Next.js is doing wrong. Next.js is just simply making sure that anything we import (and use) gets bundled up to the client-side JS for a page.

Right now, you might have the following client-side JS in a Payload config:

• Lexical / Slate components
• Custom Payload Admin UI components
• Custom Payload component CSS

This happens because you actually import custom admin UI components into your config, and bind them to the config accordingly.

Here's an example:

import type { Field } from 'payload'

import { MyFieldComponent } from './components/MyFieldComponent'

const myField: Field = {
  name: 'myField',
  type: 'text',
  admin: {
    components: {
      // As far as bundlers (Webpack, Turbopack) are concerned,
      // We are "using" this dependency by binding it to the field object
      Field: MyFieldComponent
    }
  }
}

So the problem is pretty much the fact that we allow you to import and pass React components to the config, which is great from a DX perspective.

But then, if you import your Payload config in your frontend, all these components get sent to your client-side JS, which means you might have another few hundred + KBs of first-load JS right out of the box, just by importing the Payload config.

We can't let that happen. And we've gone down a LOT of paths to find out how we can solve this problem (my Next.js PR being one solution).

But while my PR does indeed solve the issue, there is a more technically elegant solution available.

This all has to do with the import keyword.

Alternative solution

If we instead accepted a string-based path to your component, we could only load those components in the admin UI, and the Payload config would get smaller / slimmer. Like this:

import type { Field } from 'payload'

const myField: Field = {
  name: 'myField',
  type: 'text',
  admin: {
    components: {
      // Just pass a path instead of the component itself
      Field: './components/MyFieldComponent'
    }
  }
}

Pros of this approach:

• Arguably it's less code to write
• The Payload config would be truly a Node-safe file, that you can just import and use anywhere
• The config would get slimmer, would be faster to compile, and could be more portable

Another pretty large pro to this approach is that right now, Payload has to maintain a custom "loader" that can safely import the Payload config if you are using it in standalone scripts, or outside of bundler contexts. We have to maintain this loader because we need to "disregard" client-side SCSS / CSS / JPG / SVG / etc imports, because those would crash vanilla Node.

We use this "loader" for Payload bin scripts (payload generate:types / etc.)

Our loader is getting more bulletproof, and we've always needed to provide this, but it's been the source of bugs as we've worked through the transition to ESM / 3.0. If we didn't allow client files / React components to be directly imported, and instead we just accepted a string path, we could simplify here.

Overall though, this change would make the payload package and your Payload config significantly more portable / re-usable outside of Next.js.

Cons of this approach:

• You'd need to export all admin UI components as default
• The path that you pass would not be validated / hinted by your IDE
• We'd need another function in our payload.init which builds out a "map" file that the admin UI imports, which contains all custom components
• The DX is a little less shiny
• The Payload config would not report type conflicts with your React components as you bind them to the config (although you could still type them appropriately in their separate file)
• If you moved files around, your IDE could not properly update the file paths for your custom components

Conclusion

We are not sprinting toward this being our recommendation. We are still working with the Next.js team on a solution that would allow us to continue to import client-side JS in our Payload config, but I do keep having this idea pop back into my head as the team and I think through solutions here.

It's possible that even outside of Next.js, this approach would have benefits that would make it valuable anyway.

But that's why I'm here writing this - I want to hear from you. What do you think? Would you accept the DX tradeoffs of passing a string path to your component, vs. passing the component itself to the Payload config?

[HarleySalas]

The whole point of ESM and moving to next was to reduce the complexity. As long as we aren't losing any actual functionality, I see no reason as to why this shouldn't be done.

Maintaing a custom loader? The way I see it, if we can make that burden a little easier on you guys, that is more resources towards actually working on features we can all touch.

[jmikrut]

Hey @HarleySalas 👋 I really appreciate that.

The burden would certainly be lesser for sure. As a team we don't want to shy away from handling complexity internally if it produces a better DX for end users, so I think this is truly a question of DX.

Honestly, the benefits (faster compile time, more portable config, lighter-weight local API) might even outweigh the drawbacks. Thanks for the feedback!

[notchris]

I would accept the DX tradeoffs, though I don't think it changes much for our team. Type conflicts for components would still be reported after a change / save, just not automatically right? Also, curious, why do people import their payload configuration into the frontend?

[jmikrut]

You'd still get strong typing within your React component file itself - you'd type it according to the type that the Payload config would expect and be all good. Not much really changes there. Nothing on runtime would be reported, unless you typed your component wrong and then it crashed in the admin UI.

People don't import their Payload config in the browser / client-side JS, but they do import it in React Server Components. Here's an example:

import React from 'react'
import config from './payload-config'
import { getPayloadHMR } from '@payloadcms/next/utilities'

const MyServerComponent: React.FC = async () => {
  // Gotta have the Payload config in order to initialize Payload
  const payload = await getPayloadHMR({ config })
  const pages = await payload.find({ collection: 'pages' })
  
  return (
    // ...
  )
}

[notchris]

Thanks for clarification on importing config. Good to know! And we can easily check the expected type from the IDE hints.

[Arctomachine]

Why not separate base config into client config + server config? This way importing only server config for strictly server things would not load anything to client

[jmikrut]

We've thought about this quite a bit as well, but the DX would be significantly more complicated if we did that, unfortunately.

Imagine if you have a big config, with lots of blocks, and deeply nested fields. Like rows, tabs, arrays, etc. Let's say you wanted to add a custom field component to a deeply nested field.

There would need to be some way to "match up" the field from the client config to the server conflg—and that would be super strange however it'd be done.

[jacobsfletch]

To me it feels like we're passing solvable, technical limitations onto the end user. My biggest fear is that static paths won't automatically update as components are renamed and moved around, unlike import paths do. That would be a serious pain.

[notchris]

This is a solid concern

[notchris]

Say I had a custom field component I use in every collection. I'd have to consider, how often am I going to change the location of this component? If we could alias an official location for custom components that could allow for find/replace, but that's not a realistic scenario really.

[FiniteVoid]

I'm asking this as someone who's not exactly a power user of payload and also a first-timer on NextJS (which is prolly a non-insignificant chunk of Payload's userbase), but if I'm understanding this correctly, as long as I don't use custom fields, it changes nothing for me? Or does this apply for any component included in admin panel using payload.config (like the seeding component that shows up in website template)?

Former seems a non-issue whereas latter could be a little complicated to a noob like me (or maybe not if that's how the examples were set up to begin with?).

[jmikrut]

It would apply for any custom component included in the admin panel, including the seeding component which shows up in the website template and any custom fields you build. But if you don't have any custom admin components, then this would not affect you at all!

We would definitely set up all of our examples to work this way out of the box so you probably wouldn't even notice really.

Thanks for the comments!

[RiftLurker]

Disclaimer: I only picked up Payload yesterday and don't know much about how the code splitting in Next.js or Payload works.
Is there a reason a naive dynamic import (() => import('./components/MyFieldComponent')) wouldn't do the trick?

[jmikrut]

Yep - good question! We tested that approach but Webpack / Turbopack still consider dynamic imports like that as files necessary to be bundled. Would be a good solution otherwise.

[SimonVreman]

Sounds alright, decent trade off. The big downside to me is these paths not being validated and automatically refactored by my IDE. I can already see some bugs on the horizon caused by this. Perhaps it is possible to get a build time error at least, for an invalid import path? That is; a path that does not point to a file with the default export being a JS function or some similar assertion.

[jmikrut]

We could certainly throw errors in dev / build that would be extremely explicit!

[r1tsuu]

In 1 of my plugins I'm doing a strange, yet needed thing https://github.com/r1tsuu/payload-enchants/blob/df706ed8e66a4b5dbb75bba1d4b3e121aa20cf3f/packages/better-localized-fields/src/attachLocalizedField.tsx#L52
basically It additionally wraps a provided custom field component (if it exists)
i can't imagine on how I'd do that with this approach, but i know that's probably not important for most people.

Honestly in Payload 3.0 i think we already lost kinda important thing - we can't just pass functions from config to components, we need to execute them via REST / Server actions, meaning that any dynamic client-side logic, like:

• Resolving slug in realtime from function
• Resolving custom useAsTitle, example: https://github.com/r1tsuu/payload-enchants/blob/master/packages/better-use-as-title/src/index.client.ts#L23 from function

requires additional request, but i see that Real Time API could possibly solve this. Have you guys explored for a simple solution to this?

Also it can be nightmare to manage a lot of files. While for fully featured custom fields you normally have a separated file, imagine you have 10 fields and in each field you want a custom, 2 lined description component. ATM you can just create it inline and it feels good, but maybe I'm just not a fan of having a lot of files for 2 line things.

[brankodencic]

Discplaimer:
I am webpack noob, not much experience with NodeJs either.

1. But, would it help if you had a method to load config, which strips all client components from the config and returns it to be safely imported without creating dependency?

2. Or maybe similar to how generate:types creates types, it (or different command) could create a version of config that is stripped of all client components so it can be safely imported, while original config can still be referenced where custom components are needed?

It wouldnt require maintaining 2 separate configs, just carefuly choosing which one of these we import, and to me that sounds like a fair tradeoff.

[jmikrut]

Hey @brankodencic yes, an alternative solution like you proposed in (1) may be possible as a Webpack loader / Turbopack plugin. Complex, but possible. We are investigating that now, but it's kinda nasty and adds more compilation overhead vs. the suggestion I posed in this discussion which would actually reduce the amount of compilation logic necessary.

As far as (2) we went down this path heavily back on 2.0. This one would be extremely complex and likely not worth the work involved, if it were possible in a stable way at all. It would involve parsing AST and outputting files, which again would be additional compilation time. We scrapped that idea in hopes of finding a smarter way.

[rickylabs]

Hi @jmikrut,

I share the same concern as @r1tsuu pointed out ! Though I haven't worked with Payload yet (waiting for the stable 3.0 release), I wanted to share my opinion.

How would you handle scenarios in the Payload config where you want to do props drilling into your components ? Or even handle conditional rendering without making additional requests on the components themselves ?

I might be missing something, but in my opinion, we would lose a lot of flexibility (in addition to the IDE checks and refactoring issues you already mentioned) in the way we're handling custom Admin components.

Can you provide an alternative to @r1tsuu's implementation that would work with the new approach ?

Also, adding additional build/runtime checks would impact build/hot-reload times, right ?

[r1tsuu]

Just for better pointing out what i meant about "without making additional requests":

You can do it easily if you know what you want in your React component. But sometimes it's not the case if you want to control it directly from the config, and you can pass normal props, just not functions!
It's mostly probably about plugins, as they are meant to be universal, a plugin for slug field that can only generate slug by title field isn't that good.

// If you are creator of "pluginSlug", you can't pass the provided function directly to your own Field
// Instead you can create endpoint /api/slugify that uses this function and then you can call it in the Field.
pluginSlug({ slugify: (data) => `${data.title} ${data.createdAt}` })

Official Payload Plugin SEO does the same in the beta -

payload/packages/plugin-seo/src/index.tsx

Line 202 in 11b53c2

handler: async (req) => {

payload/packages/plugin-seo/src/fields/MetaTitle.tsx

Line 50 in 11b53c2

const genTitleResponse = await fetch('/api/plugin-seo/generate-title', {

While in 2.0 they were passing the resolve function directly to the custom field component, no need to expose and call any endpoint

payload/packages/plugin-seo/src/index.ts

Line 39 in 320dcc0

Field: (props) => getMetaTitleField({ ...props, pluginConfig }),

Though i can see a solution with using contexts, but it would still require to additionally create a component by user of plugin/field that needs it.

[JPrisk]

I'd be fine with this approach provided that Payload threw some proactive errors (when custom components weren't found)

It seems to decrease the surface area for bugs in other areas and provides a much needed perf improvement for the 90% user base with a little friction for the 10% using custom fields (don't quote me on those numbers, completely fictional but you get the idea)

[kamelEddinee]

As i understood this concern more payload dashboard, for me i would prefer to add my component directly, the path can be misleading especially without errors, so yeah throw us errors and will be fine

[cbratschi]

I like the single config file approach Payload uses so far. This makes it simple to write plugins which just have to modify this single config. However, I reported the issue with Lexical CSS code being added to the Next.js page on Payload 2.x quite some time ago. Separating client/server code is non-trivial and outside of Next.js different toolchains are involved (Next.js still has at least two; with Babel three). In our projects we use a single config file for the server and during build time we create the sanitized client configuration file. Probably such a pre-processor could run once while creating the TypeScript and GraphQL files. I know you already do some tricks while loading the current config.

Concerning Next.js, there are some difficulties, like the Edge runtime which easily breaks if not supported Node.js packages are imported. Edge might be replaced by a normal Node.js runtime but haven't seen any announcements yet. Using the instrumentation feature we had to set externals in the Weback config to exclude fsevents which uses native code. Next.js pre-bundles the server code and has some restrictions.

Like @r1tsuu, we are using closures to pass props to components:

            components: {
                Field: createTaxonomySummary(parentCollection, taxonomy)
            }


/**
 * Create a TaxonomySummary instance.
 *
 * @param parentCollection
 * @param taxonomy
 * @returns
 */
export function createTaxonomySummary(parentCollection: CollectionConfig, taxonomy: string) {
    return function TaxonomySummaryCollection() {
        return (
            <TaxonomySummary
                parentCollection={parentCollection}
                taxonomy={taxonomy}
            />
        );
    };
}

With strings this will no longer work. One workaround could be to move the taxonomy value to the custom property and get the collection via hooks.

I hope there is a way to pre-process the config to keep its flexibility and generate ready to use configs for all purposes.

[thgh]

How about introducing a map of field type -> imports, and then autogenerate typings for the component names?

• DX is good, as the custom field types are type checked
• Less code to write, or at least more shallowly nested
• Bundler friendly

// admin.config.ts
export const fieldTypes = {
  phone: { Field: PhoneInput },
  map: { Field: MapView }
}

// payload-types.ts (autogenerated)
type CustomFieldType = 'phone' | 'map'

// payload.config.ts
const field: Field = {
  type: 'phone'
}

To be checked if extending field.type introduces too much change and instead use this:

// payload-types.ts (autogenerated)
type CustomComponentName = 'phone' | 'map'

// payload.config.ts
const field: Field = {
  type: 'text',
  admin: { component: 'phone' }
}

[jmikrut]

Hey @thgh this type of pattern might solve for custom field components specifically, but there are so many other types of customizable components that I don't think this would be a clear enough solution, unfortunately. Consider adding custom views, labels, row labels, afterInput, etc. It would get messy quick.

I am not immediately against having a separate config, but that would make plugins more difficult to install / build as well.

We'll explore this idea for sure!

[gu-stav]

So the biggest downside I see being articulated seems to be the fact that imports may break and I am wondering if there is a way around it by keeping the import keyword at the top-level somehow, which would give users IDE support and mitigate this concern.

In theory I would expect the following to work, but I am not sure it does? If so this would also allow for named exports.

// payload.config.ts

const myField: Field = {
  admin: {
    components: {
      Field: importComponent(() => import('./components/MyFieldComponent'))
    }
  }
}

// imported from payload
function importComponent(componentCallback) {
  // in a client context the admin app needs to await all components, before rendering, because this would return a Promise
  if (typeof window !== "undefined") {
    return componentCallback();
  }

  // in a server context a module is always undefined
  return undefined
}

[r1tsuu]

it's a good thought, but even having the import('./components/MyFieldComponent') present in a Webpack app will cause the import to get bundled. We REALLY ran the gauntlet trying to figure out a way to make that work but came up empty-handed. Gotta love bundling JS!

Is there a chance that Typescript plugin would work here? It should kinda represent importComponent as import, but not actually import, so it will be typescript-only thing, under the hood importComponent can do it's own thing, not depended on a bundler. Next.js has a plugin that checks for named exports in app router pages, so maybe it could be a good reference https://github.com/vercel/next.js/blob/canary/packages/next/src/server/typescript/index.ts

Or this actually could be closer to Payload case https://github.com/mrmckeb/typescript-plugin-css-modules

[r1tsuu]

Honestly I have a really good feeling that it's possible, this is the difference between AST import and importComponent

import('./components/Test')
// produces
factory.createExpressionStatement(factory.createCallExpression(
    factory.createToken(ts.SyntaxKind.ImportKeyword),
    undefined,
    [factory.createStringLiteral("./components/test")]
  ))

importComponent('./components/Test')
// produces
 factory.createExpressionStatement(factory.createCallExpression(
    factory.createIdentifier("importComponent"),
    undefined,
    [factory.createStringLiteral("./components/test")]
  ))

So basically what we need here, is to dynamically a bit modify the AST based on importComponent identifier, then you should get autocomplete / typechecking.

[r1tsuu]

I tried to implement this and I just realized that if you'd change the AST, it will also change of course the transpiled version with normal dynamic import()..

Still, instead of replacing the AST, you can extend the autocompletions and errors, sharing my basic code, it can be already used as typescript plugin, but yeah, need to really dig into the Typescript API

import type tsModule from "typescript/lib/tsserverlibrary";

// Finds all the importComponent calls
const findImportComponentCalls = (
  node: tsModule.Node,
  ts: typeof tsModule
): tsModule.CallExpression[] => {
  const importComponentCalls: tsModule.CallExpression[] = [];

  function visit(node: tsModule.Node) {
    if (
      ts.isCallExpression(node) &&
      ts.isIdentifier(node.expression) &&
      node.expression.text === "importComponent"
    ) {
      importComponentCalls.push(node);
    }
    ts.forEachChild(node, visit);
  }

  visit(node);
  return importComponentCalls;
};

const init: tsModule.server.PluginModuleFactory = ({ typescript }) => {
  return {
    create(info) {
      const proxy: tsModule.LanguageService = Object.create(null);

      // Wrap all language service methods
      for (const k of Object.keys(info.languageService) as Array<
        keyof tsModule.LanguageService
      >) {
        const x = info.languageService[k];
        // @ts-ignore
        proxy[k] = (...args: Array<{}>) => x.apply(info.languageService, args);
      }

      // Override getCompletionsAtPosition
      proxy.getCompletionsAtPosition = (fileName, position, options) => {
        const completions = info.languageService.getCompletionsAtPosition(
          fileName,
          position,
          options
        );

        // somehow extend them here for importComponent calls?...
        // we can do completions.entries.push(...)

        return completions;
      };

      return proxy;
    },
  };
};

export = init;

[thgh]

having the import('./components/MyFieldComponent') present in a Webpack app will cause the import to get bundled

Is the problem that webpack bundles a component file or that webpack doesn't split the bundle?
Could externalsType help?

module.exports = {
  externals: [
    function ({ context, request }, callback) {
      // Only match imports from components folder
      if (/\/components\//.test(request)) {
        // Use dynamic import in this case
        return callback(null, `import ${request}`);
      }
      callback();
    }
  ]
};

[gu-stav]

@thgh I like the idea too and was thinking something similar at the time of writing, which is why I suggested the import util. This way the payload team might be able to create externals programmatically whenever it is called, instead of having to hardcode a components folder and such.

Question is whether this would be compatible with turbopack one day.

[BenAHammond]

I think this makes sense and is worth the hit to DX. I could see a plugin for the IDE to help with detecting paths exist and are properly exported, potentially paired with some something custom pre-build just to add some extra safety.