perf: deduplicate blocks used in multiple places using new config.blocks property (#10905)

If you have multiple blocks that are used in multiple places, this can quickly blow up the size of your Payload Config. This will incur a performance hit, as more data is
1.  sent to the client (=> bloated `ClientConfig` and large initial html) and
2. processed on the server (permissions are calculated every single time you navigate to a page - this iterates through all blocks you have defined, even if they're duplicative)

This can be optimized by defining your block **once** in your Payload Config, and just referencing the block slug whenever it's used, instead of passing the entire block config. To do this, the block can be defined in the `blocks` array of the Payload Config. The slug can then be passed to the `blockReferences` array in the Blocks Field - the `blocks` array has to be empty for compatibility reasons.

```ts
import { buildConfig } from 'payload'
import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'

// Payload Config
const config = buildConfig({
  // Define the block once
  blocks: [
    {
      slug: 'TextBlock',
      fields: [
        {
          name: 'text',
          type: 'text',
        },
      ],
    },
  ],
  collections: [
    {
      slug: 'collection1',
      fields: [
        {
          name: 'content',
          type: 'blocks',
          // Reference the block by slug
          blockReferences: ['TextBlock'],
          blocks: [], // Required to be empty, for compatibility reasons
        },
      ],
    },
     {
      slug: 'collection2',
      fields: [
        {
          name: 'editor',
          type: 'richText',
          editor: lexicalEditor({
            BlocksFeature({
              // Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
              blocks: ['TextBlock'],
            })
          })
        },
      ],
    },
  ],
})
```

## v4.0 Plans

In 4.0, we will remove the `blockReferences` property, and allow string block references to be passed directly to the blocks `property`. Essentially, we'd remove the `blocks` property and rename `blockReferences` to `blocks`.

The reason we opted to a new property in this PR is to avoid breaking changes. Allowing strings to be passed to the `blocks` property will prevent plugins that iterate through fields / blocks from compiling.

## PR Changes

- Testing: This PR introduces a plugin that automatically converts blocks to block references. This is done in the fields__blocks test suite, to run our existing test suite using block references.

- Block References support: Most changes are similar. Everywhere we iterate through blocks, we have to now do the following:
1. Check if `field.blockReferences` is provided. If so, only iterate through that.
2. Check if the block is an object (= actual block), or string
3. If it's a string, pull the actual block from the Payload Config or from `payload.blocks`.

The exception is config sanitization and block type generations. This PR optimizes them so that each block is only handled once, instead of every time the block is referenced.

## Benchmarks

60 Block fields, each block field having the same 600 Blocks.

### Before:
**Initial HTML:** 195 kB
**Generated types:** takes 11 minutes, 461,209 lines

https://github.com/user-attachments/assets/11d49a4e-5414-4579-8050-e6346e552f56

### After:
**Initial HTML:** 73.6 kB
**Generated types:** takes 2 seconds, 35,810 lines

https://github.com/user-attachments/assets/3eab1a99-6c29-489d-add5-698df67780a3

### After Permissions Optimization (follow-up PR)
Initial HTML: 73.6 kB

https://github.com/user-attachments/assets/a909202e-45a8-4bf6-9a38-8c85813f1312


## Future Plans

1. This PR does not yet deduplicate block references during permissions calculation. We'll optimize that in a separate PR, as this one is already large enough
2. The same optimization can be done to deduplicate fields. One common use-case would be link field groups that may be referenced in multiple entities, outside of blocks. We might explore adding a new `fieldReferences` property, that allows you to reference those same `config.blocks`.

Author: AlessioGr

.github/workflows/main.yml

@@ -283,6 +283,7 @@ jobs:
           - fields-relationship
           - fields__collections__Array
           - fields__collections__Blocks
+          - fields__collections__Blocks#config.blockreferences.ts
           - fields__collections__Checkbox
           - fields__collections__Collapsible
           - fields__collections__ConditionalLogic
@@ -293,6 +294,7 @@ jobs:
           - fields__collections__JSON
           - fields__collections__Lexical__e2e__main
           - fields__collections__Lexical__e2e__blocks
+          - fields__collections__Lexical__e2e__blocks#config.blockreferences.ts
           - fields__collections__Number
           - fields__collections__Point
           - fields__collections__Radio

docs/fields/blocks.mdx

@@ -295,6 +295,70 @@ export const CustomBlocksFieldLabelClient: BlocksFieldLabelClientComponent = ({
 }
 ```
 
+## Block References
+
+If you have multiple blocks used in multiple places, your Payload Config can grow in size, potentially sending more data to the client and requiring more processing on the server. However, you can optimize performance by defining each block **once** in your Payload Config and then referencing its slug wherever it's used instead of passing the entire block config.
+
+To do this, define the block in the `blocks` array of the Payload Config. Then, in the Blocks Field, pass the block slug to the `blockReferences` array - leaving the `blocks` array empty for compatibility reasons.
+
+```ts
+import { buildConfig } from 'payload'
+import { lexicalEditor, BlocksFeature } from '@payloadcms/richtext-lexical'
+
+// Payload Config
+const config = buildConfig({
+  // Define the block once
+  blocks: [
+    {
+      slug: 'TextBlock',
+      fields: [
+        {
+          name: 'text',
+          type: 'text',
+        },
+      ],
+    },
+  ],
+  collections: [
+    {
+      slug: 'collection1',
+      fields: [
+        {
+          name: 'content',
+          type: 'blocks',
+          // Reference the block by slug
+          blockReferences: ['TextBlock'],
+          blocks: [], // Required to be empty, for compatibility reasons
+        },
+      ],
+    },
+     {
+      slug: 'collection2',
+      fields: [
+        {
+          name: 'editor',
+          type: 'richText',
+          editor: lexicalEditor({
+            BlocksFeature({
+              // Same reference can be reused anywhere, even in the lexical editor, without incurred performance hit
+              blocks: ['TextBlock'],
+            })
+          })
+        },
+      ],
+    },
+  ],
+})
+```
+
+<Banner type="warning">
+  **Reminder:**
+  Blocks referenced in the `blockReferences` array are treated as isolated from the collection / global config. This has the following implications:
+
+  1. The block config cannot be modified or extended in the collection config. It will be identical everywhere it's referenced.
+  2. Access control for blocks referenced in the `blockReferences` are run only once - data from the collection will not be available in the block's access control.
+</Banner>
+
 ## TypeScript
 
 As you build your own Block configs, you might want to store them in separate files but retain typing accordingly. To do so, you can import and use Payload's `Block` type:

packages/db-mongodb/src/models/buildSchema.ts

@@ -3,7 +3,6 @@ import type { IndexOptions, Schema, SchemaOptions, SchemaTypeOptions } from 'mon
 import mongoose from 'mongoose'
 import {
   type ArrayField,
-  type Block,
   type BlocksField,
   type CheckboxField,
   type CodeField,
@@ -193,11 +192,12 @@ const fieldToSchemaMap: Record<string, FieldSchemaGenerator> = {
     schema.add({
       [field.name]: localizeSchema(field, fieldSchema, payload.config.localization),
     })
-
-    field.blocks.forEach((blockItem: Block) => {
+    ;(field.blockReferences ?? field.blocks).forEach((blockItem) => {
       const blockSchema = new mongoose.Schema({}, { _id: false, id: false })
 
-      blockItem.fields.forEach((blockField) => {
+      const block = typeof blockItem === 'string' ? payload.blocks[blockItem] : blockItem
+
+      block.fields.forEach((blockField) => {
         const addFieldSchema: FieldSchemaGenerator = fieldToSchemaMap[blockField.type]
         if (addFieldSchema) {
           addFieldSchema(blockField, blockSchema, payload, buildSchemaOptions)
@@ -207,11 +207,11 @@ const fieldToSchemaMap: Record<string, FieldSchemaGenerator> = {
       if (field.localized && payload.config.localization) {
         payload.config.localization.localeCodes.forEach((localeCode) => {
           // @ts-expect-error Possible incorrect typing in mongoose types, this works
-          schema.path(`${field.name}.${localeCode}`).discriminator(blockItem.slug, blockSchema)
+          schema.path(`${field.name}.${localeCode}`).discriminator(block.slug, blockSchema)
         })
       } else {
         // @ts-expect-error Possible incorrect typing in mongoose types, this works
-        schema.path(field.name).discriminator(blockItem.slug, blockSchema)
+        schema.path(field.name).discriminator(block.slug, blockSchema)
       }
     })
   },

packages/db-mongodb/src/predefinedMigrations/migrateRelationshipsV2_V3.ts

@@ -80,6 +80,10 @@ const hasRelationshipOrUploadField = ({ fields }: { fields: Field[] }): boolean
 
     if ('blocks' in field) {
       for (const block of field.blocks) {
+        if (typeof block === 'string') {
+          // Skip - string blocks have been added in v3 and thus don't need to be migrated
+          continue
+        }
         if (hasRelationshipOrUploadField({ fields: block.fields })) {
           return true
         }

packages/db-mongodb/src/queries/getLocalizedSortProperty.ts

@@ -65,18 +65,24 @@ export const getLocalizedSortProperty = ({
     }
 
     if (matchedField.type === 'blocks') {
-      nextFields = matchedField.blocks.reduce((flattenedBlockFields, block) => {
-        return [
-          ...flattenedBlockFields,
-          ...block.flattenedFields.filter(
-            (blockField) =>
-              (fieldAffectsData(blockField) &&
-                blockField.name !== 'blockType' &&
-                blockField.name !== 'blockName') ||
-              !fieldAffectsData(blockField),
-          ),
-        ]
-      }, [])
+      nextFields = (matchedField.blockReferences ?? matchedField.blocks).reduce(
+        (flattenedBlockFields, _block) => {
+          // TODO: iterate over blocks mapped to block slug in v4, or pass through payload.blocks
+          const block =
+            typeof _block === 'string' ? config.blocks.find((b) => b.slug === _block) : _block
+          return [
+            ...flattenedBlockFields,
+            ...block.flattenedFields.filter(
+              (blockField) =>
+                (fieldAffectsData(blockField) &&
+                  blockField.name !== 'blockType' &&
+                  blockField.name !== 'blockName') ||
+                !fieldAffectsData(blockField),
+            ),
+          ]
+        },
+        [],
+      )
     }
 
     const result = incomingResult ? `${incomingResult}.${localizedSegment}` : localizedSegment

packages/db-mongodb/src/queries/sanitizeQueryValue.ts

@@ -1,4 +1,10 @@
-import type { FlattenedBlock, FlattenedField, Payload, RelationshipField } from 'payload'
+import type {
+  FlattenedBlock,
+  FlattenedBlocksField,
+  FlattenedField,
+  Payload,
+  RelationshipField,
+} from 'payload'
 
 import { Types } from 'mongoose'
 import { createArrayFromCommaDelineated } from 'payload'
@@ -40,14 +46,18 @@ const buildExistsQuery = (formattedValue, path, treatEmptyString = true) => {
 // returns nestedField Field object from blocks.nestedField path because getLocalizedPaths splits them only for relationships
 const getFieldFromSegments = ({
   field,
+  payload,
   segments,
 }: {
   field: FlattenedBlock | FlattenedField
+  payload: Payload
   segments: string[]
 }) => {
-  if ('blocks' in field) {
-    for (const block of field.blocks) {
-      const field = getFieldFromSegments({ field: block, segments })
+  if ('blocks' in field || 'blockReferences' in field) {
+    const _field: FlattenedBlocksField = field as FlattenedBlocksField
+    for (const _block of _field.blockReferences ?? _field.blocks) {
+      const block: FlattenedBlock = typeof _block === 'string' ? payload.blocks[_block] : _block
+      const field = getFieldFromSegments({ field: block, payload, segments })
       if (field) {
         return field
       }
@@ -67,7 +77,7 @@ const getFieldFromSegments = ({
       }
 
       segments.shift()
-      return getFieldFromSegments({ field: foundField, segments })
+      return getFieldFromSegments({ field: foundField, payload, segments })
     }
   }
 }
@@ -91,7 +101,7 @@ export const sanitizeQueryValue = ({
   if (['array', 'blocks', 'group', 'tab'].includes(field.type) && path.includes('.')) {
     const segments = path.split('.')
     segments.shift()
-    const foundField = getFieldFromSegments({ field, segments })
+    const foundField = getFieldFromSegments({ field, payload, segments })
 
     if (foundField) {
       field = foundField

packages/db-mongodb/src/utilities/buildProjectionFromSelect.ts

@@ -123,7 +123,8 @@ const traverseFields = ({
       case 'blocks': {
         const blocksSelect = select[field.name] as SelectType
 
-        for (const block of field.blocks) {
+        for (const _block of field.blockReferences ?? field.blocks) {
+          const block = typeof _block === 'string' ? adapter.payload.blocks[_block] : _block
           if (
             (selectMode === 'include' && blocksSelect[block.slug] === true) ||
             (selectMode === 'exclude' && typeof blocksSelect[block.slug] === 'undefined')

packages/db-mongodb/src/utilities/sanitizeRelationshipIDs.ts

@@ -1,7 +1,7 @@
 import type { CollectionConfig, Field, SanitizedConfig, TraverseFieldsCallback } from 'payload'
 
 import { Types } from 'mongoose'
-import { APIError, traverseFields } from 'payload'
+import { traverseFields } from 'payload'
 import { fieldAffectsData } from 'payload/shared'
 
 type Args = {
@@ -150,7 +150,7 @@ export const sanitizeRelationshipIDs = ({
     }
   }
 
-  traverseFields({ callback: sanitize, fields, fillEmpty: false, ref: data })
+  traverseFields({ callback: sanitize, config, fields, fillEmpty: false, ref: data })
 
   return data
 }

packages/drizzle/src/find/traverseFields.ts

@@ -185,7 +185,8 @@ export const traverseFields = ({
           }
         }
 
-        field.blocks.forEach((block) => {
+        ;(field.blockReferences ?? field.blocks).forEach((_block) => {
+          const block = typeof _block === 'string' ? adapter.payload.blocks[_block] : _block
           const blockKey = `_blocks_${block.slug}`
 
           let blockSelect: boolean | SelectType | undefined

packages/drizzle/src/postgres/predefinedMigrations/v2-v3/fetchAndResave/traverseFields.ts

@@ -1,4 +1,4 @@
-import type { FlattenedField } from 'payload'
+import type { FlattenedBlock, FlattenedField } from 'payload'
 
 type Args = {
   doc: Record<string, unknown>
@@ -51,7 +51,10 @@ export const traverseFields = ({ doc, fields, locale, path, rows }: Args) => {
           Object.entries(rowData).forEach(([locale, localeRows]) => {
             if (Array.isArray(localeRows)) {
               localeRows.forEach((row, i) => {
-                const matchedBlock = field.blocks.find((block) => block.slug === row.blockType)
+                // Can ignore string blocks, as those were added in v3 and don't need to be migrated
+                const matchedBlock = field.blocks.find(
+                  (block) => typeof block !== 'string' && block.slug === row.blockType,
+                ) as FlattenedBlock | undefined
 
                 if (matchedBlock) {
                   return traverseFields({
@@ -69,7 +72,10 @@ export const traverseFields = ({ doc, fields, locale, path, rows }: Args) => {
 
         if (Array.isArray(rowData)) {
           rowData.forEach((row, i) => {
-            const matchedBlock = field.blocks.find((block) => block.slug === row.blockType)
+            // Can ignore string blocks, as those were added in v3 and don't need to be migrated
+            const matchedBlock = field.blocks.find(
+              (block) => typeof block !== 'string' && block.slug === row.blockType,
+            ) as FlattenedBlock | undefined
 
             if (matchedBlock) {
               return traverseFields({