New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block API: Server-side awareness of block types #2751

Open
aduth opened this Issue Sep 20, 2017 · 23 comments

Comments

@aduth
Member

aduth commented Sep 20, 2017

Previously: #2529, #104
Related: #886
Related Slack conversation: https://wordpress.slack.com/archives/C5UNMSU4R/p1505677036000104 (cc @jasonbahl)

When initially developing the API for implementing a block (#104), there were some competing objectives:

  • The editor representation of a block must occur in the client to preserve an ideal user experience
  • A block should ideally be easy to implement for quick prototyping (#27, e.g. defining block in a browser console)
  • General information about a block should be context unaware (title, attributes definition, category, etc need not specifically be defined in a client, server, or an other external static file)

Where we have fallen short is in:

  • The third of these objectives; As a need has arisen for server-side attribute validation (#2529), we have further fragmented the definition of a block (its attributes may be defined on the server or in the client).
  • It is not possible for the server to have reliable awareness of all blocks which would be registered in the editor.

Proposal:

To improve consistency, we should move block attributes (and other general information) to a server block registration. This would likely behave exactly as it does in #2529, including client-side attributes bootstrapping and validation, but would be applied consistently across all blocks. Therefore, every block would have a companion PHP file associated with it, which in addition to current supports defines:

  • Title
  • Icon
  • Category
  • Attributes

The trade-off here is that it is not as simple to implement a block in a single location. We could potentially leave client-defined attributes support as-is, but this could also entice developers to avoid the server registration and reintroduce all the inconsistencies/drawbacks therein.

Some previous discussion had considered the idea of a third JSON "manifest" file, similar to a package.json or composer.json, describing the static general information about a block. While this would be a simple format for what is essentially overview data, is problematic because (a) it is yet another file that a developer could need to become familiar with, (b) it is difficult to bring into the client without a build step understanding JSON imports, and (c) localization of strings would need to implement some amount of "magic" (hard-coding to properties expected to be localized).

It's worth pointing out that a plugin author already needs some amount of server-specific logic for each of their blocks, specifically that of enqueueing the JavaScript files responsible for registering the blocks in the client. By requiring block registration in the server, there may be some added benefit in handling scripts and styles.

  • A block could define its companion scripts and styles as properties of the block type (e.g. script_handle, style_handle).
  • Or: These files could be "discovered" adjacent the block registering file by naming conventions (e.g. find style.css, edit.css, block.js adjacent myblock/index.php). This could encourage developers to define their blocks in a standalone folder/file structure for easier separation overview.

A remaining challenge in moving all block attributes definitions to the server is support for attribute sources. With advent of additional source types (meta, options), we will likely need to expand this concept to cover more than just DOM-based attribute sourcing. Therefore, I would propose creating an alternative structure for representing what currently exists as the assignment of source functions:

Before:

{
	url: {
		source: attr( 'img', 'src' )
	}
}

After:

[
	'url' => [
		'source' => [
			'type'     => 'attribute',
			'selector' => 'img',
			'name'     => 'src',
		],
	],
]

The intent here is not necessarily that the server would become responsible for parsing attributes from the markup of a block (although there is nothing that precludes it from doing so), but rather to be able to represent the structure in a static format, which could then be bootstrapped into the client to recreate the current behavior.

As an example for how this extends into the concept of additional sources:

'author' => [
	'type' => 'string',
	'source' => [
		'type' => 'meta',
		'name' => 'author', // Inferred from attribute name if not specifically defined?
	],
],
'siteTitle' => [
	'type' => 'string',
	'source' => [
		'type' => 'option',
		'name' => 'name',
	],
]

Open questions:

  • While there is justification for defining block attributions in the server for the purpose of improving consistency, there is still a lack of general understanding of specific use cases for server-specific awareness of blocks, aside from a "nice to have" reasonings. Lack of these use cases is not intended as an argument for a server definition not to exist, but it would help guide specific implementation decisions.

cc @westonruter

@jasonbahl

This comment has been minimized.

Show comment
Hide comment
@jasonbahl

jasonbahl Sep 20, 2017

@aduth thanks for creating this ticket and CCing me!

there is still a lack of general understanding of specific use cases for server-specific awareness of blocks, aside from a "nice to have" reasonings. Lack of these use cases is not intended as an argument for a server definition not to exist, but it would help guide specific implementation decisions.

I'll share some use cases (some might still be considered nice to have but in my opinion should not be dismissed).

General WordPress Consistency

WordPress thrives on the hook/filter system, and defining many pieces of the block Schema on just the client limits the power of the system that essentially makes WordPress what it is.

I know there's been work on a JS implementation of similar hook/filter functionality, but that's only useful in the context of WordPress itself, and WordPress is used to power much more than just itself (I'll talk more about decoupled apps below).

Take a step back and imagine if the entire admin was rewritten in JS (think Calypso). And we statically defined Post Type and Taxonomy Registration only in JS and not on the server.

Now think about how much of the WP Ecosystem at large would not exist at all because of the lack of a server-side registration. . .a LOT. There would be no way to generate per-post-type feeds or REST endpoints. Calypso wouldn't be able to work with external WordPress sites, because it wouldn't know what post_types and taxonomies exist in the sites, because that info wouldn't be available via REST, because the REST server wouldn't have any knowledge of it. . .

Without comprehensive server-side registration, we drastically reduce the power of WordPress as we know it to interact with the blocks.

Same goes with most extendable parts of WordPress. . .image sizes, post_stati, post_types, taxonomies, admin_menus, etc. . .so many things are statically defined on the server, so not having a solid registration for Blocks server-side seems like it goes agains WP's history. I know there is a minimal schema, but think if register_post_type was just register_post_type('post') and didn't accept any additional args for labels, capabilities, etc. . .

Also consider existing conversations about how troublesome Meta Boxes in Gutenberg are going to be, mostly due to the lack of a true Fields API (server side schema for fields).

Ideally, one should be able to register a block on the server and that becomes the source of truth for the block's capabilities. The REST API could make use of it, other plugins could make use of it, other APIs (XML-RPC, WPGraphQL, etc) could make use of it, and of course the Gutenberg JS itself could make use of it.

Alternative UIs
Let's be honest. Gutenberg is not going to solve everyones's problems. No matter how awesome it is or will become, it simply will not meet everyone's needs. There will still be a market for creating content in different ways. Page builders currently completely dismiss the Post Edit screen, and these alternative post creation solutions will continue to exist post-Gutenberg.

With a solid Server Side schema, hopefully the Page Builders could at least start to adhere to a standard way of interacting with "Blocks". . .that way, if Page Builder A thinks using Modals to edit content makes more sense than the fixed-right sidebar of Gutenberg, they could build their own UI to interact with Gutenblocks, but still ensure the blocks are saved in the same way.

I think having a solid server-side schema for blocks will help standardize the Page Builder market. Page builders could essentially become "Gutenberg Themes" where the editing experience looks and feels different, but at the end of the day the data produced is the same. This probably isn't impossible without a comprehensive server side schema, but it would be much easier.

If there was a comprehensive Block schema on the server, page builders could use that to hydrate their view layer and ensure compatibility with Gutenberg (avoid the lock-in, at least to a degree, that everyone talks about with page builders).

Headless CMS / Decoupled Apps

Headless CMS with WordPress is nothing new. . .one of my first projects with WordPress was populating a Flash site with data from WordPress via XML-RPC back in ~2009. . .but now with the REST API in core (and exciting plugins like WPGraphQL), it's becoming more and more common to see folks interacting with WordPress in non-traditional Theme / WP-Admin contexts.

For any of these decoupled applications to interact with Gutenblocks, they have to have knowledge of the blocks in the same way Gutenberg itself does. . .not just the blocks that have already been created by Gutenberg inside the WP-Admin, but also the capabilities of what can be done to create/modify blocks. That way these decoupled applications can provide an experience comparable in some fashion to Gutenberg. If you edit a post on a Native Mobile App, Calypso or some other decoupled app, the data should be able to be saved back in the same format that Gutenberg saves it. But in order for said decoupled app to provide an experience that allows for data to be saved in the same format as Gutenberg, said decoupled app needs to know what the capabilities of each block are.

If everything is defined in client-side code that lives in the WP-Admin, we're limiting interaction with the blocks to just the WP-Admin.

The way I see this, is that blocks will have a statically defined registry which will act as the source of truth for what blocks are available, what their capabilities are, the attributes they can have, the types (string, int, bool, array, etc) of data their attributes are expecting to store, etc.

With this knowledge, a decoupled application could request this Block registration from the server, hydrate a client and render out a UI based on the Block registration.

The Gutenberg editor could also use this same registration to Hydrate what blocks are available, what UI elements are needed to provide editing of the available attributes per-block, etc.

Having the configuration defined on the server should also help reduce code duplication. If the server is the source of truth, then the client and server can both use that truth for their needs. The client wouldn't need to define block attributes and fields in addition to the server, just know how to read the source of truth from the server and do something with it, which should actually reduce JS file sizes because there's less info in there defining things, and instead just rendering data that gets served from the client.

Alternate Storage Options
Right now the block data is stored as a large string in the post_content (which is genius for backward compatibility), but I see alternative storage being explored by both core and plugin developers.

For example, where I work we will likely at least experiment with offloading Blocks and their associated meta to ElasticSearch, so that we can read/write individual blocks without having to alter the entire post_content to fix a typo in one block, for example. We will also use ElasticSearch to to aggregations, etc to get insight into how blocks are being used across our sites, etc (how we use ES is beyond the point, though). The point is that the server will need to know about the blocks so validation can be properly made when interacting with external systems.

ElasticSearch has a type mapping for the data that gets sent to it, and if there's a server side schema for blocks, plugins like ElasticPress, or even Jetpack's ElasticSearch implementation would be able to provide mapping for blocks to ES to ensure block data gets indexed properly and efficiently. Without a server side schema, the mapping would be created only as data is sent to ES and ES would give it's best guess on things, but we'd all lose out on the potential ES could give us if there were a server side Block registry that could be used to define mapping.

ElasticSearch aside, I imagine the core team or other developers will explore options for storing Block data in WordPress in some fashion. . .whether it's new wp_blocks and wp_blocks_meta tables or something else, I'm sure folks will experiment with storing blocks outside of a massive post_content string and as we've learned with the evolution of post_meta, the lack of a schema makes things "interesting" to say the least.

WPGraphQL

I'm the creator and maintainer of WPGraphQL (https://github.com/wp-graphql/wp-graphql) which brings a GraphQL API to WordPress. . .so I'll pitch my case, which I think is applicable to all the above in some fashion.

WPGraphQL allows for data to be declaratively fetched and for just the requested data to be sent to the client from the server.

An example of a WPGraphQL query right now would be:

{
  post(id:"someId") {
     id
     title
  }
}

And that produces a JSON response like so:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World"
    }
  }
}

As you can see, unlike REST where a request sends back a pretty large payload with the entire post object (unless you've written up some custom feature endpoints 🤢 ), the minimal amount of data needed is transferred from server to client, which is super beneficial, especially on slow mobile networks where data transfer can hold things up.

Anyway, with Gutenberg I envision WPGraphQL being a super powerful tool, especially for decoupled applications. I can picture folks using Gutenblocks with various block meta to indicate whether a block should be viewable on desktop or mobile, (or in our case WordPress even powers our newspaper print content). With WPGraphQL, a native mobile client could ask for just blocks that are meant for use in a native context.

For example, Let's say I've extended all of my Gutenblocks to have a "Context" multi-select which would allow me to apply various contexts for where my blocks should display (Web / Native Mobile / Print, for example).

I could have a post created by Gutenberg that has this HTML:

<!-- wp:core/heading {"align":"center", contexts:[web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show only on web</h3>
<!-- /wp:core/heading -->

<!-- wp:core/heading {"align":"center", contexts:[mobile-native, web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>
<!-- /wp:core/heading -->

I could query from a React native (or similar) app, and ask for just the blocks that were tagged with "mobile-native" context. . .something to this tune:

{
   post(id:"someId") {
      id
      title
      contentBlocks( where: { context: MOBILE } ) {
           ...on HeadingBlock {
               rawContent
               __typename
           }
      }
   }
}

I would receive a payload like:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World",
       contentBlocks: [
         {
             rawContent: "<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>",
              __typename: "HeadingBlock"
          }
       ]
    }
  }
}

My post contains multiple blocks, but my Native client was able to request just the blocks that were intended to be rendered in the Native context. This reduces network bandwidth as the data send to the client is limited to what should be sent to the client, reduces bugs in code as it's a schema with defined type system, reduces transformation on the client as the payload contains what's needed so the client doesn't need to parse/transform the Gutenblocks for all contexts to determine which ones it should use, etc. . .

With the current implementation of gutenberg_parse_blocks my hypothetical scenario is already partly possible with WPGraphQL, but the big missing piece (in this example), would be WPGraphQL knowing what Enum options there are for the Context select. . . WPGraphQL would have no knowledge of what options exist for the Context select field if it's defined in the client, not the server. It wouldn't even know that "context" was a possible attribute of a block/all blocks if the attributes weren't defined on the server.

=====

I'll stop here, as I probably just sound like I'm spouting a bunch of nonsense. . .but happy to elaborate, discuss, clarify anything I've pointed out. . .and who knows, maybe some things I've pointed out already have been addressed since the last time I had to dig through things??

jasonbahl commented Sep 20, 2017

@aduth thanks for creating this ticket and CCing me!

there is still a lack of general understanding of specific use cases for server-specific awareness of blocks, aside from a "nice to have" reasonings. Lack of these use cases is not intended as an argument for a server definition not to exist, but it would help guide specific implementation decisions.

I'll share some use cases (some might still be considered nice to have but in my opinion should not be dismissed).

General WordPress Consistency

WordPress thrives on the hook/filter system, and defining many pieces of the block Schema on just the client limits the power of the system that essentially makes WordPress what it is.

I know there's been work on a JS implementation of similar hook/filter functionality, but that's only useful in the context of WordPress itself, and WordPress is used to power much more than just itself (I'll talk more about decoupled apps below).

Take a step back and imagine if the entire admin was rewritten in JS (think Calypso). And we statically defined Post Type and Taxonomy Registration only in JS and not on the server.

Now think about how much of the WP Ecosystem at large would not exist at all because of the lack of a server-side registration. . .a LOT. There would be no way to generate per-post-type feeds or REST endpoints. Calypso wouldn't be able to work with external WordPress sites, because it wouldn't know what post_types and taxonomies exist in the sites, because that info wouldn't be available via REST, because the REST server wouldn't have any knowledge of it. . .

Without comprehensive server-side registration, we drastically reduce the power of WordPress as we know it to interact with the blocks.

Same goes with most extendable parts of WordPress. . .image sizes, post_stati, post_types, taxonomies, admin_menus, etc. . .so many things are statically defined on the server, so not having a solid registration for Blocks server-side seems like it goes agains WP's history. I know there is a minimal schema, but think if register_post_type was just register_post_type('post') and didn't accept any additional args for labels, capabilities, etc. . .

Also consider existing conversations about how troublesome Meta Boxes in Gutenberg are going to be, mostly due to the lack of a true Fields API (server side schema for fields).

Ideally, one should be able to register a block on the server and that becomes the source of truth for the block's capabilities. The REST API could make use of it, other plugins could make use of it, other APIs (XML-RPC, WPGraphQL, etc) could make use of it, and of course the Gutenberg JS itself could make use of it.

Alternative UIs
Let's be honest. Gutenberg is not going to solve everyones's problems. No matter how awesome it is or will become, it simply will not meet everyone's needs. There will still be a market for creating content in different ways. Page builders currently completely dismiss the Post Edit screen, and these alternative post creation solutions will continue to exist post-Gutenberg.

With a solid Server Side schema, hopefully the Page Builders could at least start to adhere to a standard way of interacting with "Blocks". . .that way, if Page Builder A thinks using Modals to edit content makes more sense than the fixed-right sidebar of Gutenberg, they could build their own UI to interact with Gutenblocks, but still ensure the blocks are saved in the same way.

I think having a solid server-side schema for blocks will help standardize the Page Builder market. Page builders could essentially become "Gutenberg Themes" where the editing experience looks and feels different, but at the end of the day the data produced is the same. This probably isn't impossible without a comprehensive server side schema, but it would be much easier.

If there was a comprehensive Block schema on the server, page builders could use that to hydrate their view layer and ensure compatibility with Gutenberg (avoid the lock-in, at least to a degree, that everyone talks about with page builders).

Headless CMS / Decoupled Apps

Headless CMS with WordPress is nothing new. . .one of my first projects with WordPress was populating a Flash site with data from WordPress via XML-RPC back in ~2009. . .but now with the REST API in core (and exciting plugins like WPGraphQL), it's becoming more and more common to see folks interacting with WordPress in non-traditional Theme / WP-Admin contexts.

For any of these decoupled applications to interact with Gutenblocks, they have to have knowledge of the blocks in the same way Gutenberg itself does. . .not just the blocks that have already been created by Gutenberg inside the WP-Admin, but also the capabilities of what can be done to create/modify blocks. That way these decoupled applications can provide an experience comparable in some fashion to Gutenberg. If you edit a post on a Native Mobile App, Calypso or some other decoupled app, the data should be able to be saved back in the same format that Gutenberg saves it. But in order for said decoupled app to provide an experience that allows for data to be saved in the same format as Gutenberg, said decoupled app needs to know what the capabilities of each block are.

If everything is defined in client-side code that lives in the WP-Admin, we're limiting interaction with the blocks to just the WP-Admin.

The way I see this, is that blocks will have a statically defined registry which will act as the source of truth for what blocks are available, what their capabilities are, the attributes they can have, the types (string, int, bool, array, etc) of data their attributes are expecting to store, etc.

With this knowledge, a decoupled application could request this Block registration from the server, hydrate a client and render out a UI based on the Block registration.

The Gutenberg editor could also use this same registration to Hydrate what blocks are available, what UI elements are needed to provide editing of the available attributes per-block, etc.

Having the configuration defined on the server should also help reduce code duplication. If the server is the source of truth, then the client and server can both use that truth for their needs. The client wouldn't need to define block attributes and fields in addition to the server, just know how to read the source of truth from the server and do something with it, which should actually reduce JS file sizes because there's less info in there defining things, and instead just rendering data that gets served from the client.

Alternate Storage Options
Right now the block data is stored as a large string in the post_content (which is genius for backward compatibility), but I see alternative storage being explored by both core and plugin developers.

For example, where I work we will likely at least experiment with offloading Blocks and their associated meta to ElasticSearch, so that we can read/write individual blocks without having to alter the entire post_content to fix a typo in one block, for example. We will also use ElasticSearch to to aggregations, etc to get insight into how blocks are being used across our sites, etc (how we use ES is beyond the point, though). The point is that the server will need to know about the blocks so validation can be properly made when interacting with external systems.

ElasticSearch has a type mapping for the data that gets sent to it, and if there's a server side schema for blocks, plugins like ElasticPress, or even Jetpack's ElasticSearch implementation would be able to provide mapping for blocks to ES to ensure block data gets indexed properly and efficiently. Without a server side schema, the mapping would be created only as data is sent to ES and ES would give it's best guess on things, but we'd all lose out on the potential ES could give us if there were a server side Block registry that could be used to define mapping.

ElasticSearch aside, I imagine the core team or other developers will explore options for storing Block data in WordPress in some fashion. . .whether it's new wp_blocks and wp_blocks_meta tables or something else, I'm sure folks will experiment with storing blocks outside of a massive post_content string and as we've learned with the evolution of post_meta, the lack of a schema makes things "interesting" to say the least.

WPGraphQL

I'm the creator and maintainer of WPGraphQL (https://github.com/wp-graphql/wp-graphql) which brings a GraphQL API to WordPress. . .so I'll pitch my case, which I think is applicable to all the above in some fashion.

WPGraphQL allows for data to be declaratively fetched and for just the requested data to be sent to the client from the server.

An example of a WPGraphQL query right now would be:

{
  post(id:"someId") {
     id
     title
  }
}

And that produces a JSON response like so:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World"
    }
  }
}

As you can see, unlike REST where a request sends back a pretty large payload with the entire post object (unless you've written up some custom feature endpoints 🤢 ), the minimal amount of data needed is transferred from server to client, which is super beneficial, especially on slow mobile networks where data transfer can hold things up.

Anyway, with Gutenberg I envision WPGraphQL being a super powerful tool, especially for decoupled applications. I can picture folks using Gutenblocks with various block meta to indicate whether a block should be viewable on desktop or mobile, (or in our case WordPress even powers our newspaper print content). With WPGraphQL, a native mobile client could ask for just blocks that are meant for use in a native context.

For example, Let's say I've extended all of my Gutenblocks to have a "Context" multi-select which would allow me to apply various contexts for where my blocks should display (Web / Native Mobile / Print, for example).

I could have a post created by Gutenberg that has this HTML:

<!-- wp:core/heading {"align":"center", contexts:[web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show only on web</h3>
<!-- /wp:core/heading -->

<!-- wp:core/heading {"align":"center", contexts:[mobile-native, web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>
<!-- /wp:core/heading -->

I could query from a React native (or similar) app, and ask for just the blocks that were tagged with "mobile-native" context. . .something to this tune:

{
   post(id:"someId") {
      id
      title
      contentBlocks( where: { context: MOBILE } ) {
           ...on HeadingBlock {
               rawContent
               __typename
           }
      }
   }
}

I would receive a payload like:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World",
       contentBlocks: [
         {
             rawContent: "<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>",
              __typename: "HeadingBlock"
          }
       ]
    }
  }
}

My post contains multiple blocks, but my Native client was able to request just the blocks that were intended to be rendered in the Native context. This reduces network bandwidth as the data send to the client is limited to what should be sent to the client, reduces bugs in code as it's a schema with defined type system, reduces transformation on the client as the payload contains what's needed so the client doesn't need to parse/transform the Gutenblocks for all contexts to determine which ones it should use, etc. . .

With the current implementation of gutenberg_parse_blocks my hypothetical scenario is already partly possible with WPGraphQL, but the big missing piece (in this example), would be WPGraphQL knowing what Enum options there are for the Context select. . . WPGraphQL would have no knowledge of what options exist for the Context select field if it's defined in the client, not the server. It wouldn't even know that "context" was a possible attribute of a block/all blocks if the attributes weren't defined on the server.

=====

I'll stop here, as I probably just sound like I'm spouting a bunch of nonsense. . .but happy to elaborate, discuss, clarify anything I've pointed out. . .and who knows, maybe some things I've pointed out already have been addressed since the last time I had to dig through things??

@westonruter

This comment has been minimized.

Show comment
Hide comment
@westonruter

westonruter Sep 21, 2017

Member

@aduth I highly support what you've proposed here. By having a language-agnostic schema defined for what a block's attributes look like, this can only open opportunities for improvements for portability of blocks across systems. If source types can all be defined in a schema then it would open the possibility for a server-side PHP processor to be able to parse and apply transformations to a block, or to parse a block in order to extract data for an indexing service, for example. It could also then be used in mobile apps for native code to do the same. The blocks API written in JS then in Gutenberg would serve as a reference implementation of a standardized/common format.

Member

westonruter commented Sep 21, 2017

@aduth I highly support what you've proposed here. By having a language-agnostic schema defined for what a block's attributes look like, this can only open opportunities for improvements for portability of blocks across systems. If source types can all be defined in a schema then it would open the possibility for a server-side PHP processor to be able to parse and apply transformations to a block, or to parse a block in order to extract data for an indexing service, for example. It could also then be used in mobile apps for native code to do the same. The blocks API written in JS then in Gutenberg would serve as a reference implementation of a standardized/common format.

@sc0ttkclark

This comment has been minimized.

Show comment
Hide comment
@sc0ttkclark

sc0ttkclark Oct 9, 2017

+1 if we could register blocks via PHP, this would greatly improve the ability for plugins like Pods to register blocks etc. Otherwise we're stuck building our own interface for it that then outputs JS objects that we have another JS file loaded to register those JS objects as blocks in the editor.

sc0ttkclark commented Oct 9, 2017

+1 if we could register blocks via PHP, this would greatly improve the ability for plugins like Pods to register blocks etc. Otherwise we're stuck building our own interface for it that then outputs JS objects that we have another JS file loaded to register those JS objects as blocks in the editor.

@sc0ttkclark

This comment has been minimized.

Show comment
Hide comment
@sc0ttkclark

sc0ttkclark Oct 9, 2017

After a short discussion, what we want for Pods integration would be PHP registration of blocks, with some way for us to provide JS callbacks for edit/save, so that we can do what we need dynamically. In addition to this, having an ability to hook into a Gutenberg save-specific filter (what data to save) or action (saving) would be very handy for us to do some of the more complicated feature integrations that Pods would need.

sc0ttkclark commented Oct 9, 2017

After a short discussion, what we want for Pods integration would be PHP registration of blocks, with some way for us to provide JS callbacks for edit/save, so that we can do what we need dynamically. In addition to this, having an ability to hook into a Gutenberg save-specific filter (what data to save) or action (saving) would be very handy for us to do some of the more complicated feature integrations that Pods would need.

@aduth

This comment has been minimized.

Show comment
Hide comment
@aduth

aduth Oct 9, 2017

Member

If we move toward registering all blocks on the server for a context-unaware definition of a block, what's left to implement in the client editor is to support the user interactions therein: edit is the most obvious. Others like toolbar controls and inspector are included here, though currently as an implementation detail but would otherwise generally belong in the client-specific behaviors as well. Less obvious are details like: icon, category, transforms, and maybe even save. I'm inclined to treat these as client-specific (despite the text of my original comment). save is a can of worms I'm hesitant to explore: The roles it serves in the browser are important to enabling compatibility / reusability toward an edit and save duality, validating blocks, and generating markup for the text editing modes. Defining this on the server would be harmful to all these points, however exposes the possibility of handling block content updates from non-Gutenberg clients.

Member

aduth commented Oct 9, 2017

If we move toward registering all blocks on the server for a context-unaware definition of a block, what's left to implement in the client editor is to support the user interactions therein: edit is the most obvious. Others like toolbar controls and inspector are included here, though currently as an implementation detail but would otherwise generally belong in the client-specific behaviors as well. Less obvious are details like: icon, category, transforms, and maybe even save. I'm inclined to treat these as client-specific (despite the text of my original comment). save is a can of worms I'm hesitant to explore: The roles it serves in the browser are important to enabling compatibility / reusability toward an edit and save duality, validating blocks, and generating markup for the text editing modes. Defining this on the server would be harmful to all these points, however exposes the possibility of handling block content updates from non-Gutenberg clients.

@aduth

This comment has been minimized.

Show comment
Hide comment
@aduth
Member

aduth commented Oct 9, 2017

@joehoyle

This comment has been minimized.

Show comment
Hide comment
@joehoyle

joehoyle Nov 20, 2017

Seems like this should happen sooner than later, as I'd imagine many are already taking advantage of the JavaScript registration API to add support in plugins etc - it would be a shame to cause duplicative work for any 3rd parties.

Is anyone owning this / have a write up on specifically what we need to do? If the code is written, perhaps it's mainly a documentation effort needed next?

joehoyle commented Nov 20, 2017

Seems like this should happen sooner than later, as I'd imagine many are already taking advantage of the JavaScript registration API to add support in plugins etc - it would be a shame to cause duplicative work for any 3rd parties.

Is anyone owning this / have a write up on specifically what we need to do? If the code is written, perhaps it's mainly a documentation effort needed next?

@aduth

This comment has been minimized.

Show comment
Hide comment
@aduth

aduth Nov 21, 2017

Member

The most recent work here is in #2854, merged in the past week. Effectively we should have all of the pieces to start moving block registration to the server, while still supporting fully client-side block capabilities (there's still some desire to enable a client editor to be fully functional without a server part).

Member

aduth commented Nov 21, 2017

The most recent work here is in #2854, merged in the past week. Effectively we should have all of the pieces to start moving block registration to the server, while still supporting fully client-side block capabilities (there's still some desire to enable a client editor to be fully functional without a server part).

@westonruter

This comment has been minimized.

Show comment
Hide comment
@westonruter

westonruter Dec 3, 2017

Member

I see this being similar to what Customizer does with registering controls in core. They usually get registered in PHP but this is just a wrapper that passes the parameters that you passed to the client for passing into the corresponding JS API for registration. This gives opportunities for PHP plugins to know about and manipulate the controls, and the same would be true for blocks.

Member

westonruter commented Dec 3, 2017

I see this being similar to what Customizer does with registering controls in core. They usually get registered in PHP but this is just a wrapper that passes the parameters that you passed to the client for passing into the corresponding JS API for registration. This gives opportunities for PHP plugins to know about and manipulate the controls, and the same would be true for blocks.

@aduth

This comment has been minimized.

Show comment
Hide comment
@aduth

aduth Dec 3, 2017

Member

I see this being similar to what Customizer does with registering controls in core. They usually get registered in PHP but this is just a wrapper that passes the parameters that you passed to the client for passing into the corresponding JS API for registration

Effectively this does already exist, currently limited to attributes but easily expandable to any / all block properties we want to support from server registration:

// Preload server-registered block schemas.
$block_registry = WP_Block_Type_Registry::get_instance();
$schemas = array();
foreach ( $block_registry->get_all_registered() as $block_name => $block_type ) {
if ( isset( $block_type->attributes ) ) {
$schemas[ $block_name ] = $block_type->attributes;
}
}
wp_localize_script( 'wp-blocks', '_wpBlocksAttributes', $schemas );

Member

aduth commented Dec 3, 2017

I see this being similar to what Customizer does with registering controls in core. They usually get registered in PHP but this is just a wrapper that passes the parameters that you passed to the client for passing into the corresponding JS API for registration

Effectively this does already exist, currently limited to attributes but easily expandable to any / all block properties we want to support from server registration:

// Preload server-registered block schemas.
$block_registry = WP_Block_Type_Registry::get_instance();
$schemas = array();
foreach ( $block_registry->get_all_registered() as $block_name => $block_type ) {
if ( isset( $block_type->attributes ) ) {
$schemas[ $block_name ] = $block_type->attributes;
}
}
wp_localize_script( 'wp-blocks', '_wpBlocksAttributes', $schemas );

@jasonbahl

This comment has been minimized.

Show comment
Hide comment
@jasonbahl

jasonbahl Dec 6, 2017

@aduth you pointed out that this preloads server-registered blocks:

// Preload server-registered block schemas.
$block_registry = WP_Block_Type_Registry::get_instance();
$schemas = array();
foreach ( $block_registry->get_all_registered() as $block_name => $block_type ) {
if ( isset( $block_type->attributes ) ) {
$schemas[ $block_name ] = $block_type->attributes;
}
}
wp_localize_script( 'wp-blocks', '_wpBlocksAttributes', $schemas );

However, it's a completely optional thing. Gutenberg still loads up all 40 (or however many) blocks whether there are any blocks registered server side or not.

With plugin developers being asked to migrate functionality to Gutenblocks, the server-side API needs to be more of a first-class citizen.

Let's take Matias demo from WCUS as an example. He showcased editing a "Book" page that had some content and an image.

Let's say the workflow for an organization is that certain user roles can edit the content, and certain user roles can edit the image.

If all block logic is handled on the client, that means any user role can effectively edit any content, as the restriction to what they can edit is just a client-side prop that could easily be changed via the browser console giving any user power to do whatever they want. . .

The server needs to be the contract between the client and the persistence layer. We can't trust the client 💯 .

I get that the current TinyMCE is basically free-reign, and any content can be placed in there without server-side validation, but that's 1 big text input. Gutenberg is introducing 100's of new text inputs, but not introducing server-side validation/auth checks for any of them.

I feel like for a long time, when learning best practices folks would say: "Look how core does it", but when folks "look how core does it" and see Gutenberg doing things exclusively on the client with no contract on the server determining what's safe and expected, folks will follow suit, and will start writing code that trusts the client explicitly, encouraging bad practices and all sorts of security vulnerabilities down the road.

If Gutenberg is trying to be "the best editor" out there, and "leapfrog medium and squarespace", etc. . .then let's make it "the best editor" by also making sure it adheres to best practices (never trust the client, handle validation on the server, etc)

jasonbahl commented Dec 6, 2017

@aduth you pointed out that this preloads server-registered blocks:

// Preload server-registered block schemas.
$block_registry = WP_Block_Type_Registry::get_instance();
$schemas = array();
foreach ( $block_registry->get_all_registered() as $block_name => $block_type ) {
if ( isset( $block_type->attributes ) ) {
$schemas[ $block_name ] = $block_type->attributes;
}
}
wp_localize_script( 'wp-blocks', '_wpBlocksAttributes', $schemas );

However, it's a completely optional thing. Gutenberg still loads up all 40 (or however many) blocks whether there are any blocks registered server side or not.

With plugin developers being asked to migrate functionality to Gutenblocks, the server-side API needs to be more of a first-class citizen.

Let's take Matias demo from WCUS as an example. He showcased editing a "Book" page that had some content and an image.

Let's say the workflow for an organization is that certain user roles can edit the content, and certain user roles can edit the image.

If all block logic is handled on the client, that means any user role can effectively edit any content, as the restriction to what they can edit is just a client-side prop that could easily be changed via the browser console giving any user power to do whatever they want. . .

The server needs to be the contract between the client and the persistence layer. We can't trust the client 💯 .

I get that the current TinyMCE is basically free-reign, and any content can be placed in there without server-side validation, but that's 1 big text input. Gutenberg is introducing 100's of new text inputs, but not introducing server-side validation/auth checks for any of them.

I feel like for a long time, when learning best practices folks would say: "Look how core does it", but when folks "look how core does it" and see Gutenberg doing things exclusively on the client with no contract on the server determining what's safe and expected, folks will follow suit, and will start writing code that trusts the client explicitly, encouraging bad practices and all sorts of security vulnerabilities down the road.

If Gutenberg is trying to be "the best editor" out there, and "leapfrog medium and squarespace", etc. . .then let's make it "the best editor" by also making sure it adheres to best practices (never trust the client, handle validation on the server, etc)

@westonruter

This comment has been minimized.

Show comment
Hide comment
@westonruter

westonruter Dec 6, 2017

Member

It would be interesting to see block-level capabilities. In the classic editor, if an admin adds a script tag or iframe and saves the post, but then an editor comes along and updates that same post, the script/iframe will get stripped by Kses.

In Gutenberg, the Custom HTML block could be used by an admin to add a script tag or iframe. If an editor user comes along and edits the post, they should be able to do so without destroying that block's contents. This could be done by sanitizing/validating block-by-block rather than the post_content as a whole. There are a couple server side attributes here: capability, validate_callback, and sanitize_callback. If the user doesn't have the required capability, then the block should be read only and locked. If the validate_callback returns a WP_Error for a given block's attributes, then the post save could be rejected from the REST API. And then the sanitize_callback would be essentially by default just do Kses like core does already.

Member

westonruter commented Dec 6, 2017

It would be interesting to see block-level capabilities. In the classic editor, if an admin adds a script tag or iframe and saves the post, but then an editor comes along and updates that same post, the script/iframe will get stripped by Kses.

In Gutenberg, the Custom HTML block could be used by an admin to add a script tag or iframe. If an editor user comes along and edits the post, they should be able to do so without destroying that block's contents. This could be done by sanitizing/validating block-by-block rather than the post_content as a whole. There are a couple server side attributes here: capability, validate_callback, and sanitize_callback. If the user doesn't have the required capability, then the block should be read only and locked. If the validate_callback returns a WP_Error for a given block's attributes, then the post save could be rejected from the REST API. And then the sanitize_callback would be essentially by default just do Kses like core does already.

@aduth

This comment has been minimized.

Show comment
Hide comment
@aduth

aduth Dec 12, 2017

Member

Some further exploring of expanding the server bootstrapping at #3962

Member

aduth commented Dec 12, 2017

Some further exploring of expanding the server bootstrapping at #3962

@jasonbahl

This comment has been minimized.

Show comment
Hide comment
@jasonbahl

jasonbahl Apr 26, 2018

So, in digging in and actually developong a site that will be using Gutenberg in production I’m even more convinced of the importance of the Server Side API being the primary way blocks are registered.

The Gutenberg client should be treated like any other client that interacts with WordPress.

The Server Side Block registry should be the source of truth and not only include a list of blocks, but should also be where blocks express what post_types they can be edited on, what users can interact with them, what attributes they can have, what values should be allowed for the attributes, what type of field (text, rich text, etc) should handle the interactions with the attribute, and yes, even how blocks should be saved and how blocks should be edited. . .

If the client determines the save mechanism, then plugin and theme developers who make custom blocks can never have their blocks interacted with on decoupled clients like the iOS/Android apps or Calypso...because it’s custom client code that only the WP Admin would know about. . .bummer.

The Gutenberg Fields Middleware project (https://github.com/rtCamp/gutenberg-fields-middleware) is a great example showcasing that blocks can be registered fully on the server, and with essentially no custom js, I can have heaps of new blocks that can now be interacted with via the Gutenberg client AND are now possible to be exposed and interacted with via REST, WPGraphQL, WP-CLI, Calypso and the native iOS and Android apps.

——

Anyway, several issues I’m having right now with using Gutenberg for a real production project all seem to come back to this ticket:

whitelisting/blacklisting

...Both for the entire editor and for nesting (@noisysocks is making great progress, but still no other client (mobile, etc) will know the rules about what blocks can nest within other blocks, etc if the server doesn’t know and can’t expose the rules to other clients)

filtering fields away from core blocks

pretty much all the inspector fields on the paragraph block need to go for our project...would be trivial to filter out (and/or filter in new fields) had the fields been added via a server side registry like what the middleware provides. . .FWIW I’d be happy with a client filter here for my immediate needs, but a server filter ultimately makes way more sense

hydrating dynamic blocks on page load

With a good Server Side API, when the page loads the data for dynamic blocks can be generated up front and printed to the window, much like __wpBlocks, etc are now, saving the client from having to do many HTTP round trips to hydrate the dynamic blocks on the client. I have working code for hydrating dynamic blocks via GraphQL on the server, and allowing the client to take over, if necessary, and re-query for more data when variables change, etc. . .it should be easily applied to REST too. Blocks should be able to declare their data dependencies on the server and when the Editor loads, it can have all the data loaded up front. Really will help with performance on several levels. . .both perceived performance and real performance.

validation

We still believe it’s important to have the server make the final call on whether something is both valid and save. For the most part Gutenberg still ensures its safe, but doesn’t ensure accuracy in any way. If a block attribute should only be allowed to be Blue or Green, and it gets changed to “red” on the client, the server should be able to catch that, fix it, and notify the client that their input was invalid. Ideally client side validation is good enough that server side validation isn’t needed often, but we still think it’s important in many cases, especially since Gutenberg isn’t the only client, as we’ve established already. . .A robust server side api would be able to validate input from any client to make sure blocks adhere to the shape they’re intended to...this even includes validating allowed blocks, etc. . .this would mean that saving of blocks would move to the server as well...but if course that needs to happen for other clients to interact with blocks anyway.

jasonbahl commented Apr 26, 2018

So, in digging in and actually developong a site that will be using Gutenberg in production I’m even more convinced of the importance of the Server Side API being the primary way blocks are registered.

The Gutenberg client should be treated like any other client that interacts with WordPress.

The Server Side Block registry should be the source of truth and not only include a list of blocks, but should also be where blocks express what post_types they can be edited on, what users can interact with them, what attributes they can have, what values should be allowed for the attributes, what type of field (text, rich text, etc) should handle the interactions with the attribute, and yes, even how blocks should be saved and how blocks should be edited. . .

If the client determines the save mechanism, then plugin and theme developers who make custom blocks can never have their blocks interacted with on decoupled clients like the iOS/Android apps or Calypso...because it’s custom client code that only the WP Admin would know about. . .bummer.

The Gutenberg Fields Middleware project (https://github.com/rtCamp/gutenberg-fields-middleware) is a great example showcasing that blocks can be registered fully on the server, and with essentially no custom js, I can have heaps of new blocks that can now be interacted with via the Gutenberg client AND are now possible to be exposed and interacted with via REST, WPGraphQL, WP-CLI, Calypso and the native iOS and Android apps.

——

Anyway, several issues I’m having right now with using Gutenberg for a real production project all seem to come back to this ticket:

whitelisting/blacklisting

...Both for the entire editor and for nesting (@noisysocks is making great progress, but still no other client (mobile, etc) will know the rules about what blocks can nest within other blocks, etc if the server doesn’t know and can’t expose the rules to other clients)

filtering fields away from core blocks

pretty much all the inspector fields on the paragraph block need to go for our project...would be trivial to filter out (and/or filter in new fields) had the fields been added via a server side registry like what the middleware provides. . .FWIW I’d be happy with a client filter here for my immediate needs, but a server filter ultimately makes way more sense

hydrating dynamic blocks on page load

With a good Server Side API, when the page loads the data for dynamic blocks can be generated up front and printed to the window, much like __wpBlocks, etc are now, saving the client from having to do many HTTP round trips to hydrate the dynamic blocks on the client. I have working code for hydrating dynamic blocks via GraphQL on the server, and allowing the client to take over, if necessary, and re-query for more data when variables change, etc. . .it should be easily applied to REST too. Blocks should be able to declare their data dependencies on the server and when the Editor loads, it can have all the data loaded up front. Really will help with performance on several levels. . .both perceived performance and real performance.

validation

We still believe it’s important to have the server make the final call on whether something is both valid and save. For the most part Gutenberg still ensures its safe, but doesn’t ensure accuracy in any way. If a block attribute should only be allowed to be Blue or Green, and it gets changed to “red” on the client, the server should be able to catch that, fix it, and notify the client that their input was invalid. Ideally client side validation is good enough that server side validation isn’t needed often, but we still think it’s important in many cases, especially since Gutenberg isn’t the only client, as we’ve established already. . .A robust server side api would be able to validate input from any client to make sure blocks adhere to the shape they’re intended to...this even includes validating allowed blocks, etc. . .this would mean that saving of blocks would move to the server as well...but if course that needs to happen for other clients to interact with blocks anyway.

@jasonbahl

This comment has been minimized.

Show comment
Hide comment
@jasonbahl

jasonbahl May 2, 2018

I’m back 😉

Another issue that comes back to having a good Server Side API: translations

I posted my thoughts here: https://make.wordpress.org/core/2018/05/01/javascript-internationalization-the-missing-pieces/

But it seems like translations can be solved by having the client ask for things like registered blocks and the block inspector fields, (labels, descriptions, etc) and the server respond with these things already translated. 🤔

jasonbahl commented May 2, 2018

I’m back 😉

Another issue that comes back to having a good Server Side API: translations

I posted my thoughts here: https://make.wordpress.org/core/2018/05/01/javascript-internationalization-the-missing-pieces/

But it seems like translations can be solved by having the client ask for things like registered blocks and the block inspector fields, (labels, descriptions, etc) and the server respond with these things already translated. 🤔

@gziolo

This comment has been minimized.

Show comment
Hide comment
@gziolo

gziolo May 2, 2018

Member

Another issue that comes back to having a good Server Side API: translations

It would only work for simple use cases. As soon as the number of plugins grows, you will want to have some of them to be lazily loaded when they are really necessary for your application to operate as expected. The same applies to translations. That's why at some point they should be controlled by the JS code and be bundled together with the corresponding logic that uses them.

Member

gziolo commented May 2, 2018

Another issue that comes back to having a good Server Side API: translations

It would only work for simple use cases. As soon as the number of plugins grows, you will want to have some of them to be lazily loaded when they are really necessary for your application to operate as expected. The same applies to translations. That's why at some point they should be controlled by the JS code and be bundled together with the corresponding logic that uses them.

@jasonbahl

This comment has been minimized.

Show comment
Hide comment
@jasonbahl

jasonbahl May 2, 2018

@gziolo with a proper server side API and the core Gutenberg client JS shaped to be an API itself (like the direction Gutenberg Gields Middleware has gone) there wouldn’t need to be custom JS for most plugins.

If Gutenberg core took what Gutenberg Fields Middleware is doing, and expanded it further, providing all the components needed to build flexible Interfaces, then users could register their blocks and block controls, their data dependencies, and even the edit experience from the server.

The Gutenberg interface is a tree of React components. If you can pass a tree of JSON from a server-side-api to the client, the client can build out whatever UI you tell it, without plugins writing a line of custom JS.

Obviously that’s not the case now, but it’s all possible if Gutenberg re-shaped things.

As it stands though, plenty of things are lazy loaded from the server, like contents of Dynamic blocks, which are already fully translated by the server. Take that a step further where all blocks are registered by the server and all blocks (not just their contents, but their field control labels, etc too) would be already translated when Gutenberg loads them.

If enough components and groups of components are mapped to a server side registry (like Gutenberg Fields Middleware allows) the only JS needed for plugins should be the js already loaded by Gutenberg core. And then custom JS from plugins could still be added for more extreme situations, but the majority of plugins could do what they need to do via server registration of their blocks (and plugins, and other parts of the Gutenberg UI)

I can’t see this happening anytime soon because it would be a major shift in things, but ultimately it will make WordPress more useful to all clients (CLI, iOS, Android, REST, and the client known as Gutenberg).

ACF Flex Fields is conceptually very similar to Gutenberg. Each flex field group is pretty much the same as a block, and when you interact with the flex fields it all happens via a JS client, but you don’t have to write a line of JS to register flex field groups or the fields within them. . .and ACF flex fields can be interacted with via any other client. So plugin developers can register any flex field you could dream of and any client can have the knowledge of it with just the initial core client code to read the API.

Gutenberg can get there.

Even the Customizer is a great example of how server side APIs can create dynamic experiences without writing any custom JS. I can register a widget area and some widgets and I now have “blocks” that I can interact with in Customizer, and external clients can interact with as well if needed without any custom JS.

I think folks should still be able to extend Gutenberg with custom JS, but it should really become the exception, not the rule.

jasonbahl commented May 2, 2018

@gziolo with a proper server side API and the core Gutenberg client JS shaped to be an API itself (like the direction Gutenberg Gields Middleware has gone) there wouldn’t need to be custom JS for most plugins.

If Gutenberg core took what Gutenberg Fields Middleware is doing, and expanded it further, providing all the components needed to build flexible Interfaces, then users could register their blocks and block controls, their data dependencies, and even the edit experience from the server.

The Gutenberg interface is a tree of React components. If you can pass a tree of JSON from a server-side-api to the client, the client can build out whatever UI you tell it, without plugins writing a line of custom JS.

Obviously that’s not the case now, but it’s all possible if Gutenberg re-shaped things.

As it stands though, plenty of things are lazy loaded from the server, like contents of Dynamic blocks, which are already fully translated by the server. Take that a step further where all blocks are registered by the server and all blocks (not just their contents, but their field control labels, etc too) would be already translated when Gutenberg loads them.

If enough components and groups of components are mapped to a server side registry (like Gutenberg Fields Middleware allows) the only JS needed for plugins should be the js already loaded by Gutenberg core. And then custom JS from plugins could still be added for more extreme situations, but the majority of plugins could do what they need to do via server registration of their blocks (and plugins, and other parts of the Gutenberg UI)

I can’t see this happening anytime soon because it would be a major shift in things, but ultimately it will make WordPress more useful to all clients (CLI, iOS, Android, REST, and the client known as Gutenberg).

ACF Flex Fields is conceptually very similar to Gutenberg. Each flex field group is pretty much the same as a block, and when you interact with the flex fields it all happens via a JS client, but you don’t have to write a line of JS to register flex field groups or the fields within them. . .and ACF flex fields can be interacted with via any other client. So plugin developers can register any flex field you could dream of and any client can have the knowledge of it with just the initial core client code to read the API.

Gutenberg can get there.

Even the Customizer is a great example of how server side APIs can create dynamic experiences without writing any custom JS. I can register a widget area and some widgets and I now have “blocks” that I can interact with in Customizer, and external clients can interact with as well if needed without any custom JS.

I think folks should still be able to extend Gutenberg with custom JS, but it should really become the exception, not the rule.

@aduth

This comment has been minimized.

Show comment
Hide comment
@aduth

aduth Sep 13, 2018

Member

For posterity's sake, and as discussed previous in #5649, #5652, and #5099, server-registered blocks will need to be able to handle common supports types in a generalized fashion (not implemented per-block): customClassName, align, etc.

Member

aduth commented Sep 13, 2018

For posterity's sake, and as discussed previous in #5649, #5652, and #5099, server-registered blocks will need to be able to handle common supports types in a generalized fashion (not implemented per-block): customClassName, align, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment