Render shortcode previews within Gutenberg

[notnownikki]

I thought we could use server side rendering to have a base settings object for shortcode content, and extend it for specific shortcodes.

Looking at the embed block as an example of how content can be rendered server-side, and speed up the creation of new blocks (the embed block supports all embeds that WP_oEmbed does, simply by using WP_oEmbed server side for rendering) I thought we could do something similar for shortcodes.

We'd need a new API endpoint to render shortcodes, and a base settings object would take values from the block properties and produce a shortcode as it would be entered in a post, e.g. [myshortcode foo="bar" bar="bing"]

The base settings' save would just construct the shortcode itself, in the same way that the embed block just saves the URL to be embedded.

Each concrete shortcode block would only have to add the UI for inputting the values needed for the shortcode, and this would allow us to create a large number of blocks in a short time. (We could even automate the creation of the blocks based on the shortcodes registered, and refine the UI manually.)

[aduth]

Related: #334

[westonruter]

Would be good to loop-in Shortcake contributors here. They've done a ton of work on adding a shortcode UI to the editor, creating what I think is a great prototype for blocks itself. They'd be able to share a lot of knowledge for how they went about previewing shortcodes in the editor, as well as providing a UI for manipulating the shortcode attributes.

[mattheu]

Hey, I'll share some of our experiences with Shortcake.

I thought we could use server side rendering to have a base settings object for shortcode content, and extend it for specific shortcodes.

We took exactly this approach. We used the WordPress core Tiny MCE views functionality and created a generic shortcode view that makes an AJAX request to fetch the shortcode preview fully rendered in PHP.

I think this solution generally works well. It means that most Shortcodes just work out of the box and developers can customise the style in their editor stylesheet if necessary.

What are the downsides?

1. Simple HTML shortcodes work great, but anything with JS is more problematic. We added hooks to our render code that allowed developers to get things working by loading their JS. Sometimes this was as simple as loading the JS dependencies and initializing the functionality. But often this wasn't so easy. For example, the front end JS often makes assumptions about the page DOM structure which doesn't hold true when rendered in TinyMCE. However I feel like this was still a reasonable solution - and it was mostly possible to get things working, only it didn't 'just work' as easily as promised.
2. Shortcodes often rely on the context of the WordPress page. e.g. they call is_singular() or get_the_ID(). Whilst it's possible to spoof this and try to get things working, its not so nice. We went as far as setting global post.. I know @westonruter has thought about this problem a bit so maybe he has some ideas :).
3. The preview was sometimes a little slow to render. I'd love for the interface to be snappier but because it had to wait for a request to complete, we had to display loading animation as the preview whilst this happened. It might be nice to allow developers to register a client side preview for shortcodes if they want to, and using the AJAX/PHP render as a default.

Other thoughts

Using the core TinyMCE Views meant we actually didn't need to do all that much to get this working. That handles some smart things such as sandboxing previews that contain JS inside an iframe and resizing them. If you're implementing this from scratch, that is probably something you'll need to think about.

[westonruter]

Shortcodes often rely on the context of the WordPress page. e.g. they call is_singular() or get_the_ID(). Whilst it's possible to spoof this and try to get things working, its not so nice. We went as far as setting global post.. I know @westonruter has thought about this problem a bit so maybe he has some ideas :).

Yeah, this is the problem space that Selective Refresh in the Customizer addresses. It ensures the proper context for a given post by doing a request to the specific URL that the partial will be rendered on, so the $wp and $wp_query globals will be set as expected, but then at template_redirect it short-circuits and just renders the partials instead of proceeding with rendering the normal template.

[aduth]

Related: Some resizable / sandboxed iframe exploration previously at #754 (since closed).

Originally sourced from Calypso: Resizable Iframe. The Calypso Shortcode component is an interesting example as well (manages server rendering and communication between sandboxed resizable iframe).

[notnownikki]

Yeah, the resizable iframe doesn't do everything, there's some deep magic going on in Calypso to make it work. We should open an issue specifically for that...

…

On 15 Jun 2017 18:13, "Andrew Duthie" ***@***.***> wrote: Related: Some resizable / sandboxed iframe exploration previously at #754 <#754> (since closed). Originally sourced from Calypso: Resizable Iframe <https://github.com/Automattic/wp-calypso/tree/master/client/components/resizable-iframe>. The Calypso Shortcode <https://github.com/Automattic/wp-calypso/tree/master/client/components/shortcode> component is an interesting example as well (manages server rendering and communication between sandboxed resizable iframe). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1054 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADKSss8_c_dgsFJX3-NOi6MhEQdQIDYKks5sEWZXgaJpZM4NyjPs> .

[notnownikki]

This is very useful, thank you!

…

On 15 Jun 2017 09:13, "Matthew Haines-Young" ***@***.***> wrote: Hey, I'll share some of our experiences with Shortcake. I thought we could use server side rendering to have a base settings object for shortcode content, and extend it for specific shortcodes. We took exactly this approach. We used the WordPress core Tiny MCE views functionality and created a generic shortcode view that makes an AJAX request to fetch the shortcode preview fully rendered in PHP. I think this solution generally works well. It means that most Shortcodes just work out of the box and developers can customise the style in their editor stylesheet if necessary. What are the downsides? 1. Simple HTML shortcodes work great, but anything with JS is more problematic. We added hooks to our render code <https://github.com/wp-shortcake/shortcake/blob/master/inc/class-shortcode-ui.php#L456> that allowed developers to get things working by loading their JS. Sometimes this was as simple as loading the JS dependencies and initializing the functionality. But often this wasn't so easy. For example, the front end JS often makes assumptions about the page DOM structure which doesn't hold true when rendered in TinyMCE. However I feel like this was still a reasonable solution - and it was mostly possible to get things working, only it didn't 'just work' as easily as promised. 2. Shortcodes often rely on the context of the WordPress page. e.g. they call is_singular() or get_the_ID(). Whilst it's possible to spoof this and try to get things working, its not so nice. We went as far as setting global post. <https://github.com/wp-shortcake/shortcake/blob/master/inc/class-shortcode-ui.php#L437>. I know @westonruter <https://github.com/westonruter> has thought about this problem a bit so maybe he has some ideas :). 3. The preview was sometimes a little slow to render. I'd love for the interface to be snappier but because it had to wait for a request to complete, we had to display loading animation as the preview whilst this happened. It might be nice to allow developers to register a client side preview for shortcodes if they want to, and using the AJAX/PHP render as a default. Other thoughts Using the core TinyMCE Views meant we actually didn't need to do all that much to get this working. That handles some smart things such as sandboxing previews that contain JS inside an iframe and resizing them. If you're implementing this from scratch, that is probably something you'll need to think about. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1054 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADKSsuYLWkEJITkMZEmh86l5Afu0zTr7ks5sEOfDgaJpZM4NyjPs> .

[mtias]

A simple first pass, and maybe all we need for now—since we want to encourage shortcode authors to port them to native blocks—is to have a "shortcode" block that gives you a textarea, maybe with [] delimiters in place that gets saved as is.

[westonruter]

@mtias I suppose something similar should be done for making legacy widgets available as blocks? In that case, we'd have the widget form that could be displayed in the inspector pane.

[dgwyer]

Is there any info yet on converting shortcodes to native blocks? I'm looking to write some new shortcodes and want them to be fully compatible with Gutenberg from the start.

Or, would it be better to focus on creating custom blocks instead?

[westonruter]

@dgwyer You should probably create both a shortcode and a block for now, but have them share as much logic as possible, such as in the rendering function.

[jasmussen]

Since shortcodes work in text blocks and Classic Text, and there's also a separate shortcode block, can we close this as addressed for now?

[notnownikki]

If we're happy with how the shortcode block renders, I guess we can. Although I would have liked to have seen rendering done in the way the embed blocks do, where instead of a placeholder, they show the rendered content as it would appear in the post. I get that it's tricky to do though, so I'll leave this up to someone else to decide.

[jasmussen]

I think that's a fine reason to keep this open. Adding an "Enhancement" label.

[robmcclel]

Team Gutenberg - My theme has a lot of shortcodes for clients so they can easily perform special formats and such (embed books, star ratings for reviews, etc). In testing Gutenberg, a couple of things jump out, and this seemed like the support thread dedicated to it.

First, I definitely think that shortcodes should render in the Gutenberg editor. I realize that is probably a huge PITA, but if the editor is going to be marketed as a more visual editor, then that's just how its going to have to be. Either visualize stuff or don't, but going halfway with it is just aggravating and confusing to everyone - in other words, it's not an improvement over the current method. So, yeah, shortcodes needs to render in Gutenberg.

The "shortcode" block is confusing - I'm not sure what its supposed to accomplish. I get that it's for inserting a shortcode, but how is that supposed to work in the context of composing a document? A lot of shortcodes are built to flow into the text, or at least wrap around them. How does the "shortcode" block assist with that? I can see such a thing being great for "click to tweet" or a similar type of shortcode insert where it is standalone and is intended to take up an entire line/block - but there are a lot of shortcodes that are intended to be a part of the text. Will there be an area in the "Block Editor" panel for when a shortcode block is activated that will hook in and list/activate/provide buttons for shortcodes? How will those be declared? It gets complicated in a hurry - tinyMCE has solved this issue already, so maybe we should look to that as a basis for shortcode insertion.

As also noted here and elsewhere - where will plugin buttons intended for the editor reside? I see there is now a "classic editor" block which brings in the tinyMCE (sort of, I see some items missing in the Gutenberg MCE vs the actual MCE - I assume it is a refinement in the hook). So, when will hooks for the Gutenberg editor be available and where will those new buttons be placed? I don't think it's a good idea to have two text block editor systems - make one that works and stick to it.

Also - I don't see this issue as an "enhancement" Shortcodes are a key part of WordPress and a lot of sites, themes, and plugins depend on them. There are few other ways to insert code into posts and pages. This needs to be addressed and, IMO, a big part of that starts with rendering shortcodes in the Gutenberg visual editor.

Just my two cents - and I have very happy to test this issue ad-nauseum, as, well, pretty much my entire business depends on getting this right. I'm betting I'm not the only one.

[amandablum]

Just a note- it would be nice to be able to add styles to a shortcode block... I can think of plenty of reasons. Since you can't, and above it was noted you can add a shortcode to a text block, I figured that would solve the problem. But it doesn't, because the shortcode content lives outside of the p style wrapper.

1. First, you can only add a shortcode to a P block in html mode. doing so in visual mode will turn it into a shortcode block.
2. If you edit a pblock in html and add a shortcode, and then add a style to the block and save it, on the front end, the p and the style will be rendered before the shortcode, not around it.
3. using a classic text block will not render the shortcode at all. (see image below)
   
4. If you convert a shortcode block to a reusable block, you can add a style. But the same thing happens as above. On front end render, the style is gone.

I think there's great utility in being able to wrap shortcode content in a div. I realize much of this content will bring in its own styles and containers, but for Gutenberg to work, we need to be able to control how that content is handled on the page. Right now, it just serves it inline.

[niranjan-uma-shankar]

Thanks for the insightful comments everyone. I'm looking into this ticket, and I am sharing some of my initial research. I read up on a few related tickets, and also went through how embed work. I've collated stuff in the document below, and I've added a few follow up questions that need clarifications.

Please check the document here : https://docs.google.com/document/d/1xIajOV9Uoh5XXOLfwCPKO_Z0_BZxls1gXEH_KWhWP9c/edit?usp=sharing

Feedback/inputs, especially on the points mentioned in the "Next Steps" section would be helpful!

[niranjan-uma-shankar]

It would be nice to have an interface in which the user can choose to preview by clicking a preview tab or button, instead of auto-rendering the preview. Something like this perhaps :

Advantages I can think of in this interface :
(1) The tabbed interface is similar to the HTML block, so no learning curve for the user. Also, reuse of existing components, leading to a uniform, consistent experience
(2) Makes the editor interface somewhat snappier to load, without having to load several previews when the page loads
(3) Most importantly, allows the user to edit shortcodes and preview again the edited output

Experience would be something like this :

Suggestions are welcome. @jasmussen

[jasmussen]

I like the idea of being able to preview shortcodes. But I think we should leverage the block design guidelines instead of using the tabbed interface, which I would consider a one-off for the custom HTML block.

The key principle is that an unselected block is a preview, and a selected block can surface extra input fields and controls. For example a button block shows a URL input field when selected.

Couldn't it work well if the shortcode block always showed a preview when unselected, and when selected it surfaced the input field for the shortcode itself?

ASCII mockup:

Selected shortcode block:

[product_category]
__________________

           
 Apple Products

Unselected shortcode block:

           
 Apple Products

[mtias]

Couldn't it work well if the shortcode block always showed a preview when unselected, and when selected it surfaced the input field for the shortcode itself?

I'm conflicted here, because it will signal users that these are just like any other content in their page, but shortcodes are edited abstractly.

[niranjan-uma-shankar]

Thanks for your thoughts, @mtias and @jasmussen!

I'm conflicted here, because it will signal users that these are just like any other content in their page, but shortcodes are edited abstractly.

I echo the same thoughts Matias - shortcodes are "different" given that they can be edited abstractly - the preview approach would signal this nicely. Would be good to hear @jasmussen's take on this. I can experiment with the block design UI too, if we feel having it makes more sense. Would be good to get opinions @notnownikki @youknowriad @aduth

@mtias @jasmussen - it would be awesome if you could review the pull request I've referenced and give it a spin :)

[jasmussen]

Matías has convinced me. It was hard to put my finger on it, but shortcodes are code, just like HTML is code, which makes it different from visual blocks. As such, perhaps your initial instinct of tabs was correct. Okay to the tab preview 👍

[niranjan-uma-shankar]

While discussing with @ehg, an idea that came up is if we can automatically convert the shortcode block to an embed block or gallery block, depending on whether the shortcode entered is [embed] or [gallery]. Similarly, auto-convert to a video or audio block, if the shortcode entered is [video] or [audio]. This would impress upon the user that they should rather be using an available block, instead of using shortcodes to achieve the same effect. Putting this out here to gather thoughts from everyone.
If the block convert idea appeals, it would be good to have UX advice from @jasmussen and @mtias on how this could be achieved. When do we prompt the user to make the block switch? Maybe when they click the preview tab, we can find the type of shortcode and ask the user if they would like to convert the block. However, what if the user doesn't click the preview tab, how do we then prompt him to switch block?
We could add an option in the overflow menu specially for shortcode blocks, though I'm not convinced that would be a good option either.

[Lewiscowles1986]

Has this solved the loading of scripts and localising? #5214 semi-solves although there can be a lot of noise from the theme and as-yet there is no front-end for gutenberg (I tested by pasting into a html block). Both things this seems to have, with less manual steps and a similar UI.

[Lewiscowles1986]

any way I can help with this?

[westonruter]

@Lewiscowles1986 review and test #4710.

[Lewiscowles1986]

Been referred back here by @niranjan-uma-shankar I agree with #1054 (comment) (I know they changed their opinion).

Design users are not always technical users, clicking the block (or clicking to edit) should allow technically competent users to alter a shortcode, but the value of a shortcode is it's output. We don't value to code of them, so I'd suggest it's easier to think about most things from an output point of view, rather than how the input is distinct.

I've been experimenting and I prefer the default to preview view, with no toolbar, even as a technical user. Even though selecting the preview itself is a little less than easy with me having to use the right control to select the block from preview mode. I checked and that is the same with or without the tabs as present in the master branch of #4710

[danielbachhuber]

@notnownikki @westonruter Can we file this to the Ideas project and close? I don't think shortcode preview is a Gutenberg v1 requirement — developers that want previews should implement block-equivalents. Server-Side Rendering seems like it addresses our immediate needs.

[strarsis]

+1 for shortcode/widget server-side preview rendering.

I found two plugins that offer this functionality, so it is possible and the code is there:

• Simple Shortcode Block
• CD2 Gutenberg Shortcode Block Preview

[Lewiscowles1986]

Hi @strarsis CD2 Gutenberg Shortcode Block Preview is Open Source and on GitHub for contributors.

Early on, it caused a lot of frustration for me as most people I was working with that used WordPress jetisoned it; then because the JS API for blocks kept moving and I didn't have the workflow to keep up.

The plugin is still tested on a demo WordPress I have, so it seems to work with modern WP; but If I'm honest I'd go a different route if your clients really need the features.

[strarsis]

@Lewiscowles1986: Sadly the CD2 Gutenberg Shortcode Block Preview block has been blocked/deprecated in the WordPress Plugin directory.

[Lewiscowles1986]

I blocked it as keeping up with the API changes and theme / plugin peculiarities for a plugin with such low uptake seems foolhardy and is frustrating. Especially if you receive no errors from the console when things fail, but a redirect loop.

5.4 update saw a number of things stop working and I kinda feel like if I'm only using your API's and things constantly seem to break, then it feels like they are being deliberately broken.

The code is still available via SVN and GitHub should you choose to build or fork the plugin it's got two parts.

Automated testing of the plugin was always a problem as with most of WordPress, you're effectively e2e testing. That takes a fair amount of effort.

Update
I got it to build and with some minor edits (using de-construction for wp.url instead of npm package) things worked just as they did before, which is odd and frustrating. I still don't recommend it's use, but I published a new release on GitHub tested on latest 5.4 with 2020 theme. YMMV and I do think it is right this has closed.