WordPress source V4 & Theme Epic

[TylerBarnes]

Intro

This epic is to track a new major version of gatsby-source-wordpress which is currently being worked on. This will be a ground-up rewrite using WPGraphQL instead of the REST API in order to enable better caching, previews, incremental builds, and an overall better developer experience.

How it will be done

The next major version of gatsby-source-wordpress is V4, and the release will coincide with the release of WPGraphQL V1. This project will inform some potential changes in WPGraphQL as a concrete example of a framework implementing many WordPress conventions and functionalities headlessly via WPGQL. There will also be a WPGatsby WordPress plugin that will modify the WPGQL schema when changes are Gatsby-specific, as well as add support for incremental builds and preview on the WP side.

Additionally, an opinionated theme, gatsby-theme-wordpress, will be built and it will compose a few new plugins together to implement many WordPress conventions and functionalities such as the template hierarchy, automatic routing (page creation using the WP permalink structure), ability to use the WP admin template selector, easy use of WP menus, and more.

Philosophy

We want to enable you to do what you're already doing with WordPress and more.

The intention behind the theme is to reduce friction as much as possible both for WordPress developers coming to Gatsby and for anyone else that's interested in using WP and Gatsby together for a wonderful and familiar admin experience. We want to lower the barrier of entry and make it as easy as possible for everyone to use Gatsby and React with WordPress. As a very large portion of the web is built on WordPress, our hope is that this will empower many more people to build React apps/sites and push the WordPress community and the internet as a whole forward.

The intention behind the source plugin is to create a scalable, cacheable integration with WordPress. It will support Gatsby cloud Preview and incremental builds and will generally be more efficient than current solutions and will have an improved DX.

Responsibilities

I will be leading the new source plugin and theme, and @jasonbahl will of course be leading WPGraphQL V1. The two of us will be leading the WPGatsby plugin together.

Github Project Links

• gatsby-source-wordpress@v4/gatsby-theme-wordpress
• WPGraphQL V1

Initial scope

This will likely change as things progress, but for now:

• gatsby-source-wordpress@v4

  • Incremental builds support and aggressive cacheing
  • Preview support
  • GraphQL introspection and full data download
    • Fetch any types that are lists of nodes
    • Fetch specific types that are not lists of nodes (such as options)
  • Use schema customization to add types from WPGQL to Gatsby via introspection
  • Recursively process schema
    • Connections of the same type as one of our Gatsby nodes should be linked to Gatsby nodes instead of pulling queried data from WPGQL connections
    • Process lists of multiple types
  • Media items
    • Fetch all media posts
    • Fetch media files only when they’re queried for
    • Download and process all media files in html
  • Add request rate limiting controls
    • Make some importable defaults for “shared”, “vps”, “dedicated”, and “local”?

• WPGatsby

  • Action monitor post type
    • a post is created for every content-related action performed in wp
    • recent actions are exposed to the WPGQL schema for cache invalidation and pulling new content in Gatsby

• gatsby-wordpress-core

  • gatsby-wordpress-permalink-structure
    • Get entire WordPress site structure and build pages using the path of each page or post
    • This needs to play nice with template sync and hierarchy
  • gatsby-wordpress-template-hierarchy
    • Needs to play nice with themes and shadowing https://www.gatsbyjs.org/blog/2018-11-11-introducing-gatsby-themes/#sub-themes-and-overriding ← this is how it will
    • Replicate https://wphierarchy.com/ using folders instead of dashes for organization. For ex “archive-post.php” would become “templates/archive/post.js”
    • Only replicate a subset of the hierarchy for now (maybe parity later if needed?)
    • Fork https://github.com/pressupinc/wphierarchy and make it into a gatsby theme for documentation purposes
  • gatsby-wordpress-template-sync
    • Makes gql mutations sending available templates back to WP
    • Need to think about how this will work with multiple frontends (staging, production, local, etc)
  • gatsby-wordpress-menus
    • Should enable using a menu slug or location in a React component to display a menu. See https://github.com/TylerBarnes/wordsby/wiki/Menus for a related implementation

[TylerBarnes]

Some work has already started in this PR

[henrikwirth]

gatsby-wordpress-permalink-structure for this, it would be great to keep internationalization in mind with common plugins like Polylang and WMPL. I had to do a lot of custom work for the site I'm working on right now, to make sure I have proper paths. Also isFrontPage would be nice to have for each separate Language etc...

In general it would be cool to see some approaches for making internationalization easier. Maybe thats what you already mean with Multi-language support.

[TylerBarnes]

@henrikwirth I agree!
Most of that depends on the schema on WPGQL side. I think the Gatsby schema for this plugin should be as close as possible to the original schema to reduce maintenance burden, while still retaining familiar Gatsby schema patterns like input arguments and root field name patterns (allXyNodeName for ex). As much as possible I'd like to not add additional fields to nodes on the Gatsby side except in transformer plugins. I believe @jasonbahl is working on adding isFrontPage to the schema and that would automatically then show up in this plugin with no additional work on the Gatsby side. I feel like for 99% of the schema, that's the strategy we should take.

For multi-language support I do mean internationalization, and that will also depend on support on the WPGQL side. Since this plugin will be automatically transforming the WPGQL schema and creating nodes out of that, just enabling support for internationalization in polylang or WPML on the WPGQL side may be enough depending on how the schema is set up. It may also not be enough and then we'll need to have some mechanism for plugins to hook in to the schema transformation, node creation, and the permalink structure plugin on the Gatsby side.

In any case, internationalization is important and needs to be added, but we're looking to get most of the core features of this integration working first. At this point I think we're not 100% sure what it should look like and if you have any ideas I'd love to hear them.

[TylerBarnes]

@henrikwirth good news!
I just tested this WPGQL polylang extension and as far as I can tell it will work out of the box with the planned source/theme. It adds all posts of all languages to each post types root field. So querying for posts in WPGQL will return posts of all languages. Since the permalink structure plugin will just loop through every available page/post and create a new page using the WP permalink path as the Gatsby page path, it will add all languages to the site by default if that plugin is activated.

It looks like there isn't a WPML plugin but it probably wouldn't take too much effort for someone to fork that Polylang WPGQL extension and make it work for WPML instead.

[TylerBarnes]

You can see a video demo of this that I made for WCUS at https://www.youtube.com/watch?v=41blWYDQafM

[henrikwirth]

You can see a video demo of this that I made for WCUS at https://www.youtube.com/watch?v=41blWYDQafM

@TylerBarnes Looks very promising. Is this already working with what is in the PR? Do you have an example repo for it? Would love to test it out.

[TylerBarnes]

@henrikwirth yep, #19101 contains the demo in examples/using-gatsby-source-wordpress, the Gatsby plugin in packages/gatsby-source-wordpress and the WP plugin in packages/WPGatsby. The example is actually broken now due to some changes I've made in switching from GQL inference to using schema customization. The WPGQL side needs some changes now to make the demo keep working as in the video.

[moonmeister]

@TylerBarnes If we're starting to move a full-stack WP to Gatsby + Headless WP right now, What can we do to optimize for this major change? What kind of choices can we make now that would mean less work later? I'm guessing Using WPGraphQL and gatsby-source-graphql for now? Then once gatsby-source-wordpress and the WPGatsby plugins are finished we can add and switch to those?

Also, any chance we have a rough timeline for when this work will be done?

Thanks,

Alex.

[michaellopez]

@moonmeister we are exploring options too. We are also creating Gatsby builds for several blogs with hundred thousands of posts. So we are also looking for recommendations

[TylerBarnes]

@moonmeister @michaellopez the plan is to have this ready for spring 2020. Even though there was a working demo, I don't consider this anywhere close to production ready as it is now. I'm working on this full time though so things should keep moving along fairly quickly. Right now I'm working on getting this draft PR into a very early stage where all the pieces of the source plugin work together properly. I'll then be breaking the remaining work up into smaller issues & PR's, so you'll be able to monitor progress and help out if you'd like. I'll post an update here when I get it to that stage.

For switching/upgrading we're trying to as much as possible keep the schema close to the WPGQL schema, so the upgrade path from using gatsby-source-graphql to gatsby-source-wordpress@v4 should be fairly painless. I think we will have some documentation on how to switch over or upgrade when the plugin is ready.

[michaellopez]

@TylerBarnes That sounds great! Looking forward to it.

We have a new years deadline so we are looking to find shortcuts of implementing the gatsby-source-wordpress@v4 workflow/implementation (I've read the source, both Gatsby plugin and WordPress plugin and it is quite impressive and elegant in its simplicity). As we have quite strict rules we can follow and eliminate quite a bit of logic that you most likely will need to have in the "public-for everyone-production-version" of the plugin. Ours can be quite restricted as it kinda only needs to handle posts and authors in a very straightforward and limited fashion.

So yeah, hopefully we can contribute back some of our code to gatsby-source-wordpress@v4 once #19101 gets to a place where contributions are more easily made. Right now I just think that we would get in your way. Better you iron out the big picture then maybe we can come in and help out when its easier to scope/define the tasks needed.

Exciting times! 🎉

[TylerBarnes]

I've released a working alpha version of the new version of gatsby-source-wordpress. If you're interested you can check it out by installing the starter at https://github.com/TylerBarnes/using-gatsby-source-wordpress-experimental. For now the "documentation" is very rough and is just the stream of consciousness README in that repo. Better docs to come! There will be breaking changes and not all features listed in this epic are supported yet, but for many it will probably already be an improvement over the existing solutions. Please try it out and let me know if you run into any problems.
If you need any help, you can find me in the WPGraphQL #gatsby slack channel https://wpgql-slack.herokuapp.com/

[moonmeister]

@TylerBarnes Excited to try this out, I can't seem to find the source code anywhere in the Gatsby repo...do you know where I can check this out? It's not in master or https://github.com/gatsbyjs/gatsby/tree/feat/gatsby-source-wordpress-v4.

[michaellopez]

@TylerBarnes Great work! We have actually been following closely since its inception and we are developing a solution on top of it currently. We have a very simple use case so this initial alpha works pretty good for us. We have done some custom code too here and there, will try and find you on Slack to discuss if our code changes are beneficial to the generic Gatsby project.

[michaellopez]

@moonmeister The Gatsby source plugin is here https://github.com/TylerBarnes/gatsby/tree/feat/source-wordpress-v4/packages/gatsby-source-wordpress-experimental and the companion Wordpress plugin is here https://github.com/TylerBarnes/gatsby/tree/feat/source-wordpress-v4/packages/wp-gatsby

[TylerBarnes]

@michaellopez that's awesome, I'd love to hear more about what you've done

[TylerBarnes]

Hey all! I've moved the repo here https://github.com/gatsbyjs/gatsby-source-wordpress-experimental and added some improved docs. We're pushing towards moving into beta soon!