Refactor internal data pipeline

[KyleAMathews]

At the heart of Gatsby is a data pipeline that extracts data from files, transforms the data, passes the data to React.js wrapper components, and finally assembles these components into an app using React Router.

But the code that does this doesn't cleanly reflect this pipeline. In thinking this through, there's also some valuable extension points that could be added. A site should be able to intervene if desired at each step of this pipeline. This refactor would also make the code much more testable.

Some interesting possibilities this refactor should unlock.

You'd be able to easily integrate with a 3rd party CMS (or perhaps legacy CMS). Gatsby right now is tied to using files for pages that are actually on your hard drive. But with the right hook, a site could fetch pages from another system and return them to Gatsby. These "virtual" files could be written to a tmp directory (Webpack only understands real files) and everything should just work. Gatsby could also expose an update hook so while developing, you could update data in your third-party system which when pushed to Gatsby would cause the mapped tmp file to be overwritten causing a hot reload.

Another idea to chew on. It might make sense to only have one Webpack loader for all static files. Right now there has to be a separate loader for each file type e.g. md, html, yaml, etc. And extending Gatsby to support a new file type isn't supported unless you fork Gatsby. If Gatsby just had one loader for all page types, then supporting a new file type as a page or to override the core behavior would just mean adding a function in your app.js and adding a wrapper component. This would make choosing which files become pages explicit e.g. in app.js (probably) would return the file types as array of extensions e.g. ['js', 'jsx', 'md', 'json']. Having our own simplified "file transform" API would be much more simple then expecting people to learn oddities of Webpack loaders.

[stale]

This issue has been automatically closed due to inactivity.