(WIP) Backend restructure - Integrations #1171
What are they?
Integrations hook into a number of different places in the CMS to replace different calls, most (all?) of which would normally go to a backend. Currently the assets store (at least) requires additional work There are currently two integration providers:
Integrations vs integration providers
Ostensibly, the integration providers are implementations of generalized APIs for a named set of integration hooks:
Each of these APIs indicates that certain methods will be available on the
There are some other methods implemented by integration providers, like
How they work currently
The integrations are added by creating an
Redux state init
Other keys are stored as provider data. The final state produced by this has two main keys:
Since this is implemented by reducing over the configured list of integrations, the precedence of integration hooks is determined by the order of the
Collection names can collide with hook names, since they're both used as keys to the same Map. (e.g., a collection named
Since the order of the integrations is dependent on the order of their listing in the
There are two ways to create hooks that apply to the whole CMS: either list
The source of the reducer described above is as follows:
Integration provider init
The actual integration providers themselves are initialized in
Whichever of these lines run first in a particular CMS setup determines when the integrations objects are set up. For most setups this probably occurs in
As described above, hooks are configured and stored as lists of strings. The strings must match both the predefined hook name and a method on the integration provider object.
A hook is called via the following process (this process is not wrapped by anything else, so every site where a hook may be called must implement
An example hook call follows:
Comparison of potential integration designs
Integrations as is
Integrations are currently a very complex API, with multiple layers of setup and configuration. Parts of the API are very general and indirected (e.g., the integration/integration provider distinction), and others are tightly coupled to specific implementations of both integration providers and backends (e.g.,
This leads to a situation where the API is simultaneously so flexible in principle that it's difficult to follow or implement, while so specific in operation that it requires indirect support across wide swaths of the codebase.
It's also inherently stateful - the list of providers instances is a singleton, and providers themselves are class instances which store information on the instance's properties.
Finally, the integrations API has sole responsibility for some concerns, meaning they cannot be implemented by backends. Search, for instance, is either done locally or through an integration - there's no ability for a backend to implement server-side search. Adding this to the backend API as well would introduce further API duplication between the backend and integration API.
Integrations as backend composition
One potential approach would be to unify backends and integrations into a single API, allowing them to be combined with normal code. For instance, an Algolia integration could be defined as a function which wraps an existing backend and calls its functions, except for
Benefits of this could include removing
Integrations as middleware
Redux allows intercepting actions before they hit reducers using middleware. This is very powerful, but it does allow unrestricted access to our Redux actions, essentially making our current action structure the public integrations API.
The text was updated successfully, but these errors were encountered:
Great breakdown here, love it.
Couple of thoughts:
The middleware option doesn't have to involve exposing raw state - we can process the middleware functions however we like. I'd expect that we'd transform the state into a shape matching our published API, allow that to travel through the middleware functions, and then transform the result from there.
The precedence problem
You make it clear that precedence matters here, as it does in almost any plugin architecture. I'm wondering if we can construct the API in such a way that a backend/integration must declare what parts of the API it handles in order to be allowed to handle those parts - e.g., Algolia, or the CMS config, declares that it handles
This allows us to statically determine handlers for each action and where overrides occur.
This could easily be an enhancement for later.
@knpwrs currently I'm working on getting the GitLab and BitBucket backends wrapped up, as well as working on some of the preliminary refactoring that'll be required before we move to a new backend API. You can follow along with that work here: #517 (GitLab) and #525 (BitBucket).
Once that work is done, the last step to prepare for combining integrations and backends will be to move the media library integrations into
As for the new backend API itself, it's still very much in the design phase. The primary issue for the backend API design is here: #1134. The best way to help out with that now is to add suggestions and critiques to that issue. As a quick intro, the core ideas of the backend restructure are as follows:
@Benaiah for when you dig into this:
Hello, is there a status update on this? I've been following #432 searching for a way to integrate NetlifyCMS with another media store such as S3 or GCP Storage. If there already exists an API (such as the one proposed in #1602 (comment)) I'd be happy to help contribute documentation and leverage Netlify Functions if a backend is needed.