Back-end Integrations

[LukeAskew]

Awesome work Cloud Four team 👏

Thought I'd toss out a question I receive a lot when using Fabricator - I'm often asked if there is a way to easily bridge the gap between Fabricator and a back-end system. I designed Fabricator to be agnostic of whatever system it might be integrated into, but often its used as an initial phase of development for a single site, not a suite of sites that use a shared design system.

Many teams want to reduce the wasted effort that comes with maintaining two separate projects. When a pattern is updated in Fabricator, they want the corresponding pattern in their .NET, Java, PHP, etc project to be updated as well. It becomes burdensome to make a change in Fabricator, then make the same exact change in the integrated code.

Have you heard the same? Any ideas on how to solve the issue?

[grigs]

OMG @LukeAskew. That is the holy grail of pattern libraries isn't it?

FWIW, I've talked to a lot of organizations about this topic and very few are doing it successfully. It is something that I'm constantly asking people about to see if others have solved it well.

Outside of Fabricator or Drizzle, the Lonely Planet and Financial Times have both written about their projects to tie their pattern library to their development environments. I was trying to find links for them, but the plane's wifi isn't working well at the moment.

Walmart has talked about moving all of their patterns into React components and then having people reuse those components as a way to keep things in sink. I believe something similar would make sense using web components.

In the long run, I think that's where all of these things go. The maintainers of the pattern library for an organization create web components that are then included in pages throughout the site. Because web components are self-contained and can reference an external source, any updates to the pattern would automatically flow to those pages.

But these are all theoretical at this point. I've tried to pick clients on tackling this problem for them, but haven't got any takers yet.

[erikjung]

Thanks @LukeAskew

This is a great question that we're currently trying to answer for our own site. We actually just began integrating https://github.com/cloudfour/cloudfour.com-patterns into our backend this week (CC @mrgerardorodriguez).

We're including the pattern library as a submodule within the backend repository, and using npm scripts to build the static output (so that the resulting CSS and JS can be referenced from above). I believe the only thing harvested at the moment is CSS.

Ideally, we'd be able to reuse the Handlebars templates as well (server-side), but that would require that they be authored with that consideration in mind:

• No hard-coded content
• No assumptions about the parent context
• No extraneous markup at all
• etc.

Definitely something to think about :)

[tylersticka]

Thanks for the kind words, @LukeAskew!

I agree with everything @grigs and @erikjung said, but wanted to add one non-theoretical point. One common motivator for building a pattern library is to foster some element of consistency across an organization with differing server-side implementations. For clients like that, the server-side integration piece is really a non-starter unless they're investing in normalizing everything across the board, so we've focused more on creating a portable toolkit of CSS and JavaScript UI components that can be leveraged by all those teams in a manner similar to frameworks like Bootstrap or Foundation.

In those cases, the greatest areas of focus have been portability of those assets (the fewer images, the better) and clarity of documentation and examples.

Moving back to the hypothetical, an idea I've tossed out in the past was to move from a single templating solution (like Handlebars) to something more configurable a la consolidate.js. This would give us more freedom to include the pattern library within a project and harvest its templates and such, though it'll make development more difficult (Handlebars helpers are so easy to write, and a few are currently dependencies). Food for thought, though! 🍔

[LukeAskew]

👍 Great stuff.

We do something similar with the toolkit as a dependency of another project. Typically its deployed to a CDN and referenced as a config on the server-side e.g. cdn.url.com/v1.0.0/assets/toolkit/scripts/toolkit.js

The closest we've gotten to a smooth server-side integration of templates is code generation via a script. We run the script agains the materials and it generates .php, .ftl, etc files. Not incredibly useful, but it saves some time in that initial hand-off.

The consolidate.js idea is interesting 🤔

[gerardo-rodriguez]

Thanks @LukeAskew! 😄

Yep, @erikjung, @tylersticka and @grigs covered it well. It's an interesting challenge for sure and it'll be awesome to discover even more efficient ways of handling this as we move into the future. Very exciting to think about. 😉

[bradfrost]

I'm excited a lot more people are talking about achieving the Holy Grail. I'm working with an organization who has their pattern library integrated into their backend system, which allows the pattern library and production environment to stay in sync. Hooray! But the downside is that in order to contribute to or manipulate the patterns in any way (for rapid prototyping, etc) you have to spin up their whole gigantic Java stack. Boo!

I'm ridiculously excited about more nimble tools like Fabricator, Drizzle, and Pattern Lab because they lower the barrier of entry for entire teams to participate in the creation/maintenance of the design system. After all, a successful design system should involve everyone in the organization, not just developers. Claudina Sarahe from Casper talked about how moving away from a heavy Ruby-based design system infrastructure allowed more people to contribute to their system. This is what I want for everyone.

The folks who have been working hard on the next version of Pattern Lab (@dmolsen, @bmuenzenmeyer, @EvanLovely, and others) have decoupled things so people can use whichever templating engine they prefer, which means they can more easily integrate Pattern Lab into their backend systems. @EvanLovely talked about how they can now integrate Pattern Lab into their Drupal builds, which I think is freaking awesome.

So yeah, I love how people are chasing the Holy Grail, and I'm hopeful we can all figure out how to do it in a way that keeps things nimble and inclusive.

[micahgodbolt]

We're doing exactly this at Red Hat right now. We're writing all of our component/layout templates in Twig (supporting both PHP and Node) and distributing them via bower to redhat.com's Drupal repo. Drupal then renders its data directly through those exact templates. So then, along with the bower repo's CSS and JS, they have everything they need to render our design system.

The challenge

The challenge comes in when you try to match your CMS/app data models to your highly modular design system. 99% of the time your data model is not going to match the data needs of our mustache/nunjucks/twig template based design system.

The solution

The solution a few of us came up with (which I just wrote about) was to inject an intermediary file (which I call a presenter, as in MVP) which can translate your data model into what's required of your view:

Here is an example of a what that presenter would look like:

{% extends 'card.twig' %}

{% block body %}
  {% include 'title.twig' with {'headline': title} only %}

  {% include 'image.twig' with {
    'image': image, 
    'align': 'right'} 
  only %}

  {% include 'content.twig' with {
    'teaser': content|truncate(35), 
    'body': content} 
  only %}
{% endblock %}

Conclusion

In the end, it doesn't matter what pattern library tool you are using. If you can write production ready templates that are (as @erikjung said):

No hard-coded content
No assumptions about the parent context
No extraneous markup at all

you can do it too. I'd be happy to talk to you about it if you want to try.

Sorry about the info dump, but this is something I've been really passionate about recently, and we're really excited about what we've been able to accomplish.

If you want to get me really talking, just ask me about JSON Schemas and component API's.

[EvanLovely]

Here's a sample of how we've done this with Drupal 8 & Pattern Lab 2 using its Twig Engine: https://github.com/phase2/drupal-lab

[rikschennink]

The current project I'm working on we're doing something similar to @micahgodbolt but instead of PHP it's Java Spring.

I setup patterns and base templates, then fill them with mock data. the templates then get auto injected in the backend system where they use JTwig to generate the HTML with live data.

I've got the benefit of being able to quickly build components without the hassle of setting up and being dependent on the entire stack. Takes away the hassle of keeping front-end and backend in sync plus gives the advantage of being able to feed all sorts of test data into the patterns and place them in different settings to see how resilient they are.

There's some small implementation differences between JTwig and Node Twig but once you're familiar with those it's super effective.

[paulmsmith]

Excellent discussion here! 👍

To chime in, I worked on a large multinational site that did achieve an organisational Holy Grail. Using Nunjucks we developed a design system that allowed templates and macros to be written once in the style-guide but consumed directly by the production environment server-side and client-side views.

Server-side was: (Node+Express+(Mongo populated from SDL Tridion)+Nunjucks)
Client-side javascript views (combination of Backbone + Cocktail + Nunjucks).

This meant we wrote one 'presenter' (to use @micahgodbolt's jolly nice terminology) and that could be required and used anywhere we needed to output markup.

For the Javascript we used a Just in Time RequireJS technique and for the CSS it was written modularly in a BEM style and built/bundled with assets.

We defined a set of component schemas which meant code coverage was really good. This also meant where we could not achieve the Absolute Holy Grail (outside orgs not on our stack or using nunjucks) they could still quickly mimic it in their own environments and we could test them.

I've since joined UK Gov and we (Government Digital Service & Cross Government Departments) are just attempting to formalise requirements and tackle this very issue. As you can imagine there is a multitude of environments, requirements, skills and capability to consider in Gov so It might be that it's pragmatically unachievable for a long time.

That said, The work GDS are doing is early days and only just got the funding so it's still discovery phase. No doubt when they're ready they will be blogging.. in the meantime I will be sharing this link with the people running the discovery, keep the excellent discussion going!

[micahgodbolt]

@paulmsmith great to hear about this approach used elsewhere! Did you use anything specific to define your component schemas? And if you did, how did you take advantage of them?

We on a whim decided to use JSON Schemas to define our presenter data needs, and it turned out to be an amazing choice. Not only can we validate the data before we send it to the templates, but we can also use the schema to mock up what a admin form would look like using this JSON Schema Editor.

Using this little JS library I was able to create an entire Drizzle'esque pattern library that we call Pattern Kit so that users could use the editor's form to modify the contents of the data being passed into the template.

Secondly, there's no reason that a pattern library couldn't support multiple template formats. You wrote yours in Nunjucks (great choice btw), but you could also create a ruby version, and twig version along side it during development. It'd create a bit more work for component creation, but it'd mean that all systems could use the same pattern library, but they'd pick the file format that worked best for them.

[paulmsmith]

@micahgodbolt We did, using SDL Web/Tridion, a very powerful content management system that ships with what is essentially a full fledged version of what it appears you've nicely done with your Pattern Kit. It comes with a UI for defining schemas and creating components out of the box and uses them as 'blue print' for other sites/markets.

See the docs for example: Creating Schemas for components

Once you've defined the schema in Tridion you use that as a basis for defining components. The beauty in Tridion is you can assemble them into groups, nest them simply by copying and pasting. The UI to Tridion ensures you can't enter anything that doesn't meet the schema for the component.

It has the notion of 'embeddable schemas' which is very nice for standardising things like key metadata that will be applicable for all components, such as: modifiers, classes, data attrs, etc

From there, the clever back-end chaps I worked with exported that data to MongoDB and it was all neatly available as JSON context data within Express.

With the beauty of Nunjucks and the declarative way you can create functional style filters, it was easy enough to expose chunks of things like lodash to manipulate the data within the presenters/views.

I'm not a Tridion salesmen by the way, it's not without it's faults but it works out of the box in the way you're describing. I'm under NDA so sadly I can't describe too much more. In-fact I remember having a conversation with Mark Perkins (@allmarkedup) on Twitter about what we were doing long before he did his excellent work on Fractal

I've been thinking of ways you could create a platform for creating component schemas and then some how transform them into consumable templates for various engines/environments. Maybe in Gov we'd define a pattern/component schema in a sort of working-group and then have the vested interests write the template/macro/presenter for their particular concerns? Who knows.

[paulmsmith]

Has anybody tried: acetate - One to add to the list perhaps? This is nunjucks based out of the box and is powerful for creating quick prototypes or perhaps being used for a pattern library.

[micahgodbolt]

@paulmsmith thanks! Excited to hear others pursuing the same ideas. I'm a big fan of Fractal. Quite a few similarities between it and Pattern Kit.

[paulmsmith]

@rikschennink - How did I miss your comment previously. Do you have an example of what you describe online? That sounds like it could be very useful to my organisation that run both Node and Java with Spring.

[rikschennink]

@paulmsmith I don't know :-)

Don't have an online repo for this. I'm no backend dev but this is what we are doing:

1. We use jTwig on the backend (Spring) and Twig.js on the front-end
2. Front-end dev (in this case me) builds templates and partials and feeds mock data into them.
3. Build is done with Gulp and Browsersync is used to run a local dev server to test the templates.
4. Templates and partials are then copied to dist folder and imported using Maven
5. On the Java/Spring side jTwig is used generate the live pages based on the templates and partials

Step 4 also does a tiny bit of reformatting to iron out some inconsistencies between jTwig and Twig.js (both are not a 100% the same a Twig).

[shawnbot]

Hi, everyone! I started hacking on a tool called meta-template recently that I think starts to get at a solution for the template engine compatibility problem. The idea is to use Nunjuck's parser to generate an AST of a template then implement formatters for other well-known templating systems. On the command line, you could imagine doing something like:

# convert Nunjucks to Handlebars
meta-template hero.njk --format hbs > hero.hbs
# or ERB
meta-template hero.njk --format erb -o hero.erb

So far I've only just implemented a Nunjucks formatter for the simplest expressions, and a very rudimentary PHP one to see if the idea holds up. I started documenting some common template features, but that table represents more of a first pass at (roughly) ranking output formats by parity with Nunjucks than an exhaustive reference. (For instance, I hadn't even heard of Twig when I made it!)

I'll admit that I didn't read this entire thread, so I may have missed something. But if this sounds relevant to anyone's need, feel free to get in touch, fork away, or file some issues. 🍻

[nickdunn]

At @wearefriday we do something very similar to @micahgodbolt. For the build of https://www.nuffieldhealth.com (about 18 months ago now) we swapped our Java CMS's templating engine from JSP to handlebars. We then developed components and page layouts using handlebars JSON "mock" files; using a simple Node/express app to serve them as a styleguide.

The JSON files represent the data interface between the CMS and the component. So a basic "card" component might be represented by:

{
  "heading": "Here's the heading",
  "description": "Here's the summary...",
  "url": "/some/page"
}

While the styleguide Node app renders a card example using this example mock data, straight up, page templates in the CMS render a card component using the presenter pattern, e.g.

{{ render_component 'card' heading=treatment.name description=treatment.shortDescription url=treatment.permalink }}

(Where "treatment" is a domain object coming from the database.)

As @micahgodbolt suggests, the presenter pattern means the component remains "pure" with attributes named after the UI. However the template (presenter) is doing the work of transposing these into attributes that are named after the domain.

The benefits are huge:

• frontend developers can develop outside of the CMS with a lighter application stack
• frontend development can start earlier (we picked handlebars and started before we'd even selected the CMS!)
• there's virtually no "integration phase" for making templates work in the CMS
• testing can start earlier, doesn't have to wait for a CMS implementation
• the result is a truly living component library — the styleguide templates and assets are the exact same files as used by the production CMS application

Obviously the single-most important thing here is choosing a platform-agnostic templating layer. Handlebars is a great choice because it has implementations across most stacks (PHP, Java, JavaScript, .NET). Settling for something platform specific such as Java's JSP or .NET's Razor (or to some extent PHP's Twig) wouldn't afford us the same flexibility.

We also found that we were restricted in the handlebars helpers that we wrote. Helpers that needed to touch the filesystem or do anything more than basic string manipulation we ended up writing twice: once for the Node styleguide, and then porting to a Java version for production. Definitely an area for improvement.

For any new site we'd definitely look into Drizzle, PatternLab or Fractal which neatly package up the similar concepts.