New Hoodie Architecture

[janl]

New Hoodie Architecture

This is going to be a long one, grab a nice beverage and enjoy the ride :) ☕ 🍵 🍸

The goal of this issue to lay out a proposal for some fundamental changes in how Hoodie is architected and gather community feedback to improve the proposal.

We are going to touch many pieces of the system and how we plan to work on them and it all goes hand-in-hand. That’s why this is a big overview discussion point. We can break it out to individual issues in their target repos later.

Table of Contents:

• Motivation
• The New Hoodie Module
• Testing
• The hoodie Module
• Plugins
• The New Hoodie Module Dependency Tree
• The New hoodie-server Layout
• Releases
• The Hoodie Internals Documentation

Motivation

The primary goal of this proposal is to make working on and with Hoodie easier and more fun.

We are attempting to get there by automating a lot more tedious tasks around software development and management than we have today.

The target audiences for this initiative are, in decending order of importance:

• End users will benefit from simpler ways to start a new app and to maintain existing apps.
• New Hoodie contributors will benefit from well documented Hoodie internals, consistently documented and laid-out sub-modules, consistent testing procedures and a lot of automated support tools that help to guide new contributors to their first contribution as fast as possible.
• The existing Hoodie development community will, in addition, benefit from more automated and more straightforward release engineering and reduced maintenance overhead.

The New Hoodie Module

There are a bunch of changes we want to make to all the modules on http://github.com/hoodiehq. The prime objective here is to unify a few things across all repositories in a way that the development experience on each module, be it a CSS library, our website or core JavaScript code, is the same everywhere.

One thing we’d like to encourage and pioneer is the use of README.md files in all subdirectories, with information what the particular sub-part of a module is doing and why. That way, our code is much easier to navigate and learn.

In addition, where it makes sense, modules will generate and expose their automatically generated API documentation for pick-up by a central documentation location and for offline viewing.

We’d like to move away from grunt as a built tool because it is a bit unwieldy. We’ll be moving to npm scripts for our built scripting needs.

We want to standardise on a single set of commit message conventions for all the types of commits we have across all repos. And we want to enforce these with pre-commit hooks that have friendly error messages for newcomers to do the right thing.

This is a bit technical, but we found a neat solution to make this easy for everyone. A pre-commit hook is a piece of software that git runs to determine whether it should allow the user to make a commit. We want to use these hooks to ensure that all commit messages follow our standard format of type(scope): message with type and scope being from a strict set that has semantic meaning down the road of the lifecycle of a module.

Pre-commit hooks are something that each user has to install for themselves, they are not part of a git clone. In order to avoid everyone having to do this manually, we propose this procedure:

1. all modules have a require new module, say hoodie-commit-hooks or hoodie-developer-tools.
2. if the require fails, we print a nice message like “please run this command: `npm install -g hoodie-commit-hooks”.
3. Now the require succeeds and it can install the pre-commit hook in the current repository.

Things we want to enforce with pre-commit hooks:

• commit messages
• package.json structure and order
• anything else that needs enforcing at that stage.

This should make the experience of contributing to a single module a lot more pleasant and remove a lot of the burden of reviewing and maintaining code from the existing developers.

Testing

Testing is important in order to ship quality software.

Our testing strategy is that each module should test anything that it does in isolation. A module can have unit tests or integration tests or whatever other tests it needs.

Tests are run with a simple npm test.

There will be a top-level hoodie module that include an integration test suite that tests the whole Hoodie system top to bottom.

Our tests are contracts for the compatibility of our modules. If a test breaks, we must assume our module changed in ways that users need to be informed about it with a new major version number.

The hoodie Module

The hoodie module is meant to tie our core components hoodie-server and hoodie-client (formerly hoodie.js, more on this below) together. It will have hoodie-server and hoodie-client as peer dependencies, because they can’t work without each other.

It is also the center point that defines the semantic version number of a full Hoodie release. The integration test suite (see previous point) will act as a guard for breaking changes, which we use to detect whether we need to release a new major version.

The hoodie module will also incorporate the existing hoodie-cli tool and hide it inside of npm scripts. hoodie start will become npm run hoodie start. The benefit here is to remove the need for a dedicated installation of hoodie-cli. It also means that all Hoodie apps come with command line clients that are compatible with the rest of the Hoodie system.

The hoodie module will also have a bundled and fixed dependency tree. That way we always know what a given Hoodie release contains down to the last dependency.

Plugins

Plugins are installed by adding them to the dependencies list of an app’s package.json and are automatically loaded from there.

Frontend plugin code now uses require() instead of Hoodie.extend().

The hoodie-client code will be served from hoodie-server in four configurations:

• hoodie.js
• hoodie.min.js
• hoodie.browserify.js
• hoodie.browserify.min.js

That way we support <script src=""> setups as well as Browserify ones, and development and production modes for both.

Since we package the full Hoodie dependency tree on release-time, that’s when we’ll also generate the hoodie*.js files. That way, they don’t have to be generated during a first-run experience and by each user again and again. Like today, hoodie-server will check for changes in an app’s plugin configuration and recompile the hoodie*.js files on the server start that follows the configuration change. Single client requests to hoodie*.js will always be fast.

We will reorganise the plugin layout:

• ./client/index.js: frontend code (formerly ./hoodie.pluginname.js)
• ./server/index.js: backend code (formerly worker.js or package.json’s main)
• ./server/hooks/* hooks, formerly ./hooks/
• ./admin-dashboard remains
• (as per require() rules, these could also be ./client.js, ./server.js ./server/hooks.js, but those are discouraged)

That way all parts of a plugin can be require()’d as require('hoodie-plugin-foo/server') on the server or require('hoodie-plugin-foo/client') on the client, etc.

The New Hoodie Module Dependency Tree

The new dependency tree looks like this:

         hoodie-app-template-ember                                                                            
                                                 +-----------> hoodie-plugin-users                            
         hoodie-app-template-angular             |                                                            
                                                 +-----------> hoodie-plugin-email                            
         hoodie-app-template-*                   |                                                            
                                                 +-----------> hoodie-plugin-appconfig                        
         hoodie-app-template (default)           |                                                            
          +                                      +-----------> admin-dashboard +-------> admin-dashboard-uikit
          |                                      |                                                            
          |                                      +-----------> server                                         
          |                                      |                                                            
          |                                      |                                                            
          v                                      |                                                            
+------> app +-------------+----------> hoodie +-+                                                            
                           |                     |                                                            
                           |                     +-----------> client (details omitted)                                        
                           |                                                                                  
                           |                                                                                  
                           |                                                                                  
                           |                                                                                  
                           |                                                                                  
                           +----------> hoodie-plugin-foo                                                     

The changes at a glance:

• new module names for hoodie.js
• core plugins are now peer dependencies to hoodie-server
  • app-plugins are still dependencies of the user’s app
• my-first-hoodie is now split into two modules:
  • hoodie
  • hoodie-app-template (or any of the other variants)
  • hoodie-plugins-manager and hoodie-plugins-api are now part of hoodie-server and not standalone modules. The nodejitsu and localtld features will be removed.

This neatly abstracts all Hoodie-responsibilities into the hoodie-module.

This also allows us to run npm install hoodie to, well, install Hoodie. How we deal with the existing hoodie package on npm will be explained at a later stage (cc @boennemann).

Since this is almost entirely a new dependency tree, there is very little need to update existing modules. We can start work on this right away in new repositories. The old module’s repositories and packages can be retired eventually.

Releases

Releases will happen through the hoodie module. A new version is determined by the highest type of change of any of the dependent modules. If there is a breaking change in one or more of the dependencies, hoodie gets a new major version number. New features: new feature version, and a bugfix release for bugfixes.

Each module will make use of release tags in npm. We’ll be using the two tags latest and next. All modules tagged latest is what we include in a proper Hoodie release. A next version can be installed by early adopters for testing, e.g. new features. Think “release channels” for browsers.

In order to release a new Hoodie version, all we need to do is move the latest tag of all dependencies to the desired version number, and use next for any new ones coming in after that.

That way releases are absolutely immutable and there is no monkeying around with 4.3.2-beta12 type releases.

The hoodie module itself will get a new next release each time any of the sub-dependencies gets a new release. hoodie’s full integration test suite will make sure that we introduce no full-system errors on a module level.

We can then decide on what basis to release hoodie versions. We can do timed releases (every week/month, after X new features, or manually after a desired set of fixes and features landed.)

The Hoodie Internals Documentation

All of the above is copiously documented. In particular, there is documentation for our various processes:

• design/decision making process (RFCs like Ember)
• development process (PRs, commit messages, review checklists, security reporting etc.)
• release process (semantic versioning, upgrades, migrations)
• Hoodie architecture (kinda this document)

---

This all is the combined work of @gr2m, @boennemann, @chistophwitzko & myself. I’m just writing it up, most credit is to the others.

---

We are now very interested in your feedback.

Despite the length, this whole thing is a bit sparse on details, so if you have any questions or need any clarification, just ask here on the issue.

If you violently agree or disagree, also let us know here.

Thanks for reading and looking forward to hear from you! 🎈

[gr2m]

ha, just finished my feedback on your original text: https://gist.github.com/gr2m/40330a80a7405968977d/revisions

peer dependencies

Will hoodie have the peerDependencies or hoodie-server, hoodie-client etc? Could you make this more clear? If I understand you correctly, a hoodie app's npm dependencies would look like this:

myapp
├─ hoodie
│    ├─ server
│    ├─ client
│    └─ ...
├─ hoodie plugin 1
├─ hoodie plugin 2
├─ some lodash, async and whatnot ...
└─ ...

That's what I'd love to have very much <3

motivation,

I'd say "Hoodie contributors", not "New Hoodie contributors". I'd also like to add "plugin developers". The barrier to entry should be as low as possible, a vital eco system of hoodie plugins will be crucial for our future milestone(s), where we will focus morn on hoodie users / UX developers

Enforcing Commit Messages

I'm not 100% convinced. From my experience with these commit conventions over the past year, I think they are great in combination with semantic-release. But for that we only need fix:, feat: and Breaking changes. The rest can be what ever makes sense, and we don't need to discuss if a commit is chore, or style, or refactor ... not even we agree on these.

And I don't want to enforce these commit message formats while working on pull requests. On pull request, I think it's better to just quick & dirty add new commit messages, without squashing all the time, as it tends to remove existing comments on the PR discussion. And I'm not sure if we can enforce these commit styleguides on master only, and prevent a merge of a feature branch if it doesn't follow our conventions.

"The New Hoodie Module" vs "The hoodie Module"

these titles are somewhat confusing :) Maybe we can rename "The New Hoodie Module" to "The New Hoodie Architecture"?

Module Dependency Tree

Just want to add, based on what we discussed so far. With all the work done on the current milestone, the dependency tree will look like this

myapp
├─ hoodie
│    ├─ hoodie server
│    ├─ hoodie client
│    ├─ hoodie account
│    ├─ hoodie store
│    ├─ hoodie task
│    ├─ hoodie admin
│    └─ ...
├─ hoodie plugin 1
└─ ...

• hoodie account / store / task / admin will be core modules just like hoodie server / client. They will be built like plugins, but are not meant to be replaced at this point, because they are too entangled. Decoupling them better could be something for future milestones.
• related: hoodie admin / admin dashboard does not need be exposed on a different url / port anymore, as we don't use cookies anymore.

[daleharvey]

Just commenting in terms of enforcing commit message style, at pouchdb we have generally taken the approach that PR's come with whatever style they want, we review them based on the code alone and when it becomes time to merge the committer will fix the commit message to follow our format, occasionally mention it when the commit is merged. This way new contributors arent put off by what are somewhat arbitrary style choices (we do this with most stylistic changes), regular contributors learn the style choices (and generally agree on them) and committers are responsible for keeping them and get the benefit of them being kept

[gr2m]

I very much prefer PouchDB's approach. It makes life simpler for (new) contributors, and it usually only takes seconds for us to fix commit messages if needed

[boennemann]

@janl Thanks for writing this up. This is a very accurate representation of what we discussed.

Releases will happen through the hoodie module. A new version is determined by the highest type of change of any of the dependent modules. If there is a breaking change in one or more of the dependencies, hoodie gets a new major version number. New features: new feature version, and a bugfix release for bugfixes.

This isn't 100% what we discussed. We wanted to let the highest type of every component release determine hoodie's version number, except for breaking changes. Which means the component's release numbers only help us to differentiate between patch and feature releases. For Breaking Changes we rely on a thorough integration test suite on the public api with which we use breaking change detection (cracks).

Will hoodie have the peerDependencies or hoodie-server, hoodie-client etc? Could you make this more clear? If I understand you correctly, a hoodie app's npm dependencies would look like this: […]

@gr2m yes

Just want to add, based on what we discussed so far. With all the work done on the current milestone, the dependency tree will look like this […]

What we (@janl, @christophwitzko) discussed didn't include account, store, task and admin, because they're not in existence in that form yet. We can discuss where to fit them once they're there and we have decent heuristics of where to put them now.

Enforcing Commit Messages

I'm strictly against modifying the commit messages conventions. They're "a thing", just like "standard" is a thing. You can refer to it and people (will) know what you're talking about. I don't want to be in the commit message business, just like I don't want to be in the linting rules business, because it's mostly bikeshedding. We have a ready made thing, it has a name, a good description/spec, parsers etc, and I don't want to think about these things myself.

I'm glad @daleharvey chipped in. I've suggested rewriting commit-messages for PRs a few times and I think it's a pretty good solution for most of the problems you brought up and it's not as offensive as it sounds in the first moment. We can document that we're doing this in the CONTRIBUTING.md. A necessity I see here is that we need to explain our changes on the PR, so people don't feel offended and make that very friendly and empathetic. We should even have a standard response template for that.

[boennemann]

This also allows us to run npm install hoodie to, well, install Hoodie. How we deal with the existing hoodie package on npm will be explained at a later stage (cc @boennemann).

Currently hoodiehq/hoodie.js is published as hoodie on npm, taking up the name we want to publish the main hoodie thing under in the future. Just publishing something else there would break existing apps.

Here is my proposed process to make this as painless as possible, even though it will still be painful.

1. Change the name field of hoodiehq/hoodie.js's package.json to hoodie-client
2. Republish it (maybe even all versions) with exactly the same version number
3. use npm deprecate to mark hoodie as deprecated with a message to use hoodie-client instead
4. Make my-first-hoodie use hoodie-client
5. Inform existing users that they should change hoodie to hoodie-client.

[… some time goes by and hoodie with the new architecture is ready…]

1. npm unpublish hoodie --force
2. Immediately republish the new hoodie entry point as hoodie

There is one extra that we could discuss:

hoodiehq/hoodie.js is the repo that has all the watchers and stars on github right now. We could use the current hoodie.js and not rename it, but create a new repo and push the repo there again.
The hoodie.js repo will then be renamed to hoodie, and we push a completely new repo there, that contains the new hoodie entry point. That way we could preserve all the stars and watchers and also have them in the right place, so they get informed about future releases in the github feed.

[svenheden]

[… some time goes by and hoodie with the new architecture is ready…]

1. npm unpublish hoodie --force
2. Immediately republish the new hoodie entry point as hoodie

You should be aware that if you're gonna re-use the hoodie package name for a new package you're not gonna be able to use the same version numbers that's been used earlier. Maybe you didn't plan on doing that but I just wanted to point it out. You can read more about it here.

[HipsterBrown]

We can document that we're doing this in the CONTRIBUTING.md. A necessity I see here is that we need to explain our changes on the PR, so people don't feel offended and make that very friendly and empathetic. We should even have a standard response template for that.

A thorough and clear CONTRIBUTING.md that's shared across the Hoodie modules & projects is going to very important to anyone looking to help out and get involved. Clarifying the standards expected by every contributor submission and easing the process for anyone from any experience background. The Atom text editor project has a great example of this. It will be a great group effort to cover all aspects of the Hoodie community through this doc.

P.S. I love this discussion doc and open feedback.

[lewiscowper]

@HipsterBrown there is a shared CONTRIBUTING.md floating around, I just think we need something that can add it to all repos.

https://github.com/hoodiehq/hoodie-dotfiles/blob/master/static/CONTRIBUTING.md

[boennemann]

@JonathanP yeah thanks, the version numbers are gone, we should just start the new hoodie at 1337.x.x or something

[gr2m]

Okay damn that's tempting :P

Consider starting the commit message with an applicable emoji

— https://github.com/atom/atom/blob/master/CONTRIBUTING.md#git-commit-messages

[gr2m]

I've a question regarding the hoodie cli

The hoodie module will also incorporate the existing hoodie-cli tool and hide it inside of npm scripts. hoodie start will become npm run hoodie start. The benefit here is to remove the need for a dedicated installation of hoodie-cli. It also means that all Hoodie apps come with command line clients that are compatible with the rest of the Hoodie system.

How will I create a new hoodie app?

[boennemann]

@gr2m I think the globally installed cli should still be a thing, but for certain actions it could just invoke the app's own commands.

[gr2m]

I think the globally installed cli should still be a thing, but for certain actions it could just invoke the app's own commands.

Perfect 👍

[gr2m]

@boennemann can you describe how we plan to do the releases? If I remember correctly from our discussion, we'd only have unit tests in the internal hoodie modules, and release based on them, then commit to the hoodie main module (which only pulls together sub modules and has CI tests) and then release a new version, unless the CI tests, break, right? I'm not 100% clear on that yet, but it sounds pretty awesome, and I'd like to try the same setup for kazana

[boennemann]

@gr2m Individual modules aren't technically limited to unit tests, but it's what I think makes sense on that level. So yeah unit tests for individual modules.
As you said the top-level hoodie module pulls them together and has a test suite that will be the actual definition of what the hoodie is. I think having a good integration test suite that tests all parts in combination is a good idea here. We're using that to release the main module then.

[gr2m]

it’s all done now https://github.com/hoodiehq/hoodie/tree/master/server#architecture ✌️

We don’t (yet) have a CLI, but there is no reason not to create one in future