monorepo multi-team/multi-project folder structure

[Whoaa512]

The hygen FAQ says that it was built "to solve developer effectiveness in a multi-team, multi-module monorepo", however I'm curious about how to use it for that? 

From reading through the docs, and your medium post I didn't get a sense of how one could have templates co-located within the different projects, but still be able to call hygen from the top level.

Would something similar to this structure work?

package.json
apps/
    project-1/
        _templates/
        .hygen.js
    project-2/
        _templates/
        .hygen.js
    project-3/
        _templates/
        .hygen.js

Or does hygen need to run within the same directory as the .hygen.js file?

Thank you again for some a declarative approach to code gen tooling.

[anithri]

hygen looks for the first .hygen.js file it finds walking up from your current directory to the root directory.

Sharing as in multiple directories providing generators to all, then no, not yet, but I'm working on finding the right way to allow for that. Duplicate generator names is the sticking point there.

Using HYGEN_TMPLS you can set any directory to contain your templates, which is how I share templates between my projects.

Otherwise the philosophy is to use hygen-add as a source of templates, but that templates should live in the repo they are used in, so they can be modified for specific uses.

In your example, and assuming you didn't set HYGEN_TMPLS, if you are in or below apps/project-2/, then you'd be using apps/project-2/.hygen.js and apps/project-2/.hygen.js

If you had HYGEN_TMPLS=~/code/mono/_templates and this structure...

package.json
_templates/
.hygen.js
apps/
    project-1/
    project-2/
        _templates/
        .hygen.js
    project-3/

Then project-1 ad project-3 would both use the top level .hygen.js, while project-2 uses the one in it's own directory. All 3 projects would use the top level _templates.

[jondot]

Yup just to re-confirm, what @anithri explained is spot on.
As a general statement hygen tries to do the intuitive thing, and if your team(s) have a special requirement to make hygen more intuitive we're excited to hear

[madeleineostoja]

Just ran up a similar issue myself. I'm using Hygen in a monorepo setup where I'd like to share some templates (eg: React component generator), and also have templates scoped to packages (eg: Gatsby page generator). If there was some way Hygen could crawl up the tree until it finds a suitable template that would be awesome.

Using the above example it would be great if you called a generator from within project-2 that first looks for templates in /apps/project-2/_templates and if it doesn't find a match then looks in /_templates

[jondot]

Happy to push this suggestion forward. Looks like a few solutions can be had:

1. Conventional: hygen picks up templates in CWD, so run it per team, inside the folder of the team (no sharing upwards)
2. Crawl up: pick up current in current folder, then pick up upper level templates (and accept some confusing use cases such as conflicting names / new templates popping up that your team did not create)

thoughts?

[madeleineostoja]

Yeah I think option 2 would be great. I think accidentally calling a generator you didn’t create from higher in the tree is a very niche edge case, and IMO duplicated template names are a non-issue because it’ll call the first it finds. This is the same as the configs for other popular tools — eg: Babel, uses the first babelrc it finds from the cwd it was called from.

You could actually consider that a feature, for example if I had a top level component template that I used in most sub projects, and a specific custom component template in one sub project, I would expect Hygen to automatically use that scoped template when called from in the subproject.

[Whoaa512]

@seaneking I think you're describing the first option @jondot wrote, which is to stop looking for template files or a config file after you've found one from wherever you started.

I would tend to favor that as well.

If we opt for crawling up and inheriting anything upwards it could make for some confusing cases since at what point do we stop looking for templates. It also begs the question about what is the stop condition.

[madeleineostoja]

Hmm how I read 1. was that Hygen would stop looking beyond the CWD if it doesn't find an appropriate template. Maybe @jondot can clarify 😅

I agree that inheriting would be weird and confusing. But I don't think searching for templates up the tree until it find the appropriate generator falls under that scenario.

[jondot]

So to sum up:

1. Today, hygen looks for a single templates folder -- that one which it can find either in CWD or that someone supplied via ENV variables
2. We want: hygen to bubble up and find the shortest path to the closest _templates folder. once it finds it, it will stop.

[madeleineostoja]

Ah, I was thinking multiple _template folders. Ie: Hygen would look for the closest generator specified. I could have a root level _templates/component and a subproject _templates/page and use both.

That said I think the behaviour outlined in 2 (searching up the tree for _templates rather than using an env variable) is an improvement regardless.

[Whoaa512]

Yea searching up is definitely an improvement. Would also be nice to keep the env variable method of specifying as well.

I think what's also missing today is a way to specify a .hygen.js via env var.

[luan007]

+1 for this, we have a giant monorepo with multiple realms of dev-tasks,
each top-level folder contains a template dir (for server / client-side / or libs & so on), thus setting env manually is actually quite heavy given the dev usually works on multiple directories at once (reason for using mono-repo)

thus:
mashed up a tiny cli real quick:

#!/usr/bin/env node

const fs = require('fs');
const path = require('path');
const cwd = process.cwd();
const signature = "./_templates";
var base = cwd;
var defaultTemplates = '';
while (true) {
    var p = path.resolve(base, signature);
    if (fs.existsSync(p)) {
        defaultTemplates = p;
        break;
    }
    else {
        if (base == '/') { throw new Error("Templ folder not found!"); }
        base = path.resolve(base, '../');
    }
}

process.argv.splice(0, 2);
var p = require('child_process').spawn('hygen', process.argv, {
    env: {
        ...process.env,
        'HYGEN_TMPLS': defaultTemplates
    }
});
process.stdin.pipe(p.stdin)
p.stdout.pipe(process.stdout)
p.stderr.pipe(process.stderr)

searches parent directory for signature & injects into HYGEN_TMPLS, works for me now..
does not solve the 'inheritance' problem, but I believe it can be done by including hygen as a lib?

hope this helps

[nareshbhatia]

@jondot, is there any update on this request? I was looking at Use Generators From a Single Place and was wondering if that's the solution for this.

Overall, it would be great to have:

1. A good monorepo solution where global templates are at the root, with potentially more templates in subfolders to override/extend.
2. Ability to issue all commands from the root with a parameter that indicates which workspace we are targeting. This is similar to npm and yarn workspaces where you can target a workspace using --workspace (see here)
3. The concept of defining workspaces, so that we can have workspaces in multiple places such as under /apps and /packages.

[airtonix]

Hmm how I read 1. was that Hygen would stop looking beyond the CWD if it doesn't find an appropriate template. Maybe @jondot can clarify sweat_smile

I agree that inheriting would be weird and confusing. But I don't think searching for templates up the tree until it find the appropriate generator falls under that scenario.

This is how typescript, eslint and babel work btw. I think if you document it, then people wont find it confusing at all.

[airtonix]

@jondot, is there any update on this request? I was looking at Use Generators From a Single Place and was wondering if that's the solution for this.

Overall, it would be great to have:

1. A good monorepo solution where global templates are at the root, with potentially more templates in subfolders to override/extend.
2. Ability to issue all commands from the root with a parameter that indicates which workspace we are targeting. This is similar to npm and yarn workspaces where you can target a workspace using --workspace (see here)
3. The concept of defining workspaces, so that we can have workspaces in multiple places such as under /apps and /packages.

We actually don't need anything specific to monorepos.

all we need here is for the hygen cli allow us to override the cwd but still traverse upwards looking for a config file.

i'd suggest that you migrate hygens configuration system to cosmiconfig. it handles all the problems of finding config files and allows us to choose what ever fileformat we want (but i'd hope you choose to implement the typescript loader).

merging configs is upto the repo custodians, and is as simple as using js/ts configs combined with lodash/merge.

[vim-daniel]

I was able to achieve a custom template search logic using .hygen.js file in the root of my monorepo

for example, this logic searches for templates from the current directory up to parent folders

const fs = require('fs');
const path = require('path');
const cwd = process.cwd();
const signature = './_templates';
let base = cwd;
let defaultTemplates;

while (base !== __dirname) {
  var p = path.resolve(base, signature);
  if (fs.existsSync(p)) {
    defaultTemplates = p;
    break;
  } else {
    base = path.resolve(base, '../');
  }
}
let exportsObj = {};

if (defaultTemplates) {
  exportsObj.templates = defaultTemplates;
}
module.exports = exportsObj;

[Robbie-Cook]

Just to give an example of calling hygen with HYGEN_TMPLS set to one level above:

HYGEN_TMPLS=../_templates hygen self --help

works great for my use case of a template system outside of a repo

[svallory]

Hey guys, future user of hygen here. I just read the docs, and I'm already a fan of hygen!

I'm thinking of migrating moon-launch to hygen, and I'm checking if I'm not going to paint myself into a corner (again). So far, I'm loving everything about it! One key thing for me is the ability to register new template locations. In moon that can be done by setting an array of template locations in workspace.yml.

I think that's the best approach. We won't need hygen to discover templates at all if we can tell it where all of them are :)

Here's my suggestion for a new templateLocations (or for a templates)

type templates = string | Array<string | {
    path: string;
    prefix?: string;
}>

What about conflicts?

If two locations have generators with the same name and using the same prefix, there will be a conflict and it needs to be resolved. But first...

Why handle conflicts instead of forcing uniqueness via prefix?

It allows for a very powerful mechanism that promotes reusability. Imagine there's an awesome template package that does a lot, but I want to replace one of its generators. Instead of forking it, understanding its code, and modifying it, I can simply add a generator to my project and setup hygen to use

Proposed solution

Instead of trying to come up with clever algorithms that will be confusing and take ages to get right (as in any package manager ever), let's just let the user decide what to do.

Specifically, suggest the addition of another property to .hygen: conflictResolutionStrategy. The property value would be an enum with 3 possible values:

• fail: [default] stops the generation and lets the user know that templates are in conflict, and it needs to pick a strategy
• override: keeps the template that appears last in the array
• skip: keeps the template that appeared first skipping the ones that conflict

I think this strategy should be fairly easy to implement. In the future, a custom strategy may be added to allow specifying a function that takes 2 template paths and returns a map of their paths to { name: string, prefix: string } for the templates.

btw, sorry if this was already proposed. I looked but didn't find it anywhere.