Updatable templates for template hierarchies and reuse #934

sisp · 2023-01-23T14:05:49Z

Is your feature request related to a problem? Please describe.

I'd like to create a Copier base template (e.g. a general Python project template) and Copier child templates (e.g. a Python web app project template using FastAPI, a Python ML library template using PyTorch, etc.). The Copier child templates would mostly extend the Copier base template (e.g. add new files, add content to existing files from the Copier base template) but might also overwrite files/folders in the Copier base template (e.g. modify content in existing files from the Copier base template) or even delete files from the Copier base template (although this might be a rare case). Naively, I could create independent Copier templates, but then maintenance wouldn't scale because I'd have to sync the common parts manually. Instead, I would like to update a Copier child template with the changes of its Copier base template in the same way as I can update a generated project with the changes of its associated Copier template.

Describe the solution you'd like

I know this topic has been discussed a few times already (#79, #402, #416), but I haven't found a satisfactory solution and came up with a (to my knowledge) new approach that I'd like to suggest and discuss as it doesn't seem possible out of the box.

To clarify possibly confusing terminology upfront, I'd like to explicitly distinguish between Copier templates (i.e. project templates) and Jinja templates (i.e. content templates used in text files or file/folder names).

So far, I've seen this topic under the term "template inheritance" which (to me) sounds similar to the idea of, e.g., OverlayFS where multiple filesystem sources are merged, possibly enhanced with Jinja inheritance such that an overwritten file may actually be an extension of the base file rather than a complete replacement. I believe #79 tried to implement such an approach, but it was not merged for apparently good reason.

I've been giving this problem some thought for a while and believe that the word "inheritance" might be misleading. Instead of trying to inherit from a Copier base template, I think a Copier child template should be a (modified) copy of the Copier base template containing a reference to the Copier base template similar to how a generated project has a reference to its corresponding Copier template in .copier-answers.yml. Then, a very similar update mechanism as for updating a generated project could be used to update a Copier child template with the changes of its associated Copier base template:

graph TD
%% nodes ----------------------------------------------------------
base_template_repo("base template repository")
base_template_current("/tmp/base_template<br>(current tag)")
base_template_latest("/tmp/base_template<br>(latest tag)")
child_template_regen("/tmp/child_template<br>(fresh, current version)")
child_template_current("current child template")
child_template_half("half migrated<br>child template")
child_template_updated("updated child template")
child_template_applied("updated child template<br>(diff applied)")
child_template_full("fully updated<br>and migrated child template")
update["update current<br>child template in-place<br>+ run tasks again"]
compare["compare to get diff"]
apply["apply diff"]
diff("diff")
%% edges ----------------------------------------------------------
        base_template_repo    --> |git clone| base_template_current
        base_template_repo    --> |git clone| base_template_latest
        base_template_current --> |copy and run tasks| child_template_regen
       child_template_current --> compare
       child_template_current --> |apply pre-migrations| child_template_half
         child_template_regen --> compare
          child_template_half --> update
         base_template_latest --> update
                       update --> child_template_updated
                      compare --> diff
                         diff --> apply
       child_template_updated --> apply
                        apply --> child_template_applied
       child_template_applied --> |apply post-migrations| child_template_full
%% style ----------------------------------------------------------
classDef blackborder stroke:#000;
class compare,update,apply blackborder;

This approach would offer significant flexibility for customizing a Copier child template because anything can be changed without requiring the developer of the Copier base template to anticipate extension points (i.e. Jinja blocks).

In contrast to generating projects from a Copier template, generating a Copier child template doesn't involve a questionnaire or Jinja templating. Currently, I think only the following settings (from copier.yml) remain relevant:

answers_file (But since there are no questions to ask for generating Copier child templates, this setting might be renamed or removed entirely and a hardcoded file name might be used instead.)
migrations
min_copier_version
skip_if_exists
tasks

So perhaps a new file, e.g. copier-template.yml, should be introduced where the settings for generating Copier child templates are put. The Git rev (_commit: <rev>) of the Copier base template might be written to a file like .copier-extension.yml (or a different file name; subject to discussion).

Let's make this idea more concrete.

The filesystem layout of a Copier base template might look like this (with _subdirectory: template in copier.yml):

.
├── copier.yml
├── copier-template.yml
└── template
    ├── {{_copier_conf.answers_file}}.jinja
    ├── pyproject.toml.jinja
    └── ...

And the filesystem layout of a Copier child template (which itself could serve as a Copier base template again) might look like this:

.
├── .copier-extension.yml
├── copier.yml
├── copier-template.yml
└── template
    ├── {{_copier_conf.answers_file}}.jinja
    ├── Dockerfile.jinja
    ├── pyproject.toml.jinja
    └── ...

In order to speed-up the adoption of this idea, it might be possible to make copier-template.yml optional and assume sane defaults (i.e. no migrations, no tasks, ...) when this file is not present.

Finally, two new CLI subcommands would need to be added to the Copier CLI, one for generating a new Copier child template from a Copier base template and one for updating an existing Copier child template from its associated Copier base template:

copier template <base_template_src> <child_template_path>
copier template update

Describe alternatives you've considered

Both YAML includes of several copier.yml files and appyling multiple templates to the same subproject don't solve this problem.

I also thought about using Git submodules to include a Copier base template repository in a Copier child template repository, but this approach doesn't work either for several reasons:

Git submodules cannot be pinned to tags (at least not with SSH URLs).
The approach would follow the idea of WIP - Folder inheritance #79 (if I'm not mistaken) while missing a lot. For instance, I don't see how Copier would fall back to copying and rendering all files and folders from the Copier base template (included via a Git submodule) unless the Copier child template contained files and folders with the same name, thus taking precedence over those in the Copier base template. And in any case, WIP - Folder inheritance #79 seems like the wrong approach.

Additional context

I think it would be a huge win for Copier to support Copier template hierarchies. If others agree with this idea or we refine it to a promising proposal, I'd be happy to work on a draft PR.

The text was updated successfully, but these errors were encountered:

yajo · 2023-01-26T19:23:58Z

Seems interesting indeed. As you know, I always try to keep it as simple as possible here. So I've been thinking about a possible solution for your problem that doesn't require a lot of refactoring.

Quoting from https://copier.readthedocs.io/en/stable/#basic-concepts:

Copier doesn't replace the DRY principle... but sometimes you simply can't be DRY and you need a DRYing machine...

So maybe it's time for Copier to DRY itself!

Have you thought about having a metatemplate? A template that generates templates? This way:

Child templates are just normal copier templates that have other templates applied.
You get full reproducibility and updatability, just with the well known copier update... but from the child template itself.
Still doesn't replace all the well known DRY methods you already mentioned that Git, Copier and Jinja provide you. But it also fills the gap where they're still not enough.

I have made a proof of concept in https://github.com/yajo/copierception. According to my tests, the only problem is that the child template doesn't get the copier.yaml file created as it should. This is because it's explicitly skipped in Copier for obvious reasons. But we can safely no skip it if there's a _subdirectory specified, because in that case anything inside of it is put there on purpose.

Seems like an easy solution. What do you think?

sisp · 2023-01-27T11:57:23Z

That's a great idea! 👏 Also, thanks for the proof of concept! 🙏 (And great name copierception and meme! 😂)

I think this could work indeed. 🎉 I've created another proof of concept with a chain of meta-templates to check whether/how this could be done:

flowchart TD
    copier-python-meta --> copier-python
    copier-python --> python-project
    copier-python-meta --> copier-python-django-meta
    copier-python-django-meta --> copier-python-django
    copier-python-django --> python-django-project

For this to work, I think the workflow looks like this:

`copier-python-meta`

The filesystem layout of a (simplified) generic Python meta-template might look like this:

copier-python-meta
├── copier.yml
└── meta-template
    ├── [[ _copier_conf.answers_file ]].j2
    ├── copier.yml
    └── template
        ├── {{ _copier_conf.answers_file }}.jinja
        ├── {{ project_name }}
        │   └── __init__.py
        └── pyproject.toml.jinja

As you mentioned, copier.yml needs to be removed from the _exclude list in /copier.yml.

`copier-python`

Using copier-python-meta, a regular Copier template copier-python for generic Python projects can be generated:

$ copier copier-python-meta copier-python
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+b21a356
    create  .
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  .copier-answers.yml
    create  copier.yml

And the resulting filesystem layout of the copier-python template looks like this:

copier-python
├── .copier-answers.yml
├── copier.yml
└── template
    ├── {{ _copier_conf.answers_file }}.jinja
    ├── {{ project_name }}
    │   └── __init__.py
    └── pyproject.toml.jinja

Using this template, a generic Python project (e.g. python-project) can be generated as usual.

`copier-python-django-meta`

Now, let's create a more specific meta-template for a (simplified) Python web project that uses Django based on copier-python-meta.

The following filesystem layout needs to be created manually:

copier-python-django-meta
├── copier.yml
└── meta-template
    └── [[ _copier_conf.answers_file ]].j2

Then, the content of the folder meta-template/ is further populated by the generated output of the copier-python-meta template:

$ copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+b21a356
 identical  .
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  .copier-answers-meta.yml
    create  copier.yml

Now, just for the sake of this example, we could add the file meta-template/template/manage.py.jinja, that Django uses, to exemplify that the child template is an extension of the copier-python(-meta) template. The resulting filesystem layout looks like this:

copier-python-django-meta
├── copier.yml
└── meta-template
    ├── .copier-answers-meta.yml
    ├── [[ _copier_conf.answers_file ]].j2
    ├── copier.yml
    └── template
        ├── {{ _copier_conf.answers_file }}.jinja
        ├── {{ project_name }}
        │   └── __init__.py
        ├── manage.py.jinja
        └── pyproject.toml.jinja

Note the flag --answers-file .copier-answers-meta.yml which is important to avoid a conflict between the answers file created by copier-python-meta and the answers file [[ _copier_conf.answers_file ]].j2 that is rendered when generating the copier-python-django template (see below). On the other hand, .copier-answers-meta.yml shouldn't be copied into the generated child template copier-python-django, so it needs to be excluded via _exclude in /copier.yml.

When the copier-python-meta meta-template changes and we would like to update this meta-template accordingly, we would run the following command:

copier --answers-file .copier-answers-meta.yml update copier-python-django-meta/meta-template

Again, note the flag --answers-file .copier-answers-meta.yml.

`copier-python-django`

Using copier-python-django-meta, a regular Copier template copier-python-django for Python web projects using Django can be generated:

$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post1.dev0+04c5d90
    create  .
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/manage.py.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  .copier-answers.yml
    create  copier.yml

The resulting filesystem layout looks like this:

copier-python-django
├── .copier-answers.yml
├── copier.yml
└── template
    ├── {{ _copier_conf.answers_file }}.jinja
    ├── manage.py.jinja
    ├── {{ project_name }}
    │   └── __init__.py
    └── pyproject.toml.jinja

Using this template, a Python web project using Django (e.g. python-django-project) can be generated as usual.

WDYT, @yajo? If you agree with this extended proof of concept:

Do you think we should automatically allow copying a copier.yml file (i.e. remove it from the _exclude list) in a template when _subdirectory is used and is not "" or "."?
Should we add a proper guide in the docs that describes this workflow?

sisp · 2023-01-27T12:40:34Z

We could even create a template for meta-templates that generates this filesystem layout. 🤣

yajo · 2023-01-28T17:22:47Z

Note the flag --answers-file .copier-answers-meta.yml which is important to avoid a conflict between the answers file created by copier-python-meta and the answers file [[ _copier_conf.answers_file ]].j2 that is rendered when generating the copier-python-django template (see below). On the other hand, .copier-answers-meta.yml shouldn't be copied into the generated child template copier-python-django, so it needs to be excluded via _exclude in /copier.yml.

IIUC this isn't really a big problem because both the meta-template and the sub-template use _subdirectory. Thus, the sub-template's answers file (which points to the version in the meta-template) would be outside that subdirectory, so it would be ignored when producing a project based on the sub-template.

Do you think we should automatically allow copying a copier.yml file (i.e. remove it from the _exclude list) in a template when _subdirectory is used and is not "" or "."?

Sure. Actually we should completely remove any default value if using a subdirectory.

Should we add a proper guide in the docs that describes this workflow?

Definitely. Template reusability strategies are a common subject in forums, so it makes sense to make them a chapter in docs.

sisp · 2023-01-28T19:26:23Z

IIUC this isn't really a big problem because both the meta-template and the sub-template use _subdirectory. Thus, the sub-template's answers file (which points to the version in the meta-template) would be outside that subdirectory, so it would be ignored when producing a project based on the sub-template.

The conflict is between the answers file of the parent meta-template copier-python-meta and the (templated) answers file of the child meta-template copier-python-django-meta. If I ran

-copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template
+copier copier-python-meta copier-python-django-meta/meta-template

then copier-python-django-meta's filesystem layout would be

 copier-python-django-meta
 ├── copier.yml
 └── meta-template
-    ├── .copier-answers-meta.yml
+    ├── .copier-answers.yml
     ├── [[ _copier_conf.answers_file ]].j2
     ├── copier.yml
     └── template
         ├── {{ _copier_conf.answers_file }}.jinja
         ├── {{ project_name }}
         │   └── __init__.py
         ├── manage.py.jinja
         └── pyproject.toml.jinja

and generating python-copier-django using this meta-template would show a conflict for .copier-answers.yml

$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post2.dev0+c422bf7
    create  .
    create  .copier-answers.yml
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/manage.py.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml
    create  copier.yml

because meta-template/.copier-answers.yml (which points to the version of the copier-python-meta meta-template) and meta-template/[[ _copier_conf.answers_file ]].j2 after rendering have identical file names. It looks like meta-template/[[ _copier_conf.answers_file ]].j2 overwrites meta-template/.copier-answers.yml, but this behavior is ambiguous even if its implementation is deterministic. Do you see my point?

I think it would be great if there was a way to avoid a custom answers file name for pointing to the parent meta-template, but I don't see one at the moment. Any idea?

Do you think we should automatically allow copying a copier.yml file (i.e. remove it from the _exclude list) in a template when _subdirectory is used and is not "" or "."?

Sure. Actually we should completely remove any default value if using a subdirectory.

👍 I'd be happy to send a PR next week.

Should we add a proper guide in the docs that describes this workflow?

Definitely. Template reusability strategies are a common subject in forums, so it makes sense to make them a chapter in docs.

👍 I'd be happy to write a draft and send a PR.

sisp · 2023-02-10T13:04:34Z

Perhaps a slightly better alternative to the --answers-file .copier-answers-meta.yml flag in the command

copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template

is

copier --answers-file ../.copier-answers.yml copier-python-meta copier-python-django-meta/meta-template

which

doesn't introduce a different name for the answers file, and
moves the answers file related to the parent meta-template (copier-python-meta) to the root folder of the child meta-template (copier-python-django-meta), which seems better to me.

The resulting filesystem layout of copier-python-django-meta then looks as follows:

copier-python-django-meta
├── .copier-answers.yml
├── copier.yml
└── meta-template
    ├── [[ _copier_conf.answers_file ]].j2
    ├── copier.yml
    └── template
        ├── {{ _copier_conf.answers_file }}.jinja
        ├── {{ project_name }}
        │   └── __init__.py
        └── pyproject.toml.jinja

yajo · 2023-02-21T15:55:17Z

I'm trying to understand but I really don't get what's the problem of using just the default copier answers file.

According to my tests, after #941 was merged, the meta-template sample works as expected. Check out this shell session:

$ subtpl=$(mktemp -d)
$ dst=$(mktemp -d)

# Create a sub-template from the meta-template
$ copier -r 84e05b9ae447a4e36e9e678270af8cd3e0cad026 copy https://github.com/yajo/copierception $subtpl
🎤 python (bool)
   Yes
🎤 pre_commit (bool)
   Yes

Copying from template version 0.0.0.post2.dev0+84e05b9
 identical  .
    create  template
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  template/{% if pre_commit %}.pre-commit.yaml{% endif %}
    create  template/pyproject.toml.jinja
    create  template/README.md.jinja
    create  copier.yaml
    create  .copier-answers.yml


$ cd $subtpl

$ git init
Inicializado repositorio Git vacío en /tmp/tmp.eC3Bv4Ijmq/.git/

$ git add -A

$ git commit -m 1
[main (commit-raíz) 6499725] 1
 6 files changed, 55 insertions(+)
 create mode 100644 .copier-answers.yml
 create mode 100644 copier.yaml
 create mode 100644 template/README.md.jinja
 create mode 100644 template/pyproject.toml.jinja
 create mode 100644 template/{% if pre_commit %}.pre-commit.yaml{% endif %}
 create mode 100644 template/{{ _copier_conf.answers_file }}.jinja

# Generate a final project from the sub-template
$ copier copy $subtpl $dst
/nix/store/asr4mbhx1va7ha0kn3cvx4n0qlxmg8vk-python3.10-copier-7.1.0a0.dev20230221141200+nix-git-ab72507/lib/python3.10/site-packages/copier/vcs.py:155: ShallowCloneWarning: The repository '/tmp/tmp.eC3Bv4Ijmq' is a shallow clone, this might lead to unexpected failure or unusually high resource consumption.
  warn(
No git tags found in template; using HEAD as ref
🎤 project_name
   my project
🎤 build_system
   poetry

Copying from template version 0.0.0.post1.dev0+6499725
 identical  .
    create  .copier-answers.yml
    create  pyproject.toml
    create  README.md

# Update the sub-template getting last changes from the meta-template
$ copier update
No git tags found in template; using HEAD as ref
Updating to template version 0.0.0.post3.dev0+fea6c75
No git tags found in template; using HEAD as ref
🎤 python (bool)
   Yes
🎤 pre_commit (bool)
   Yes

Copying from template version 0.0.0.post3.dev0+fea6c75
 identical  .
 identical  template
 identical  template/{{ _copier_conf.answers_file }}.jinja
 identical  template/{% if pre_commit %}.pre-commit.yaml{% endif %}
 identical  template/pyproject.toml.jinja
 identical  template/README.md.jinja
 identical  copier.yaml
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml

# Check that the sub-template was properly updated
$ git diff
diff --git a/.copier-answers.yml b/.copier-answers.yml
index bb3d46a..55ab783 100644
--- a/.copier-answers.yml
+++ b/.copier-answers.yml
@@ -1,5 +1,5 @@
 # Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
-_commit: 84e05b9
+_commit: fea6c75
 _src_path: https://github.com/yajo/copierception
 pre_commit: true
 python: true

sisp · 2023-02-21T16:20:52Z

Thanks for sticking with me and challenging me! 🙏 I really appreciate your time invest into this discussion! 🙏

In your shell session example, you have only one meta-template (copierception) from which you generate the template in $subtpl from which you generate the project in $dst. In this scenario, there's no problem.

But I'm considering an extended meta-template scenario in which there are two meta-templates: copier-python-meta and copier-python-django-meta where copier-python-django-meta is based on the generated sub-template from copier-python-meta (and possibly edited, which is fine because changes in the meta-template copier-python-meta can be applied by the regular update mechanism). Only this scenario causes a conflict of the answers files.

If you go through the example workflow in #934 (comment) and run

copier copier-python-meta copier-python-django-meta/meta-template

instead of

copier --answers-file .copier-answers-meta.yml copier-python-meta copier-python-django-meta/meta-template

or (as I later improved IMO)

copier --answers-file ../.copier-answers.yml copier-python-meta copier-python-django-meta/meta-template

then you'll see

$ copier copier-python-django-meta copier-python-django
No git tags found in template; using HEAD as ref

Copying from template version 0.0.0.post2.dev0+c422bf7
    ...
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml
    ...

which is the result of copier-python-django-meta/meta-template/.copier-answers.yml and copier-python-django-meta/meta-template/[[ _copier_conf.answers_file ]].j2 conflicting after rendering the latter.

I know it's a brain twist with the many meta-templates and sub-templates generated into other meta-templates. 🤣

Can you reproduce the problem now? 🤞

yajo · 2023-02-25T07:45:10Z

Haha I need a meta-brain to follow you! 🤭

Could you prototype something and give me a code snippet to reproduce it please? I guess it'll be easier to understand.

sisp · 2023-02-26T09:31:25Z

Sure, I'll prepare something next week. 🙂

sisp · 2023-02-27T20:24:47Z

I've created an example with two meta-templates based on the example above:

copier-python-meta is a simplified Python base meta-template. You can generate a regular template and a project from the regular template like this:
```
copier https://gitlab.com/copier-meta-templates-example/copier-python-meta copier-python
copier copier-python pyproj
```
Everything is fine, keep reading. 😉

copier-python-django-meta is a simplified Python Django meta-template whose child-template is a (minor) extension of the Python base meta-template's child-template. I have made two releases:

v1.0.0 contains a child-template that was initialized without the --answers-file flag:

copier https://gitlab.com/copier-meta-templates-example/copier-python-meta copier-python-django-meta/meta-template/

So the answers file related to the copier-python-meta meta-template is meta-template/.copier-answers.yml. Now, when you generate the regular template, you'll see the following conflict:

$ copier --r v1.0.0 https://gitlab.com/copier-meta-templates-example/copier-python-django-meta copier-python-django

Copying from template version 1.0.0
    ...
  conflict  .copier-answers.yml
 overwrite  .copier-answers.yml

v1.0.1 contains a child-template that was initialized with the --answers-file flag:

copier --answers-file ../.copier-answers.yml https://gitlab.com/copier-meta-templates-example/copier-python-meta copier-python-django-meta/meta-template/

So the answers file related to the copier-python-meta meta-template is .copier-answers.yml. Now, when you generate the regular template, you'll see no conflict:

$ copier --r v1.0.1 https://gitlab.com/copier-meta-templates-example/copier-python-django-meta copier-python-django

Copying from template version 1.0.1
    create  .
    create  template
    create  template/{{ project_name }}
    create  template/{{ project_name }}/__init__.py
    create  template/pyproject.toml.jinja
    create  template/manage.py.jinja
    create  template/{{ _copier_conf.answers_file }}.jinja
    create  template/README.md.jinja
    create  .copier-answers.yml

Do you see the difference and why passing the --answers-file flag (and IMO ideally with the value ../.copier-answers.yml) is needed (or rather recommended - it works despite overwriting but is sensitive to which file takes precedence)?

Kludex · 2023-03-24T11:57:55Z

This would be useful for FastAPI projects as well. 👍

Any alternative way to achieve this before this idea gets in?

sisp · 2023-03-24T13:57:21Z

@Kludex Do you mean a more native/direct way without the meta-template "detour"?

Kludex · 2023-03-24T14:53:17Z

I didn't get your question, but what I want is to be able to create the full structure, and then "components" on subdirectories. Those components can be different copier projects...

yajo · 2023-03-24T15:07:48Z

IIUC you're going too deep @sisp. As it always has been, with Copier 1 template = 1 repo. If you want to use several levels of meta-templates, each one of them must produce a template by itself, which should be pushed to a separate repo.

So, a meta template of 3 levels would be:

copier.yml
<< _copier_conf.answers_file >>.jinja1
top-template/
  copier.yml
  [[ _copier_conf.answers_file ]].jinja2
  middle-template/
    copier.yml
    (( _copier_conf.answers_file )).jinja1
    final-template/
      {{ _copier_conf.answers_file }}.jinja
      launch_rocket.py.jinja

Thus you copy it like this:

# Use top-template to generate a middle one and publish it
copier copy https://gitlab.com/example/top-template ./middle
cd middle
git init
git add -A
git commit -m 'middle template'
git remote add origin https://gitlab.com/example/middle-template
git push -u origin main
cd ..

# Use middle-template to generate the final one and publish it
copier copy https://gitlab.com/example/middle-template ./final
cd final
git init
git add -A
git commit -m 'final template'
git remote add origin https://gitlab.com/example/final-template
git push -u origin main
cd ..

# Use final-template to generate a... real project!
copier copy https://gitlab.com/example/final-template ./real
cd real
git init
git add -A
git commit -m 'build: init from template'
git remote add origin https://gitlab.com/example/real
git push -u origin main
./launch_rocket.py

In such scenario, these will always be true:

.copier-answers.yml provides answers to the parent template
tpl/{{ _copier_conf.answers_file }}.jinja (or equivalent syntax) instructs copier where to put answers for the next meta-generation.
End users will only care about https://gitlab.com/example/final-template. Only template maintainers will know that there exists a meta-template (and a meta-meta-template) that makes their template maintenance easier.

Reaching this point probably means that you're doing something wrong. But it'd work as long as:

All are git-tracked.
All are pushed to different repos.
All are configured with different jinja markers.
All use the _subdirectory option.

sisp · 2023-03-25T12:03:37Z

I didn't get your question, but what I want is to be able to create the full structure, and then "components" on subdirectories. Those components can be different copier projects...

@Kludex I don't think I understand what you'd like to achieve exactly. Could you provide a simple example?

sisp · 2023-03-25T12:09:51Z

@yajo I'm not trying to build a meta-template with 3 levels but rather two meta-templates that each generate a regular template but one meta-template extends the other. This is important because it enables meta-templates to extend one another while being developed by independent authors in independent projects. I even think a meta-template with 3 levels covers a different use case than what I'm suggesting.

It seems it's difficult to convey the approach here. Do you think it would make sense to have a brief voice chat about it some time?

sisp · 2023-04-19T19:56:05Z

@yajo and I had a video chat today, and after a while he suggested to simply fork a parent template and update the child template from the parent template using git merge/git pull, so neither a meta-template nor a new Copier feature would be needed. Indeed, this seems to work, but it requires keeping the parent template's history in the child template, otherwise Git isn't able to determine the correct patch. This may be a bit confusing though because it may include commits like "docs: Update changelog for version 0.4.0" which, in this case, contains a reference to a release version of the parent template which has no meaning in the child template. Also, it might make sense to ignore some files in the parent template like CHANGELOG.md because the child template would always have its own changelog. WDYT, @yajo?

I think a kind of squash-merge approach might be better. But then, since the parent template's history would not be present in the child template's repo, we can't simply perform a pull/merge from the parent template's repo because Git can't determine the correct patch. That's why we'd need to compute the patch manually which means we'd need to keep a record of the parent template's version on which the child template is built. The manual workflow might look like this:

Create a child template based on a parent template, e.g. https://github.com/pawamoy/copier-poetry:
1. Clone the repo into a folder named after the new project, e.g. copier-django, and include only the tagged commit:
```
git clone --depth 1 --branch 0.3.2 https://github.com/pawamoy/copier-poetry.git copier-django
```
2. Reinitialize the Git repository to delete its history, tags, etc., and make a new initial commit:
```
$ rm -rf .git
$ git init
$ git add .
$ git commit -m "chore: initialize with gh:pawamoy/copier-poetry#0.3.2"
```
3. Extend the template according to your needs, commit the changes, and make a release:
```
# Make changes, and then ...
$ git add .
$ git commit
$ git tag v0.1.0
```

When the parent template evolves, e.g. 0.4.0 gets released, apply the changes to the child template:

Compute the patch for 0.3.2 -> 0.4.0:

$ tmpdir=$(mktemp -d)
$ git clone --depth 1 https://github.com/pawamoy/copier-poetry.git $tmpdir
$ pushd $tmpdir
$ git fetch --no-tags origin tag 0.3.2 0.4.0
$ git diff 0.3.2 0.4.0 > 0.3.2-0.4.0.patch
$ popd

Apply the patch to the child template:
```
git apply -3 $tmpdir/0.3.2-0.4.0.patch
```

Resolve any potential merge conflicts and commit the update:

git commit -m "chore: update to template gh:pawamoy/copier-poetry#0.4.0"

This feels like something Copier should be able to do for us. So Copier could offer the following new commands:

copier template create [-r,--vcs-ref] parent_template_src child_template_destination_path
copier template update [-r,--vcs-ref] child_template_destination_path

And as mentioned above, we'd need to keep a record of the parent template's version (and URL, actually) on which the child template is built. Copier might keep this information in a file like .copier-template.yml:

src: https://github.com/pawamoy/copier-poetry.git
ref: 0.3.2  # commit hash or tag name

This extension would be non-intrusive because (a) self-contained templates would remain unchanged, only child templates would contain this extra file, and (b) any existing template could serve as a parent template out of the box. WDYT, @yajo?

There's one legal aspect I'm not sure about though. Is it legally allowed (with regard to license and copyright) to effectively squash-merge a parent template (update)? Would it suffice to keep the parent template's license and add the child template's author as an additional copyright holder? In the fork approach, the commit history of the parent template would also exist in the child template's repo, so there is proper attribution of the changes coming from the parent template. When they are squashed, this information is gone. Does anybody know? @yajo @pawamoy

This is another long comment, @yajo. Sorry ... 🙈 😄

yajo · 2023-04-23T06:35:59Z

I think this is a different thing than copier. 😅 Let me explain. Copier is made to bootstrap projects based on a template and keep them updated. In the Unix philosophy, that's the one thing it does well. 🎯 What you're looking for, though, is a replacement for `git pull --squash`. Let's say we solve it in Copier. I think that all you want is a copy mode where rendering is disabled. We could solve that with a `--render=no` flag for both copy and update. At first it seems easy, but in reality it goes against all that copier is. For example, Copier would need to ignore the copier.yml file in the template, because its instructions can be templated. But we're not rendering, so chances are that the raw copier.yml produces failures. But if we ignore that file, how are you supposed to say that you want to skip CHANGELOG.md if it exists? How are we supposed to infer the default path for the answers file? The answers file would be simplified because it would contain no answers. However it still needs the src url, commit and a special new `_render: false` answer. However, without rendering, how are we supposed to convert `{{_copier_conf.answers_file}}.jinja` into the answers file? Even if fixing #983 as has been discussed there, it still makes no sense without reading the copier.yml file. And if we managed somehow to read it, many settings would become confusing. For example, we don't want to use the subdirectory setting, even if found, because we want to clone the full repo always. But that would change the semantics of the template manager internally completely. Also, why would we have to bother any upstream repo about having a copier.yml file to better support forks? Git pull works just fine with any git repo regardless of its content. Why would a replacement tool be worse than that? This tool should work entirely on the side of the fork, without having to ask for anything upstream. After all, it's the fork who cares about i.e. the CHANGELOG problem, and you could even decide to solve it with a CHANGELOG_DOWNSTREAM.md file. Upstream shouldn't care annoy any of those. Why would we need copier's complex update diff algorithm at all? There's no rendering, so there's no need to replay last render, right? But then, why do we even call the answers file like that, if it has no answers? And the list of nonsense (for copier) keeps growing if you think about it. But I think we have enough... So, my friend, if git pull is still not enough for you, indeed you need something else. But it seems to me that something is not copier. 😄 It's more close to mrchef, but still it isn't the same. I think it's a totally new tool (or maybe it exists already? 🧐). There's one legal aspect I'm not sure about though. Is it legally allowed

(with regard to license and copyright) to effectively squash-merge a parent template (update)? Would it suffice to keep the parent template's license and add the child template's author as an additional copyright holder? In the fork approach, the commit history of the parent template would also exist in the child template's repo, so there is proper attribution of the changes coming from the parent template. When they are squashed, this information is gone. Does anybody know? @yajo <https://github.com/yajo> @pawamoy <https://github.com/pawamoy>

AFAIK FLOSS licenses cover the software source and binary distributions, but not the way it's built or credited. It would be a concern for the user of your new tool I think, not for you. Anyways, IANAL.

…

Message ID: ***@***.***>

yajo · 2023-04-23T06:43:17Z

You're not the first one with these needs:

sisp added the enhancement label Jan 23, 2023

yajo added the documentation Issue that requires updating docs label Jan 28, 2023

yajo added this to the Community contribution milestone Jan 28, 2023

sisp mentioned this issue Jan 30, 2023

feat: ignore default excludes when using subdirectory #941

Merged

sisp mentioned this issue Feb 14, 2023

docs: add chapter on template hierarchies with meta-templates #969

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updatable templates for template hierarchies and reuse #934

Updatable templates for template hierarchies and reuse #934

sisp commented Jan 23, 2023 •

edited

Loading

yajo commented Jan 26, 2023

sisp commented Jan 27, 2023 •

edited

Loading

sisp commented Jan 27, 2023 •

edited

Loading

yajo commented Jan 28, 2023

sisp commented Jan 28, 2023

sisp commented Feb 10, 2023 •

edited

Loading

yajo commented Feb 21, 2023

sisp commented Feb 21, 2023 •

edited

Loading

yajo commented Feb 25, 2023

sisp commented Feb 26, 2023

sisp commented Feb 27, 2023 •

edited

Loading

Kludex commented Mar 24, 2023

sisp commented Mar 24, 2023

Kludex commented Mar 24, 2023

yajo commented Mar 24, 2023

sisp commented Mar 25, 2023

sisp commented Mar 25, 2023

sisp commented Apr 19, 2023

yajo commented Apr 23, 2023 via email

yajo commented Apr 23, 2023

Updatable templates for template hierarchies and reuse #934

Updatable templates for template hierarchies and reuse #934

Comments

sisp commented Jan 23, 2023 • edited Loading

yajo commented Jan 26, 2023

sisp commented Jan 27, 2023 • edited Loading

copier-python-meta

copier-python

copier-python-django-meta

copier-python-django

sisp commented Jan 27, 2023 • edited Loading

yajo commented Jan 28, 2023

sisp commented Jan 28, 2023

sisp commented Feb 10, 2023 • edited Loading

yajo commented Feb 21, 2023

sisp commented Feb 21, 2023 • edited Loading

yajo commented Feb 25, 2023

sisp commented Feb 26, 2023

sisp commented Feb 27, 2023 • edited Loading

Kludex commented Mar 24, 2023

sisp commented Mar 24, 2023

Kludex commented Mar 24, 2023

yajo commented Mar 24, 2023

sisp commented Mar 25, 2023

sisp commented Mar 25, 2023

sisp commented Apr 19, 2023

yajo commented Apr 23, 2023 via email

yajo commented Apr 23, 2023

sisp commented Jan 23, 2023 •

edited

Loading

sisp commented Jan 27, 2023 •

edited

Loading

`copier-python-meta`

`copier-python`

`copier-python-django-meta`

`copier-python-django`

sisp commented Jan 27, 2023 •

edited

Loading

sisp commented Feb 10, 2023 •

edited

Loading

sisp commented Feb 21, 2023 •

edited

Loading

sisp commented Feb 27, 2023 •

edited

Loading