Unable to make copier work recursively; bug? better way?

[mogul]

I am attempting to generate a single project using sub-templates, aka "composition" as described in #934 . However, I don't see how to make this work as things stand, and believe me, I've been trying! (Maybe this discussion can lead to the sorely-needed content over here?)

Consider:

• A - Copier template for projectA that tries use the other two by calling them via tasks
• B - Copier template for subproject B
• C - Copier template for subproject C

Below is a bash script to test this pattern. This seems like it should work... Each of the three directories is generated from a different, disjoint template, and each directory has a distinct .copier-answers.yml file in it that refers to the source template. So copy seems to work fine. During that part of the script you'll see output that looks like this:

[...]
Current directory: [...path to your test directory...]
_folder_name is: projectA
[...continues...]

However, during the update operation, the working directory is not the same, and you will see an error like:

[...]
Current directory: [...path to your test directory...]
_folder_name is: copier._main.old_copy.l0p2l2_s
Error: Argument of destination_path expected to be ExistingDirectory, not 'subprojectB':
    ValueError('subprojectB is not a directory')
[...errors out...]

The docs say the default value for working_directory should be the destination directory. So why do I see this _folder_name? Shouldn't this be working?

The test script is below.

---

#!/usr/bin/env bash
set -euo pipefail

# Cleanup the directories from the last time you ran this
function cleanup() {
  echo "Cleaning up the directories we'll use..."
  rm -fr A B C projectA
}

function setup() {
  echo "Setting up the three copier templates..."

  # Make three Copier templates, each with an empty `copier.yml`
  # and the standard `{{_copier_conf.answers_file}}.jinja` content:
  mkdir {A,B,C}
  for dir in {A,B,C} ; do
    echo "# This is the $dir template" > $dir/copier.yml
    echo "{{ _copier_answers|to_nice_yaml -}}" > $dir/'{{_copier_conf.answers_file}}.jinja'
  done

  # For template A, add a `_tasks` block that appropriately calls
  # B and C in both the `copy` and `update` modes.
  cat >> A/copier.yml <<-EOT
_tasks:
  - when: "{{ _copier_operation == 'copy' }}"
    command: |
      echo "Current directory: $PWD"
      echo "_folder_name is: {{_folder_name}}"
  - when: "{{ _copier_operation == 'copy' }}"
    command: uvx copier copy --trust -r HEAD ../B subprojectB
  - when: "{{ _copier_operation == 'copy' }}"
    command: uvx copier copy --trust -r HEAD ../C subprojectC

  - when: "{{ _copier_operation == 'update' }}"
    command: |
      echo "Current directory: $PWD"
      echo "_folder_name is: {{_folder_name}}"
  - when: "{{ _copier_operation == 'update' }}"
    command: uvx copier update -r HEAD --trust subprojectB
  - when: "{{ _copier_operation == 'update' }}"
    command: uvx copier update -r HEAD --trust subprojectC
EOT

  # Commit each template into its own respective repository
  for dir in {A,B,C}; do
    (cd $dir && git init . && git add . && git commit -m "" --allow-empty-message) > /dev/null 2>&1
  done
}

function copy_and_commit_the_result() {

  echo "Now, we copy from template A to projectA, which will also"
  echo "copy from templates B and C into subdirectories, then commit the result."
  uvx copier copy -r HEAD --trust A projectA

  echo "This is what the resulting directory structure looks like:"
  tree -a -I '.git' projectA

  echo "And here's what the content of the .copier-answers.yml files looks like:"
  find projectA -name .copier-answers.yml -print | while read -r file; do
    echo "==> $file <=="
    cat "$file"
    echo
  done
  echo "Finally, we commit the result in projectA to its own repository."
  (cd projectA && git init . && git add . && git commit -m "" --allow-empty-message) > /dev/null 2>&1
}

function modify_and_try_updating {
  echo 
  echo "Now, let's modify each of the templates..."
  for dir in {A,B,C} ; do
    echo "# Here's a new file in $dir" >> $dir/file.txt
    (cd $dir && git add . && git commit -m "Update template $dir") > /dev/null 2>&1
  done

  echo "...then try to update projectA using the modified templates."
  if ! uvx copier update -r HEAD --trust projectA; then
    # If the update fails, tell the user how to reproduce the error so they can iterate
    echo
    echo "Reproduce this error by running: "
    echo "  uvx copier update -r HEAD --trust projectA"
    echo "But note that updating subprojectB directly will work:"
    echo "  cd projectA && uvx copier update -r HEAD --trust subprojectB"
    exit 1
  fi

  echo "Here's what the updated directory structure looks like:"
  tree -a -I '.git' projectA

  echo "And here's what the updated .copier-answers.yml files look like:"
  (cd projectA && git add . && git diff)
  # find projectA -name .copier-answers.yml -print | while read -r file; do
  #   echo "==> $file <=="
  #   cat "$file"
  #   echo
  # done
}

cleanup
setup
copy_and_commit_the_result
modify_and_try_updating

[mogul]

(cc @yajo @sisp @francesco086 btw)

[sisp]

Composition using tasks is an experimental idea with some known quirks, it's not officially supported by Copier. With the following changes, your specific example will work:

   - when: "{{ _copier_operation == 'copy' }}"
     command: uvx copier copy --trust -r HEAD ../C subprojectC

-  - when: "{{ _copier_operation == 'update' }}"
-    command: |
-      echo "Current directory: $PWD"
-      echo "_folder_name is: {{_folder_name}}"
-  - when: "{{ _copier_operation == 'update' }}"
-    command: uvx copier update -r HEAD --trust subprojectB
-  - when: "{{ _copier_operation == 'update' }}"
-    command: uvx copier update -r HEAD --trust subprojectC
+_migrations:
+  - command: |
+      echo "Current directory: $PWD"
+      echo "_folder_name is: {{_folder_name}}"
+  # Commit changes from updating to A's new version.
+  - command: |
+      git  add .
+      git commit -m "update A"
+  - command: |
+      uvx copier update -r HEAD --trust subprojectB
+      git add .
+      git commit -m "update subprojectB"
+  - command: |
+      uvx copier update -r HEAD --trust subprojectC
+      git add .
+      git commit -m "update subprojectC"
 EOT
 
   # Commit each template into its own respective repository

How is this different?

1. Copier internally generates fresh projects based on the old and new template versions during an update. Those projects are generated using essentially copier copy .... Configuring a task with when: "{{ _copier_operation == 'update' }}" means that it runs only during an update for each internal copier copy ... call. This isn't what you want. Instead, you want to run copier update for subproject B and C when running copier update for project A (after project A has been updated to the new version of template A). For this, you need to use a migration task which runs after the update (default behavior unless you explicitly specify when). Admittedly, the difference between a task with when: "{{ _copier_operation == 'update' }}" and a migration task isn't well documented. We've discussed and summarized it in feat: Add _operation variable #1733 (comment).
2. The new version of template A contains a change itself, so the repository of project A is dirty before running the migration tasks that run copier update for subprojects B and C. copier update cannot be run on a dirty project though. So, we need to commit the changes in the first migration task before running copier update for subprojects B and C.
3. Similarly, running copier update in each of the migration tasks results in a dirty project, so the changes need to be committed before running the next copier update command.

While this modified demo script works, a few problems with this approach are revealed:

• A full update involves multiple "partial" updates. The changes of each "partial" update must be committed automatically, else the next "partial" update would be applied to a dirty project which isn't supported by Copier. This means, each "partial" update has a dedicated commit with a commit message that the Copier user cannot (easily) control.
• Each of the "partial" updates may have merge conflicts which cannot be resolved. Instead, the conflicts are automatically committed which might even break subsequent "partial" updates altogether.

If Copier supported this kind of template composition natively, it might be possible to interrupt the update process upon merge conflicts, allow the user to resolve them, and then resume the update process – similar to how, e.g., git rebase works. Also, it might be possible to only make temporary commits for all "partial" updates and soft-reset them at the end, so that the changes of the full update are neither committed nor staged. I think it's an idea worth pursuing. But this is most likely a non-trivial feature implementation which needs a proper design proposal to start with. If anybody is willing to give it a try, I'm happy to help.

/cc @francesco086

[mogul]

Thanks for asking! After more consideration of what a monorepo would need to look like (eg a collection of YAML files in .github/workflowseach contributed by a different template) I ended up going the route recommended in the docs, with all templates targeting the top-level directory. The result is still in progress, but here if you want to take a look.

I do want to see these methods explored more in the docs, but my particular project is unblocked.

[francesco086]

Uhm... interesting! I think what you have done is a mixture of what I have done and the route recommended in the docs 😆 you used the tasks, like me, and used them to instantiate the various template that compose your complete template.

I believe that docs suggest to manage all the various templates "manually", meaning applying copier copy etc. for each template manually, just applying them to the same folder.

Interesting approach, I will keep it in mind for this docs PR

[mogul]

Yes, I'm using the main, top-level hello-ato template to gather information in a non-redundant fashion, then pass it on to the other implicated templates so they don't have to ask for everything again. I am however pointing them all at the same top-level path, so they can populate everything including .github as warranted. Not obvious here is that they're all explicitly authored so as to avoid collisions.

[mogul]

Oh, and: I'm putting the responsibility for the name of the answers file on each of the sub-templates, like so.

[mogul]

Though now I'm rethinking that, since the knowledge of the name of the answers file is now spread between the parent and the sub-templates. 🤔

Anyway, a work in progress still. I think I will be working on it again this week.