How to update a project with changes from its cookiecutter?

[aroig]

Hi,

I maintain several personal cookiecutters, and I found that I spend a lot of time keeping projects in sync with updates on the cookiecutter.

I started trying a workflow to deal with this, as follows:

1. Every project I render is assumed to be versioned by git. In the project repo, there is a branch template, that contains the result of cookiecutter rendering, without any other changes.
2. Every cookiecutter renders a file .cookiecutter.json at the root of the project containing the original context used to render that project, in json format.
3. Every project has a Makefile containing at least an update-template target. This make target does the following:
   • Create a temporary checkout (via git worktree) of the template branch.
   • Use the stored context in .cookiecutter.json to regenerate the project on top of the temporary checkout, overriding previous content.
   • Commit the newly rendered project in the template branch.

This way, the template branch tracks the changes from the cookiecutter, and I can use git-merges into master to update the current project.

The key thing is that by using git merges, I can update template files that got havily modified in the project. Most of the time the merge is clean, but from time to time a conflict will arise, which is easily resolved.

I've used it for a while and works pretty well. There are at least a couple of pain points though, that could benefit from changes in cookiecutter upstream, which may be of independent interest.

1. The template I use to generate a .cookiecutter.json is a bit hacky. I'd like to do something like this:

   {{ cookiecutter | jsonify }}
   

   Would you accept distributing this jsonify as a default extension? I can provide the code in an other PR.

2. In order to get updates from the cookiecutter, the project stores its url in a _template field on its context. Right now, I hard-code this url in the cookiecutter itself. I would love to free my cookiecutters from hard-coding their url, That could be achieved if cookiecutter itself would inject a _template field into the context with the url used at the time of rendering. I submitted Include template directory or url in the context dictionary #774 to that effect.

Additionally, maybe a functionality like the update-template I described above could be moved into cookiecutter upstream. That would require some thinking, I guess...

Of course, any suggestions, or other ways in which a workflow like this could be better supported by cookiecutter would be very welcome!

[hackebrot]

Hi @aroig! 👋 Thank you for opening this issue.

Sorry for the late response, we are currently very short on time. 😢

Can you please provide a link to such this jsonify jinja2 extension, given it already exists?

[hackebrot]

There is something you may want to have a look at. Cookiecutter already dumps the used context as a JSON file. It's overwritten though if you use the same template again.

You could have a post_gen_project hook to copy that file though, if you know the templates name (_template for instance).

See dump context.

[aroig]

@hackebrot Thanks for the feedback!

I submitted a PR with what I meant in #791.

It didn't occur to me to use a post_gen hook for that. I still prefer the jsonify extension. Using post_gen would mean I have to hardcode the path to the cookiecutter-generated json files in the cookiecutter itself.

[gnestor]

I'd like to chime in with a solution I came up with:

cookiecutter REPO_URL --output-dir .. --config-file .cookiecutter.yaml --no-input --overwrite-if-exists

• --output-dir .. allows us to run the cookiecutter in the repo root vs. the parent directory because cookiecutter will always render output in a child directory of the output-dir.
• --config-file cookiecutter.yaml will run the cookiecutter with the original inputs which are persisted to .cookiecutter.yaml in the repo root (this is generated by the cookiecutter).
• --no-input will skip the cookiecutter prompts.
• --overwrite-if-exists will replace our previous cookiecutter output with the most up-to-date output.

[rmedaer]

@gnestor I'm doing quiet the same excepted that (such as @aroig ) I'm using Git to applu the upgrade in a separate branch and then a merge.

Furthermore I implemented the following option: "strip directory" #894. When option is given it apply the template directly on output-dir instead of creating a sub-directory in output-dir.

[aroig]

@gnestor I know you can re-render your project on top of the working directory. The problem with this approach is that this is useless once your project modifies any of the rendered files, as they will be overwritten by cookiecutter, and would need manual intervention.

The point of my original post was using git to handle this situation via merges from the template branch. I can add a lot of content to a rendered file, and as long as the lines touched in the project, do not overlap with the lines touched in a cookiecutter update, the merges are clean (no manual intervention). That was quite an improvement!

Oh, by the way, a quick update of the situation to anyone interested:

1. The jsonify extension is merged. You can just use {{ cookiecutter | jsonify }} to render the context in json format from within the cookiecutter.

2. Injecting the _template url into the context is not merged yet. As far as I know there is no pending things to do on my part, but I understand injecting stuff into the context is a bit ugly. I have no other solution besides hard-coding urls into the cookiecutter, which is also ugly.

3. It would be very nice to integrate the git boilerplate into cookiecutter itself. That would remove the need for a python script on every repo I render to achieve that purpose. And I believe this workflow is of general interest.

I'm not sure what would be the best way to integrate this git boilerplate with cookiecutter though, and I haven't had time to think about it. If anyone is interested in the scripts I use for that, can take a look at aroig/cookiecutter-latex-paper for instance. Look at the end of the Makefile and cookiecutter-update.py

[rmedaer]

@aroig I'm building a HTTP API which is using Git to manage directory content and Cookiecutter to template it. I'm doing quiet the same than your cookiecutter-update.py script for the upgrade.
Let me explain the process I'm implementing (WIP)...

Context

My HTTP API is managing a bunch of "projects" which contains various content. Each project is a Git repository. Each time you're doing an API call, it will do some magic and commit all changes made.

Projects are really similar. Thus I'm using Cookiecutter to make some project templating.

Templating

All changes made by the API are done in master branch. When user ask to "install" a template (understand "apply a cookiecutter magic"), the API will do the following...

Template install

1. Create a new orphaned branch template in a new worktree.
2. In this new-empty branch I'm running cookiecutter with extra_context (given by the API).
   I dump the context into .cookiecutter.json (thks to @aroig for his jsonify PR).
   I do NOT put _template in context (see point 4)
3. API commit all changes made by cookiecutter. (a)
4. API is adding a Git Note with cookiecutter template reference.
   This note is attached to commit made in third operation.
5. I'm removing the worktree (unlink + prune)
6. In my initial worktree, I checkout into master and merge template branch. (b)

Template upgrade

For an upgrade of the template, the API will do the following:

1. Create a worktree with HEAD on template branch but without checkout !!
2. Running cookiecutter:
   • template is found in original Git Note.
   • extra_context is coming from both user and context dumped in .cookiecutter.json (priority to user changes).
3. Commit changes. (c)
4. Removing the worktree (unlink + prune)
5. In my initial worktree (master branch) I merge template changes. (d)
   Since there is a common ancestor (it could be a 3-way merge). In this case conflicts could quickly append... Because I know content of my repository, I'm writing Git merge drivers to avoid conflicts.

Template detection

As you understood, I'm using Git features (branch and note) to detect if a template is already installed (branch) or to know which one and from where (note).

Illustration

template:        a-----------c
                  \           \
master:       -----b---(...)---d----->

If you have any advice, idea or whatever about Git/Cookiecutter workflow be my guest !

[aroig]

@rmedaer That sounds cool! and the git boilerplate, quite similar to what I do.

A couple of comments:

I'm even more convinced now that having this git boilerplate in cookiecutter as a low level thing, might be a good idea. Maybe something like a cookiecutter --update-branch option, that given a cookiecutter.json and a working directory, does the git trick if it finds a cookiecutter/template branch, and complains if it does not.

Doing an update without a template branch is not useful, because cookiecutter has no way to know whether the files have been modified in the project or not. Then maybe a cookicutter --init-branch option, to create the cookiecutter/template branch if it does not exist.

I do not have time right now to do it, but if cookiecutter devs would consider something like it, I may put it in my mid-term TODO list.

Oh, and that git note thing is interesting, it had not ocurred to me. But I imagine you cannot do that entirely in cookiecutter, your API that knows the cookiecutter repo url may do that, isn't it? It would be nice if it could be done, for instance from a cookiecutter hook, but then I think we are back to injecting the url into the context...

[rmedaer]

@aroig I agree with you ! And btw, now I'm reading myself xor your comments, here some additional idea/comment...

Assuming we are implementing update feature exclusively with Git, why should we dump the context in .cookiecutter.json since it's only used by update mechanism. I mean we could put the context in Git Note as well. In this case, each "pre-merge-commit" (what I'm calling "apply template", a and c on my illustration) could have its own Git Note attached with dumped context.

Then the upgrade procedure would be:

1. Create worktree
2. Find "root" Git Note from namespace cookiecutter/template (on my illustration: a)
3. Find previous commit Git Note from namespace cookiecutter/context (on my illustration: c)
4. Apply cookiecutter/template (2) with preloaded context cookiecutter/context (3).
   Note that this context could be override or complete. Use case: replace a template parameter, a template parameter has been added with new template version.
5. Commit changes.
6. Remove worktree.
7. Merge in master (or whatever branch you want).

[rmedaer]

I would suggest the following arguments:

• --git: To apply cookiecutter with Git features: cookiecutter --git (...).
• --git-template-branch=<branch>: by default it SHOULD use branch template, but I guess some of you would like to use another one.
• --update: Update using template branch or raise error.
  Furthermore, we should be able to give a update reference (tag, branch, release).

Otherwise we could also implement a wrapper around cookiecutter like I'm doing in my API. It's effortless for Cookiecutter developers and maybe more "flexible"...

[aroig]

why should we dump the context in .cookiecutter.json since it's only used by update mechanism

After thinking a bit more about it, I'm not sure I like sticking stuff in git notes. I think I prefer having it in files within the repo. But that might just be a personal bias.

Anyway, there are pros and cons: sticking it in the repo, implies that other people cloning it may be able to use cookiecutter with it (assuming the template url is public). On the othe hand, the git notes thing may enable you to use a private cookiecutter template without letting everyone know.

I think I would like to find a minimal set of changes to cookiecutter that enable those workflows, are as generic as possible, and leave the specifics of the workflow to cookiecutter hooks.

For example, I think that the following

• let cookiecutter use a json file as context (which could come from git notes),
• allow rendering the output into a non-checked out git branch, either newly created or update an existing branch,

would be enough to let us implement our particular workflow via hooks. It has the advantage it does not commit cookiecutter to a particular way of doing things, plus those two features may be of independent interest to other people.

But I'm not really sure...

[rmedaer]

Well as you an see in Github references, I made a small POC. I don't want to start developing a lot of crappy stuff in Cookiecutter, so I created the following project: milhoja. I saw you're coming from Spain and I like cooking with cookiecutter, so I called that Milhoja. ;-)

In the CLI module you'll find the options I identified. In "main" module I already wrote procedures I described.

[rmedaer]

@aroig To answer your comment, even we are dumping context in a Git Note, you still be able to dump it in .cookiecutter.json file.

[yajo]

So is there a way to update a cookiecutter project then?

[patcon]

fwiw, i recall doing similar things (though not as complex) without using a template branch, mostly because i leaned on git add --patch for stepping through each patchset that I wanted to commit, and not adding patches that override customized bits. the "s" (split patch) and "e" (edit patch) options when walking through the patchsets are critical to this feeling natural :)

So it's basically:

cd my/project/to/update
make update-from-template # this makefile is part of initial template. checks that git workspace is clean.
git status # shows lots of overridden files
git add . -p # walk through patchsets, selecting files for adding
git commit -m "Updated from template."

This has worked fine for me. It's non-destructive, and walking through patchsets seem to be as easy as resolving merge conflicts from an actual branch. The one downside would be file deletions, which need to be done manually.

[yajo]

After thinking a bit more about it, I'm not sure I like sticking stuff in git notes. I think I prefer having it in files within the repo. But that might just be a personal bias.

Anyway, there are pros and cons: sticking it in the repo, implies that other people cloning it may be able to use cookiecutter with it (assuming the template url is public). On the othe hand, the git notes thing may enable you to use a private cookiecutter template without letting everyone know.

If you want your private cookiecutter, just add .cookiecutter.json to .gitignore, or use a private repo that even if somebody knows about it, it cannot be used by them. I think I prefer the files approach too. It seems easier to grasp by anybody.

[chrish42]

Would have a need for this also. What's needed for this to move forward? Any opposition from the cookiecutter maintainers for integrating a solution to this into cookiecutter itself? Because at the moment, many people seem to be reinventing that wheel.

Assuming there's no opposition to the idea of adding this feature, do we still need to discuss the approach to take here, or would we be at the stage where a pull request would be the best way to move this forward? Thanks!

(One implementation of this feature is here: https://github.com/senseyeio/cupper)

[quetzai]

Hello,
There is several tools and project that sits on top of cookiecutter and can update a generated project.
The PR I'm proposing would add this feature directly inside cookiecutter.
Given the number of users, I can't possibly think of all the use cases, but merged this feature could live and be enhanced to be useful to more and more people.
As this is the beginning, any comments are welcomed :)

PR: #1173

[baurmatt]

Hey, just wanted to leave two links which might be interesting in this context:

• https://github.com/voxpupuli/modulesync
• https://github.com/puppetlabs/pdksync

Both projects are used in the Puppet universe to repeatedly update standardized files in all lot of Puppet module repositories while still allowing some customization of those files. If think this come close to what we're looking for in cookiecutter.

[jhgoodwin]

Does anyone have a suggested workflow to handle git delete/ git mv events for merging template updates into previously applied templates using git merges as suggested by this issue?

[yajo]

Just use copier.

[megazone87]

Just use copier.

This is totally another alternative than cookiecutter, but without as many templates right now. While crupt is like a feature add-on for cookiecutter. That means I still can favor plenty of templates and existed projects without more effort. So I made my choice: CRUPT.

[samj1912]

https://github.com/timothycrosley/cruft/ has recently been updated to provide more robust cookiecutter updates. It is a 100% compatible with existing cookiecutters and is a thin layer over the cookiecutter api to provide support for updating. Out of all the options I have tried, I have found it to work the best.

Disclaimer - I maintain this project so I am biased towards it.

[ssbarnea]

While risking to upset quite a few people, I think is better to close this ticket because is likely that will never happen, mostly because code generators will never be able to fully update code that was likely modified. It is more likely to have AI writing production that seeing it done.

[chrish42]

Err, nobody's asking for a magic, do-what-I-mean solution. This is a three-way merge. You do the merges you can, and then flag the conflicts for human review. This general concept is already implemented in multiple other places, including Debian for config files in /etc, and more specifically here, other projects built on top of cookiecutter (like cruft: https://github.com/timothycrosley/cruft/). It's fine for the project to decide this is out of scope. But it's clearly possible, because this problem is being solved elsewhere.

[achillesrasquinha]

This is my workaround.

1. First, generate a fresh new template.

cookiecutter COOKIECUTTER_TEMPLATE --output-dir TMP_PATH --config-file CONFIG_FILE --no-input

2. Initialize a git repository for this said template.

cd TMP_PATH/PROJECT
git init
git add . && git commit -m "Initial Commit"

3. Within your already rendered template directory, merge changes.

git remote add template TMP_PATH
git fetch template
git merge --allow-unrelated-histories template/master

You now can also track conflicts.

[samj1912]

Alternatively just use cruft https://cruft.github.io/cruft/

[iuiu34]

hi,
regarding this topic, i agree is not about a magic-tool/magic-ai that will solve all the code conflicts. But rather providing a method, to check code updates in the template efficiently, to decrease (yet not avoid) time in manual code updates.

For this, i would propose, that
given an existing_repository and a cookiecutter new-template.
You would

• render the new-template inside existing_repository/.cookiecutter folder (with the proper args)
• add .cookiecutter to the .gitignore
• with git, compare the 2 folders git diff --no-index existing_repository existing_repository/.cookiecutter --output cookiecutter.diff
• manually, review file cookiecutter.diff. Deleting what is not convenient.
• apply differences, git apply cookiecutter.diff