CookieKeeper: templates as reusable, enduring artifacts rather than project generators

[glyph]

(This is a feature request / design discussion rather than a bug report, so, template omitted.)

Cookiecutter's primary use seems to be at the point where one generates a new project. However, the files that Cookiecutter templates generate (tox.ini, travis.yml, etc) frequently need to be updated over time, and best practices evolve, so templates rarely stand still.

The most illustrative example I can think of for this is https://github.com/audreyr/cookiecutter-pypackage/blame/master/README.rst - features like auto-release to PyPI were added after the template was originally created, so projects initially created by that template would miss out. Projects may want to migrate from bumpversion to versioneer to towncrier. New projects might want to start off with flit rather than setup.py, but migrate when they get bigger or need more advanced features.

I maintain dozens of projects which already exist, and sadly few (if any) were originally created using these templates. But I want to stop submitting individually-reviewed PRs to do stuff like dropping python 2.6 (or 3.3) in the travis.yml, the tox.ini (etc) of 17 different projects. I want to have one master template and then just re-cut the cookies across the 25 or so repos on https://github.com/twisted.

I don't know exactly what cookiecutter needs to do in order to support this workflow. Perhaps this is already possible with some sort of discipline around how these files are written and maintained; so my main request here is that the docs examine this use case, and explore how to deal with the passage of time, and the updating of 1 template across N projects. But I believe that in many cases some kind of merging support is required; for example, given the case of PyPI uploads, if some modification needs to be made to that workflow, the travis secrets need to be maintained in the travis.yml, so that needs to be parsed and re-emitted.

[pydanny]

This is something that was considered from the first day the project launched. In fact, myself and Audrey have cooked up in-house tools to "kinda sorta" make this happen for ourselves on various projects. But our requirements are specific for us only, and open sourcing these tools doesn't make any sense.

To make a tool of this sort usable by others is an enormous amount of sophisticated work. It's the kind of effort that involves at least a hundred hours of work. We've wanted to do this, but when we asked the Cookiecutter community (including companies with deep pockets) if they were interested in our working on this kind of thing, it fell on deaf ears. While some individuals voiced interest in having such a feature or submitting pull requests, no one was willing to pull out the proverbial checkbook. Or even escalate it to management.

To summarize: We want to do it but the core devs can't invest the time to make it happen without appropriate funding.

[glyph]

Is there an existing funding structure for cookiecutter where one might make a directed donation?

[pydanny]

Thank you @glyph, the current place is https://patreon.com/hackebrot.

[pawamoy]

Hi @glyph, following is how I am partially solving this issue. It's not perfect. I got inspiration from someone else but can't remember who or where (surely in another issue here).

Explained in a few words:

• when project is generated, user-input is saved in .cookiecutterrc yaml file
• also immediately checkout a new branch called cookiecutter
• this branch is used to update the project when the cookiecutter changes
• to update, checkout the cookiecutter branch, re-generate the project with config in .cookiecutterrc (overwrite), and merge into master

One limitation is that is doesn't work well when the project or the cookiecutter has already changed a lot: the re-generated branch will be more recent than a lot of master changes, so things might get lost when merging. And if the cookiecutter has new parameters, you have to manually add them into the .cookiecutterrc file before updating.

Here is the script I'm using: https://github.com/Pawamoy/cookiecutter-pydjama/blob/master/%7B%7Bcookiecutter.repo_name%7D%7D/scripts/update.sh

[glyph]

Hi @pawamoy - any chance you could rewrite this shell script in Python? :)

[pawamoy]

@glyph I can give it a try yes! Good idea!

[glyph]

Thanks!

[CurtLH]

I'm interested in this as well. I've followed a similar process as outlined by @pawamoy, but I had some trouble moving the output of the regenerated project into the correct location. The process I attempted to take is below:

When the initial cookcutter runs, store the .cookiecutterrc file as a hidden file in the project that is generated.

When time to update:

1. Checkout the cookiecutter branch of the project
2. Read in the .cookiecutterrc file for that project
3. Delete everything from the cookiecutter branch
4. Run the cookiecutter again, using the .cookiecutterrc file as the input parameters

The problem that I had with this approach is that the output of the cookiecutter was a directory, but I wanted the contents of the directory. I guess I just need to move from the output directory back one level, but I wasn't able to get this to work as smoothly as I would have liked with all projects.

[ppg]

Would it be possible to take small steps that moves towards this instead of trying to solve the hard problem of handling all peoples cases?

For example, I think one big feature that would go a long ways towards this would be if I could indicate files that should overwritten vs. files that should only be created if they don't exist. This would let me have files that cookiecutter 'controls' and they can call out to other files meant to be user-editable that I'd want to bootstrap with the right entry points and some notes for users. On first run it would create all those files; on subsequent runs I'd bring the 'control' files into conformance, but I'd leave the user-editable files alone. That's a form of partial overwrite the project, but seems like it wouldn't be monumental to support?

A similar but finer grained feature that might just go the rest of the way would be to expose either the existing file contents, or even better a part of the existing file contents between tags, to the jinja templating context; then I could (re)inject that into the file I write, thus preserving user-edits; i.e. I'd love to be able to do something liket this for my .gitignore template:

*.swp

### COOKIECUTTER START SAVE ###
{{ cookiecutter.saved_content }}
### COOKIECUTTER END SAVE ###

Even better you could use a custom jinja tag to hide the starting and ending tags. Again, while not trivial, doing something like reading the existing file and putting in the jinja context, or even reading it and parsing it and putting part of it in the jinja context, don't seem overly large to tackle.

Preemptively, if the answer is go make PRs I'm pretty sure I could figure out the second request, but for the first request is there somewhere to read/understand the config system better? I don't feel like I understand enough to know where that type of control/indication should go.

[merwok]

But I want to stop submitting individually-reviewed PRs to do stuff like dropping python 2.6 (or 3.3) in the travis.yml, the tox.ini (etc) of 17 different projects.

I have found https://github.com/asottile/pyupgrade/ recently (but am not using it at the moment), which upgrades Python code for you. I wonder what @asottile would think of extending it to also update build tools and service configs!

[asottile]

it's ~slightly out of scope for pyupgrade -- but, for what it's worth, I use all-repos to automate those sorts of tasks for myself -- you can see some of my sample invocations here

the workflow is roughly:

1. install all-repos
2. set up a github token in a configuration
3. all-repos-clone
4. (optional) write an autofixer (or reuse one of the existing ones, or use all-repos-sed for simple cases)
5. run the fixer (I usually start in dry-run mode with a few) -- it will create PRs for you
6. merge an you're on your way!

---

an aside: I'm working (a little privately -- unfortunately not much to show yet) on a "detect the convention of this project and normalize it" sort of tool (similar to the ideas of cookiecutter / yeoman / etc.) but more aimed at "making every project less special" (and then runnable with all-repos) -- for now it just wraps a bunch of tools (pre-commit autoupdate / pyupgrade / setup-cfg-fmt) -- but I eventually plan to have it manage all of these files in some regard: .gitignore, .coveragerc, .pre-commit-config.yaml, azure-pipelines.yml, LICENSE, README.md, requirements-dev.txt, setup.cfg, setup.py, tox.ini, +/- a few more depending on the effort

[KostyaEsmukov]

@asottile have you by chance heard about https://scaraplate.readthedocs.io/ ? It shares a somewhat similar goal (with regards to "making every project less special"). Perhaps it might be of use for you.

(disclaimer: I'm maintaining it).

[yajo]

FTR I've been recently contributing with @pykong to the upcoming 3.0.0 release of Copier, to support using it as a project maintenance tool instead of just a project bootstrapping tool, out of the box.

[sinoroc]

Maybe related: https://github.com/jaraco/skeleton

[asottile]

(I'm unsubscribing this thread, I don't really have interest in suggestions -- if you need my specific input please @ me directly -- thanks!)

[ionelmc]

FYI I did cookiepatcher some time ago deal with the problem. I run it and then pick/discard changes in git gui. But scaraplate does look interesting, hmmm. I wonder if it can be made to work with the already existing .cookiecutterrc

@pawamoy you "inherited" the .cookiecutterrc thing from cookiecutter-pylibrary :)

[pawamoy]

@ionelmc yes I know, thank you for that! I generated a lot of my projects from your cookiecutter, then slowly derived it to my own preferences 😊 I learned a lot of things from it!

[samj1912]

This seems related to #784. My suggestion there was to use https://github.com/cruft/cruft/

[glyph]

Thank you for this recommendation! This looks very encouraging…