Support for .copierignore file similar to .gitignore

[entelecheia]

Is your feature request related to a problem? Please describe.
I'm finding it cumbersome to handle the exclusion of certain files and directories when using Copier. Currently, there's no built-in support for an ignore file like .gitignore, which makes it difficult to manage a list of files or directories to be skipped during the copying process.

Describe the solution you'd like
I'd like to have a .copierignore file that works similarly to a .gitignore file. This file should allow users to list files and directories that should be excluded from the copying process. Copier should automatically recognize the .copierignore file and exclude the specified files and directories without the need for additional command line arguments.

Describe alternatives you've considered
As a workaround, I have been using a bash script to read a .copierignore file and pass the files to Copier as --skip arguments:

@bash -c 'args=(); while IFS= read -r file; do args+=("--skip" "$$file"); done < .copierignore; copier "$${args[@]}" --answers-file .copier-config.yaml gh:entelecheia/hyperfast-python-template .'

However, this solution is not ideal, as it requires extra manual effort and may not be as intuitive for users unfamiliar with bash scripting.

Additional context
Having a built-in solution for ignoring files and directories would streamline the copying process and improve the user experience. This feature would be particularly useful for projects with numerous files or directories that need to be excluded, allowing users to easily manage their exclusion lists.

[sisp]

Have you tried the _exclude setting in copier.yml? I believe it does what you're looking for.

[sisp]

Or _skip_if_exists depending on your needs

[entelecheia]

Thank you for pointing out the _exclude setting in copier.yml. While this option does provide a way to exclude files and directories during the copying process, it may not be the ideal solution for all use cases.

The main reason for suggesting a separate .copierignore file is to distinguish between the roles of template creators and template users. A template creator may have certain files or directories that they want to exclude by default, and they can do so using the _exclude setting in copier.yml.

However, template users might have their own unique requirements for excluding additional files or directories when using the template. These users may not want to modify the copier.yml file directly, as it is part of the template's source repository. By introducing a separate .copierignore file, we can offer template users a more convenient and flexible way to manage their own exclusion lists without affecting the template's original configuration.

This approach ensures that template creators can define default exclusions in copier.yml, while template users can easily customize their own exclusion preferences using a separate .copierignore file.

[yajo]

Makes total sense. However, it seems to me that just dealing with ignore is a bit small-scoped. The same need could happen with any other conf. We should be able to provide user conf. Maybe something like .config/copier.yaml.

[yajo]

Half-duplicate of #235.

[lhupfeldt]

I would like to have support for .gitignore files

_exclude_from:
  - file: .gitignore

I basically (or maybe exacltly) have the same rather long list of ignored files twice.
--exclude-from is used by tar and rsync.

[sisp]

@entelecheia I see how this would work for copier update when the userland exclude list is present in the project. But how would it work for copier copy? Would you expect a user to create a new folder and add a file that contains the exclude list before running copier copy $src .? IMO, this workflow would have a suboptimal UX. I imagine we could initialize the exclude list with the filenames provided via the --exclude flag; see below for more details.

@lhupfeldt I'm not sure I understand the use case behind using .gitignore. Could you clarify it? My understanding is that the requested feature is a userland version of exclude. This means, any file listed there would not be generated, and this means you don't need an entry in .gitignore because the file wouldn't exist anyway. Am I missing something?

@entelecheia @yajo @lhupfeldt We need an agreed design to get actionable, so let me offer a suggestion:

1. I like these filenames (imagining the adjacent .copier-answers.yml file):

   • .copier-config.yml (a bit similar to .pre-commit-config.yaml)
   • .copier.yml

2. I think we need to support a separate userland config file per template when using template composition, i.e. it should be possible to exclude different files for each template. This might be useful when, e.g., two templates would generate the same file and a user would like to use this file from template 1 and exclude it from template 2. To allow this, we could use an approach inspired by the answers file management:

   • Add an optional, templatable setting to copier.yml, e.g., _config_file: .copier-config.<name>.yml (defaults to .copier-config.yml; or whatever the decision on the filename is).

   • Provide a Jinja variable that contains the config filename which would be used like this: {{ _copier_conf.config_file }}.jinja

   • We could populate the config file with the exclude list passed to Copier via --exclude:

     {{ _copier_config|to_nice_yaml -}}

     When a user ran copier copy --exclude <file1> --exclude <file2> $src $dst, then the generated config file would be:

     exclude:
       - <file1>
       - <file2>

     And subsequent runs of copier update could use the exclude list from there.

3. When the exclude list gets edited manually, merge conflicts might occur upon copier update. So the recommended workflow would be to use copier copy --exclude ... for populating and copier update --exclude ... for updating the exclude list.

4. To allow extending the exclude list instead of overriding the one in the template's copier.yml file, we could add another switch, e.g., --extend-exclude VALUE:str (inspired by Ruff's extend-exclude setting).

And we might need to offer support for skip_if_exists in the userland config as well, which could follow the same approach as exclude.

WDYT?

[lhupfeldt]

To me the only time it makes sense to use exclude is when working on the template, and running directly from the checked out sources, in which case .gitignore should be exactly what is needed.
If template files are kept in a template subdir separate from other files, then I don't see any other need for using exclude.

[yajo]

I like these filenames (imagining the adjacent .copier-answers.yml file):

* `.copier-config.yml` (a bit similar to `.pre-commit-config.yaml`)

* `.copier.yml`

.copier.yml looks too similar to copier.yml, so we better use another thing...

I kinda like the initiative from https://dot-config.github.io/, so I feel more inclined to default to something like this, going from best to worst match: .config/copier-config.yaml:.copier-config.yaml:$XDG_CONFIG_HOME/copier/config.yaml. This way we fix #235 as a side effect. Gotta see how that behaves on windows (I think there'll be something like %AppData%\copier\config.yaml or similar).

2. I think we need to support a separate userland config file per template when using template composition, i.e. it should be possible to exclude different files for each template.

I also thought about that. However, we're talking about user configs. Thus, I'm not sure we really want to hardcode all this stuff. Different users might need different things. Maybe we can leave that for later if needed. If a user has different templates, it's reasonable to think that all of them (if not most) will require the same default user configs. It's also reasonable IMHO to ask the user to specify a different config if the default one doesn't apply, the same as happens with answer files.

So maybe this is something that we simply shouldn't need to care for now.

• When a user ran copier copy --exclude <file1> --exclude <file2> $src $dst, then the generated config file would be:

Config file support and config file management are 2 different things.

We should focus on using configs for now, so we narrow the scope and make it simpler.

Later, if needed, for config file management, I think it's better to have a specific subcommand copier config $whatever, just like git has thousands of flags, but only you write the config file with git config.

4. To allow extending the exclude list instead of overriding the one in the template's copier.yml file, we could add another switch, e.g., --extend-exclude VALUE:str

Isn't that the default behavior of copier when you use --extend already? 🤔

copier/tests/test_copy.py

Lines 198 to 211 in f3639ee

	def test_exclude_extends(tmp_path_factory: pytest.TempPathFactory) -> None:
	"""Exclude argument extends the original exclusions instead of replacing them."""
	src, dst = map(tmp_path_factory.mktemp, ("src", "dst"))
	build_file_tree({src / "test.txt": "Test text", src / "test.json": '"test json"'})
	# Convert to git repo
	with local.cwd(src):
	git("init")
	git("add", ".")
	git("commit", "-m", "hello world")
	copier.run_copy(str(src), dst, exclude=["*.txt"])
	assert (dst / "test.json").is_file()
	assert not (dst / "test.txt").exists()
	# .git exists in src, but not in dst because it is excluded by default
	assert not (dst / ".git").exists()

[lhupfeldt]

I just realized why this conversation seemed very confusing to me.
It seems you have been talking about support for a .copierignore (and extended to a general config) in standard user config directories.
I think it is a really bad idea to have a user wide ignore file (user wide config for some other setting might make sense). Which files to ignore should be determined only be the template creator. If the template expansion contains files which should not be there, then the template should be corrected.
To be clear, my request for .gitignore support was for a .gitignore contained in the repository.

[sisp]

.copier.yml looks too similar to copier.yml, so we better use another thing...

👍

I kinda like the initiative from https://dot-config.github.io/, so I feel more inclined to default to something like this, going from best to worst match: .config/copier-config.yaml:.copier-config.yaml:$XDG_CONFIG_HOME/copier/config.yaml.

I wasn't aware of this initiative, sounds reasonable. It tends to be better to stick to "standards" than invent something custom. 👍

Gotta see how that behaves on windows (I think there'll be something like %AppData%\copier\config.yaml or similar).

We can use platformdirs for cross-platform support. For instance, DVC uses platformdirs in a similar way as you suggest.

I also thought about that. However, we're talking about user configs. Thus, I'm not sure we really want to hardcode all this stuff. Different users might need different things. Maybe we can leave that for later if needed. If a user has different templates, it's reasonable to think that all of them (if not most) will require the same default user configs. It's also reasonable IMHO to ask the user to specify a different config if the default one doesn't apply, the same as happens with answer files.

So maybe this is something that we simply shouldn't need to care for now.

👍 It should be possible to extend to per-template user configs in a backwards compatible way if/when needed.

Config file support and config file management are 2 different things.

We should focus on using configs for now, so we narrow the scope and make it simpler.

Later, if needed, for config file management, I think it's better to have a specific subcommand copier config $whatever, just like git has thousands of flags, but only you write the config file with git config.

I think global configs with config management only make sense when those settings only affect the local environment. Having a global .gitignore wouldn't make much sense either. I'm still a bit unhappy with the UX when a user runs copier copy --exclude <file> $src $dst, which excludes <file>, and then copier update $dst, which adds <file> (I just tested this). Specifically the exclude setting doesn't seem particularly suitable as a global setting IMO because it would require a maintainer team to sync their global settings to have consistent copier update results. So coming back to the dot-config initiative, I'm not yet sure which settings would make sense in a global config, but exclude doesn't seem like one that does.

Isn't that the default behavior of copier when you use --extend already? thinking

Ah, you're right. I confused the --exclude switch with the _exclude setting in the template, which overrides the default excludes:

copier/copier/template.py

Lines 303 to 314 in f3639ee

	@cached_property
	def exclude(self) -> Tuple[str, ...]:
	"""Get exclusions specified in the template, or default ones.
	See [exclude][].
	"""
	return tuple(
	self.config_data.get(
	"exclude",
	DEFAULT_EXCLUDE if Path(self.subdirectory) == Path(".") else [],
	)
	)

[sisp]

@lhupfeldt I also realized that there was some confusion about "template-land" vs. userland config / ignore file. I also misunderstand the OT at first. So just to fully clarify your request: You'd like the _exclude setting in copier.yml to default to the content of the .gitignore file in the template repo in addition to the DEFAULT_EXCLUDE list. Correct? That would make a lot of sense to me.

[lhupfeldt]

To be honest I didn't remember that there was a DEFAULT_EXCLUDE list. I actually think that if the config explicitly specifies _exclude / _exclude_from, the DEFAULT_EXCLUDE should be ignore. Maybe with an option to append. But if the current default is append, then it may be better to keep that and add a no-append option.
I would prefer a new option _exclude_from taking a list of file names, instead of changing the current one. If you want to e.g. add only few patterns to those in the .gitignore, then it will be more convenient to list the patterns in the config instead of in an additional external file. Adding a new option would also make it completely backwards compatible.

[yajo]

Which files to ignore should be determined only be the template creator

Sorry, I don't understand this comment.
The template creator can do that already. See #1131 (comment) and #1131 (comment).

I actually think that if the config explicitly specifies _exclude / _exclude_from, the DEFAULT_EXCLUDE should be ignore.

Yes, that's how it works currently. See the docs.

You'd like the _exclude setting in copier.yml to default to the content of the .gitignore file in the template repo in addition to the DEFAULT_EXCLUDE list. Correct?

If you're talking about the template .gitignore file, it's already working like that.

If you talk about the .gitignore file in the subproject, that cannot be done. The only purpose of secret questions is to output files in git-ignored paths. We cannot drop support for secret questions.

It should be possible to extend to per-template user configs in a backwards compatible way if/when needed.

Ha! Contradicting myself completely, I just got an idea about how to do this 😁

We can assume that any answers file will contain the word "copier-answers" somewhere in its name. Thus, if a file with the same name, but replacing that with "copier-config' is next to it, use it by default. Easy!

Valid examples:

• .copier-answers.yml + .copier-config.yml
• .my-copier-template.copier-answers.yaml + .my-copier-template.copier-config.yaml
• .config/copier-answers.yaml + .config/copier-config.yaml

As a final note, we should be careful about what cnfigs are supported there. A malicious template could provide that file with unsafe: true. That should only be handled in per-user configs, not per-subproject. Also we should warn template creators that, in general terms, it makes no sense for a template to produce such file.

[entelecheia]

Ha! Contradicting myself completely, I just got an idea about how to do this 😁

We can assume that any answers file will contain the word "copier-answers" somewhere in its name. Thus, if a file with the same name, but replacing that with "copier-config' is next to it, use it by default. Easy!

Valid examples:

• .copier-answers.yml + .copier-config.yml
• .my-copier-template.copier-answers.yaml + .my-copier-template.copier-config.yaml
• .config/copier-answers.yaml + .config/copier-config.yaml

As a final note, we should be careful about what cnfigs are supported there. A malicious template could provide that file with unsafe: true. That should only be handled in per-user configs, not per-subproject. Also we should warn template creators that, in general terms, it makes no sense for a template to produce such file.

I'm glad to hear that you're considering this idea! Your approach of looking for a copier-config file in the same directory as the copier-answers file seems promising. It would allow users to manage their unique configurations easily and efficiently. For instance, users might want to provide the same answers (like personal information) across various templates, and a user-specific configuration file would streamline this process. In my experience, I've been using copier-config.yaml as my answers file, which I initially thought was a type of user configuration file.

Much like the .gitignore file, a user configuration file should be uncomplicated. A file that is easy to understand and modify would ensure a better user experience, especially for those who regularly use the templates in an unattended manner. I agree that the current format of the answers file is good enough for the proposed purpose. It's already familiar to users and can effectively hold the necessary configuration details.

[sisp]

You'd like the _exclude setting in copier.yml to default to the content of the .gitignore file in the template repo in addition to the DEFAULT_EXCLUDE list. Correct?

If you're talking about the template .gitignore file, it's already working like that.

I don't think the template .gitignore file content is automatically appended to the exclude list.

Otherwise, what you are suggesting sounds good to me, @yajo.

But what do you think about the problem that copier copy --exclude <file> $src $dst excludes <file> and copier update $dst adds it? I think the solution is to generate a user config file in the project populated with <file> in the exclude list. I wouldn't manage it through Copier from that point on, just initialize it like this.

[yajo]

Hold on a moment.🤔 Sorry for throwing out so many conflicting thoughts.😁

1. I copy a project.
2. I delete one file.
3. I commit.
4. I update.

This would be equivalent to excluding the file on copy, with no configs involved.

What should happen? 🧐 According to our update logic, the file deletion should be preserved in updates unless it changes too in the template, in which case you'd be getting a conflict (a perfectly valid one).

So, the use case you discovered @sisp could maybe be the only thing important here. That's a bug. If we fix it, we'd be providing an implicit fix for this issue: you don't want some files? Just delete them. You'll only get conflicts in next updates if they evolve too in the template, but that could happen not so often... so as to make this feature request unnecessary?

[lhupfeldt]

@sisp

My layout is

repo_root/
    .gitignore
    ...
    template/
        .gitignore
        ...

What I'm proposing is to allow _exclude_from to be any list of files. I would then personally reference top/.gitignore, in order to avoid including gitignored files when running copier from the checked out template which I'm working on.