-
-
Notifications
You must be signed in to change notification settings - Fork 285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Supporting non-root VCS ignore #304
Comments
The current behavior also doesn't account for global .gitignore files, i.e. core.excludesFile and/or ~/.gitconfig. I haven't thought hard enough but perhaps a guess at a solution would be using |
Amongst other reasons, to also specifically avoid pypa/hatch#304.
Flit uses git ls-files and I can confirm that it more closely matches user expectations. :) |
I can add more ignore file support but will never use Please keep in mind that a VCS checkout is not guaranteed and is in fact the rarest of all situations. If a project is only published on PyPI in CI/CD then that assumption is met but then fails as soon as a user installs the sdist or attempts are made to build for other ecosystems like Conda or Fedora that use either the sdist on PyPI or the tarball release artifact on GitHub. |
Note the issue is not a feature request to strictly mimic the behavior of git. It describes a misbehavior in specific conditions, i.e. when ignore files are not at the root of the project. The problem mostly appears when using sdists so using Maybe Hatch has more context and is more likely to be used from a VCS checkout. So an option could be added there. Supporting per-directory ignore files would be an additional feature. But it could be added as part of the same solution. |
FWIW, I've almost shipped a package with a Would it be possible to maybe statically add certain dirs that never should be shipped like I know you're opposed to magic, but this would reduce the amount of package screwups significantly. |
|
Can I somehow convince you to add Because as I wrote: my Git is ignoring it by default and I suspect I'm not the only one so this can lead to a lot of |
Can you find out how Git is ignoring those? It's not for me |
Yeah I've found out. Both directories are created with an For mypy you may have to delete the directory and run it again because it doesn't seem like it's adding the files if the dir has been created by an older version. I guess the trick for both would be to evaluate |
Workaround for <pypa/hatch#304 (comment)>
Workaround for <pypa/hatch#304 (comment)>
This was the only Hatch-related pain point / surprise I encountered while migrating a large codebase (aesara-devs/aesara#1384). It would be great to support recursive |
Given that |
I was hit by this (unintended?) behaviour. By the way I noticed because the source dist was not reproducible, due to the cached files. For now I add [tool.hatch.build]
exclude = [
".*_cache",
] in every While the best option would be to honor |
More sensible default behavior would be desirable here. This bit me too, I don't keep a .gitignore in the sourcetree unless it's excluding stuff generated specifically by that project. Excludes for .venv, pyc etc all go in global gitignore. In pytest-dev/pytest-freezer#9 it was pointed out that the sdist on PyPI had a whole venv inside. Oops, embarrassing.. :) |
For everyone running into these problems I’d like to propose https://github.com/hynek/build-and-inspect-python-package for your consideration to build and check your packages in a clean CI with introspection – and then ideally upload from CI using the new PyPI trusted publisher support. This bug in Hatch has finally pushed me over the edge to take this on a few months ago and it’s basically just adding a workflow to your repo and a bit of configuration on PyPI’s side: https://github.com/python-attrs/attrs/blob/main/.github/workflows/pypi-package.yml It’s very liberating. |
@hynek Thank you for that, really nice. Just tried it out and found a couple of interesting things: it didn't work on a windows runner (meh?), and the PKG-INFO inside the sdist didn't get the source timestamp like all the other files (also meh?) As for hatch, after trying+failing multiple times in a row to configure contents in the .tar.gz and the .whl correctly, I think I'm giving up on it for the moment. I'm all for explicit configuration, but the default configuration should not do things that are obviously wrong. |
@wimglenn What is not working with your explicit configuration? The default file inclusion for each target is well documented:
Based on that your project seems to have perfectly fine defaults for wheels. Therefore, you should not be using the global |
I have to disagree here. It is not a perfectly fine default to ship files to PyPI that aren't part of the repo and aren't shown in |
I'm not sure where that response came from? I was talking specifically about that user's project. |
I'm sorry, maybe I've misunderstood you both. |
Just because I'm noticing this now,
I'm not sure where this comes from,from what I see the last @ofek opined on the matter was
so AFAICT, @ofek correct me if I'm wrong, but the status of this issue is still "PR welcome to add an option, at least for part of the problem"? |
FWIW I have a PoC patch against hatch-vcs that uses setuptools_scm to determine the file listing from git/hg: ofek/hatch-vcs#40 It seems to be held up by a setuptools_scm refactor, but it can be used now. |
that is correct |
@ofek There is nothing wrong with that explicit configuration: when using My complaint was that the default configuration is not clever enough to know that packing User error? Yes. But also perhaps a bit too easy to make mistakes without noticing. |
I've been scratching my head at something related to this... I think. I'm working in a monorepo where subpackages live under a
but when I run FTR, I've tried various combinations of exclude patterns and also tried Maybe that's intended behavior based on the docs note that |
Yes, it's the same issue. It's not literally that the file is copied (so it doesn't matter if it's excluded). Hatch reads the rules, and applies them to a newly-generated .gitignore file. The effect is the same. There are at least a couple issues with this, discussed above: the behavior is surprising; and rules meant for a different path are applied without correction, so it's also wrong. Although I originally reported this, I'm unlikely to work on this issue because I'm not using Hatch anymore. |
pypa/hatch#304 would make it unnecessary.
I've accidentally noticed that my .vscode/settings.json got included into source distribution, even though it's ignored by git as configured in my global config. This came as a surprise, but appears to be a design decision (see pypa/hatch#304). Switch to flit which by default only includes the package contents, readme and license.
I was exploring the content of my project's sdist, and discovered this issue as well. In my case my editor's config, which is excluded from git globally, got sucked in. It had been configured this way for so long I didn't even remember about it. While it's true that this behavior is documented under VCS section, I unconsciously made the assumption that only tracked files are selected. It seems it's not just me who made this assumption, so I'd appreciate if you could add a note there explaining this limitation. There's also a contradictory information given in sdist builder's documentation:
It links to the mentioned VCS section, but still, this statement introduces some confusion. It's not VCS who determines if the file is ignored, but Hatch itself. |
What's the status here? It seems to me that respecting .gitignore, but not embedded .gitignore is a major miss-feature. I agree that requiring git at run-time is not great so:
would a PR for (1) be considered? NOTE: I took a quick look at the source, and can't find where in the code the filelist for the sdist is created -- so I'll need a pointer ... |
.gitignore files don't have to live at the root of the working directory:
Hatchling assumes that there is at most one .gitignore file, and that it describes paths relative to the project root. This mismatches the behavior of git in a couple of ways:
Example
.gitignore might contain the pattern
foobar/ignore_me
, which would be effective enough for git. But Hatchling would not be able to apply the pattern to exclude ignore_me. If a file with the same name happened to be in the innerfoobar
directory, it would be accidentally excluded.I don't use Mercurial anymore, but I imagine that 1 would apply there as well. Mercurial restricts .hgignore files to live at the root of the repo, so there can't be multiple ones. But you could have a project in a sub-directory of a hg repo.
It would be nice if Hatch/Hatchling could do the same thing the VCS does.
I don't know if a solution for 1 would involve, at sdist building time, to rewrite .gitignore to make relative patterns (those with
/
not at the end) relative to the project directory (or write the modified patterns into the sdist pyproject.toml instead of including the .gitignore file).Per-directory .gitignores complicate things a bit. It implies that when traversing parents you cannot stop at the first .gitignore you find, because there can be a more general one up above. Traversal should stop when you reach the root of the working tree (i.e. when you find
.git
). But when you unpack an sdist you don't have a .git.And I don't even want to start thinking about submodules.
The text was updated successfully, but these errors were encountered: