Write a .gitignore file into htmlcov

[nedbat]

To keep people from accidentally committing the HTML report to git, or to keep them from having to edit their .gitignore to prevent it, the html command can write a .gitignore file into the directory.

It should only write the file if it doesn't already exist. This would let people edit the .gitignore if they did want to commit the report.

Some people suggested only writing the file if it's a git repo, but that's over-engineered. One more tiny dot file inside the report directory won't hurt.

[nedbat]

This is implemented in ea6906b

[nedbat]

This is now released as part of coverage 6.1.

[emcd]

It should only write the file if it doesn't already exist. This would let people edit the .gitignore if they did want to commit the report.

Hi @nedbat , the functionality described in the initial post to this issue does not match what was implemented in coverage/html.py as part of ea6906b . I.e., the implemented functionality unconditionally overwrites an existing .gitignore. Was this intentional?

(As someone who had a preexisting .gitignore in my HTML report directory, I would've preferred not to have mine overwritten.)

Thanks.

[nedbat]

@emcd Sorry about that, I see your point. I can make an adjustment. Can you tell me more about how you used it? What was in your .gitignore file originally?

BTW, very impressive workflow file, in a few ways: https://github.com/edaa-org/pyEDAA.ProjectModel/blob/master/.github/workflows/Pipeline.yml

[nedbat]

I've changed the code in b471e55 to not overwrite an existing file. I'd still like to understand more about your use case though.

[emcd]

Thanks for the quick response, @nedbat .

The use case is basically that I prepopulate an artifacts directory skeleton with .gitignore files so that its subdirectories are available on clone. Any contributor, who uses the included tooling for running unit tests / gathering coverage reports, generating Sphinx docs, building sdists and wheels, publishing to PyPI, etc... will not have to worry about the configured target directories not existing and the resultant errors that their non-existence might cause in the tooling. The .gitignore files serve as placeholders so that Git is "aware" of the directories and doesn't elide them, since it holds empty directories in low esteem.

Thanks for sharing the link to that pyEDAA pipeline (https://github.com/edaa-org/pyEDAA.ProjectModel/blob/master/.github/workflows/Pipeline.yml). I might borrow some of their gh-pages publishing logic. What specifically stood out to you in it, if I may ask?

[nedbat]

@emcd Oh, I thought that pyEDAA link was yours! :) It was @umarcor fixing a similar concern.

That workflow: 1) uses emojis on everything including the matrix, 2) does in one file what other people do with multiple workflows, 3) detects tags with an if-clause rather than an "on" clause, 4) probably some other things... :)

[umarcor]

@emcd, you might want to consider using a different file name as a placeholder. You are effectively overloading the functionality of the .gitignore filename, so you are likely to hit similar issues in the future. In this case, I do agree with you that coverage should not ovewrite an existing file. I think your demand and @nedbat's adjustement to the feature are correct. However, as a rule of thumb, it is better to use files names .gitkeep or .gitempty or .empty for the sole purpose to having placeholders for git to preserve subdirs.

I might borrow some of their gh-pages publishing logic

See similar but distinct solutions for using GitHub Actions and pushing to GitHub Pages:

• https://github.com/ghdl/ghdl-cosim/blob/5d4f6be22a3cc41a9c0ebeec060b7e8bb583852b/.github/workflows/doc.yml#L19-L28 (just use Action buildthedocs)
• https://github.com/edaa-org/pyEDAA.ProjectModel/blob/master/.github/workflows/Pipeline.yml#L360-L377 (use Action buildthedocs with a local container image)
• https://github.com/ghdl/ghdl/blob/56735b0d1e92b4246ef53442236e89afe07aacdc/.github/workflows/Test.yml#L27-L63 (use Action buildthedocs with a local container image, and with "public" content copied raw to the site)
• https://github.com/hdl/containers/blob/93776734a6535deaba1039fa58e726195a244e3a/.github/workflows/doc.yml#L49-L68 (use asciidoctor instead of sphinx, and push to gh-pages manually using the default credentials)
• https://github.com/VUnit/vunit/blob/18e9c58654cad7a66dc38f002f9c66e0eadce65f/.github/workflows/docs.yml#L29-L58 (run sphinx without using buildthedocs, and push to GitHub Pages in a different repo manually using an SSH key)

---

@nedbat, that workflow is actually joint work with @Paebbels. He was the one who wrote most of that Pipeline.yml, while I helped him with syntax details, since I'm more familiar with GitHub Actions. You will find almost the same workflow used in several Python projects we co-maintain:

• https://github.com/edaa-org/pyEDAA.ProjectModel/blob/master/.github/workflows/Pipeline.yml
• https://github.com/edaa-org/pySVModel/blob/main/.github/workflows/Pipeline.yml
• https://github.com/VHDL/pyVHDLModel/blob/master/.github/workflows/Pipeline.yml
• https://github.com/pyTooling/pyAttributes/blob/master/.github/workflows/Pipeline.yml
• https://github.com/pyTooling/pyTooling/blob/main/.github/workflows/Pipeline.yml
  • https://github.com/pyTooling/pyTooling.CallBy/blob/master/.github/workflows/Pipeline.yml
  • https://github.com/pyTooling/pyTooling.Exceptions/blob/master/.github/workflows/Pipeline.yml
  • https://github.com/pyTooling/pyTooling.MetaClasses/blob/master/.github/workflows/Pipeline.yml
  • https://github.com/pyTooling/pyTooling.CommonClasses/blob/master/.github/workflows/Pipeline.yml
  • https://github.com/pyTooling/pyTooling.GenericPath/blob/master/.github/workflows/Pipeline.yml

In fact, we discussed about creating a reusable workflow (see https://docs.github.com/en/actions/learn-github-actions/reusing-workflows) or a "Composite Action" (see https://docs.github.com/en/actions/creating-actions/creating-a-composite-action). Not something we can tackle in the shortest term; nonetheless, please let me know if you want us to keep you updated.

You can find other "fancy" Actions workflows in the following repos:

• https://github.com/ghdl/ghdl/blob/master/.github/workflows/Test.yml (icons/symbols and job dependencies everywhere). E.g.: https://github.com/ghdl/ghdl/actions/runs/1454956157
• https://github.com/stnolting/neorv32/blob/master/.github/workflows/Implementation.yml and https://github.com/stnolting/neorv32/blob/master/.github/workflows/Windows.yml (dynamically generated matrices)
• https://github.com/im-tomu/fomu-workshop/blob/master/.github/workflows/test.yml (ubuntu, macos, windows, msys2, multiple containers...)

1. uses emojis on everything including the matrix,

Since those are unicode symbols, some maintainers don't like them. Moreover, each symbol might have a different width, depending on the editor/font. This is to say that there is some slightly increased complexity to deal with, apart from the obviousness of having a more verbose workflow file.

Anyway, I believe it is worth. As people with a hardware design background, the symbols along with the "all in one" pipeline approach allow us to more easily spot the problems and understand the cross-talking.

2. does in one file what other people do with multiple workflows,

As said, this is basically so that we can have the big picture drawn for us, instead of requiring maintainers/contributors to mentally construct that abstraction. Overall, we slightly increase the cognitive complexity of the workflow files, in order to reduce the everyday maintenance effort.

3. detects tags with an if-clause rather than an "on" clause

IMHO, having multiple workflows is abused in GitHub Actions, without a clear motivation. Most other CI services cannot handle multiple workflow files. There is typically a single one. Therefore, having a single Pipeline and handling conditions per-job or per-step feel more natural to us, coming from GitLab or Travis or others. Don't take me wrong: there are use cases for having multiple workflows, and it's a desirable capability in GitHub Actions; but most of the projects might get along with a single one, as proven by the examples above.

[nedbat]

Wow, @umarcor, thanks for the write-up. It will take some time to absorb the detail in those links.

However, as a rule of thumb, it is better to use files names .gitkeep or .gitempty or .empty for the sole purpose to having placeholders for git to preserve subdirs.

I like putting a README.txt into an intentionally empty directory, to explain why it's there.

[emcd]

@emcd Oh, I thought that pyEDAA link was yours! :) It was @umarcor fixing a similar concern.

No worries. I thought it might have been a case of mistaken identity. Sadly, I've been mostly out of open source development for the past 8 years, so I don't have any active projects at the moment. (That is about to change though.)
And, I definitely need to catch up on the state-of-the-art for open source CI pipelines. :)

That workflow: 1) uses emojis on everything including the matrix,

@nedbat: it has been interesting to watch how tools, such as Slack, (with some help from the Unicode standard) have started to evolve written English from an almost purely alphabetic language towards a mixed ideographic one. Somewhat of a reversal from the historical evolution of ideographic writing (a CISC architecture?) to alphabetic writing (a RISC architecture?).

As output markers, I definitely prefer emoji over ANSI SGR colorization, because emoji are agnostic of color schemes and also render less painfully in build pipeline logs, when tools don't perform TTY detection. (Probably also a bonus for people who have various forms of color blindness.)

I like putting a README.txt into an intentionally empty directory, to explain why it's there.

Fair point, though one could argue that, in some cases, if it needs to be explained, then maybe it shouldn't be there.

[emcd]

@emcd, you might want to consider using a different file name as a placeholder. You are effectively overloading the functionality of the .gitignore filename, so you are likely to hit similar issues in the future.

Thanks for the suggestion, @umarcor. However, I am actually using the functionality of .gitignore and not overriding it, since when artifacts are deposited into the skeleton directory, I don't want them to show up as untracked files to commands like git status.

In this case, I do agree with you that coverage should not ovewrite an existing file. I think your demand and @nedbat's

To be clear, I was not demanding anything. :) I simply asked whether the implementation was intentional and then stated a personal preference. I don't think it is good idea to demand things of people who voluntarily share good software with the rest of humanity.

See similar but distinct solutions for using GitHub Actions and pushing to GitHub Pages:

[snipped]

Thanks for the links! I will give these a good study.

[umarcor]

Thanks for the suggestion, @umarcor. However, I am actually using the functionality of .gitignore and not overriding it, since when artifacts are deposited into the skeleton directory, I don't want them to show up as untracked files to commands like git status.

My bad. I misunderstood.

To be clear, I was not demanding anything. :) I simply asked whether the implementation was intentional and then stated a personal preference. I don't think it is good idea to demand things of people who voluntarily share good software with the rest of humanity.

Agree. My bad again. I'm not a native speaker, so maybe "demand" was not the best verb. I meant your "feature request", or "request for clarification".

Thanks for the links! I will give these a good study.

Feel free to ping if you have any question!

[aalok-sathe]

This gitignore file caused hours of wondering why my CI workflow was completely disregarding coverage reports deployment to github pages. my usage of coverage.py is

coverage html -d codecov MY_MODULE/**/*.py 

and so in the process the folder is created empty first, with a .gitignore.

I think rather than coverage.py imposing the ignore rule, users should themselves ignore it if needed, and as always, it is not recommended to add all files without actually checking to your git history, so the onus is not on coverage.py to prevent this behavior. If a user "accidentally" adds the html directory due to git add ., that's on them.

I request that we disable this behavior unless explicitly requested using a flag, or just not at all.

[nedbat]

I'm sorry for the confusion. I still think the right default is to create the .gitignore file, since most people will not want to commit their HTML reports.

[zhao-li]

I too just got snagged by this unexpected behavior. Luckily, everything was local so I did not need to spend an entire day figuring things out.

@nedbat would you be open to providing a flag or configuration to disable auto-creating a .gitignore file within the htmlcov/ folder? It would afford some flexibility for those edge case situations where a .gitignore file is undesirable.

Thank you for making such a wonderful product 🙏

[nedbat]

@zhao-li You can already prevent the creation of .gitignore: the file will only be created if coverage.py created the directory. If you mkdir htmlcov before creating the html report, you won't get the file. Does that work for you?

[zhao-li]

@nedbat thanks for that tip. I just gave it a try by leaving htmlcov/.keep in the folder. But coverage html still generated the .gitignore file.

In case it helps, this is my .coveragerc:

[report]
show_missing = true

[html]
directory = artifacts/htmlcov/

Having this behavior would be nice, however, having a flag/setting would be slightly more explicit.

I'm okay leaving the .gitignore file behavior since it's not blocking me, but if you want me to help troubleshoot why .gitignore is still being generated even with the existing folder, I'm happy to help.

Have a fantastic day 🌈

[nedbat]

Hmm, I don't understand why you still got a .gitignore file. Coverage.py checks if the directory is empty, and only if it is does it write a .gitignore file. (I misspoke in the previous comment: it's not based on whether coverage.py created the directory, but about whether the directory already had files in it.)

Your .keep file should have been enough to stop coverage.py from making the file. If you really want to dig in, you could debug starting here: https://github.com/nedbat/coveragepy/blob/6.3.3/coverage/html.py#L231

[zhao-li]

Thanks for the link. I gave it a little bit more effort by adding a file that's not considered hidden, like fake.txt instead of .keep, but same behavior.

I'm gonna stop here, but leaving the details here in case anyone else cares.

Thanks again @nedbat for your time 🙏

[nedbat]

@zhao-li Is your project public? I would like to understand why you are seeing different behavior than I am.

[zhao-li]

hmmm...the one I'm currently working on is not public. When I get some time, I'll try and see if I can replicate the same issue on another project that is public. I'll post back when I get a chance to do that. Thanks for continuing to share your time 🙏