Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Ensure files are committed with eol=lf via .gitattributes #3527

Conversation

polarathene
Copy link
Member

@polarathene polarathene commented Sep 5, 2023

Description

Slight project improvement, mainly beneficial if developing from a Windows machine.

Open to alternatives:

  • A simpler file containing only * text=auto.
  • Rejection due to existing EC linter (preventing CRLF check-in is probably a good thing though?).

Although our EditorConfig linter check can catch this mistake, it's probably useful to have a .gitattributes file to automatically convert to LF when necessary? (Default allows checking in CRLF into the repo, as this non-obvious CI test failure running on Linux demonstrates).

The current commit is a bit overkill / verbose:

  • We could trust git to make the decision with just * text=auto and rely on tests and linting to communicate failures? πŸ€·β€β™‚οΈ
  • Or drop most rules for niche files (notably those that are extensionless), I believe git will decide if a file is binary or not by it's content, and since we already git diffs generated for all relevant text files, those are not treated as binary.

The inline docs at the bottom, I'm not sure what your opinion is on keeping them. It's harmless and not really going to bother anyone, while providing a decent refresher/overview of the file format and rules (vs the upstream docs). They are revised from an earlier .gitattributes I wrote years ago.

Additional reference:

Do note that if you do add more rules regarding text files, it will only affect new commits, it won't apply to the prior history, for that you'd need to rewrite the history, so it's good to commit this with new repos at the start of a project.

Github has a good resource on the topic, which also details how to rewrite history of the repo. At the end of the page the "Further reading" section links to a quality article: "Mind the End of your Line" :)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Improvement (non-breaking change that does improve existing functionality)

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • New and existing unit tests pass locally with my changes

Both of these files support the `.yml` extension. Normalize on that.
Avoids accidentally committing files with `CRLF` when they're created on Windows. Or worse, if some editors don't detect `LF` and would introduce mixed line-endings with `CRLF`.

Shouldn't be a problem in practice as we already have a linting check to catch this via CI during PRs. This file is complimentary, in that it should automate that concern away.
@polarathene polarathene added area/ci kind/improvement Improve an existing feature, configuration file or the documentation kind/bug/fix A fix (PR) for a confirmed bug labels Sep 5, 2023
@polarathene polarathene added this to the v13.0.0 milestone Sep 5, 2023
@polarathene polarathene self-assigned this Sep 5, 2023
Copy link
Member

@georglauterbach georglauterbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually do quite like the changes! πŸ‘πŸΌ

@casperklein
Copy link
Member

casperklein commented Sep 8, 2023

Disclaimer: Not going to block this / discuss to death πŸ˜†

I read a bit about .gitattributes and have mixed feelings. Personally I don't like it, when files are automatically changed (LF/CRLF in this case) on commit/checkout. If there are erroneous line endings, I prefer to get to know this by a failed lint test. So that I am informed that something did not fit and I can make it better in the future - instead of getting it silently fixed.

Every modern editor should be able to handle LF/CRLF correctly. It feels weird, to convert/alter all text files on checkout to another line ending (CRLF, for example on windows). And later after the work is done, convert it back to LF when commiting. Instead, a proper editor should be used, so that converting beforehand is not necessary at all.

I believe git will decide if a file is binary or not by it's content

If the first 1K bytes contains a null byte, it's considered binary.

BTW: Any idea, why the mentioned test failure wasn't caught by eclint?

In .editorconfig we have

[*]
end_of_line = lf

I would have expected, that all text files are checked for the correct (LF) line ending.

@polarathene
Copy link
Member Author

TL;DR:

  • .gitattributes sanitizes for the repo, only eol=lf should change check-out conversion, but this is enforced when it should be LF.
  • On my current system (Windows), it seems to checkout without conversion, but it was also happy to commit without conversion, where VSCode was introducing the CRLF (apparently for edited files sometimes too).
  • Nobody will want mixed line-endings, and for this project there's really no reason to want to commit with CRLF. This change should really only affect you if you're on Windows, but in a positive way.
  • ECLint is doing it's job perfectly fine catching this. CI will skip it if ShellCheck fails first though.

I read a bit about .gitattributes and have mixed feelings. Personally I don't like it, when files are automatically changed (LF/CRLF in this case) on commit/checkout.

I had read a lot on the topic myself and have had a few experiences where it was problematic over the years.

To be fair, I've been very verbose/explicit in the config here so you aren't really getting any surprises. It's similar to the linting that would catch it, but if you commit the change then the CI won't pick up on it via PR as the issue was prevented.

This just ensures that line endings are converted to LF during commit, as you really don't want CRLF (especially mixed) in the repo. If the file does not explicitly have eol=lf, but only text this allows for the local environment to checkout with CRLF if it wants to.

Especially since this is a project focused on files being copied into a linux container image, you have very little reason to want CRLF there. It will cause weird bugs like .sh scripts unable to find the binary from shebang hint due to that invisible CRLF, or as I recently experienced a test failure that told me CONTENT is not CONTENT πŸ€”


If there are erroneous line endings, I prefer to get to know this by a failed lint test. So that I am informed that something did not fit and I can make it better in the future - instead of getting it silently fixed.

The lint should work that way if you run it before committing. I often forget to, but since we contribute via PR instead of direct push to master it can be prevented sneaking into master.

I had no idea when the CI tests failed initially as the lint test was bailing at ShellCheck before EditorConfig, plus some tests were intermittent/unrelated test failures, so line nedings didn't cross my mind until EC Lint pointed it out.

That said I'm working on rewriting a docs page. Normally these say LF, but I just noticed that it's reporting CRLF in the editor right now, possibly it's mixed endings and after rewriting more than half of the page it detects as CRLF now, or perhaps it changed when I saved it πŸ€·β€β™‚οΈ

If you're not working on Windows you're not going to have this surprise gotcha, it's better to prevent it for those that do, lints are great, but leveraging this git feature to fix all files when they're committed is even better.


Every modern editor should be able to handle LF/CRLF correctly. It feels weird, to convert/alter all text files on checkout to another line ending (CRLF, for example on windows). And later after the work is done, convert it back to LF when commiting. Instead, a proper editor should be used, so that converting beforehand is not necessary at all.

I'm using VSCode, while for git it's one of the few other dev tools I like to use a GUI for instead of the CLI, I use GitKraken there. I consider both of these modern and quite capable.

In both cases I think they're just respecting the Git defaults, which is to use CRLF unless told otherwise. .gitattributes here as mentioned will only change the normalize to LF on commit. I could probably install an extension in VSCode for EditorConfig that might help prevent files getting changed to CRLF (I thought it was only when I create new ones, but it seems to being doing that when editing docs too...).

Regardless, we get other contributors and can't expect it'd be any better on their end. The CI lint tests will still catch that, but I can't think of a reason where .gitattributes isn't good for this project. Nothing should really need CRLF when it's all being used during a docker image build (which I doubt converts back to LF?).


BTW: Any idea, why the mentioned test failure wasn't caught by eclint?
I would have expected, that all text files are checked for the correct (LF) line ending.

It did catch it, like mentioned above. Just requires the linter to reach that check before failing.

https://github.com/docker-mailserver/docker-mailserver/actions/runs/6070286374/job/16466057026

The test run before that one skipped ECLint due to ShellCheck failure:

https://github.com/docker-mailserver/docker-mailserver/actions/runs/6068011319/job/16460416288

While the test-case was failing it wasn't for that same reason as I hadn't yet adjusted for the white-space difference, so I wasn't confused at that point either.

@polarathene polarathene merged commit ad8b618 into docker-mailserver:master Sep 8, 2023
7 checks passed
@casperklein
Copy link
Member

Thanks for the detailed explanation πŸ‘

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ci kind/bug/fix A fix (PR) for a confirmed bug kind/improvement Improve an existing feature, configuration file or the documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants