gradlew check fails checksum validation in a fresh clone on Windows

[sabi0]

Description

By default Git uses CR LF line breaks in checked out text files on Windows. And this causes mismatch of the checksums that were calculated for files with LF line breaks.

Either checksum calculation should be made line breaks agnostic. Or "proper" LF line breaks should be pre-enforced on Windows too, e.g. via .gitattributes:

*.java text eol=lf
*.py text eol=lf
*.g4 text eol=lf

* Where:
Script 'C:\Development\Projects\lucene\gradle\generation\regenerate.gradle' line: 187

* What went wrong:
Execution failed for task ':lucene:backward-codecs:generateForUtil84ChecksumCheck'.
> Checksums mismatch for derived resources; you might have modified a generated resource (regenerate task: generateForUtil84):
  Current:
    lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene84/gen_ForUtil.py=6e47288c8d05515d3b99764c976b72a7984cc643

  Expected:
    lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene84/gen_ForUtil.py=e3c0b1d1d9fcc9f04ae8870e40aab44133321272

  Input files for this task are:
    C:\Development\Projects\lucene\lucene\backward-codecs\src\java\org\apache\lucene\backward_codecs\lucene84\gen_ForUtil.py

  Files generated by this task are:
    C:\Development\Projects\lucene\lucene\backward-codecs\src\java\org\apache\lucene\backward_codecs\lucene84\ForUtil.java

Gradle command to reproduce

gradlew check

[sabi0]

In addition to the checksum problem gradlew tidy in a fresh clone on Windows results in 5147 modified files!
I guess forcing the LF line breaks is the best fix for both these issues.

[dweiss]

I'm sorry, but I disagree with this. I'm using Windows myself all the time but the automatic conversion of line ends is the most weird and annoying feature ever conceived. It can be turned off, please do turn it off, then clone the repository again.

[sabi0]

To work with this repository the files must have LF line breaks. Why not make that part of the repository itself?
And forget about describing this quirk in the contributing guide, answering issues like this, etc.

There is a .dir-locals.el for Emacs users. Why not .gitattributes for Windows users?
What harm do you see in adding those .gitattributes rules?

[sabi0]

With core.autocrlf set to false line breaks normalization when committing new files is disabled too.
Then contributors on Windows will be pushing files with CR LF line breaks to the server?

[sabi0]

At the very least the contributing guide needs to mention that one should set the config while cloning the repo: git clone --config core.autocrlf=false https://github.com/apache/lucene

[sabi0]

Apparently, gradlew check also requires Perl, which contributing guide fails to mention.

* What went wrong:
Execution failed for task ':lucene:documentation:changesToHtml'.
> A problem occurred starting process 'command 'perl''

[dweiss]

I won't stand in the way if others want those .gitattributes - I just expressed my opinion. Do yourself a favor and switch it off globally, really. This should be the default on Windows. I've spent countless hours trying to figure out why git checked out something that really wasn't there (at least not for anybody else on the planet) - this happens over and over again if you have automatic conversions in place.

Then contributors on Windows will be pushing files with CR LF line breaks to the server?

At least they know that what they've checked in is the same as what they have locally, byte-to-byte. As a Windows user myself, I don't see this as a bigger harm than git replacing stuff for me automatically. And nobody can convince me otherwise, sorry.

[dweiss]

Apparently, gradlew check also requires Perl, which contributing guide fails to mention.

Perl and python3 (in many cases, not sure whether it's required for tasks check depends on).

[dweiss]

What harm do you see in adding those .gitattributes rules?

I don't like these conversions, that's about it. I like to get out what I put in. Consider this example if you need one (hexdump it) -
https://github.com/dweiss/tmp-crlf/blob/master/Foo.java

[sabi0]

I'm with you on not liking this behavior, actually.
Having a bunch of repositories converted from Subversion with a mix of line breaks from different contributors it is always a hassle from humongous diffs to what not.

Shall I submit a PR for contributing guide with a "remark for Windows users" and a "Pre-requisites" section?

[dweiss]

I'm with you on not liking this behavior, actually.

I think any programmer on Windows has been through this. I don't know why git defaults are such on Windows - maybe git folks want Windows users to be miserable (as if being forced to use ntfs wasn't enough)... Again - I can be sour about it because I'm actually a Windows user myself (or rather - multi-system user).

Shall I submit a PR for contributing guide with a "remark for Windows users" and a "Pre-requisites" section?

Please do and thank you!