CRLF Line Endings

[kazkansouh]

Hello.

I have noticed that the line endings in 212 of the 5144 .txt files are CRLF. I guess this is by accident and not intended? I find it can make scripting a little more inconvenient when reading these files as there is one more item to think about.

Usually, the GIT approach would be to rely on core.autocrlf or .gitattributes to manage the translation. That is, store with LF endings, and allow git to add CRLF during checkout when needed, e.g. on Windows.

If its ok, I can submit a simple PR that adds a .gitattributes file (to help standardise future commits) and standardise the existing line endings by running dos2unix on them?

[L1ghtn1ng]

Would be good to add a .gitattributes file

[g0tmi1k]

Love a PR! <3

[ItsIgnacioPortal]

I find it can make scripting a little more inconvenient when reading these files as there is one more item to think about.

How so? Any half decent library/tool that parses files should already have the capability of seamlessly parsing CRLF or LF endings. If you can point a specific example I think this could be fixed with a CI/CD pipeline using a Github Action.

[molangning]

@g0tmi1k I think this can be closed, also #977 renormalised the line endings.

[ItsIgnacioPortal]

Currently the only file remaining that still has line-endings problems is Discovery/Web-Content/trickest-robots-disallowed-wordlists/top-10000.txt. Possible solutions are being discussed in ItsIgnacioPortal#16