Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Normalize line endings. #78

Closed
wants to merge 62 commits into
from

Conversation

Projects
None yet

After setting text=auto in #75, it would make sense to once normalize all files that are not normalized yet. Thereafter all new files will always have the same line endings which will help the diff and is also good for consistency between platforms.

funnelweb and others added some commits Oct 30, 2012

@funnelweb funnelweb Deliberate build break to test continuous build process 92fcb78
@funnelweb funnelweb Revert deliberate build break b9a7269
@funnelweb funnelweb Merge branch 'master' of https://github.com/fsharp/fsharp 67a9e4e
@funnelweb funnelweb Fix bootstrap FSharp.SRGen.Build.Tasks so it is next to te FSHarp.Cor…
…e.dll it likes
5ee4b44
@funnelweb funnelweb Use strong names for bootstrap compiler to help with compile on windows
switch to strong names for compiler binaries and update the bootstrap
compiler with those names. The strong names are based on
src\fsharp\test.snk

this means the strong name for FSharp.Compiler.dll changes, but no one
should be depending on that and the new names are useful as the won't
collide with the visual studio versions of these DLLs on windows. The
monodevelop binding is neutral to the strong names used.

we still compile FSharp.Core as
delay-signed-with-the-microsoft-key-and-then-mono-key-signed which Mono
recognizes as strong named but i think windows does not. We have to use
this identity for FSHarp.Core to keep binary compatibility with
Windows-compiled thing like type provider DLLs.

We may have to use the Microsoft FSHarp.Core.dll on Windows, i think it
gets picked up automatically from the GAC but need to check tomorrow.

also fixes bugs in ilsupp.fs in strong nae signing.

tested by bootstrapping, and manually examining the names on the dlls.
5f82267
@funnelweb funnelweb More work aiming to get build working on windows bc4b488
@funnelweb funnelweb make "xbuild fsharp-build.proj /t:Clean work 1cecc53
@funnelweb funnelweb typo 200d1f4
@funnelweb funnelweb Add strong-name signed bootstrap FSharp.Core 2524743
@funnelweb funnelweb update to previous checkin 826b899
@funnelweb funnelweb add FSharp.Core project refrence to FShap.Compiler.fsproj c917759
@funnelweb funnelweb update bootstrap to use signed FSharp.Core
The FSharp.Core in lib/bootstrap/4.0 needs to be properly strong name
signed if we'r going to run on Windows without VS installed and without
FSharp.Core in the GAC
7f6e2e4
@funnelweb funnelweb typo in FSharp.Compiler.fsroj 6332c76
@funnelweb funnelweb Look for fsc.exe alongside FSharp.Build.dll 0175d4f
@funnelweb funnelweb Update FSharp.Build.dll to probe for fsc.exe alongside it bb17ae2
@funnelweb funnelweb fix banner string 5484c08
@funnelweb funnelweb Update README.md e344f05
@funnelweb funnelweb Make mono/xbuild F# resource naming a bit more consistent with Visual F#
See #50

The comments in FSharpBuild/Fsc.fs explain a bit more. It is realy hard
to work out how to make this perfectly consistent. Essentially, be
careful using resources in folders if you want your projects to be
portable across VisualStudio/MSBuild and mono/MonoDevelop/XBuild

We hit this bug in fsharp/fsharpbinding, but we've moved that to no
longer put resources in folders for now.
1a62236
@funnelweb funnelweb Remove print statements from last checkin 9b24dd1
@knocte knocte Reenable pkgconfig check for mono, but reducing bloat
In case you need a clarification wrt bloat:
http://tirania.org/blog/archive/2012/Oct-20.html
5f46a4f
@knocte knocte configure.ac: Mono 2.11.5 == 3.0
Mono version 2.11.5 was never released and was actually renamed to 3.0
a307d66
@funnelweb funnelweb protect against a null condition if no DefineConstants is present 9a25e57
@dsyme dsyme merge update to F# power pack original sources 6d3d2e4
@funnelweb funnelweb reduce noise during build 7af51f8
@funnelweb funnelweb fix build break a347714
@funnelweb funnelweb copy and install xml files
This gives documentation for FSharp.Core.dll in MonoDevelop
68523b1
@vishnumenon vishnumenon Fixed a typo c3b4704
@knocte knocte Merge pull request #58 from vishnumenon/master
Fixed a typo in the README
92aaa16
@funnelweb funnelweb Install docs for FSharp.Core 2.0 44874e5
@funnelweb funnelweb Build and install FSharp.Core 2.1 profile by default
This includes adding dependencies for the Mono profile 2.1 binaries we
need to reference
1d95bb4
@funnelweb funnelweb Merge branch 'master' of https://github.com/fsharp/fsharp c17d7c2
@funnelweb funnelweb Add notes to README.md about what's produced by build 03dac4e
@funnelweb funnelweb Fix token used for XML documentation GAC entries
FSharp.Core is delay-signed and uses token b03f5f7f11d50a3a

All other DLLs use token f536804aa0eb945b
d89a3ef
@funnelweb funnelweb Fix resource file processing in bootstrap when using MSBuild
The FSStrings.resources was being dropped. This is because of
differences in resource file processing for MSBuild and XBuild. This
detects which is being used and adjusts resource file processing
accordingly.
ac56e9e
@funnelweb funnelweb Adjust fix to targets to treat resources correctly for both msbuild a…
…nd xbuild
310402e
@funnelweb funnelweb Adjust configuration message to give better guidance
On mac this now reports the following more helpful message:

configure: WARNING: /usr/local/bin/mono not found. Prefix should
normally be set to the mono installation path. Please re-run with
   ./autogen.sh
--prefix=/Library/Frameworks/Mono.framework/Versions/3.0.1
ed38ca3
@funnelweb funnelweb Allow build using xbuild and Mono
xbuild requires absolute paths for tools
79b1809
@funnelweb funnelweb Merge branch 'master' of https://github.com/fsharp/fsharp f914396
@funnelweb funnelweb Suppress generation of DebuggerBrowsable if not in mscorlib
Mono 2.1 profile does not have DebuggerBrowsable attribute. This means
wwe currently give an invalid FSHarp.Core.dll for Mono 2.1 profile.

Suppress the generation of DebuggerBrowsablee if the attribute does not
occur in mscorlib.dll
610b7e3
@funnelweb funnelweb Fix mistake in suppressing generation of DebuggerBrowsable f2e558f
@funnelweb funnelweb Fix unnecessary rebuild of DLLs when using msbuild ade5737
@funnelweb funnelweb Update Microsoft.FSharp.targets in bootstrap to handle resources corr…
…ectly
a0ea33c
@funnelweb funnelweb Fix READMEs to document build on windows bf2979e
@funnelweb funnelweb Fix build on windows to put binaries in same places as build on unix
Binaries built by msbuildd and xbuild now go in
lib/release/4.0/...

etc. as described in the README
50313a0
@migueldeicaza migueldeicaza [fsharpi]: add kill-to-end-of-line (control-k), fix control-p and con…
…trol-n (F# uses decimals, not octals), and make control-d delete char when not at the end
049fe24
@funnelweb funnelweb Merge branch 'master' of https://github.com/fsharp/fsharp a0c4eb8
@deneuxj deneuxj Disabled LF normalization 92f4a16
@funnelweb funnelweb Merge pull request #67 from deneuxj/e33e4fc6af3772a044737d9ee1b882b30…
…a25e739

Disabling LF normalization
2e26ddd
@funnelweb funnelweb Undo line ending changes in FSharpSource.targets a7bb788
@funnelweb funnelweb Add fsiAnyCpu.exe to the xbuild/msbuild
This is in Visual F# 3.0 and is for use on 64-bit systems.

We still need to add Makefile, install etc. scripts for this
8fed39b
@funnelweb funnelweb Make .NET 4.0 compiler align with Silverlight capabilitiies
The Microsoft.FSharp.Compiler.Silverlight DLL has a class that lets you
create a hosted F# Interactive session. This adds that capability to the
.NET 4.0 build.
f077153
@funnelweb funnelweb Add build of fsiAnyCpu.exe to Makefile a6694ff
@funnelweb funnelweb fix build break for sl5-compiler 7ca459c
@funnelweb funnelweb Fix build break on windows dddede2
@funnelweb funnelweb Fix build break on windows (part 2)
build break detected by teamcity continuous build
62e5f9a
linux use name Microsoft.FSharp.Targets instead of Microsoft.FSharp.target 227c81b
@funnelweb funnelweb Drop fsi.exe.config etc.
Fix #73
bc6308a
@ony ony Respect --with-gacdir=/path/to/gac
Use parameter of --with-gacdir and MONODIR as a source for ROOTDIR in
gacutil.
01de35e
@ony ony use only relative symlinks
Many linux distributives uses DESTDIR for sandbox and then relocate all
files under root causing absolute (to DESTDIR) symlinks to be broken.
977fab7
@funnelweb funnelweb Add sample project created using MonoDevelop FSharpBinding 3.2.8 cons…
…ole app template
3980bf3
@funnelweb funnelweb Merge pull request #74 from ony/master
fix symlinks for linux distributives and respect --with-gacdir
7e03a7f
@forki forki Merge pull request #76 from funnelweb/tb2
use Microsoft.FSharp.Targets to fix #75
8168462
Contributor

knocte commented Dec 2, 2012

Good intentions. The problem is that this effectively kills all the history (wrt git-blame).

I see, didn't think of that. It might be possible to rewrite the initial commit in such a way, that those line endings would be fixed there and then rebase master on that. Want me to try it out?

Contributor

knocte commented Dec 2, 2012

That would be the ideal solution but it also has a big drawback: if you do that, it means rewriting history, and I think that the consequence of that is that people will not be able to pull later, they will need to do a clean checkout on their trees. But maybe I'm mistaken.

Contributor

knocte commented Dec 2, 2012

s/tree/systems/

There is also the option of using git blame -w to not lose that history, but it's kinda a weak solution, too.

To the best of my knowledge, btw, it's only a problem if someone made a branch because the SHA-1 of HEAD or master would change, so their base would not be valid anymore. If, however you then rebase your branch, too, that should not be a big deal.

It basically looks like someone deleted the master and made a new one. Just a pull, though should work, I think.

Contributor

knocte commented Dec 2, 2012

Yeah, if you're right, go ahead, test it, and propose it. Only thing is I guess you cannot propose rebasing-changes in a pull-request, hence you would need merge rights in the repo (I just mention it in case you don't have them already, I don't know).

I'll look into playing around with git then. Maybe I can put together a git script that would do such an operation and someone with rights can try it. Going on vacation soon, so not quite sure when it'll be. Please don't just close the request if I can't connect.

Contributor

cdrnet commented Dec 2, 2012

Starting Point:
git filter-branch -f --prune-empty --tree-filter 'git ls-files -z | xargs -0 dos2unix --skipbin' -- --all

Contributor

knocte commented Dec 2, 2012

So, one second: is the purpose of this to convert ALL files in the repo to Unix line endings?

While that would benefit some contributors (like me, as am in Linux), that would be a nightmare for people who want to contribute from a Windows box. (For example, MSBuild files are more correct if they have Windows EOLs, and even MonoDevelop respects this when running in Linux/Mac)

What I think should happen is: normalize to Windows line endings, and adopt a git commit hook that barfs when someone tries to commit introducing unix line endings.

Contributor

knocte commented Dec 2, 2012

(Another disadvantage of converting everything to Unix line endings which I forgot to mention is: not being able to generate a clear diff to compare this repo with the upstream's CodePlex one.)

text=auto does set everything to Unix line endings in the repo and checks them out as your line endings in your working copy. That way your local text editor works with Unix or Windows line endings, respectively, whilst in the repo it is Unix line endings. Also, on every commit, all the files committed get changed upon commit to be Unix, therefore Codeplex changes will be very clear, too.

Contributor

knocte commented Dec 2, 2012

I don't understand why Codeplex changes would be very clear if this repo is converted to Unix line endings: is CodePlex repo using Unix line endings? I don't think so.

What am I missing?

That to see the changes, you still commit them to some git repo and
therefore the line endings get normalized prior to a git diff

On 2 December 2012 16:24, Andres G. Aragoneses notifications@github.comwrote:

I don't understand why Codeplex changes would be very clear if this repo
is converted to Unix line endings: is CodePlex repo using Unix line
endings? I don't think so.

What am I missing?


Reply to this email directly or view it on GitHubhttps://github.com/fsharp/fsharp/pull/78#issuecomment-10930814.

the trouble with not normalizing we see occasionally on files just randomly differ on every single line. It's not a huge problem, more like a caveat.

I did the
git filter-branch -f --prune-empty --tree-filter 'git ls-files -z | xargs -0 dos2unix' -- --all

and it made the history really pretty, check it out in my repo

Contributor

knocte commented Dec 2, 2012

@DanielFabian : agreed, but what I have a problem is what to normalize to? IMHO it should be Windows line-endings, otherwise you would translate the same problem to the other way around: when you change something in an msbuild file (.sln, .csproj, .fsproj) via MonoDevelop, you will see the whole file changing.

Cannot you use text=auto but using Windows line endings?

well, no. What it auto does locally what's for you and in the repo it does LF. You do can set CRLF explicitly, but then it just is CRLF on all systems, including Unix. Or you can set it to LF for all systems. But you can't set CRLF in the repo and auto elsewhere. (It would have no effect, though, because you'd still get your line endings locally)

Do play a bit with my branch, maybe https://github.com/DanielFabian/fsharp/commits/master it really did clean up the history. Maybe play a bit with git blame and a bit with git diff etc...

@ghost

ghost commented Dec 6, 2012

What's the status here? Do people think this should be merged?

I think its good to normalize line the endings in GitHub - is this commit the best way to do it?

Member

rneatherway commented Dec 6, 2012

I agree that the line endings should be normalized. I think the only concern is what they will be when checked out. This will default, as @DanielFabian says, to the line ending of your platform, but can be changed if desired in your .gitconfig. The only reason I can think of for doing this would be to do a diff with Codeplex for example, and this isn't too bad for a one-off event. Rebasing cleans up the history, so this really solves the problem for good.

@knocte: Monodevelop could only be a problem if it reads a .fsproj with unix line endings and then saves it with windows line endings. Surely this isn't the behaviour?

Contributor

knocte commented Dec 6, 2012

@rneatherway : that is the behaviour, it's what I've tried to explain

Contributor

cdrnet commented Dec 6, 2012

*.fsproj eol=crlf in .gitattributes will make sure it is still normalized in the repo but always in CRLF in the working directory/checkout, even on Linux. Hence MonoDevelop should work fine, unless I'm missing something? Or does it save all files in CRLF, not just the project file?

Contributor

knocte commented Dec 6, 2012

You would need that for SLN files as well.

Member

rneatherway commented Dec 6, 2012

Right, that wasn't clear before. So Monodevelop does this for all files including *.cs, *.fs? It would seem pretty strange behaviour for it to only change *.{fsproj,csproj,sln}.

Contributor

knocte commented Dec 6, 2012

It only changes msbuild files because that's what it needs to do to be fully interoperable with VS. VS saves all msbuild files with CRLF, but it doesn't do it with the rest unless you tell it to.

Member

rneatherway commented Dec 6, 2012

Thanks, I see. So two options then:

  1. Normalize the repository, but set eol=crlf for all msbuild files.
  2. Have all files be CRLF everywhere (no normalization) and reject any pull requests that violate this policy.
Contributor

knocte commented Dec 6, 2012

There can be a thrid option derived from the 2nd that is not so harsh:
3. Have all files be CRLF everywhere (no normalization), and include a git pre-commit hook to unix2dos all files so we don't get any pull requests that violate this policy.

Contributor

knocte commented Dec 6, 2012

And when I said "all files" I meant "all files modified in the commit". Just in case someone is thinking this solution hurts performance...

Member

rneatherway commented Dec 6, 2012

This seems like a reasonable solution.

@ghost

ghost commented Dec 6, 2012

@knocte I don't yet repro this behaviour where MD saves all .fsproj files as CRLF. Maybe I'm doing something wrong, but I tried this on OSX with 3.0.5:

  • ran dos2unix on an .fsproj (not under a git repo) and checked its length
  • opened it in MD
  • renamed a file in solution pad but kept the name the same length
  • closed MD

The file stayed the same "unix" length after the edit. I then did the same on the "dos" file and it seemed to convert to "unix" endings (the file became shorter)

In any case, I don't think this repo should use CRLF line endings just because of the behaviour of one particular development tool (MD), which to be honest isn't even used very much AFAIK when editing the compiler.

To me, "auto" line endings look like the simplest and best choice to keep things sane on the different systems.

Contributor

knocte commented Dec 6, 2012

@dsyme : TBH I haven't tried with *.fsproj files specifically. I just know this is the behaviour of MD with msbuild files in general (which maybe only applies to *.csproj files, and the FSharpBinding decided not to mimic this).

In regards to the argument of MD not being used much in fsharp compiler: fair enough. But maybe this argument doesn't apply so much to other modules such as FSharpBinding?

Contributor

knocte commented Dec 6, 2012

BTW, if FSharpBinding indeed has this behaviour, I would consider it a bug (otherwise it would generate lot of noise when working with F# projects which were originally created in Windows).

Contributor

cdrnet commented Dec 6, 2012

Throwing in some technical aspects of Git itself.

(I personally prefer 1) but that doesn't matter here)

Note that the normalization of the complete git history as discussed above and tried by @DanielFabian causes some havoc among all forks as well, although that's just once, then it is done and fine. Forks essentially need to reset to the new master. It is still possible to cherry-pick from the old history afterwards (e.g. for pending pull requests), but we should never merge between them.

  1. Enable repository-wide normalization by git:
  • Trivial to implement by adding the following lines to .gitattributes:
*.sln text eol=crlf merge=union
* text=auto

A complete .gitattributes could then look like:

*.cs text diff=csharp
*.sln text eol=crlf merge=union
*.fsproj text merge=union
*.csproj text merge=union
* text=auto
  • Automatically enforced (auto-fixing if needed, with warning)
  • Well supported by git and git tools
  • Supports custom working copy formats depending on client config out of the box (e.g. on Windows, text files would be checked out as CRLF by default, but this is locally configurable).
  • Repository format does not depend on how git was installed or configured locally
  1. Enable clone-wide normalization by git (status quo):
  • Trivial to implement: do nothing, i.e. do not specify any text handling in .gitattributes
  • Normalization depending on local core.autocrlf settings, not enforceable repository wide
  • Works quite well in Windows-only projects, provided all users have installed and configured git the same way
  • If core.autocrlf is enabled, can cause a lot of files to be marked as changed right on a clean checkout if others have not enabled it or configured differently. Merging can quickly become a nightmare.
  • Deprecated in git (exactly because of above points) and replaced with .gitattributes text handling some years ago.
  1. Disable any text normalization by git, accept random line endings
  • Trivial to implement by adding the following line to .gitattributes:
* -text
  • Works quite well in Windows-only projects as the randomness is zero :) but this is not the case here
  • Diff and Merge can get messy once files start being converted between the two randomly by the editors; gitk allows to "not show" such changes, but otherwise git tools are not very good at ignoring white space and line endings in my experience. But at least it doesn't flag files as changed right after checkout. But it might do so after a file has been opened and saved in your favorite editor even if no change was made.
  • Some primitive tools or editors might complain (or show redundant characters) when confronted with uncommon line endings for that platform
  1. Try to normalize to CRLF by other means
  • Not standard behavior, so it is unclear how it will be supported by tools
  • Hooks are not enforceable unless we can set up server-side hooks (client-side hooks are neither automatically distributed nor enforced by git)
  • Server-side hooks cannot fix anything, just block the operation until it is fixed manually by the contributor, which can become tedious
  • clean/smudge filters could work and can be configured by .gitattributes, but it is unclear to me what the advantage would be over using the standard text normalization behavior as in 1).
Contributor

funnelweb commented Dec 9, 2012

Then let's just go for (1). That means merging this pull request and adjsuting the .gitattribute, right?

@DanielFabian - do you know why this pull request is labelled "cannot be automatically merged"?

I'm not 100% sure, but most likely, because I ran the script I mentioned
earlier and it rewrote every commit from the beginning of the git history,
thus also rewriting the parent of the branch.

Maybe just run my script locally and make a 'push -f' to force an overwrite
(took about 4 minutes of processing locally in my case).
EDIT: the script mentioned is:
git filter-branch -f --prune-empty --tree-filter 'git ls-files -z | xargs -0 dos2unix' -- --all

Contributor

funnelweb commented Dec 9, 2012

The repository has now been normalized to Unix and "text auto", and a .gitattributes like put in place like the one described by @cdrnet. Thanks everyone and lets see how this goes.

Contributor

funnelweb commented Dec 9, 2012

closing this since it has been don by direct commit

@funnelweb funnelweb closed this Dec 9, 2012

@fsgit fsgit pushed a commit that referenced this pull request Jul 2, 2014

@latkin latkin Adjust parser to disallow vertical pipes in active pattern case ident…
…ifiers.

Related to #78 - without this restriction, the typechecker can't distinguish between valid pipes separating cases and invalid pipes that are embedded in a case identifier.  The result is that nonsense like this compiles, when it should not:

let (|``Foo|Bar``|) x =
    match x with
    | 0 -> Foo
    | _ -> Bar

match 42 with
| Foo -> "foo"
| Bar -> "bar" (changeset 1282321)
1936874

@fsgit fsgit pushed a commit that referenced this pull request Jul 2, 2014

@latkin latkin Fix for #78 - allow space characters in active pattern case identifie…
…rs (changeset 1282327)
9e55115
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment