Add a Clean/Smudge Filter for Windows UTF-16 files

[kismert]

The native encoding for Microsoft Windows is UTF-16 (UCS-2 little-endian). The Windows distribution of msysGit does not handle this format as text, but instead treats it as binary.

As a fix, msysGit Google group members suggested a clean/smudge filter, which does a very good job of handling UTF-16. This would take little effort to include in msysGit:

1. Distribute iconv.exe and support files in \Program Files\Git\bin for the Windows version.
2. Add the following to~\Git\etc\gitconfig:

[filter "winutf16"]
        clean = iconv -f utf-16 -t utf-8
        smudge = iconv -f utf-8 -t utf-16
        required

1. Add documentation to tell users how to setup their global ~/Git/etc/gitattributes or local ~/.gitattributes files, for example:

    *.txt filter=winutf16

I think this would be a valuable enhancement for msysGit that would allow Windows users to quickly configure their repositories to better work with the increasingly common UTF-16 format.

Note that Mercurial and Bazaar do not handle UTF-16 properly either, and this would give msysGit based products a leg up on their peers.

Thanks,
kismert

[sschuberth]

Just out of curiosity, what Windows program stores text file in UTF-16 by default? The only occasion where I saw this recently for the first time is when you pipe e.g. the output of "wmic process get executablepath,processid" to a file. Also, is there a risk of this filter breaking the user experience non-UTF-16 text files?

[dolmen]

Regedit.

[sschuberth]

Well, yeah. If you choose to export to .txt files instead of the usual .reg files (note that the suggested filter only applies to .txt files). Quite an academic example.

[kismert]

@sschuberth:
I'm trying to write a source export plug-in for Microsoft Access. Starting with 2007, Form and Report files are exported as UTF-16.
Visual Studio 2012 supports UTF-16 as a file format.
The default unicode conversion option in SQL Server Management Studio 2008 is UTF-16.
If you save a text file as 'Unicode' in Notepad, you get UTF-16.
Windows uses UTF-16 encoding for its internal functions.
Obviously, Microsoft doesn't use UTF-16 as a file-encoding standard, and UTF-8 has increasing support. Still, there is a fair chance you will run into UTF-16 files in Windows development.

Regarding breaking the user experience, the filter would be off by default. The user would have to manually configure their gitattributes to handle UTF-16 files as required.

[kusma]

@zachriggle No, they are not, at least not in Visual Studio 2008.

[dscho]

@kismert since you did almost all the work (apart from @patthoyts' patch to include iconv.exe by default), how about opening a pull request?