.gitignore file does not work when saved as UTF-8 + BOM

[Download]

Removing the BOM makes it work.
This is very unfortunate as it prevents us from writing out the BOM for .gitignore files, effectively making applications guess the encoding i.s.o. being able to detect it.
The Unicode spec mandates conforming applications to accept the BOM.
If Git does not support Unicode this issue can be closed (but that would be another issue :)

[jnavila]

At its core, Git uses UTF-8 to encode pathnames. .gitignore makes heavy use of path names and so is expected in UTF-8. There's no further need to add a BOM, as the Unicode guidelines describes.

[Download]

The Unicode spec requires all software that supports Unicode to correctly deal with the BOM.

When a process interprets a byte sequence which purports to be in a Unicode character
encoding scheme, it shall interpret that byte sequence according to the byte order and
specifications for the use of the byte order mark established by this standard for that
character encoding scheme.

_{(Unicode Standard 5.0, ch.3, C11, emphasis mine)}

There's no further need to add a BOM

There is. Without it, many editors will, once opened, save the file in the wrong format afterwards. Because as long as there aren't any characters used outside of US-ASCII, there is no way for an editor to know what the encoding is. Many then guess latin.

So:

1. Save a .gitignore file as Unicode (but without BOM) that has only ASCII text in it
2. Open in some editor... it assumes latin (it has to make assumptions as there is no BOM)
3. Type some text outside of US-ASCII
4. Save

Your file is now encoded in Latin i.s.o. UTF-8.

There is no way to prevent this. The only thing we can do (and I'm trying to) is convince the whole world to, when in doubt, assume Unicode.... AND: Write out the BOM.

I try to convinc people to do both... I keep telling them that the BOM, as it is required to be supported by the Unicode standard, should never have a negative impact. It should never hurt... and then software like Git comes along that just breaks when you save with the BOM.

Without a BOM, software has to guess at the encoding. HTTP has a Content-Type header, but what about the file system? Or database fields? MySQL allows you to specify the encoding when creating the table etc etc. But really, instead of trying to add metadata about the encoding to all systems that store text content, we should use this mechanism, standardized at the lowest possible level, the Unicode standard itself, to supply metadata about the encoding. The BOM.

I know that the Unicode FAQ does not recommend adding a BOM. That page is pretty old. The recommendation is there to help UTF-8 text that only contains ASCII to be correctly interpreted by programs that do not support Unicode... And indeed if you are from the US and ASCII is enough for you than that's good advice. But the world has changed since then and by now almost all programs correctly handle Unicode and the BOM and the bigger problem is the encoding getting lost when we move text from one system to another, open with different progams etc. The BOM is a perfect solution for this so I recommend everybody to use it.

Anyway I am not asking you to change your habits. Just make .gitignore NOT CHOKE on the BOM so I can use it the way it is supposed to work.

[jnavila]

Even if I still don't bite the usefulness of a BOM when UTF-8 exhibits no byte-order problem, you have perfectly the right to raise this issue. Anyway, I have searched for the issue in the mailing-list history and it seems to have been already fixed.

http://thread.gmane.org/gmane.comp.version-control.git/267288/focus=267399

Have you tried with the latest version?

[Download]

Ah I didn't even know how to search that list! Thank you for digging this up, let me look into it.

[sxlijin]

@Download is this issue still a problem? Otherwise I think it's safe to close this.

[Download]

@sxlijin I just tried it and it seems to be working fine now:

C:\ws\test>git --version
git version 2.7.4.windows.1

C:\ws\test>git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   .gitignore

no changes added to commit (use "git add" and/or "git commit -a")

It is (correctly) showing .gitignore as modified and not any of the files in the ignore list.
So I guess it's fixed! Yay!