Support for Unicode encodings with BOM

[notpeelz]

Variants of common Unicode encodings aren't supported, namely, UTF-8 w/ BOM, UTF-16 w/ BOM, etc.

I feel like Atom should natively support all Unicode encodings, even the most esoteric ones (e.g. UTF-32)

[kevinsawicki]

Can you link to a file that is currently failing to auto detect properly?

[notpeelz]

The problem is not so much as to how files are read, but how they're saved; upon saving a AutoHotkey script, the file is saved in regular UTF-8. This is where the issue lies—it seems that the AHK engine has a non-standard reading-byte order.

[kevinsawicki]

Can you link to an AutoHotkey script in a repo somewhere that Atom is corrupting when it is saved?

[notpeelz]

Expected result (UTF-8 w/ BOM):

Actual saved file (created within Atom):

The issue can be easily reproduced by copying & pasting the "ù" (C3 B9) character (or any codepoint using 2+ bytes) into Atom and then saving it; the BOM sequence clearly doesn't appear upon saving, as it is atypical for the UTF-8 encoding.

For what it's worth, here's the relevant script shown in the screenshots:

#SingleInstance, Force
#UseHook, On
#NoTrayIcon

!+u::
!u::
  If (GetKeyState("Capslock", "T") ^ GetKeyState("Shift", "P"))
    SendInput, Ù
  Else
    SendInput, ù
Return

!`;::`

Just for clarifications: Atom isn't actually corrupting anything, it just doesn't have inherent support for creating files in certain Unicode encodings.

[heennkkee]

Hi,
Is there anything planned to fix this issue?
Having some problems with asp classic and Atom because of this right now, and would really like to use Atom for these projects but unable to at the moment due to that.

[imamatory]

Unable to work with atom cause this issue. Failing with cyrillic symbols, with BOM that's ok. Very hope it planned to fix this.

[nicktimko]

I hit this bug when attempting to view a PowerShell redirection logfile (presumably they're all the same encoding?). Atom doesn't recognize the file as UTF-16 LE

PS> echo test > utf16le-with-bom.txt

The annoying part was that even though it looked kinda OK as UTF-8 (except for the BOM that it rendered), the null characters between every printable character totally broke searching.

[rkrv]

This is really problematic when working with .Net and Razor templates. Our project uses UTF-8 with BOM and IIS simply freaks out when it sees international characters in views if the file wasn't saved with BOM.

[samjgalbraith]

SQL server does not deal well with BOM at the start of scripts, and this causes deploy problems for us. I have to open and save in a different text editor that explicitly allows UTF-8/UTF-8 BOM distinction so that I can make sure there's no BOM there otherwise the deploy fails. So for us this is an irritating inconvenience.

[nicktimko]

Maybe of-interest is that Visual Studio Code (also an Electron/JS+Chromium app) detects them correctly. Any intrepid JS programmers want to do some copy-pasting?