Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with "default_encoding": "UTF-8 with BOM" #1074

Open
alextretyak opened this issue Jan 27, 2016 · 9 comments

Comments

@alextretyak
Copy link

commented Jan 27, 2016

When I set "default_encoding": "UTF-8 with BOM" in settings file, I can no more use Sublime Text 2 to edit his own settings files!
Just checked with latest portable versions (for Windows), and this bug is still there.

To reproduce it just do the following 2 steps:

  1. Open from Menu: Preferences -> Settings - User.
  2. File -> Save with Encoding -> UTF8 with BOM.

After step 2 I get this (on Windows 7 x64 SP1):
In Sublime Text 3:
img-2016-01-27-09-57-31

In Sublime Text 2:
img-2016-01-27-09-57-31_cr

@FichteFoll

This comment has been minimized.

Copy link
Member

commented Jan 27, 2016

What happens after you saved the file with BOM and dismissed the error message?

Saving the file without BOM again should solve the problem.

PS: Using the BOM für UTF-8 is discouraged and instead UTF-8 should be assumed as the default, for all files.

@alextretyak

This comment has been minimized.

Copy link
Author

commented Jan 27, 2016

What happens after you saved the file with BOM and dismissed the error message?

If I do not close Sublime Text editor, then I can resave settings file with UTF8 (or ANSI-like encoding) and all is ok.
But when I close Sublime Text, then it can not successfully startup any more: firstly it shows Message Box with this Error, and after that it just hangs in the background (without showing its main window), so it can be killed only manually via "End Process" sublime_text.exe in Windows Task Manager.
(This is in the latest Sublime Text 2. But Sublime Text 3 starts normally after this error message.)

Saving the file without BOM again should solve the problem.

I don't mind saving sublime-settings files in UTF8 or ANSI. But... HOW? (I mean, now I can do this only manually by clicking "File -> Save with Encoding -> UTF8" each time after I open any sublime-settings file.)
And I still need "default_encoding": "UTF-8 with BOM" for my other files I work with!

(For example, Notepad++ does not change encoding after just open and immediately saving that file. And I think, this behavior for Sublime Text would be fine too.)

// Encoding used when saving new files, and files opened with an undefined
// encoding (e.g., plain ascii files). If a file is opened with a specific
// encoding (either detected or given explicitly), this setting will be
// ignored, and the file will be saved with the encoding it was opened
// with.
"default_encoding": "UTF-8 with BOM",

I suppose, the problem is sublime-settings files are opened with an undefined encoding.
But why not just use fallback_encoding in that case?

// The encoding to use when the encoding can't be determined automatically.
// ASCII, UTF-8 and UTF-16 encodings will be automatically detected.
"fallback_encoding": "Cyrillic (Windows 1251)",

Actually, this wording looks somewhat confusing. I mean I can not immediately understand the difference between "encoding can't be determined automatically" and "undefined encoding". So, I suggest to use default_encoding only when saving new files [but not the already existing ones].

@vovkkk

This comment has been minimized.

Copy link

commented Jan 27, 2016

This looks like bug.

Steps (Win8.1, 3095):

  1. Save file with UTF-8.
  2. Close and open the file.
  3. In console run view.encoding(), result 'Undefined'
  4. Save the file with UTF-8 with BOM
  5. Close and open the file.
  6. In console run view.encoding(), result 'UTF-8 with BOM'

Expected result is opposite, since it says ‘ASCII, UTF-8 and UTF-16 encodings will be automatically detected.’


@adam-aveli You may save following code in Packages\User as py script:

import sublime_plugin


class SaveSettingsAsUTF8(sublime_plugin.EventListener):
    def on_pre_save(self, view):
        if view.file_name().endswith('.sublime-settings'):
            view.set_encoding('utf-8')

endswith supports tuple of strings so you may extend it for other extensions if need (e.g. .sublime-menu, .sublime-keymap, etc).

@alextretyak

This comment has been minimized.

Copy link
Author

commented Jan 27, 2016

It works!
Thanks, @vovkkk!
(But I am still hoping, that this issue will be fixed anyway for all files, regardless of their type or extension.)

@FichteFoll

This comment has been minimized.

Copy link
Member

commented Jan 27, 2016

@adam-aveli

But when I close Sublime Text, then it can not successfully startup any more: firstly it shows Message Box with this Error, and after that it just hangs in the background (without showing its main window), so it can be killed only manually via "End Process" sublime_text.exe in Windows Task Manager.
(This is in the latest Sublime Text 2. But Sublime Text 3 starts normally after this error message.)

ST2 will very likely not be updated anymore considering that ST3 is a thing, so I'd mark this issue off as "fixed". It also matches what I remembered.

@vovkkk

Expected result is opposite, since it says ‘ASCII, UTF-8 and UTF-16 encodings will be automatically detected.’

I suppose that this section is talking specifically about UTF with BOM, which both UTF-8 can and UTF-16 ... (can) have. So in that sense, the second result is expected. The first result is a bit unclear in that "Undefined" leads you to check back to the global default_encoding setting. I do not know how one would distinguish whether ST is using the default_encoding or the fallback_encoding in this case.


@adam-aveli I was under the impression that ST would be able to detect files using UTF8 with BOM automatically, but I believe you specifically want to have ST use UTF8 with BOM for new files, correct?

Probably related: https://forum.sublimetext.com/t/how-does-sublime-detect-file-encodings/16194/2

@alextretyak alextretyak changed the title Preferences.sublime-settings in UTF8 with BOM Problem with "default_encoding": "UTF-8 with BOM" Jan 27, 2016

@alextretyak

This comment has been minimized.

Copy link
Author

commented Jan 27, 2016

I was under the impression that ST would be able to detect files using UTF8 with BOM automatically

I suppose, it was the original title of this issue, which confuses you. Ok, I've renamed it to be more correct.
This is different thing, and actually it is not a big problem, if ST can not read its own preferences encoded in UTF-8 with BOM.
Problem is files in UTF8 without BOM (actually ANSI) implicitly saved as UTF-8 with BOM with setting "default_encoding": "UTF-8 with BOM".

but I believe you specifically want to have ST use UTF8 with BOM for new files, correct?

Yes. This is just what "default_encoding": "UTF-8 with BOM" setting is supposed to do.

@FichteFoll

This comment has been minimized.

Copy link
Member

commented Jan 27, 2016

In that case, I suggest using this plugin instead until gets changed in core:

import sublime_plugin


class NewFileWithUTF8BOM(sublime_plugin.EventListener):
    def on_new(self, view):
        view.set_encoding('UTF-8 with BOM')

and change default_encoding back to UTF-8.

@alextretyak

This comment has been minimized.

Copy link
Author

commented Jan 27, 2016

Thanks!
It does exactly what I needed.
So now I can return setting "default_encoding" back to "UTF-8".

@FichteFoll

This comment has been minimized.

Copy link
Member

commented Aug 28, 2019

To give some more information, as I was able to figure out in a forum thread: when a file only uses ASCII-encoded characters, ST will save the file as UTF-8 with BOM if you configured that for fallback_encoding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
You can’t perform that action at this time.