Encoding issue #95
Comments
|
Without the env var set: $stdin.external_encoding # => #<Encoding:US-ASCII>With the env var set: $stdin.external_encoding # => #<Encoding:UTF-8> |
|
According to opengroup, which cites IEEE Std 1003.1-2001 (might be this, and is almost certainly this, except you can't read it without signing up or something... w/e)
There's a more extensive explanation on that site, including how to parse and make sense of the values, it's just prior to section 8.3 |
|
Looks like MRI's hit this, too: |
|
Seems to have stemmed from JoshCheek/atom-seeing-is-believing#24 but I would like to fix it in SiB (or at least guess the most likely answer in the event that the invoking context got it wrong), b/c this may be what is affecting #92, and it confusingly looks like a bug in SiB |
|
Hey, @avdi, I read your post, it was great! It all made a lot of sense to me except for the utf-8 issue. I tried to express it in a coherent manner, but instead I think I just destroyed my brain >.< In an abstract sense (ie conclusions instead of explanations) it's:
I guess an example would be that a user is actually using a different encoding, then we want to avoid transcoding it since that can lose information (I assume there are encodings with info that is not encompassed by utf-8, though I haven't been able to find an example). So we should do everything in the users's encoding, and translate our internal strings into their encoding and write the file in their encoding. To figure out which is going on, I've been trying to recreate your issue. I'm pretty sure I can reproduce each of your examples from the blog, but it would be helpful for me if you could describe to me the encoding issue you experienced in #92 Eg:
|
|
More later, but to be clear, I never experienced an encoding issue from
within emacs. The problems were bigger than just encoding issues when I
first set out to make the tests pass on windows.
That said, I encountered many, many encoding issues in the test suite
itself.
On Windows, Ruby is going to believe that text data it is receiving from
files and STDIN is IBM437, and if it is wrong, things will break. And
unless told otherwise it will always transcode internal utf-8 to IBM437
when it writes, which WILL lose information.
There's no magic sniffing here - it can't know if, say, a cooperating
process is actually piping it utf-8 unless that process somehow tells it so
(or it just assumes by hardcoding an encoding).
…--
Avdi Grimm
http://avdi.org
On Dec 15, 2016 02:59, "Josh Cheek" ***@***.***> wrote:
Hey, @avdi <https://github.com/avdi>, I read your post, it was great! It
all made a lot of sense to me except for the utf-8 issue. I tried to
express it in a coherent manner, but instead I think I just destroyed my
brain >.< In an abstract sense (ie conclusions instead of explanations)
it's:
If the environment set everything correctly, it should work without
needing to care about encodings. So if it blew up, then either SiB got out
of whack somewhere, or it's an environment issue. If it's an SiB thing,
overriding the defaults, then we should find / fix it. If it's an env
thing, we should try to guess a few common environments, and if not we
should explain to the user what the problem is
I guess an example would be that a user is actually using a different
encoding, then we want to avoid transcoding it since that can lose
information (I assume there are encodings with info that is not encompassed
by utf-8, though I haven't been able to find an example). So we should do
everything in the users's encoding, and translate our internal strings into
their encoding and write the file in their encoding.
To figure out which is going on, I've been trying to recreate your issue.
I'm pretty sure I can reproduce each of your examples from the blog, but it
would be helpful for me if you could describe to me the encoding issue you
experienced in #92
<#92>
Eg:
-
Did you run SiB from emacs?
-
What encoding do you think the file was? (eg what does Emacs say it
is?)
-
How was it erroring? stacktrace / mojibaked / something else? If
stacktrace, can you provide it? If mojibaked, can you screenshot the
incorrect chars?
-
What was the text that caused the issue? (as screenshot in order to
avoid encoding issues here, too :P)
-
Did you have any of the environment variables LANG, LC_ALL, LC_COLLATE,
LC_CTYPE set? If so, what were their values?
-
If you're able to invoke Ruby the same manner that you invoked SiB,
can you try invoking this and let me know what it says?
require 'pp'
r, w = IO.pipe
pp [
[Encoding, Encoding.default_internal, Encoding.default_external, Encoding.locale_charmap,
Encoding.find("external"), Encoding.find("internal"), Encoding.find("locale"), Encoding.find("filesystem"),
],
[$stdin, $stdin.internal_encoding, $stdin.external_encoding],
[$stdout, $stdout.internal_encoding, $stdout.external_encoding],
["pipe read", r.internal_encoding, r.external_encoding] ,
["pipe write", w.internal_encoding, w.external_encoding],
["__ENCODING__", __ENCODING__],
[String, "".encoding],
["ENV[LANG]", ENV["LANG"]],
["ENV[LC_ALL]", ENV["LC_ALL"]],
["ENV[LC_COLLATE]", ENV["LC_COLLATE"]],
["ENV[LC_CTYPE]", ENV["LC_CTYPE"]],
]
-
Also, if you have an ability to run bash on that machine, can you run
this:
forEachEncoding() {
code="$1"
echo "$code"
echo -n ' LANG= | '
LANG= ruby -e "p $code"
echo -n ' LANG=en_US.IBM437 | '
LANG=en_US.IBM437 ruby -e "p $code"
echo -n ' LANG=de_CH.IBM437 | '
LANG=de_CH.IBM437 ruby -e "p $code"
echo -n ' # encoding: IBM437 | '
ruby -e "# encoding: IBM437p $code"
echo -n ' --internal-encoding IBM437 | '
ruby --internal-encoding IBM437 -e "p $code"
echo -n ' --external-encoding IBM437 | '
ruby --external-encoding IBM437 -e "p $code"
echo -n ' -Ks | '
ruby -Ks -e "p $code"
}
forEachEncoding 'Encoding.default_internal'
forEachEncoding 'Encoding.default_external'
forEachEncoding 'Encoding.locale_charmap'
forEachEncoding 'Encoding.find("locale")'
forEachEncoding 'Encoding.find("external")'
forEachEncoding 'Encoding.find("internal")'
forEachEncoding 'Encoding.find("locale")'
forEachEncoding 'Encoding.find("filesystem")'
forEachEncoding 'Encoding.find("filesystem")'
forEachEncoding '$stdin.internal_encoding'
forEachEncoding '$stdin.external_encoding'
forEachEncoding '$stdout.internal_encoding'
forEachEncoding '$stdout.external_encoding'
forEachEncoding 'IO.pipe.first.internal_encoding'
forEachEncoding 'IO.pipe.first.external_encoding'
forEachEncoding 'IO.pipe.last.internal_encoding'
forEachEncoding 'IO.pipe.last.external_encoding'
forEachEncoding '__ENCODING__'
forEachEncoding '"".encoding'
I'm piping it through this monstrosity to try and make sense of all
the info:
ruby -rpp -e 'rows_of_cols = $stdin.read.split(/^(?=\S)/).map { |row| row_name, *cols = row.lines.map(&:chomp); [row_name, cols.map { |col| col.split("|").map(&:strip) }] }; colnames = rows_of_cols.first.last.map(&:first); rownames = rows_of_cols.map(&:first); rows = rows_of_cols.map(&:last).map { |cols| cols.map(&:last) }; format = "%-#{rownames.max_by(&:length).length}s | #{[colnames, *rows].transpose.map { |cs| "%-#{cs.max_by(&:length).length}s" }.join(" | ")}\n"; printf format, "", *colnames; printf format, *format.scan(/\d+/).map { |n| "-" * n.to_i }; rownames.each_with_index { |name, y| printf format, name, *rows[y] }'
[image: screenshot 2016-12-15 01 32 27]
<https://cloud.githubusercontent.com/assets/77495/21215369/6475961a-c266-11e6-86bf-73a3fad45e76.png>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#95 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAD1jC2yt2AR7XjbeGav4XqbRiI_HVZks5rIPNYgaJpZM4LNg0a>
.
|
|
Oh, and running Bash scripts is probably meaningless because the
environment inside bash-on-windows tools is likely to skew from baseline
windows enough to render answers which aren't generally applicable.
I think you're right that there's probably a more general solution to be
had here. I'll be thinking about it. But I think it's also true that a lot
of the assumptions most people rely on (or just don't think about) having
to do with encodings in Ruby are wrong. And operating in a UNIX environment
just makes those assumptions invisible.
…--
Avdi Grimm
http://avdi.org
On Dec 15, 2016 05:54, "Avdi Grimm" ***@***.***> wrote:
More later, but to be clear, I never experienced an encoding issue from
within emacs. The problems were bigger than just encoding issues when I
first set out to make the tests pass on windows.
That said, I encountered many, many encoding issues in the test suite
itself.
On Windows, Ruby is going to believe that text data it is receiving from
files and STDIN is IBM437, and if it is wrong, things will break. And
unless told otherwise it will always transcode internal utf-8 to IBM437
when it writes, which WILL lose information.
There's no magic sniffing here - it can't know if, say, a cooperating
process is actually piping it utf-8 unless that process somehow tells it so
(or it just assumes by hardcoding an encoding).
--
Avdi Grimm
http://avdi.org
On Dec 15, 2016 02:59, "Josh Cheek" ***@***.***> wrote:
> Hey, @avdi <https://github.com/avdi>, I read your post, it was great! It
> all made a lot of sense to me except for the utf-8 issue. I tried to
> express it in a coherent manner, but instead I think I just destroyed my
> brain >.< In an abstract sense (ie conclusions instead of explanations)
> it's:
>
> If the environment set everything correctly, it should work without
> needing to care about encodings. So if it blew up, then either SiB got out
> of whack somewhere, or it's an environment issue. If it's an SiB thing,
> overriding the defaults, then we should find / fix it. If it's an env
> thing, we should try to guess a few common environments, and if not we
> should explain to the user what the problem is
>
> I guess an example would be that a user is actually using a different
> encoding, then we want to avoid transcoding it since that can lose
> information (I assume there are encodings with info that is not encompassed
> by utf-8, though I haven't been able to find an example). So we should do
> everything in the users's encoding, and translate our internal strings into
> their encoding and write the file in their encoding.
>
> To figure out which is going on, I've been trying to recreate your issue.
> I'm pretty sure I can reproduce each of your examples from the blog, but it
> would be helpful for me if you could describe to me the encoding issue you
> experienced in #92
> <#92>
>
> Eg:
>
> -
>
> Did you run SiB from emacs?
> -
>
> What encoding do you think the file was? (eg what does Emacs say it
> is?)
> -
>
> How was it erroring? stacktrace / mojibaked / something else? If
> stacktrace, can you provide it? If mojibaked, can you screenshot the
> incorrect chars?
> -
>
> What was the text that caused the issue? (as screenshot in order to
> avoid encoding issues here, too :P)
> -
>
> Did you have any of the environment variables LANG, LC_ALL, LC_COLLATE,
> LC_CTYPE set? If so, what were their values?
> -
>
> If you're able to invoke Ruby the same manner that you invoked SiB,
> can you try invoking this and let me know what it says?
>
> require 'pp'
> r, w = IO.pipe
> pp [
> [Encoding, Encoding.default_internal, Encoding.default_external, Encoding.locale_charmap,
> Encoding.find("external"), Encoding.find("internal"), Encoding.find("locale"), Encoding.find("filesystem"),
> ],
> [$stdin, $stdin.internal_encoding, $stdin.external_encoding],
> [$stdout, $stdout.internal_encoding, $stdout.external_encoding],
> ["pipe read", r.internal_encoding, r.external_encoding] ,
> ["pipe write", w.internal_encoding, w.external_encoding],
> ["__ENCODING__", __ENCODING__],
> [String, "".encoding],
> ["ENV[LANG]", ENV["LANG"]],
> ["ENV[LC_ALL]", ENV["LC_ALL"]],
> ["ENV[LC_COLLATE]", ENV["LC_COLLATE"]],
> ["ENV[LC_CTYPE]", ENV["LC_CTYPE"]],
> ]
>
> -
>
> Also, if you have an ability to run bash on that machine, can you run
> this:
>
> forEachEncoding() {
> code="$1"
> echo "$code"
> echo -n ' LANG= | '
> LANG= ruby -e "p $code"
> echo -n ' LANG=en_US.IBM437 | '
> LANG=en_US.IBM437 ruby -e "p $code"
> echo -n ' LANG=de_CH.IBM437 | '
> LANG=de_CH.IBM437 ruby -e "p $code"
> echo -n ' # encoding: IBM437 | '
> ruby -e "# encoding: IBM437p $code"
> echo -n ' --internal-encoding IBM437 | '
> ruby --internal-encoding IBM437 -e "p $code"
> echo -n ' --external-encoding IBM437 | '
> ruby --external-encoding IBM437 -e "p $code"
> echo -n ' -Ks | '
> ruby -Ks -e "p $code"
> }
>
>
> forEachEncoding 'Encoding.default_internal'
> forEachEncoding 'Encoding.default_external'
> forEachEncoding 'Encoding.locale_charmap'
> forEachEncoding 'Encoding.find("locale")'
> forEachEncoding 'Encoding.find("external")'
> forEachEncoding 'Encoding.find("internal")'
> forEachEncoding 'Encoding.find("locale")'
> forEachEncoding 'Encoding.find("filesystem")'
> forEachEncoding 'Encoding.find("filesystem")'
>
> forEachEncoding '$stdin.internal_encoding'
> forEachEncoding '$stdin.external_encoding'
> forEachEncoding '$stdout.internal_encoding'
> forEachEncoding '$stdout.external_encoding'
>
> forEachEncoding 'IO.pipe.first.internal_encoding'
> forEachEncoding 'IO.pipe.first.external_encoding'
> forEachEncoding 'IO.pipe.last.internal_encoding'
> forEachEncoding 'IO.pipe.last.external_encoding'
>
> forEachEncoding '__ENCODING__'
> forEachEncoding '"".encoding'
>
> I'm piping it through this monstrosity to try and make sense of all
> the info:
>
> ruby -rpp -e 'rows_of_cols = $stdin.read.split(/^(?=\S)/).map { |row| row_name, *cols = row.lines.map(&:chomp); [row_name, cols.map { |col| col.split("|").map(&:strip) }] }; colnames = rows_of_cols.first.last.map(&:first); rownames = rows_of_cols.map(&:first); rows = rows_of_cols.map(&:last).map { |cols| cols.map(&:last) }; format = "%-#{rownames.max_by(&:length).length}s | #{[colnames, *rows].transpose.map { |cs| "%-#{cs.max_by(&:length).length}s" }.join(" | ")}\n"; printf format, "", *colnames; printf format, *format.scan(/\d+/).map { |n| "-" * n.to_i }; rownames.each_with_index { |name, y| printf format, name, *rows[y] }'
>
> [image: screenshot 2016-12-15 01 32 27]
> <https://cloud.githubusercontent.com/assets/77495/21215369/6475961a-c266-11e6-86bf-73a3fad45e76.png>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#95 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAAD1jC2yt2AR7XjbeGav4XqbRiI_HVZks5rIPNYgaJpZM4LNg0a>
> .
>
|
It should be okay, the only data we insert are the markers and object inspections. Markers use the chars
For our event stream, we can explicitly set the pipe to UTF-8. No strings provided by the user pass directly through it, only string literals generated within SiB, user data is marshalled and base 64'd, so it passes through the pipe as an ascii subset, and the encoding is preserved on the other side.
Aye, I deleted that from the issue after I realized it.
Definitely. Having AppVeyor and users in that env is awesome ^_^ |
|
Well if the event stream is over TCP, rather than a pipe, the point is
moot. We don't need to worry about that aspect.
The elephant in the room is the encoding of code-to-be-evaluated.
Here's what happens now, without any fixes:
1. The user writes some code in an editor. Presumably, the editor uses a
unicode format internally. Maybe they include some non-ASCII characters in
their code.
2. They either pipe the code to SiB, or write it to a file. Either way, it
PROBABLY is written out by the editor to stream or disk as UTF-8, but this
may not be true depending on their locale.
3. So far, so good.
4. Long before SiB evaluates the user-provided code, it reads it in from
either file or STDIN. Windows tells it it is reading IBM437-encoded text,
and it doesn't know any better, so it transcodes to internal UTF-8,
mangling any non-ASCII characters on the way.
5. The game is already up, even before we set up any subprocess
communication. Any non-ASCII in the code-to-be-evaled is permanently
mangled.
Here's what my definitely-inadequate hacked-up patch does:
...
3. So far, so good.
4. Long before SiB evaluates the user-provided code, it reads it in from
either file or STDIN. We hardcode the assumption that the code was written
to disk as UTF-8, and force that encoding. It "transcodes" external UTF-8
to internal UTF-8, mangling nothing.
5. ...unless we were wrong about that code being in UTF-8 (maybe it was
written to disk in Shift-JIS, for instance). In which case, text is
mangled, and the game is up :-(
I can think of a few potential scenarios for what OUGHT to happen instead,
but I'm still mulling them over.
…On Thu, Dec 15, 2016 at 12:55 PM, Josh Cheek ***@***.***> wrote:
On Windows, Ruby is going to believe that text data it is receiving from
files and STDIN is IBM437, and if it is wrong, things will break. And
unless told otherwise it will always transcode internal utf-8 to IBM437
when it writes, which WILL lose information.
It should be okay, the only data we insert are the markers and object
inspections. Markers use the chars # =>~!., which are all used by Ruby,
so should be transcodable. Inspections were obtained from the user's code,
so barring an encoding issue that exists regardless of SiB, we should be
okay to embed them into a comment. If not, then elide the result or
mojibake it, but definitely don't explode.
There's no magic sniffing here - it can't know if, say, a cooperating
process is actually piping it utf-8 unless that process somehow tells it so
(or it just assumes by hardcoding an encoding).
For our event stream, we can explicitly set the pipe to UTF-8. No strings
provided by the user pass directly through it, only string literals
generated within SiB, user data is marshalled and base 64'd, so it passes
through the pipe as an ascii subset, and the encoding is preserved on the
other side.
Oh, and running Bash scripts is probably meaningless because the
environment inside bash-on-windows tools is likely to skew from baseline
windows enough to render answers which aren't generally applicable.
Aye, I deleted that from the issue after I realized it.
I think it's also true that a lot
of the assumptions most people rely on (or just don't think about) having
to do with encodings in Ruby are wrong. And operating in a UNIX environment
just makes those assumptions invisible.
Definitely. Having AppVeyor and users in that env is awesome ^_^
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#95 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAD1otiqh-sCpEnXO5KN3xvsf2AU6ldks5rIX78gaJpZM4LNg0a>
.
|
|
It may be the case that we don't want to give the subsidiary Ruby process
any special encoding help. Since running code un-tagged with an encoding
comment under SiB should probably have the same outcome (even if it's a bad
outcome) as just running the regular Ruby interpreter on it.
If that's the case, then it may be sufficient to just make sure that the
entire chain of SiB reading in the source code and then writing it back out
is done in binary mode, treating it as purely opaque data.
…On Thu, Dec 15, 2016 at 1:30 PM, Avdi Grimm ***@***.***> wrote:
Well if the event stream is over TCP, rather than a pipe, the point is
moot. We don't need to worry about that aspect.
The elephant in the room is the encoding of code-to-be-evaluated.
Here's what happens now, without any fixes:
1. The user writes some code in an editor. Presumably, the editor uses a
unicode format internally. Maybe they include some non-ASCII characters in
their code.
2. They either pipe the code to SiB, or write it to a file. Either way, it
PROBABLY is written out by the editor to stream or disk as UTF-8, but this
may not be true depending on their locale.
3. So far, so good.
4. Long before SiB evaluates the user-provided code, it reads it in from
either file or STDIN. Windows tells it it is reading IBM437-encoded text,
and it doesn't know any better, so it transcodes to internal UTF-8,
mangling any non-ASCII characters on the way.
5. The game is already up, even before we set up any subprocess
communication. Any non-ASCII in the code-to-be-evaled is permanently
mangled.
Here's what my definitely-inadequate hacked-up patch does:
...
3. So far, so good.
4. Long before SiB evaluates the user-provided code, it reads it in from
either file or STDIN. We hardcode the assumption that the code was written
to disk as UTF-8, and force that encoding. It "transcodes" external UTF-8
to internal UTF-8, mangling nothing.
5. ...unless we were wrong about that code being in UTF-8 (maybe it was
written to disk in Shift-JIS, for instance). In which case, text is
mangled, and the game is up :-(
I can think of a few potential scenarios for what OUGHT to happen instead,
but I'm still mulling them over.
On Thu, Dec 15, 2016 at 12:55 PM, Josh Cheek ***@***.***>
wrote:
> On Windows, Ruby is going to believe that text data it is receiving from
> files and STDIN is IBM437, and if it is wrong, things will break. And
> unless told otherwise it will always transcode internal utf-8 to IBM437
> when it writes, which WILL lose information.
>
> It should be okay, the only data we insert are the markers and object
> inspections. Markers use the chars # =>~!., which are all used by Ruby,
> so should be transcodable. Inspections were obtained from the user's code,
> so barring an encoding issue that exists regardless of SiB, we should be
> okay to embed them into a comment. If not, then elide the result or
> mojibake it, but definitely don't explode.
>
> There's no magic sniffing here - it can't know if, say, a cooperating
> process is actually piping it utf-8 unless that process somehow tells it
> so
> (or it just assumes by hardcoding an encoding).
>
> For our event stream, we can explicitly set the pipe to UTF-8. No strings
> provided by the user pass directly through it, only string literals
> generated within SiB, user data is marshalled and base 64'd, so it passes
> through the pipe as an ascii subset, and the encoding is preserved on the
> other side.
>
> Oh, and running Bash scripts is probably meaningless because the
> environment inside bash-on-windows tools is likely to skew from baseline
> windows enough to render answers which aren't generally applicable.
>
> Aye, I deleted that from the issue after I realized it.
>
> I think it's also true that a lot
> of the assumptions most people rely on (or just don't think about) having
> to do with encodings in Ruby are wrong. And operating in a UNIX
> environment
> just makes those assumptions invisible.
>
> Definitely. Having AppVeyor and users in that env is awesome ^_^
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#95 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAAD1otiqh-sCpEnXO5KN3xvsf2AU6ldks5rIX78gaJpZM4LNg0a>
> .
>
|
|
If you want to play with any of this stuff yourself, MS has free VirtualBox images: https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hopefully related to the binary/utf-8 issue in #92
This code:
def π; endExplodes in Atom, but works correctly in the shell and TextMate2. I looked at the env vars, andLANGwas set in TM but not in Atom.When I opened Atom's console (cmd+opt+i) and ran
process.env.LANG = 'en_US.UTF-8', it then worked correctly.When I deleted it again:
delete process.env.LANG, it then broke again. Stacktrace:The text was updated successfully, but these errors were encountered: