fix encoding of substrings when the original string contains binary (i.e. > 0x7f) bytes. #223

merged 1 commit into from Jul 21, 2012


None yet
3 participants

jvshahid commented Jul 8, 2012

A new rubyspec was added that demonstrates the issue (

John Shahid
[encoding] ascii strings that originate from a binary string should b…
…e ascii/utf-8 encodable without throwing an exception.

A new rubyspec was added that demonstrates the issue (

This comment has been minimized.

Show comment Hide comment

headius Jul 21, 2012


This doesn't look bad, but I know the transcoding subsystem is being rewritten by @enebo right now. We will keep this open and revisit it after his work has landed.


headius commented Jul 21, 2012

This doesn't look bad, but I know the transcoding subsystem is being rewritten by @enebo right now. We will keep this open and revisit it after his work has landed.

enebo added a commit that referenced this pull request Jul 21, 2012

Merge pull request #223 from jvshahid/substring_encoding
fix encoding of substrings when the original string contains binary (i.e. > 0x7f) bytes.

This is a good fix and one I have not done.

@enebo enebo merged commit 3b92ea0 into jruby:master Jul 21, 2012

iwadon pushed a commit to iwadon/nokogiri that referenced this pull request Jan 7, 2014

[java] make sure that Xml::Document::encoding is a string with utf-8 …

In order to be consistent with the C-Nokogiri, the encoding string has to be 'utf-8' encoded
(per xml_document.c:178). This change fixes Nokogiri::XML::TestDocumentEncoding::test_encoding,
but also require a patch to JRuby. For more details see jruby/jruby#223

eregon added a commit that referenced this pull request Apr 27, 2016

Squashed 'spec/ruby/' changes from f5d943e..32e22d6
32e22d6 Ruby 2.4 raises a TypeError for invalid step args
8466e7c Fixed issues in Kernel#rand specs
3e0334a Removed using BigDecimal in rand specs.
9b8927d String#dup returns copy when no modification made.
a722922 Adds Numeric#step keyword arguments specs.
5114a66 Fixed edge case where small BigDecimal ranges were erroring
ad9be02 Fix float ranges where diff of 0
c0e852a Fixed rand() for degenerative empty range
bf42261 Only run frozen string literals specs on Ruby >= 2.3
faea24f Add two specs for new arrays with simple Fixnums
bf9ad3a Ensure String#byteslice normalizes indices by byte length, not character length.
b892ade BasicObject#instance_eval evaluates string with given filename and linenumber
946967e yield uses captured block of a block used in define_method
3e09f4e Spec that frozen string literals don't change what defined? returns, to protect against implementations that desugar the call away.
3f436dc Where we're testing the freeze command line flag, we shouldn't call .freeze.
47690be Where we're testing the freeze magic comment, we shouldn't call .freeze.
3d062e3 Add a spec for string literal freeze magic comment, where there is the same literal but in a file without the comment.
b9c349e Add command-line specs for --enable-frozen-string-literal.
b6c95d1 Add a spec for string literal freeze magic comment, where the literals are in different files but have different encodings.
db22ba5 Add a spec for string literal freeze magic comment, where the literals are in different files.
f8bfdb1 Add a spec for string literal freeze magic comment with two literals but the same content.
8a76900 Add a spec for string literal freeze magic comment with one literal.
b70af78 Add a spec for the special String#freeze syntax.
b3866cb New spec testing that compare_by_identity Hash do not call #hash.
0ade4ee Wait for the main thread to be interrupted in Thread#abort_on_exception spec
30cdf15 Remove bad rb_thread_call_without_gvl2 spec.
75c8c21 Merge pull request #248 from odaira/myContribution
99d20dc In AIX, when a pipe is readable, select(2) returns even the write side of the pipe as "readable"
b60523a Merge pull request #247 from nobu/overridable-OBJDIR
c5a98ec Make OBJDIR overridable
68dd505 Show our build status on Windows
d7a5a7b changes permissions of existing files on MRI on Windows
2077fec Try to write a more sensible spec for IO#write on Windows
5dae610 Try to read in text mode for IO#write spec on Windows
55cb889 Use local variables in Module#ancestors
2b3bf9e Merge pull request #245 from wied03/ancestors
8ad6780 Better test Module#ancestors cases
e5765b1 Merge pull request #246 from wied03/class_to_s
723e4c9 Merge pull request #244 from wied03/master
7bf3a56 Test singleton class cases with `Class#to_s`
2341203 Test a strange module case compare edge case
4b6dd97 Merge pull request #243 from wied03/master
9c14562 Fill spec change from e60792ca2fe08ff2bb2f00eff7c6d70547cfc42e crashes Opal, so don't execute on that platform
ac7f701 Merge pull request #241 from wied03/master
a274931 First cut of Pathname#relative_path_from
4879393 Merge pull request #242 from znz/fix-typo
1a34bb3 Fix a typo
bc2614f Cleanup and reorganize a bit extensions spec_helper
231f0ec Merge pull request #240 from nobu/no-capi-signatures
1829feb Fix .gitignore file to only consider files at the root
31252bd Move extension files to ext/RUBY_NAME/RUBY_VERSION
4834b1a No CAPI signature files
663c7a0 Add sentinel
71d6bde Remove mri.h as MRI should have all C-API functions spec'd in ruby/spec
d294c56 Remove rb_inspecting_p and rb_protect_inspect specs since it is not standard
e60792c Remove deviates_on and use numeric helpers for Array#fill
b85c5f2 Remove extended_on and give proper describe blocks to rb_thread_blocking_region and such
0a9af62 Remove rb_str_ptr and rb_str_ptr_readonly since they are Rubinius-specific
d6c3234 Remove extended_on for missing functionality
28c64c8 rb_obj_frozen_p exists on other implementations as well
1c00935 Merge pull request #239 from odaira/myContribution2
e2065d8 Merge pull request #238 from odaira/myContribution
7805965 AIX does not allow mkdir(2) to set a sticky bit
c45706f Fix typo: 1755 intended to be 01755
ebfaf9d Exempt AIX in core/file/lchmod_spec.rb because it does not support lchmod(2)
260cdc3 Try a third way to pass the system spec and no issue console warnings
7462137 adapt expectation for exit status after SIGKILL on Windows
0bb8a49 Merge pull request #237 from iliabylich/add-tests-for-lbrace-arg-cases
a3dc36f quarantine problematic name resolving spec
ec8e369 SO_REUSEADDR seems to have a slightly different meaning in Windows
7784171 A getaddrinfo spec is also unsupported on Windows
bfc9162 resolv with localhost seems not working on Windows
6d7ea9a Try to fix a system spec on Windows
8c87aba IO.copy_stream with IO with offsets is not supported on Windows
6c66468 File::Stat#nlink seems to be unreliable on Windows
b73e686 Add tests for method calls that have a space between method name and parentheses.
82f728c Merge pull request #236 from iliabylich/add-specs-for-arity/parameters-for-attr_reader/writer
c54cc3a Added specs for Method#arity and Method#parameters for methods generated by attr_reader/attr_writer.
9b45a0d Merge pull request #235 from wied03/master
0a570d9 Add Pathname#join specs, enhance ::new
d6a94f7 Shell variable expansion in Kernel#system is spec on Windows.
64dba27 Remove libgmp from .travis.yml as ruby-head should not need it anymore.
fa55c43 Merge pull request #234 from odaira/myContribution
8b77012 Default error message returned by stderror(3) is "Error <errno> occurred" on AIX
ec4c0d0 Allow both US_ASCII and ASCII_8BIT if fs encoding is unset (LANG=C)
d9dd997 Simplify File::Stat#inspect and do not expect birthtime in the output on Windows.
11544a8 Do not try to change the {a,m}time of an opened file in File::Stat#<=> specs
b3ff895 Try to create files first in File::Stat#<=>.
37df92f Try to simplify the File#new spec with permissions.
a8363e9 chown is no-op on Windows
bd23a74 Math.lgamma(-0.0) is only guaranteed on 2.4 for now.
e2e203b Merge pull request #233 from nobu/feature/Math#lgamma-0.0
a1e623f Math.lgamma returns [infinity, -1] on -0.0
d5e21c8 Move the #include triggering #included outside fixtures.
0e11702 Merge pull request #232 from wied03/super_again
0683a53 Fix another super case
bbe5d48 Merge pull request #231 from wied03/more_super
a081c74 Test more super edge cases
c18e55b Merge pull request #229 from wied03/super_stuff
b04d2fe Merge pull request #228 from nobu/trailing-spaces
bd89b79 Merge pull request #227 from wied03/master
cd06cdd super enhancement
9059448 remove trailing spaces
a87ca8a Cover cases where the backslash itself is escaped
932fa26 Update Coverage specs to not filter the result if Ruby >= 2.4.
45935f3 Merge pull request #225 from nobu/bug/Dir.pwd-C
6d5daba Merge pull request #226 from ruby/try_with_gmp
7f04132 Fix Dir.pwd encoding
f3a55e1 libgmp-devel apt package
f90428f UNIX sockets only on UNIX
dfdb6d0 Make clear that not.existing is not an extension
b0bedcb Avoid warning by redirecting stderr on Windows
2ceb3bd Avoid changing LC_ALL, which has also no effect on Windows
26086ad Math.frexp returns an unspecified value for exp if value is NaN.
6cf4f32 File.expand_path does not care about the external Encoding on Windows
49701ab Try reading from the pipe to avoid warning on Windows
8d6f12a Use platform-specific @rootdir in File.expand_path spec
4e60d3d Avoid nonexistent pipe warning on Windows
24c26f9 Fix Dir.pwd encoding spec: it is the filesystem encoding.
e5406c1 Avoid newlines in File#open spec.
d21886d Disable autoclose if does not raise an error
1f5ccb6 Fix namespace.
d7c8746 Take in account imprecision of Math.gamma on Windows.
ec58c97 Use a simpler way to know if Kernel has a private method with -n.
4f8ed56 is not available on Windows
548e3c1 Avoid should_not raise_error.
40265a4 UNIXServer is not available on Windows
b0a70f1 Merge pull request #224 from ruby/vais/windows
4c24384 Pass Dir.mktmpdir spec on Windows
a0a0378 Adjust Socket constants and error classes for Windows
0d7762f not_supported_on is only for ruby implementations
9e87701 Adapt exception to Windows in recv_nonblock
bf0e472 Refactor IO#{read,write}_nonblock_spec when there is no data available
edaa5e7 Fix describe in Dir.mktmpdir spec
e6f91c7 IO#expect and the exepct stdlib is not available on Windows
3bb7369 Workaround the problem to test respond_to?(:fork) and being able to call it publicly.
e69370a Changing the time zone with Continent/City is unsupported on Windows
c81e224 Try to include private methods in the fork #respond_to? test
096cb4c Revert "Process.respond_to?(:fork) seems to be true now on Windows"
c6fcd2f Process.respond_to?(:fork) seems to be true now on Windows
c31500a Process.maxgroups is not on implemented on Windows
0a2ff59 Windows guards for symlinks
b5c2ed6 Guard UNIX-specific Signal.trap specs
d194a30 Windows guards in IO#write_nonblock
1f701d9 More IO on Windows fixes
774fb8f Only notify on status change for Travis
b8a8d3a Some IO on Windows fixes
c3c16bb More File on Windows fixes
618fa06 Fix File specs on Windows.
f5d45a0 Windows raises EISDIR for
dbe045a Open files with write mode for writing in File#open spec
2ea1128 Add Windows guards in File.{new,open}
cd048ee Fix shared stat specs.
53b8dbc with a directory is not supported on Windows.
d1a895e Skip incompatible Process#kill specs on Windows
6dd62b3 Separate Process#kill specs which can be supported on Windows
3dae14d Deduplicate definition of native newline.
130a11d Merge pull request #223 from ruby/vais/windows
86a2514 Pass Kernel#spawn spec on Windows
15b803c Pass Kernel#exec with a command array spec on Windows
e10c091 Merge pull request #222 from ruby/elia/explicit-fcall-as-default-arg
682e3c3 Use an ordinary Object instead of a String
18b9172 More explicit specs for fcall as default argument
f13db61 Merge pull request #221 from ruby/vais/windows
c716825 Pass Kernel#exec spec on Windows
80e5be3 Clearly separate non-compliant behavior on Windows using platform blocks
15c6b74 Pass Kernel#system spec on Windows
9049a75 Use a cross-platform alternative to system("false")
6fa70d1 Use a cross-platform alternative to system("true")
37677eb Pass spec on Windows
62e4e19 Pass IO#readlines spec on Windows
d84d68a Pass IO.foreach spec on Windows
1b2cdcf Fix grouping of #slice_when spec.

git-subtree-dir: spec/ruby
git-subtree-split: 32e22d667b12f10655701487312926f394d1e2e9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment