Crash in mrb_str_lines

[clayton-shopify]

The following input causes a crash:

("\n" * 5000).lines { break }

This issue was reported by https://hackerone.com/bee13oy

[EiNSTeiN-]

The crash occurs in mrb_class but this seems to be a red herring. I fixed the initial bad dereference like this:

diff --git a/include/mruby/class.h b/include/mruby/class.h
index ce953af..1af5dc5 100644
--- a/include/mruby/class.h
+++ b/include/mruby/class.h
@@ -45,6 +45,7 @@ mrb_class(mrb_state *mrb, mrb_value v)
   case MRB_TT_CPTR:
     return mrb->object_class;
   case MRB_TT_ENV:
+  case MRB_TT_UNDEF:
     return NULL;
   default:
     return mrb_obj_ptr(v)->c;

It occured because v is a MRB_TT_UNDEF.

Digging further it seems the root cause is mrb_method_missing ends up calling mrb_any_to_s(mrb, self) with self being a MRB_TT_UNDEF which ends up calling mrb_obj_classname with a NULL class.

[EiNSTeiN-]

The crash occurs in mrb_class but this seems to be a red herring. I fixed the initial bad dereference like this:

diff --git a/include/mruby/class.h b/include/mruby/class.h
index ce953af..1af5dc5 100644
--- a/include/mruby/class.h
+++ b/include/mruby/class.h
@@ -45,6 +45,7 @@ mrb_class(mrb_state *mrb, mrb_value v)
   case MRB_TT_CPTR:
     return mrb->object_class;
   case MRB_TT_ENV:
+  case MRB_TT_UNDEF:
     return NULL;
   default:
     return mrb_obj_ptr(v)->c;

It occured because v is a MRB_TT_UNDEF.

Digging further it seems the root cause is mrb_method_missing ends up calling mrb_any_to_s(mrb, self) with self being a MRB_TT_UNDEF which ends up calling mrb_obj_classname with a NULL class.

[clayton-shopify]

Another thing I noticed is that lines should terminate after the first break but it seems to carry on.

[clayton-shopify]

Another thing I noticed is that lines should terminate after the first break but it seems to carry on.

[clayton-shopify]

It looks like self gets garbage collected halfway through mrb_str_lines.

[clayton-shopify]

It looks like self gets garbage collected halfway through mrb_str_lines.

[matz]

The source of the problem is mrb_yield_argv() does not support break from the block.
String#lines (with a block) should be implemented in Ruby, but I don't have an efficient implementation yet.

[matz]

The source of the problem is mrb_yield_argv() does not support break from the block.
String#lines (with a block) should be implemented in Ruby, but I don't have an efficient implementation yet.

[clayton-shopify]

OK. Do you have a suggestion for how to stop the break from corrupting program state?

[clayton-shopify]

OK. Do you have a suggestion for how to stop the break from corrupting program state?

[matz]

Not yet, sorry.

[matz]

Not yet, sorry.

[ksss]

I tried to write String#each_line by Ruby.

class String
  def each_line(rs = $/, &block)
    return to_enum(:each_line, rs) unless block
    return block.call(self) if rs.nil?

    unless rs.respond_to?(:to_str)
      raise TypeError, "no implicit conversion of #{rs.class} into String"
    end
    rs_str = rs.to_str
    unless rs_str.kind_of?(String)
      raise TypeError, "no implicit conversion of #{rs.class} into String"
    end

    paragraph_mode = false
    if rs_str.empty?
      rs_str = "\n\n"
      paragraph_mode = true
    end

    off = i = 0
    this = dup
    len = length
    rs_len = rs_str.length
    while i < len
      i = index(rs_str, off) || len
      i += rs_len
      if paragraph_mode
        i += 1 while this[i, 1] == "\n"
      end
      block.call(this[off, i - off])
      off = i
    end

    self
  end
end

You know, It was slow about x10 over.

def benchmark
  t = Time.now
  yield
  puts "#{Time.now - t}s"
end

str = "\n" * 30000

benchmark do
  str.lines{}
end

benchmark do
  str.each_line{}
end

0.008395s
0.094594s

But, It works.

1.times do
  ("\n" * 5000).each_line { |i| p i; break }
end
# => "\n"

String#lines with block is deprecated method in CRuby. (Probably from 2.0.0 https://bugs.ruby-lang.org/issues/6670)

$ ruby -we '("\n").lines{}'
-e:1: warning: passing a block to String#lines is deprecated

This is why I implement String#each_line.
And also String#lines will be able to write like this.

class String
  def lines(rs = $/, &block)
    if block
      each_line(rs, &block)
    else
      each_line(rs).to_a
    end
  end
end

I hope this helps. Thanks.

[ksss]

I tried to write String#each_line by Ruby.

class String
  def each_line(rs = $/, &block)
    return to_enum(:each_line, rs) unless block
    return block.call(self) if rs.nil?

    unless rs.respond_to?(:to_str)
      raise TypeError, "no implicit conversion of #{rs.class} into String"
    end
    rs_str = rs.to_str
    unless rs_str.kind_of?(String)
      raise TypeError, "no implicit conversion of #{rs.class} into String"
    end

    paragraph_mode = false
    if rs_str.empty?
      rs_str = "\n\n"
      paragraph_mode = true
    end

    off = i = 0
    this = dup
    len = length
    rs_len = rs_str.length
    while i < len
      i = index(rs_str, off) || len
      i += rs_len
      if paragraph_mode
        i += 1 while this[i, 1] == "\n"
      end
      block.call(this[off, i - off])
      off = i
    end

    self
  end
end

You know, It was slow about x10 over.

def benchmark
  t = Time.now
  yield
  puts "#{Time.now - t}s"
end

str = "\n" * 30000

benchmark do
  str.lines{}
end

benchmark do
  str.each_line{}
end

0.008395s
0.094594s

But, It works.

1.times do
  ("\n" * 5000).each_line { |i| p i; break }
end
# => "\n"

String#lines with block is deprecated method in CRuby. (Probably from 2.0.0 https://bugs.ruby-lang.org/issues/6670)

$ ruby -we '("\n").lines{}'
-e:1: warning: passing a block to String#lines is deprecated

This is why I implement String#each_line.
And also String#lines will be able to write like this.

class String
  def lines(rs = $/, &block)
    if block
      each_line(rs, &block)
    else
      each_line(rs).to_a
    end
  end
end

I hope this helps. Thanks.

[clayton-shopify]

Implementing lines in Ruby won't solve the problem because the are other things that call mrb_yield_with_class, for instance instance_exec, Class.initialize and Kernel.initialize. Passing a block containing break to any of these can also cause a crash.

[clayton-shopify]

Implementing lines in Ruby won't solve the problem because the are other things that call mrb_yield_with_class, for instance instance_exec, Class.initialize and Kernel.initialize. Passing a block containing break to any of these can also cause a crash.

[matz]

@clayton-shopify FYI, currently String#lines is an only C implemented method that calls mrb_yield repeatedly. Other methods do not have the problem. Of course, we need to address the root cause for the future use-case.

[matz]

@clayton-shopify FYI, currently String#lines is an only C implemented method that calls mrb_yield repeatedly. Other methods do not have the problem. Of course, we need to address the root cause for the future use-case.

[clayton-shopify]

@matz There seems to be trouble in other places that use mrb_yield though. These inputs all demonstrate segfaults:

loop do
  Struct.new { break }
  break
end
puts [0].to_s

loop do
  instance_exec { break }
  break
end
puts [0].to_s

loop do
  ObjectSpace.each_object { break }
  break
end
puts [0].to_s

loop do
  Class.initialize { break }
  break
end
puts [0].to_s

loop do
  Module.initialize { break }
  break
end
puts [0].to_s

[clayton-shopify]

@matz There seems to be trouble in other places that use mrb_yield though. These inputs all demonstrate segfaults:

loop do
  Struct.new { break }
  break
end
puts [0].to_s

loop do
  instance_exec { break }
  break
end
puts [0].to_s

loop do
  ObjectSpace.each_object { break }
  break
end
puts [0].to_s

loop do
  Class.initialize { break }
  break
end
puts [0].to_s

loop do
  Module.initialize { break }
  break
end
puts [0].to_s

[matz]

Those programs now do not crash. But some of them behave differently.
I will investigate.

[matz]

Those programs now do not crash. But some of them behave differently.
I will investigate.

[matz]

I think I got the idea to solve the issue, but it would take some time, probably after 1.3 release.

[matz]

I think I got the idea to solve the issue, but it would take some time, probably after 1.3 release.