Ruby 2.7 support

[deepj]

Is there any plan to add new Ruby 2.7 goodies in near future? In my list are these:

1. REPL improvements (the colors and syntax highlighting in IRB!!!)
2. pattern matching (+ possible some performance optimization behind MRI implementation)
3. Enumerator.produce

[eregon]

As written in
https://github.com/oracle/truffleruby/blob/master/doc/contributor/workflow.md#running-specs-for-ruby-27-features

we welcome pull requests for Ruby 2.7 features as long as they don't conflict significantly with Ruby 2.6 semantics.

So adding 2.7 features as long as they don't conflict significantly with Ruby 2.6 semantics is fine.

Regarding 1, I guess it's mostly changes the irb gem. So the work there seems mostly to support the latest irb gem. Trying gem install irb; irb shows we need Ripper.lex (#1747, #1585)

Regarding 2, it seems quite some work and it's unclear if much code uses it yet.
Also it's experimental in 2.7:

$ ruby -ve 'case [3]; in [e]; p e; end'
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
-e:1: warning: Pattern matching is experimental, and the behavior may change in future versions of Ruby!
3

So PRs for that are definitely welcome, but I guess it's not used in many gems yet, and so not so important for compatibility in general (yet).

Regarding 3, that's probably easy to implement in pure Ruby.

For each of these it would be helpful to have a separate issue so we can prioritize and triage it as needed.

---

Finally, regarding general plans for 2.7 at the moment I'm hesitant because of keyword arguments changes.
Ruby 2.7 has broken *args-delegation and *args, **kwargs-delegation is not working in 2.7 (so the only thing that works is ruby2_keywords, details). As a result, it's hard to migrate and fix warnings in Ruby 2.7.
So I'm not sure if much production apps will ever run on 2.7, it's unclear to me if they would rather wait for 2.8/3.0 which fixes *args, **kwargs-delegation and has much clearer semantics for keyword arguments in general.

Implementing the keyword arguments semantics of 2.7 seem a lot of work (at least to support the warnings), for semantics that only last for one Ruby version (semantics are significantly simpler in Ruby 2.8+). So we might want to skip the 2.7 "transition release" and go straight to Ruby 2.8 in terms of reported RUBY_VERSION. I've not made my mind yet though and I'mhappy to hear opinions on this (e.g., significant adoption of 2.7 would change things).

Individual features from 2.7 that are compatible are all welcome though.
I guess it doesn't make sense to report as RUBY_VERSION 2.7 if the keyword arguments changes are not implemented though.

[eregon]

We're now planning to target Ruby 2.7 for the GraalVM/TruffleRuby 21.0.0 release.

[eregon]

I merged initial support for Ruby 2.7 in 08187f1.
That updated the stdlib, default and bundled gems, and some basic fixes needed to get the tests running.
The new IRB seems to work well.
Bundler is stdlib is now Bundler 2 like MRI.

Remaining:

• Implement keyword arguments changes (https://github.com/ruby/ruby/blob/v2_7_0/NEWS#the-spec-of-keyword-arguments-is-changed-towards-30-) @eregon
• Implement def m(**nil) (9aaf6be allows to parse it but treats it like **kw)
• Implement the new *_kw functions in the C API.
• Implement beginless Ranges. @LillianZ Implement beginless range #2155
• Implement pattern matching (d58c943, a3a2c85) @razetime
• Remove tainting since it seems deprecated and noop in 2.7. @bjfish

New tagged specs:

• de71fac
• 107863c
• 48cb4bd
• cfedc31
• 0434964
• 89f2119

New excluded MRI tests:

• c58c460
• 94da437
• 3fa29ed

[eregon]

I'll also copy the checklist from ruby/spec#745 as it might be useful to track progress, and to show individual things to fix.
Some of these items might already be done.

NOTE: https://rubyreferences.github.io/rubychanges/2.7.html gives more details for many features and changes.
From https://github.com/ruby/ruby/blob/v2_7_0/NEWS:

NEWS for Ruby 2.7.0

This document is a list of user visible feature changes made between
releases except for bug fixes.

Changes since the 2.6.0 release

Language changes

Pattern matching

• @razetime Pattern matching is introduced as an experimental feature. Feature #14912

case [0, [1, 2, 3]]
in [a, [b, *c]]
  p a #=> 0
  p b #=> 1
  p c #=> [2, 3]
end

case {a: 0, b: 1}
in {a: 0, x: 1}
  :unreachable
in {a: 0, b: var}
  p var #=> 1
end

case -1
in 0 then :unreachable
in 1 then :unreachable
end #=> NoMatchingPatternError

json = <<END
{
  "name": "Alice",
  "age": 30,
  "children": [{ "name": "Bob", "age": 2 }]
}
END

JSON.parse(json, symbolize_names: true) in {name: "Alice", children: [{name: name, age: age}]}

p name #=> "Bob"
p age  #=> 2

JSON.parse(json, symbolize_names: true) in {name: "Alice", children: [{name: "Charlie", age: age}]}
#=> NoMatchingPatternError

See the following slides for more details:

• https://speakerdeck.com/k_tsj/pattern-matching-new-feature-in-ruby-2-dot-7
• Note that the slides are slightly obsolete.

• The warning against pattern matching can be suppressed with
  {-W:no-experimental option}[#label-Warning+option].

The spec of keyword arguments is changed towards 3.0

• Non-symbols are allowed as keyword argument keys if the method accepts
  arbitrary keywords. Feature #14183

  Non-Symbol keys in a keyword arguments hash were prohibited in 2.6.0,
  but are now allowed again. Bug #15658

def foo(**kw); p kw; end; foo("str" => 1) #=> {"str"=>1}

• **nil is allowed in method definitions to explicitly mark
  that the method accepts no keywords. Calling such a method with keywords
  will result in an ArgumentError. Feature #14183

def foo(h, **nil); end; foo(key: 1)       # ArgumentError
def foo(h, **nil); end; foo(**{key: 1})   # ArgumentError
def foo(h, **nil); end; foo("str" => 1)   # ArgumentError
def foo(h, **nil); end; foo({key: 1})     # OK
def foo(h, **nil); end; foo({"str" => 1}) # OK

• Passing an empty keyword splat to a method that does not accept keywords
  no longer passes an empty hash, unless the empty hash is necessary for
  a required parameter, in which case a warning will be emitted. Remove
  the double splat to continue passing a positional hash. Feature #14183

h = {}; def foo(*a) a end; foo(**h) # []
h = {}; def foo(a) a end; foo(**h)  # {} and warning
h = {}; def foo(*a) a end; foo(h)   # [{}]
h = {}; def foo(a) a end; foo(h)    # {}

• Above warnings can be suppressed also with {-W:no-deprecated option}[#label-Warning+option].

Numbered parameters

• Numbered parameters as default block parameters are introduced.
  Feature #4475

  [1, 2, 10].map { _1.to_s(16) }    #=> ["1", "2", "a"]
  [[1, 2], [3, 4]].map { _1 + _2 }  #=> [3, 7]

  You can still define a local variable named _1 and so on,
  and that is honored when present, but renders a warning.

  _1 = 0            #=> warning: `_1' is reserved for numbered parameter; consider another name
  [1].each { p _1 } # prints 0 instead of 1

proc/lambda without block is deprecated

• Proc.new and Kernel#proc with no block in a method called with a block is
  warned now.

  def foo
    proc
  end
  foo { puts "Hello" } #=> warning: Capturing the given block using Kernel#proc is deprecated; use `&block` instead

  This warning can be suppressed with {-W:no-deprecated option}[#label-Warning+option].

• Kernel#lambda with no block in a method called with a block raises an exception.

  def bar
    lambda
  end
  bar { puts "Hello" } #=> tried to create Proc object without a block (ArgumentError)

Other miscellaneous changes

• A beginless range is experimentally introduced. It might be useful
  in case, new call-sequence of the Comparable#clamp,
  constants and DSLs. Feature #14799

  ary[..3]  # identical to ary[0..3]
  
  case RUBY_VERSION
  when ..."2.4" then puts "EOL"
  # ...
  end
  
  age.clamp(..100)
  
  where(sales: ..100)

• Quoted here-document identifiers must end within the same line.

  <<"EOS
  " # This had been warned since 2.4; Now it raises a SyntaxError
  EOS
  

• The flip-flop syntax deprecation is reverted. Feature #5400

• Comment lines can be placed between fluent dot now.

  foo
    # .bar
    .baz # => foo.baz

• Calling a private method with a literal self as the receiver
  is now allowed. Feature #11297 Feature #16123

• Modifier rescue now operates the same for multiple assignment as single
  assignment. Bug #8279

  a, b = raise rescue [1, 2]
  # Previously parsed as: (a, b = raise) rescue [1, 2]
  # Now parsed as:         a, b = (raise rescue [1, 2])

• yield in singleton class syntax is warned and will be deprecated later. Feature #15575.

  def foo
    class << Object.new
      yield #=> warning: `yield' in class syntax will not be supported from Ruby 3.0. [Feature #15575](https://bugs.ruby-lang.org/issues/15575)
    end
  end
  foo { p :ok }

  This warning can be suppressed with {-W:no-deprecated option}[#label-Warning+option].

• Argument forwarding by (...) is introduced. Feature #16253

  def foo(...)
    bar(...)
  end

  All arguments to foo are forwarded to bar, including keyword and
  block arguments.
  Note that the parentheses are mandatory. bar ... is parsed
  as an endless range.

• Access and setting of $SAFE is now always warned. $SAFE
  will become a normal global variable in Ruby 3.0. Feature #16131

• Object#{taint,untaint,trust,untrust} and related functions in the C-API
  no longer have an effect (all objects are always considered untainted), and are now
  warned in verbose mode. This warning will be disabled even in non-verbose mode in
  Ruby 3.0, and the methods and C functions will be removed in Ruby 3.2. Feature #16131

• Refinements take place at Object#method and Module#instance_method. Feature #15373

Command line options

Warning option

The -W option has been extended with a following :, to manage categorized
warnings. Feature #16345 Feature #16420

• To suppress deprecation warnings:

  $ ruby -e '$; = ""'
  -e:1: warning: `$;' is deprecated
  
  $ ruby -W:no-deprecated -e '$; = //'
  

• It works with the RUBYOPT environment variable:

  $ RUBYOPT=-W:no-deprecated ruby -e '$; = //'
  

• To suppress experimental feature warnings:

  $ ruby -e '0 in a'
  -e:1: warning: Pattern matching is experimental, and the behavior may change in future versions of Ruby!
  
  $ ruby -W:no-experimental -e '0 in a'
  

• To suppress both by using RUBYOPT, set space separated values:

  $ RUBYOPT='-W:no-deprecated -W:no-experimental' ruby -e '($; = "") in a'
  

See also Warning in {Core classes updates}[#label-Core+classes+updates+-28outstanding+ones+only-29].

Core classes updates (outstanding ones only)

Array

• Added Array#intersection. Feature #16155

• Added Array#minmax, with a faster implementation than Enumerable#minmax. Bug #15929

Comparable

• Comparable#clamp now accepts a Range argument. Feature #14784

    ```ruby
    -1.clamp(0..2) #=> 0
     1.clamp(0..2) #=> 1
     3.clamp(0..2) #=> 2
    # With beginless and endless ranges:
    -1.clamp(0..)  #=> 0
     3.clamp(..2)  #=> 2
    ```
  

Complex

• Added Complex#<=>.
  So 0 <=> 0i will not raise NoMethodError. Bug #15857

Dir

• Dir.glob and Dir.[] no longer allow NUL-separated glob pattern.
  Use Array instead. Feature #14643

Encoding

• Added new encoding CESU-8. Feature #15931

Enumerable

• Added Enumerable#filter_map. Feature #15323

    [1, 2, 3].filter_map {|x| x.odd? ? x.to_s : nil } #=> ["1", "3"]
  

• Added Enumerable#tally. Feature #11076

    ["A", "B", "C", "B", "A"].tally #=> {"A"=>2, "B"=>2, "C"=>1}
  

Enumerator

• Added Enumerator.produce to generate an Enumerator from any custom
  data transformation. Feature #14781

    require "date"
    dates = Enumerator.produce(Date.today, &:succ) #=> infinite sequence of dates
    dates.detect(&:tuesday?) #=> next Tuesday
  

• Added Enumerator::Lazy#eager that generates a non-lazy enumerator
  from a lazy enumerator. Feature #15901

    a = %w(foo bar baz)
    e = a.lazy.map {|x| x.upcase }.map {|x| x + "!" }.eager
    p e.class               #=> Enumerator
    p e.map {|x| x + "?" }  #=> ["FOO!?", "BAR!?", "BAZ!?"]
  

• Added Enumerator::Yielder#to_proc so that a Yielder object
  can be directly passed to another method as a block
  argument. Feature #15618

Fiber

• Added Fiber#raise that behaves like Fiber#resume but raises an
  exception on the resumed fiber. Feature #10344

File

• File.extname now returns a dot string for names ending with a dot on
  non-Windows platforms. Bug #15267

      File.extname("foo.") #=> "."
  

FrozenError

• Added FrozenError#receiver to return the frozen object on which
  modification was attempted. To set this object when raising
  FrozenError in Ruby code, FrozenError.new accepts a :receiver
  option. Feature #15751

GC

• (MRI-specific) Added GC.compact method for compacting the heap.
  This function compacts live objects in the heap so that fewer pages may
  be used, and the heap may be more CoW (copy-on-write) friendly. Feature #15626

  Details on the algorithm and caveats can be found here:
  https://bugs.ruby-lang.org/issues/15626
  

IO

• Added IO#set_encoding_by_bom to check the BOM and set the external
  encoding. Bug #15210

Integer

• Integer#[] now supports range operations. Feature #8842

     0b01001101[2, 4]  #=> 0b0011
     0b01001100[2..5]  #=> 0b0011
     0b01001100[2...6] #=> 0b0011
     #   ^^^^
  

Method

• Method#inspect shows more information. Feature #14145

Module

• Added Module#const_source_location to retrieve the location where a
  constant is defined. Feature #10771

• Added Module#ruby2_keywords for marking a method as passing keyword
  arguments through a regular argument splat, useful when delegating
  all arguments to another method in a way that can be backwards
  compatible with older Ruby versions. Bug #16154

• Module#autoload? now takes an inherit optional argument, like
  Module#const_defined?. Feature #15777

• Module#name now always returns a frozen String. The returned String is
  always the same for a given Module. This change is
  experimental. Feature #16150

NilClass / TrueClass / FalseClass

• NilClass#to_s, TrueClass#to_s, and FalseClass#to_s now always return a
  frozen String. The returned String is always the same for each of these
  values. This change is experimental. Feature #16150

ObjectSpace::WeakMap

• ObjectSpace::WeakMap#[]= now accepts special objects as either key or
  values. Feature #16035

Proc

• Added Proc#ruby2_keywords for marking the proc as passing keyword
  arguments through a regular argument splat, useful when delegating
  all arguments to another method or proc in a way that can be backwards
  compatible with older Ruby versions. Feature #16404

Range

• Added Range#minmax, with a faster implementation than Enumerable#minmax.
  It returns a maximum that now corresponds to Range#max. Bug #15807

• Range#=== now uses Range#cover? for String arguments, too (in Ruby 2.6, it was
  changed from Range#include? for all types except strings). Bug #15449

String

• Update Unicode version and Emoji version from 11.0.0 to
  12.0.0. Feature #15321

• Update Unicode version to 12.1.0, adding support for
  U+32FF SQUARE ERA NAME REIWA. Feature #15195

• Update Unicode Emoji version to 12.1. Feature #16272

Symbol

• Added Symbol#start_with? and Symbol#end_with? methods. Feature #16348

Time

• Added Time#ceil method. Feature #15772

• Added Time#floor method. Feature #15653

• Time#inspect is separated from Time#to_s and it shows
  the time's sub second. Feature #15958
  (we don't support Rational subseconds)

UnboundMethod

• Added UnboundMethod#bind_call method. Feature #15955

  <code>umethod.bind_call(obj, ...)</code> is semantically equivalent
  to <code>umethod.bind(obj).call(...)</code>.  This idiom is used in
  some libraries to call a method that is overridden.  The added
  method does the same without allocation of an intermediate Method
  object.
  
      class Foo
        def add_1(x)
          x + 1
        end
      end
      class Bar < Foo
        def add_1(x) # override
          x + 2
        end
      end
  
      obj = Bar.new
      p obj.add_1(1) #=> 3
      p Foo.instance_method(:add_1).bind(obj).call(1) #=> 2
      p Foo.instance_method(:add_1).bind_call(obj, 1) #=> 2
  

Warning

• Added Warning.[] and Warning.[]= to manage emitting/suppressing
  some categories of warnings. Feature #16345 Feature #16420

$LOAD_PATH

• Added $LOAD_PATH.resolve_feature_path. Feature #15903 Feature #15230

Stdlib updates (outstanding ones only)

• Pathname.glob now delegates 3 arguments to Dir.glob
  to accept base keyword. Feature #14405

Compatibility issues (excluding feature bug fixes)

• The following libraries are no longer bundled gems.
  Install corresponding gems to use these features.
  • CMath (cmath gem)
  • Scanf (scanf gem)
  • Shell (shell gem)
  • Synchronizer (sync gem)
  • ThreadsWait (thwait gem)
  • E2MM (e2mmap gem)

Proc

• The Proc#to_s format was changed. Feature #16101

Range

• Add a tool to generate the native configuration #777 Range#minmax used to iterate on the range to determine the maximum.
  It now uses the same algorithm as Range#max. In rare cases (e.g.
  ranges of Floats or Strings), this may yield different results. Bug #15807

Stdlib compatibility issues (excluding feature bug fixes)

• Promote stdlib to default gems
  • The following default gems were published on rubygems.org
    • benchmark
    • cgi
    • delegate
    • getoptlong
    • net-pop
    • net-smtp
    • open3
    • pstore
    • readline
    • readline-ext
    • singleton
  • The following default gems were only promoted at ruby-core,
    but not yet published on rubygems.org.
    • monitor
    • observer
    • timeout
    • tracer
    • uri
    • yaml

• The did_you_mean gem has been promoted up to a default gem from a bundled gem

pathname

• Kernel#Pathname when called with a Pathname argument now returns
  the argument instead of creating a new Pathname. This is more
  similar to other Kernel methods, but can break code that modifies
  the return value and expects the argument not to be modified.

C API updates

• Many *_kw functions have been added for setting whether
  the final argument being passed should be treated as keywords. You
  may need to switch to these functions to avoid keyword argument
  separation warnings, and to ensure correct behavior in Ruby 3.

• The : character in rb_scan_args format string is now
  treated as keyword arguments. Passing a positional hash instead of
  keyword arguments will emit a deprecation warning.

• C API declarations with ANYARGS are changed not to use ANYARGS.
  See Delete ANYARGS ruby/ruby#2404

[ssnickolay]

I'll start with Added Complex#<=>., if you do not mind //cc @eregon

[norswap]

I'll take a stab at the rescue parser bug.

[norswap]

I'm marking "The flip-flop syntax deprecation is reverted." as done, since it doesn't seem we emitted the deprecation warning in the first place.

[LillianZ]

Could you also mark enumerator#produce as done per #2160 ? Thanks!

[gogainda]

taking Integer#[] now supports range operations

[LillianZ]

Pathname.glob now delegates 3 arguments to Dir.glob to accept base keyword. Feature #14405 already occurs in TruffleRuby, and thus can be removed/ticked off. The examples in the Redmine post al function.

[chrisseaton]

Are there specs for it? If so do they need to be untagged, if not we should add at least a couple.

[eregon]

Very few things left from the list above:

• Pattern matching is partially implemented, but in 2.7 it's experimental so likely not used much.
• Keyword arguments changes were done, except for some edge cases which make more sense to fix when targeting Ruby 3 (it would be work that would be basically thrown away when moving to Ruby 3)
• The new encoding CESU-8, needs support in JCodings.
• Fiber#raise
• IO#set_encoding_by_bom
• C-API *_kw functions

[gogainda]

I looked into C-API *_kw functions but didn't figure out how to approach this works. @eregon Can we do 1 function so I can work on the rest by the example?

[eregon]

@gogainda

truffleruby/src/main/c/cext/class.c

Lines 79 to 82 in 169bce9

	VALUE rb_class_new_instance_kw(int argc, const VALUE *argv, VALUE klass, int kw_splat) {
	// Ignoring kw_splat for now
	return rb_class_new_instance(argc, argv, klass);
	}

is an example.
The *_kw functions can be found by git grep -E 'rb_.+_kw\b' lib/cext/include/ruby.

[gogainda]

I thought that // Ignoring kw_splat for now should be replaced by actual code, no?

[eregon]

@gogainda At this point it seems unnecessary, because of how we handle keyword arguments in TruffleRuby.
I think we'll only need to deal with this when migrating to Ruby 3.0.
So it's OK to just ignore it and add this comment as a reminder when we move to Ruby 3.

[eregon]

I think it's time to mark this issue as solved, we already support most of 2.7 since 21.0.0/21.1.0.

Remaining from the list is:

• Pattern matching is partially implemented, but in 2.7 it's experimental so likely not used much.
• The new encoding CESU-8, needs support in JCodings.
• C-API *_kw functions, part of them are already there.

Those don't seem used much so I think we'll just pragmatically add them when we see an app/gem needing them.