Add a spec-runner configuration that tests all of Ruby Core

[lopopolo]

It runs to completion!

Passed 4500, skipped 1097, not implemented 774, failed 5188 specs.

Gist with the full error log: https://gist.github.com/lopopolo/d7f758a4482a64ede6bb8a77918baba6.

There are still a few segfaults in there, but as configured, the specs reliably execute to completion.

This change adds a self-referential Hash check to Hash#inspect to prevent a stack overflow.

This change modifes mrb_convert_type to prevent a recursive attempt to raise a TypeError when attempting to convert a type to String that doesn't implement #to_s. This was fixed upstream in mruby/mruby@01a8d84.

The all-core-specs.toml configuration enables all top-level specs in spec-runner/vendor/spec/core except:

• IO (attempts to instantiate a binary Regexp that aborts the spec-runner)
• Kernel (infinite loop somewhere in there)
• Mutex
• ObjectSpace
• Process
• Thread
• ThreadGroup
• TracePoint

This config also copies the permitted spec list from enforced-specs.toml for Array. Array#initialize specs have an exception that crashes the spec-runner.

[lopopolo]

Hi @eregon 👋

This PR adds a configuration profile for Artichoke's spec runner that tests a wider swath or Ruby Core APIs.

I was curious if you'd be willing to add a note to https://eregon.me/blog/2020/06/27/ruby-spec-compatibility-report.html that points folks at this PR for a significant update.

[eregon]

In the output above, does it mean 4500 spec examples (it) ran successfully, and then skipped/not implemented/failed are all added on top, so the sum of all of those is how many specs were attempted (all of core/ minus exclusions)?
There were 20841 core specs when I did the report, so I guess the exclusions above account for about half of the core specs.
A nice gain indeed if Artichoke now passes around 22% of core specs.

It'd be great to have this run in CI if possible, and ideally separate for language/core/library specs.

[lopopolo]

In the output above, does it mean 4500 spec examples (it) ran successfully, and then skipped/not implemented/failed are all added on top, so the sum of all of those is how many specs were attempted (all of core/ minus exclusions)?

@eregon yes that's correct.

There were 20841 core specs when I did the report

Artichoke vendors the spec repo as of artichoke/spec@eebd7ab which might change the number of specs as well.

It'd be great to have this run in CI if possible, and ideally separate for language/core/library specs.

Yes this is something I'd like to do, but haven't gotten around to it. Linking #101 and #142.

[lopopolo]

Tweaking the config a bit, I get these numbers

Passed 4669, skipped 1125, not implemented 873, failed 5335 specs.

but also a reliable segfault on interpreter teardown.

all-core-specs.toml

# This config file lists the ruby/specs that should pass for Artichoke Ruby.
#
# Valid values for `include` are:
#
# - all - run all specs. When `include` is set to `all`, the optional `skip`
#   field may list specs to skip.
# - none - run no specs, equivalent to the section not being present in this
#   config file.
# - `set` - run an enumerated set of specs. When `include` is set to `set`, the
#   set of specs must be listed in the required `specs` field, which is a list
#   of strings.

## Ruby Core

[specs.core.argf]
include = "all"

[specs.core.array]
include = "all"
skip = [
  "initialize",
  "new",
  "pack",
]

[specs.core.basicobject]
include = "all"

[specs.core.binding]
include = "all"

[specs.core.builtin_constants]
include = "all"

[specs.core.class]
include = "all"

[specs.core.comparable]
include = "all"

[specs.core.complex]
include = "all"

[specs.core.dir]
include = "all"

[specs.core.encoding]
include = "all"

[specs.core.enumerable]
include = "all"

[specs.core.enumerator]
include = "all"

[specs.core.env]
include = "all"

[specs.core.exception]
include = "all"

[specs.core.false]
include = "all"

[specs.core.fiber]
include = "all"

[specs.core.file]
include = "all"

[specs.core.filetest]
include = "all"

[specs.core.float]
include = "all"

[specs.core.gc]
include = "all"

[specs.core.hash]
include = "all"

[specs.core.integer]
include = "all"

[specs.core.io]
include = "none"

[specs.core.kernel]
include = "none"

[specs.core.main]
include = "all"

[specs.core.marshal]
include = "all"

[specs.core.matchdata]
include = "all"

[specs.core.math]
include = "all"

[specs.core.method]
include = "all"

[specs.core.module]
include = "all"

[specs.core.mutex]
include = "none"

[specs.core.nil]
include = "all"

[specs.core.numeric]
include = "all"

[specs.core.objectspace]
include = "none"

[specs.core.proc]
include = "all"

[specs.core.process]
include = "none"

[specs.core.queue]
include = "all"

[specs.core.random]
include = "all"

[specs.core.range]
include = "all"

[specs.core.rational]
include = "all"

[specs.core.regexp]
include = "all"

[specs.core.signal]
include = "all"

[specs.core.sizedqueue]
include = "all"

[specs.core.string]
include = "all"

[specs.core.struct]
include = "all"

[specs.core.symbol]
include = "all"
skip = [
  "element_reference",
]

[specs.core.systemexit]
include = "all"

[specs.core.thread]
include = "none"

[specs.core.threadgroup]
include = "none"

[specs.core.time]
include = "all"

[specs.core.tracepoint]
include = "none"

[specs.core.true]
include = "all"

[specs.core.unboundmethod]
include = "all"

[specs.core.warning]
include = "all"

FWIW, this is why I don't like prioritizing getting the spec-runner in CI. I don't like chasing segfaults and I'd rather spend my time working toward removing Artichoke's dependency on mruby altogether instead of making the FFI bits less segfaulty.

[eregon]

I added a sentence in the blog post:

Update (February 2021): Artichoke now passes an estimated 22% of core library specs when trying to run all core specs with this PR, however those are not run in CI and might regress.

I think all other Rubies actually run specs in CI (important to avoid regressions), and in that blog post I relied on Ruby implementations untagging passing specs rather than trying to run all specs blindly and ignore errors, so I wanted to clarify how these numbers differ from the rest.

[lopopolo]

Thanks @eregon!

I know Artichoke doesn't integrate with ruby/spec and MSpec the way it's intended.

It's still early days for Artichoke; when the time is right, these bits of polish will get prioritized.

[lopopolo]

I think all other Rubies actually run specs in CI (important to avoid regressions), and in that blog post I relied on Ruby implementations untagging passing specs rather than trying to run all specs blindly and ignore errors, so I wanted to clarify how these numbers differ from the rest.

Hi @eregon 👋

I'm excited to share that Artichoke runs ruby/spec on every commit as of #1328, #1329, #1330, #1331, #1332, #1333.

I've filed ruby/spec#844 to reflect this upstream in the ruby/spec README. 🚀

Artichoke doesn't have File so I've kinda had to roll my own tagging. You can see the current set of tags here: https://github.com/artichoke/spec-state/blob/trunk/reports/tagged/latest.json.

[eregon]

That's great!
Is there a place I can look at to easily find out how many language, core, and library are passing?
That would be quite useful for the next time I do an update like https://eregon.me/blog/2020/06/27/ruby-spec-compatibility-report.html

[lopopolo]

Is there a place I can look at to easily find out how many language, core, and library are passing?

Not yet. I'm planning on turning the raw data into a webpage at some point.

The suite currently only runs a subset of core specs. I'm working on being able to expand this to include some library specs #1325, #1327.

I've never run any language specs before, but I imagine getting them running wouldn't be too tricky.

Note: this isn't my biggest priority, but I am excited that this bit of infrastructure is in place to do things automatically. The hardest part is done I think 😄