New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The future of RubySpec #3403

Closed
eregon opened this Issue May 19, 2015 · 13 comments

Comments

Projects
None yet
4 participants
@eregon
Contributor

eregon commented May 19, 2015

rubyspec/rubyspec archive branch has been merged with nurse/rubyspec into https://github.com/ruby/rubyspec.
MRI as well as JRuby now contribute to that repository and run the specs on every commit.
Rubinius has its own copy in spec/ruby and they keep improving as well.

In that context, we would like to open a discussion on what to do in the hope of sharing efforts rather than divide them.

Are you, the Rubinius team and community, willing to use and contribute to a shared RubySpec maintained at ruby/rubyspec, for the mutual benefit of all implementations?

Or are interested in sharing some specs both ways? In that case, how do you see that process happening?

A requirement for ruby/rubyspec is that specs committed pass on MRI or are reported as bugs on the issue tracker and tagged in between. I am sure there are expectations for both sides.
If we aim for fully merging both sets of spec, it means we should comply to each other's requirements.

From a technical point of view, JRuby uses a git-subtree, which is much easier to manage than a submodule or a raw copy and patches. It allows to maintain a pseudo-repository with only the specs, so it's easier to merge both ways and to work on the specs.

@jemc

This comment has been minimized.

Member

jemc commented May 19, 2015

The only possible sticking point I see from my point of view is a facet of the contribution policy that https://github.com/ruby/rubyspec has adopted:

Do you receive pull requests?

Yes. If the pull request won't break anything, it will be merged.

I'm all for having an inclusive contribution policy, but I also think it's important to maintain some kind of standard of quality for the tests, both in code style and in substance. I know the issue of who was judging this quality and by what standards was a source of part of the strife in recent events regarding RubySpec.

I see no issue with sharing specs written in Rubinius' repository upstream into https://github.com/ruby/rubyspec, but it sounds like sharing downstream into Rubinius would require some quality review, which may be more of a time and effort commitment.

These are just my thoughts on the matter as a Rubinius contributor, not as an official spokesperson for Rubinius.

EDIT: I should add that I personally am glad to see MRI and JRuby continuing work on the RubySpec project and I hope that it continues to work toward becoming a more complete specification for Ruby, including new features of MRI as they are implemented.

@YorickPeterse

This comment has been minimized.

Member

YorickPeterse commented May 19, 2015

While we currently still use the same structure as when RubySpec was around there's no guarantee for this to stick around forever. In fact, both @brixen and I have been discussing potential alternatives to the RubySpec based setup we're using today, although no definitive choices have been made to date.

Are you, the Rubinius team and community, willing to use and contribute to a shared RubySpec maintained at ruby/rubyspec, for the mutual benefit of all implementations?

Or are interested in sharing some specs both ways? In that case, how do you see that process happening?

Due to the above: most likely not. This may come as a surprise to some but we've decided to have our tests focus on what we need for Rubinius instead of making sure they also work for every other implementation out there. Again for now this won't make a huge difference but in the long term we might end up using a completely different testing setup.

If we aim for fully merging both sets of spec, it means we should comply to each other's requirements.

This may come off as negative, but I don't see why this would suddenly work out if it didn't work out for the past ~6 years or so. Simply changing ownership and making people abide by the laws of MRI isn't going to solve the problems that were present before.

So to summarize: most likely not (unless both @brixen and I suddenly change our minds), although pull requests going both ways are not out of the question until we change our spec setup (if we even want to do this). This should be discussed on a per case basis as some PRs might not make sense for us or ruby/rubyspec.

@eregon

This comment has been minimized.

Contributor

eregon commented May 19, 2015

@jemc I agree that a certain quality of specifications is required to keep the project maintainable. In my experience, this is an effort of both the requester and the person in charge of merging, since some changes are just simpler to do directly than asking for it and wait another round.
This is currently in discussion for ruby/rubyspec.

@YorickPeterse

This comment has been minimized.

Member

YorickPeterse commented May 19, 2015

One thing worth mentioning that is already somewhat incompatible: we're working on removing all ruby_version and ruby_bug guards as we're using separate branches for Ruby versions that we implement. This alone will already make synchronization more difficult.

@eregon

This comment has been minimized.

Contributor

eregon commented May 19, 2015

@YorickPeterse Of course, if the spec setup changes we will have to revisit this. But currently we are both using a fairly compatible version of MSpec, so I am confident most existing specs would work on both sides without too much efforts, setup-wise.

Due to the above: most likely not. This may come as a surprise to some but we've decided to have our tests focus on what we need for Rubinius instead of making sure they also work for every other implementation out there.

Not every other implementation, just the reference implementation.
I do not think this goal of focusing on what you need for Rubinius is incompatible with specs also passing on MRI. It just means that Rubinius-specific specs would not be shared.

This may come off as negative, but I don't see why this would suddenly work out if it didn't work out for the past ~6 years or so. Simply changing ownership and making people abide by the laws of MRI isn't going to solve the problems that were present before.

Yeah, "abide by the laws" is rather strong. What we want is to keep compatibility with the reference implementation for specified behavior, for various reasons.
One criticism of MRI is actually some of the existing specs specify behavior they think is implementation-dependent and should not be specified in a common suite of specs (or with more margin). So they also aspire for specs that truly define the semantics of Ruby and are not implementation-dependent.

For one thing, the problem of MRI not running the updated specs does not exist anymore.
Also, even though it was not great in the recent years, I think this could change if the contributions were coming from different sides and the burden of maintaining the specs was not the job of mostly a single person.

@eregon

This comment has been minimized.

Contributor

eregon commented May 19, 2015

@YorickPeterse

One thing worth mentioning that is already somewhat incompatible: we're working on removing all ruby_version and ruby_bug guards as we're using separate branches for Ruby versions that we implement. This alone will already make synchronization more difficult.

We are also considering removing ruby_bug guards, in favor of tags.
MRI would like to keep specs running on 2.0, since it is still maintained. A branch would be a possibility as there significant differences between 2.0 and 2.1+.
Most recent versions (2.1+) are very similar though in that they have very few version guards.

I did a good part of the merge between rubyspec/rubyspec and nurse/rubyspec, including the removal of quite a few version guards and while it was not trivial, it did not generate huge conflicts either for the majority of the specs.

@YorickPeterse

This comment has been minimized.

Member

YorickPeterse commented May 19, 2015

But currently we are both using a fairly compatible version of MSpec, so I am confident most existing specs would work on both sides without too much efforts, setup-wise.

Based on the troubles we've had in the past I have serious doubts about this. Using something like git submodule or git subtree wouldn't really work in our case as we often get patches on rubinius/rubinius that we then have to sync back to upstream RubySpec (which neither submodule/subtree really helps with if I'm not mistaken).

For one thing, the problem of MRI not running the updated specs does not exist anymore.

It might not exist for now, but looking at the past I find it hard to believe it will stay up to date this time. Of course I could be wrong, but I'm very skeptical.

Also, even though it was not great in the recent years, I think this could change if the contributions were coming from different sides and the burden of maintaining the specs was not the job of mostly a single person.

Various people have made claims that the troubles were due to specific individuals, other claimed they were met with "undue difficulty" by maintainers. If this was the case I doubt anything would change as we (= Rubinius) are not going to exclude any of our contributors because of this.

In short, we may have interest again in RubySpec in the far future but we're not going to focus on synchronization and compatibility (with RubySpec as a project that is). Of course if one of us were to report a bug to MRI they might include a RubySpec style test, but RubySpec in general is not a priority for us any more.

@eregon

This comment has been minimized.

Contributor

eregon commented May 19, 2015

@YorickPeterse

git subtree wouldn't really work in our case as we often get patches on rubinius/rubinius that we then have to sync back to upstream RubySpec

Yes, we do have the same case in JRuby, and it worked fine so far. git subtree is designed to extract the history of a subdirectory.

It might not exist for now, but looking at the past I find it hard to believe it will stay up to date this time. Of course I could be wrong, but I'm very skeptical.

We might be talking of different things. I meant MRI will keep running ruby/rubyspec specs, and right now we just regularly sync between the JRuby specs and the main repository. I am not be skeptical about this process.
If you meant something else, please elaborate.

Various people have made claims that the troubles were due to specific individuals, other claimed they were met with "undue difficulty" by maintainers. If this was the case I doubt anything would change as we (= Rubinius) are not going to exclude any of our contributors because of this.

There is no plan to exclude any contributor and I am not making any claim on this.
My point in that sentence is that maintaining RubySpec in good shape (that is reviewing contributions, global code cleanup, etc) takes a significant amount of work and if it can be shared between multiple persons, it would likely make the task more bearable.

In short, we may have interest again in RubySpec in the far future but we're not going to focus on synchronization and compatibility (with RubySpec as a project that is).

Does Rubinius have interest in existing specs contributed to ruby/rubyspec?

How would this be if you do not need to care about synchronization?

Do you mean compatibility with MRI?

@YorickPeterse

This comment has been minimized.

Member

YorickPeterse commented May 19, 2015

We might be talking of different things. I meant MRI will keep running ruby/rubyspec specs, and right now we just regularly sync between the JRuby specs and the main repository. I am not be skeptical about this process.

Very similar claims have been made in the past, for example around the time of Ruby 1.9. Seeing how things have gone since I have little faith in RubySpec being adopted properly this time. By this I don't just mean running the specs, I mean actively working on it and considering it just as important as MRI's current test suite (or even better: as the test suite).

There is no plan to exclude any contributor and I am not making any claim on this.
My point in that sentence is that maintaining RubySpec in good shape (that is reviewing contributions, global code cleanup, etc) takes a significant amount of work and if it can be shared between multiple persons, it would likely make the task more bearable.

In the past very few bothered to contribute to RubySpec. In fact, the amount of people actively working on it can be counted on one hand. I don't see how an MRI fork of RubySpec is going to make this any easier unless said fork also includes different quality standards.

Does Rubinius have interest in existing specs contributed to ruby/rubyspec?
How would this be if you do not need to care about synchronization?

We might pull specs from ruby/rubyspec once in a while if they're suitable. However, depending on our needs/preferences we might still have to adjust them accordingly. As such it's hard to judge until such a case presents itself.

Do you mean compatibility with MRI?

No, I specifically referred to compatibility with ruby/rubyspec. With this I mean us writing and submitting specs that can be run directly in ruby/rubyspec and vice-versa. Compatibility with MRI is not directly tied into RubySpec, RubySpec is simply a tool that can be used to assert compatibility between Ruby implementations.

Having said that I'm going to summarize/wrap this up as following:

For the foreseeable future Rubinius will not be actively involved in ruby/rubyspec. We might submit one time specs or pull them into rubinius/rubinius, but we're not going to actively synchronize specs as we did with rubyspec/rubyspec. We might run the specs from ruby/rubyspec once in a while, but our primary test suite is the one as found in our current spec/ directory.

Perhaps in the future we might change our minds, but this is simply not a priority for us (any more). Instead we want to focus on things such as getting the JIT, GC and C API in a better shape, wrapping up Ruby 2.x support and working towards Rubinius 3. Using ruby/rubyspec might help with this but as mentioned it will probably be in the form of cherry picking what we need into our own repository.

Having said all that I'll close this issue [1], but feel free to ask for more information if needed.

[1]: This way we don't litter the list of active issues, which is already big enough.

@eregon

This comment has been minimized.

Contributor

eregon commented May 19, 2015

By this I don't just mean running the specs, I mean actively working on it and considering it just as important as MRI's current test suite (or even better: as the test suite).

For the foreseeable future I do think MRI will keep its existing test suite. Both MRI tests and specs are useful on their own and I think even if one is less maintained it does keep an important value such that it is worth testing against it given the large coverage and the different merits.

Thank you for your detailed answers.
With that, it is clear that Rubinius is not interested anymore in contributing directly to a common set of "Ruby specs".

@eregon

This comment has been minimized.

Contributor

eregon commented May 20, 2015

I would like to hear @brixen on this subject though, as he is the main contributor of RubySpec.

@devinus

This comment has been minimized.

devinus commented May 24, 2015

As an outside observer, this has been fascinating. I was brought here from this discussion. It really seems that Rubinius has had a major political victory (i.e. it sounds like MRI is now doing exactly what Rubinius wanted them to do in the first place), but @YorickPeterse it doesn't sound like you're celebrating and using it to your advantage. In fact,

We might run the specs from ruby/rubyspec once in a while, but our primary test suite is the one as found in our current spec/ directory.

...this is the exact attitude MRI Ruby had that both yourself and @brixen railed against. My head is spinning from the role reversal right now.

@YorickPeterse

This comment has been minimized.

Member

YorickPeterse commented May 25, 2015

@devinus We have clearly stated our opinion on this matter. We've spent years on this and we've moved on as it was a huge drain of time. I'm locking this issue to prevent any further bike shedding.

@rubinius rubinius locked and limited conversation to collaborators May 25, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.