New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minor: remove spec files from gem #1555

Closed
e2 opened this Issue Jan 7, 2015 · 26 comments

Comments

Projects
None yet
6 participants
@e2
Copy link
Contributor

e2 commented Jan 7, 2015

There little benefit in having 'spec' files installed with RuboCop every time.

@bbatsov

This comment has been minimized.

Copy link
Collaborator

bbatsov commented Jan 7, 2015

Most gems seem to ship their specs and that why we do so as well. I don't have a strong preference either way.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Jan 7, 2015

TL;DR - shipping specs is the wrong thing to do, no matter how you look at it. Lots of people are doing it wrong, unfortunately.

I've spent hours trying to figure out what "development" means for Ruby gems... and I finally "got it".

(I should write a blog about it ... haven't go to it yet).

In a nutshell, add_development_dependency is obsolete - nowadays it serves no purpose except as useless metadata on rubygems.org (long story). It especially doesn't make sense since RuboCop has bundler as a development dependency anyway.

And if you have 'bundler' as a development dependency, it makes sense to put all the dev deps in the 'Gemfile'.

So, the 'pendantic' thing to do would be: move dev deps (except 'bundler') from the gemspec to the Gemfile (into a :development group). This makes the gemspec a pure "runtime"/"release" thing used only for BUILDING (!) gems.

Next, since contributing to RuboCop requires a pull request, you need the source anyway (unless you expect people to somehow ... craft ... PRs ... from ... a ... diff ??).

And, even if tests fail for a released gem, the response is almost always "do they work on master?".

Most gems seem to ship their specs

That's because people are trying to do the right thing by copying other people instead of understanding what's going on (which is actually pretty tough given all the legacy stuff involved here).

The result is lots of wasted bandwidth, disk space (in terms of allocated space), access times (find/grep take ages), wasted developer time (trying to work out the right setup for Travis using --without=development) ... just because once upon a time (in a galaxy far, far away) ... in a pre-bundler/pre-github world ... people used to run gem install --dev.

(While their ancestors used to install gems using setup.rb.)

@bbatsov

This comment has been minimized.

Copy link
Collaborator

bbatsov commented Jan 11, 2015

Good arguments! I'm ok with updating the gemspec provided @jonas054 and @yujinakayama agree as well.

@jonas054

This comment has been minimized.

Copy link
Collaborator

jonas054 commented Jan 11, 2015

I agree too, based on the arguments presented by @e2, which seem thoroughly thought through. My own knowledge of specifying dependencies or shipping gems is limited, I find.

@swatosh

This comment has been minimized.

Copy link

swatosh commented Jan 11, 2015

One argument for including the specs (maybe not relevant to rubocop) is to provide actual working examples of usage.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Jan 11, 2015

TL;DR - even including docs in gems used to be a good idea, but now it's more of a problem

@swatosh - that should be in the README or in a examples subdir.

Specs are actually better for documenting behavior (e.g. doc formatting in RSpec) or testing edge cases - so they rarely make good usage guidelines. (Especially with isolated testing).

Acceptance tests are better - to the point that they are much prettier online (e.g. for relisapp.com). Regardless, reading updated documentation online is better anyway - especially since good docs coverage is often really heavy.

For documentation and examples, ri and rdoc should be better suited - and even then people put gem: --no-ri --no-rdoc in their ~/.gemrc to prevent them being generated locally. (For offline docs, gem server was the solution).

Not to mention: navigating to a installed gem location is troublesome.

I now think of gems as a 'versioned snapshot of code intended for apps, not humans'.

RubyGems wanted to be "more" a long time ago (doc, tests, dependencies, rubyforge), but that didn't work out well in the end.

If a gem's specs have to be read to get a working example - then that project seriously needs a PR for the Readme ;)

@bbatsov

This comment has been minimized.

Copy link
Collaborator

bbatsov commented Jan 25, 2015

@e2 You have our blessing to do the changes you suggested. :-)

@mvz

This comment has been minimized.

Copy link
Contributor

mvz commented Feb 2, 2015

As a counter-argument, Debian uses the tests/specs from the gem to run the tests when creating a .deb package.

As an aside, @e2, why do you conclude that add_development_dependency is obsolete? I recently read a discussion on a GitHub ticket for RubyGems (I can probably find it again), where the conclusion was that it specifies the gems needed for building the gem (e.g., rake-compiler).

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Feb 2, 2015

@bbatsov - I also removed the *.eps files. Makes the gem less "huge".

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Feb 2, 2015

TL;DR - why unit tests don't make sense when packaging debs + why dev deps are obsolete

@mvz - I can discuss the issue in depth, so feel free to ask me to explain myself more

Tests/specs basically don't make sense from a maintainer's point of view because:

  1. they are mostly unit tests, so they don't test dependencies with other packages or arch-specific stuff (we're talking about Ruby here)
  2. a maintainer needs (only) feature/acceptance or system integration tests - if those are not provided by the gem author, it's up to the gem author to deal with those issue (adding unit tests / acceptance tests)
  3. a maintainer's job is only to make sure the package is installed correctly with the right dependencies - unless they are building from a repository which has no official releases (which is not the case here)

So in this case, it makes no sense to run a gem's unit tests from a released gem. For a CI - it's better to clone the repo by tag anyway (since additional dev tools or steps may be needed for testing anyway).

"released" means "ready for use", which implies all the tests pass on every environment. Otherwise, you'd be using rubocop tests to test the environment, not the gem itself. (Instead it would make more sense to run the rubygems and bundler gem tests - which would be more thorough).

As for add_development_dependency, basically the meaning of "development" has changed over the years. It depends on the individual whether e.g. "testing" is part of "development" or not. Is packaging considered development? Usually, very few people can release a gem - so building only really makes sense for those 2-3 people (compared to the millions of downloads).

Historically, add_development_dependency is a pre-bundler thing, where you installed gems using gem install --dev. What it really should be called is: add_build_dependency - but that's not going to change any time soon. Again - is "building" considered "development"? If so, then PRs cannot be considered "development".

The point is - development workflows changed much faster than Rubygems can deal with (because it has to be backward-compatible with even the most ancient gems out there) - which is why add_development_dependency will likely never be deprecated.

And with bundler as a dev dependency anyway, it just makes more sense to put things like rake-compiler in the Gemfile.

As an analogy - you don't get a full development kit and manufacturing tests when you buy a smartphone. You get just the smartphone. Even if you're a re-seller.

As a side note, test_files was meant for gem install --test, which is no longer a valid option. That's because at first there was just Ruby's unit/test - then came all the various test frameworks (which made the option useless and it's why it was removed).

My point - learning by "copying others" doesn't work well where there's too much legacy baggage to deal with.

P.S. gem list rake-compiler shows nothing on my system - even though I have currently over 261 gems installed. Probably because I'm not an owner of a gem that needs compiling extensions - and if I was, I'd be the one creating the official gem in the first place - so why would I put gem build tool deps inside a gem that only I can really build? (the exception here are gems which cannot be built by Bundler from source, like zeus - because it needs golang build tools).

P.S.2 - I know people "hack" on code in installed gems to debug issues. A better approach is to immediately use Bundler with path sources and a checked out master branch - because the hacking/debugging is more likely to result in a good PR.

@bbatsov bbatsov closed this in 5e8199e Feb 28, 2015

@mvz

This comment has been minimized.

Copy link
Contributor

mvz commented Mar 2, 2015

@e2 your comment was indeed TL, so I DR it until now ;-).

Most of your arguments are based on an ideal world where gem authors test their gems on all platforms and versions of Ruby (1.9.3 on little-endian ARM, perhaps?), and where there is a perfect separation of unit and feature/integration tests. This world, however, does not exist.

In reality, tests are a very useful tool to find out hidden bugs and dependencies before they hit the end-user of a package. If Debian were to make CRuby 2.2 their default Ruby implementation, how else are they going to find out if the existing packages at their existing versions will still work?

@bbatsov

This comment has been minimized.

Copy link
Collaborator

bbatsov commented Mar 2, 2015

Btw, bundler no longer includes a tests stub for new gems, so I guess there consensus that the tests don't belong there. People repackaging gems can always make use of the original test suite IMO.

@andremedeiros

This comment has been minimized.

Copy link

andremedeiros commented Mar 2, 2015

Actually, if I may drop my $0.02 here:

@bbatsov: Bundler will generate a tests stub, if you so choose it. We don't force it onto you, the same way we don't force a license or the code of conduct anymore. You choose how you want it.

As for the whole To Include or not To Include [specs] argument, I'm in favour of including them. There's a lot to be learned from a gem's specs (even if you don't run them), and as an avid user of bundle open, I tend to read a lot of them. Sorry, let me rephrase that. I tend to read a LOT of them.

As for moving away from add_development_dependency, I mostly agree with you that it should be within a development group in the Gemfile. As it stands tho, that only works with Rails because groups are loaded depending on what environment you're on (which is why :development and :test work right out of the box), and so, it has to be explicitly loaded. The way we do that separation now consists on the scenario you're working from:

  • Installing gems as a dependency of something else: in this case, bundler will hit the dependency API, and the gems that are necessary to run the app are the ones considered a dependency (i.e. runtime, not development.) Have a look here to see what I'm talking about.
  • Installing gems locally: if you happen to run bundle on a locally checked out version of a gem, it will then install all dependencies (runtime and development.)

I hope this makes you reconsider your stance on shipping the specs.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Mar 2, 2015

TL;DR - "conveniently" having tests in a gem is the wrong solution for distro packaging problems

@mvz - in a nutshell, testing a gem "works" (as opposed to "builds") on ARM seems like "dumb testing". I would make as much sense as testing a gem on every possible version of ruby (including patchsets - you never know).

The focus should be on preventing errors, not integration testing for them. Otherwise, it would take sometimes even days to test a single gem. A "convenient quick test" is not the same as a thorough test.

Also, most gems come with unit tests - not feature tests or integration tests. And some thing just have to be tested manually, otherwise they would require complex CI setups with Virtual Machines (and thus eating up all the RAM on any CI machine).

Most gems are platform agnostic. Those that aren't could have separate packages for "sanity" testing (dependencies mostly). Also, most gems can be used in various ways - it's impossible to set up a "perfect distro". That's why every Rails project has it's own locked set of gems.

Summary: using a gem's unit tests for complex integration tests (that even the owner may not know about) is just fooling oneself. It's the wrong tool for the job. Distro management is a tough job - that's why it needs better solutions than just "gem unit tests", which may find problems, but don't provide ANY certainty about whether even the basic functionality will work.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Mar 2, 2015

TL;DR - 1) the situation with using specs as docs has become ridiculous (explained below) and 2) it's the Rakefile's job to call Bundler/setup to e.g. install/use proper development dependencies in non-Rails projects (proper solution below)

@andremedeiros - first, I cannot opt-out of having test files installed with a gem (unlike ri and rdoc docs).

Most users don't need test files, so why should they take up precious time/space/bandwidth on deployment environments? Sure disk space may be cheap, but you're polluting the FS cache and thus eating the most precious hosting asset: RAM.

Sure, developer time is expensive - but the balance shifts when you scale (e.g. millions of users wasting time and resources to help just one developer avoid checking out the source).

There's a lot to be learned from a gem's specs (even if you don't run them), and as an avid user of bundle open, I tend to read a lot of them.

Wrong tool, for the wrong job.

It's ridiculous, because nowadays developers turn off ri and rdoc generation for installed gems, because it's more of a nuisance ... and yet, while there's not rubygems command to get the source of a gem ...

... developers are using a Bundler (!) command to find the installation directory of a gem to hack the source and/or read the specs - instead of e.g. using hub clone foo to get the source and use Bundler's :path option, and have access to not only the specs, but to quickly be able to fix/test and send a PR.

(If you need to read the specs, something is wrong already - and a PR is the best fix to help others).

Maybe a bundle hack foo command would be better? (but ironically there's no RubyGems metadata to help clone/download the source of a gem - there used to be a RubyForge link, but that died as a feature).

It's no surprise that even casual users are often terrified (or discouraged) to do some debugging/hacking themselves.

As it stands tho, that only works with Rails because groups are loaded depending on what environment you're on (which is why :development and :test work right out of the box), and so, it has to be explicitly loaded.

Actually, bundler always (AFAIK) always resolves all dependencies and skips the sets provided (e.g. with the --without option). This is how non-rails projects can skip development gems in a CI environment - either within CI config, or in a Rakefile (recommended) - where the Bundler.setup happens.

I hope this makes you reconsider your stance on shipping the specs.

So there are 2 arguments for keeping the specs, and both seem flawed:

  1. to replace frequently insufficient documentation = wrong tool for the job, since you should download the source, which includes other frequently used features, such as version history, diffs, blames, switching between version, checking unmerged PRs, etc.
  2. to test a gem with minimal setup = also wrong tool for the job especially if you're a developer ("use the source, Luke!"). One example is: there may be special versions of tools needed to run the tests without problems (e.g. unreleased gems via a source ref). And this means possibly installing new/updated versions of gems (or downgrading), which may affect (read: break) your app anyway.

Summary: gems are for production/usage, source is for development - mixing the two for "convenience" is acceptable locally, but forcing that upon millions of end-users and production setups - doesn't make sense.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Mar 2, 2015

@bbatsov - specifically for RuboCop (as opposed to "general" practices described in the comments) I don't see how including specs makes sense for a computation-heavy gem like RuboCop (testing Cop edge cases seems more like just a lame way to test Ruby's base classes, and those specs with edge cases probably aren't good docs for RuboCop itself).

@mvz

This comment has been minimized.

Copy link
Contributor

mvz commented Mar 2, 2015

@e2, you misunderstood me. It is indeed dumb for gem developers to test on all platform combinations. This is why it is better to delegate such testing to parties that care about this, such as people packaging gems for a specific distribution.

I still do not see why you keep going on about unit vs integration tests. As far as I can tell, RuboCop's integration tests are located in the spec/ directory, which has now been completely removed from the distributed gem.

Throwing out (some) automated tests because you can't test everything automatically doesn't seem like a reasonable course of action.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Mar 2, 2015

TL;DR - RuboCop's unit tests actually do nothing to test if a distro/package is valid (because that's not their job or scope) - at best they just create a false sense of certainty

@mvz - sorry, I wasn't clear about what I meant by "integration tests" - those generally don't exist for most gems (with the exception of gems such as Bundler, Cucumber, etc.).

unit tests != distro testing (The two are completely unrelated - there's really no overlap.)

Also the test environment can be very different from the distro environment (stubs/mocks, special gem versions, temporary directories, custom environment variables and overrides).

RuboCop (and it's dependencies) basically use just pure Ruby. By running RuboCop's unit tests, you're pretty much just testing minor parts of Ruby internals (for no reason - assuming you ran Ruby's unit tests already and they pass) or you're testing RubyGems and the environment (custom GEM_HOME paths, etc.).

Running unit tests to test a distro is meaningless - if something does come up, the issue is elsewhere - either bin/rubocop is properly found and run (distro env is ok), or it isn't (Ruby problem, RubyGems problem, etc.).

(If you're packaging for JRuby, you may not even be able to install the test/dev deps anyway - and even if they fail, it may not mean anything - e.g. many Travis projects ignore failures for Rubinius and JRuby, but that's often just because the test frameworks often have issues, while the gem itself works fine).

RuboCop doesn't have "integration tests" (from a distro point of view), because it doesn't test that RuboCop fails (or works) with a given version of Parser or Ast or Rainbow in a specific setup. The specs don't check if bin/rubocop properly handles the LOAD_PATH, uses Bundler correctly, or runs the Cli correctly, with the right platform-specific exit code, etc.

(Case in point - there's no spec checking if bin/rubocop actually even returns a non-zero exit code on failure).

Sure, there are runtime version specifications in the gemspec, but those are not covered by any "distro-testing integration tests".

Not to mention - the dependencies for running RuboCop (to be included in the distro) may conflict with the gems needed to test RuboCop (meaning - the environment you end up with for testing may not match what users will actually have in their distro!).

So the best way to test RuboCop on a new distro is to ... run it manually - or - with a project that depends on RuboCop and is expected to pass/fail in a certain way (checking exit code, output, etc.).

Throwing out (some) automated tests because you can't test everything automatically doesn't seem like a reasonable course of action.

From a distro point of view - those tests aren't testing "everything", they're actually testing ... nothing. And there's nothing "thrown out", because you can download the source and test by tag anytime you want.

The argument is about trying to include unit tests into a gem - just to create a false sense of security by "pretending" those tests actually do test something worthwhile. Those tests may not even detect that bin/rubocop is missing (which can happen as a mistake if you're packaging RuboCop yourself).

Conclusion: nothing will break if those files are removed, while the source is easy to download. There is no (reasonable) upside to keeping them - maybe just so 5 people on the planet can avoid downloading the source?

@mvz

This comment has been minimized.

Copy link
Contributor

mvz commented Mar 2, 2015

From a distro point of view - those tests aren't testing "everything", they're actually testing ... nothing.

Debian's Ruby package maintainers do not agree with this assertion.

And there's nothing "thrown out", because you can download the source and test by tag anytime you want.

This is extra work that would be avoided by including the tests in the gems.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Mar 2, 2015

TL;DR - none of the 2 arguments (package maintainers disagreeing and how downloading is extra work) seem to make sense for RuboCop specifically, and generally I see communication issues (between packagers and gem owners) for which keeping tests seems like the wrong solution.

@mvz -

Debian's Ruby package maintainers do not agree with this assertion.

What exactly are they finding by running RuboCop specs? Bugs in RSpec? In MRI? In Bundler? RuboCop handles internal (Cop) crashes pretty well.

If gem owner are not properly supporting the needs of Ruby package maintainers, it's a communication problem, not a "lack of unit tests attached to releases" problem.

I don't see how "package maintainers do not agree" is a valid technical argument.

This is extra work that would be avoided by including the tests in the gems.

As opposed to time spent waiting for mostly meaningless unit tests to finish? (most RuboCop tests are in no way affected by distro specifics any more than Ruby is).

(And how is hub clone bbatsov/rubocop considered a lot of work?)

Package maintainers have different needs than gem developers and contributors have, so it makes sense to have separate "distro" tests - though this makes little sense for RuboCop itself.

Proper "distro" tests would test a gem from the "outside" - from within the whole distro, not within a "isolated test environment".

P.S. Package maintainers do some of the most unappreciated work in software development - it's important to voice and clarify with specific examples instead of unexplained "general workarounds" which confuse everyone else. I get that unit tests can sometimes find distro-specific bugs, but they are not designed for that. As a dev, I'd rather work on solving the root problem instead of arguing about keeping some testing "side-effect" that is somehow assumed to "help" with some obscure, unclear (to everyone else) challenges package maintainers have.

@mvz

This comment has been minimized.

Copy link
Contributor

mvz commented Sep 6, 2015

What exactly are they finding by running RuboCop specs? Bugs in RSpec? In MRI? In Bundler?

None of the above.

They mainly find incompatibilities between RuboCop (or any other gem) and the versions of its dependencies packaged in Debian. See mvz/gir_ffi#25 for an example of the kind of issues that might be found.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Sep 6, 2015

@mvz

That specific issue is related to FFI, which is beyond the scope of Rubocop's responsibility. Further more, the bug is about distro differences. If you need RuboCop specs to find such errors, there are obviously tests lacking on an entirely different level.

I don't see how a parser working in a terminal is responsible for detecting distro-specific binary/FFI issues.

I get that it can save some time for overworked and unappreciated package maintainers during some obscure scenarios - but is the price the rest of the world is paying (bandwidth, install time, disk space, disk cache) - given RuboCop's growing popularity - really worth it?

Besides, having a full checkout allows checking various tags/versions and master - many more potential issues discovered (even alongside expected failures), without needing RuboCop to release with test files.

Suggestion:

How about package managers provide guidelines on how to make gems more testable and convenient for package managers? I'm guessing there are tests that no one else needs EXCEPT package maintainers. I'd gladly support a "compliant" rake task for running tests in multiple Docker images (in multiple Ubuntu/Debian/Redhat/CentOS versions) before even releasing - I just don't what to test for in gems that don't use FFI or compiled extensions (like RuboCop).

(A failing Docker image would make it easier for gem authors to fix those distro-specific issues themselves).

@mvz

This comment has been minimized.

Copy link
Contributor

mvz commented Sep 6, 2015

I don't see how a parser working in a terminal is responsible for detecting distro-specific binary/FFI issues.

This was just an example from another gem, of the kind of issues that may be found. In the case of RuboCop, it might be a sudden unintended incompatibility of parser.

I get that it can save some time for overworked and unappreciated package maintainers during some obscure scenarios - but is the price the rest of the world is paying (bandwidth, install time, disk space, disk cache) - given RuboCop's growing popularity - really worth it?

That is a slightly different matter. I was merely addressing the usefulness of running any gem's specs as part of the package building process.

How about package managers provide guidelines on how to make gems more testable and convenient for package managers?

As I understand it, they find it convenient to automatically run gems' tests and specs as part of the package building process. If those tests and specs are missing from the gems' distributions, they use the git repository or other source if available.

(A failing Docker image would make it easier for gem authors to fix those distro-specific issues themselves).

Given the slower speed of packaging it's probably more effective to have the packagers detect them first.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Sep 6, 2015

If those tests and specs are missing from the gems' distributions, they use the git repository or other source if available.

Why not do the reverse? Download the repo (at given version) and use the gem only as a last measure (if there's e.g. no source and/or proprietary binaries)? Surely it should be possible to automate this to the point where there's no difference.

(This order makes even more sense when patching - since it's easier to quilt/apply and manage not-yet-released upstream patches).

Rubygems is for distributing, not for development. You wouldn't expect C/C++ unit tests to be included with every library (especially when preparing a distro with no compiler by default). If a gem uses something like a headless webkit browser for testing, it's only adding more unnecessary version constraints for maintainers to deal with (and it's testing a development/packaging environment - not the environment the user is going to have, where only "slimed-down"/non-dev versions of libraries may be available).

To me this seems like expecting RedHat package maintainers to include original tests in every .rpm, simply because Debian package maintainers find it more convenient to convert .rpm to .deb packages using e.g. alien - even if both .rpm and .deb packages could be both built from the original source repository (and not from each other).

This reminds me of an old analogy about opening a can of paint with a screwdriver ... and then using that screwdriver to mix the paint - just because it's "conveniently at hand" (even though you end up with a messed up screwdriver when you need it for something else).

Given the slower speed of packaging it's probably more effective to have the packagers detect them first.

What I meant by using Docker is that it helps gem authors reproduce issues - and prevent related issues without asking maintainers "does it work now?"

Anyway, thanks for responding. I think I get it: this expectation of including tests is really just meant to catch broken abstractions of portability (when compiled extensions/FFI, bundled versions of interpreters, etc. are used). I don't think RuboCop is close enough to a point of failure for this to make sense.

If RuboCop is somehow an exception (because of how crappy tests of other gems are), then using it's source to get tests shouldn't be an issue at all. If you need tests for a minimalistic busybox-like environment (where even shell/math operations will fail), you may not even have the possibility to actually run all the tests at all (due to Ruby features disabled, missing libraries, etc.) - and you'll need custom tests anyway (and likely already packaged to even be able to install them).

Special Packaging/Integration Tests would be the answer, not unit tests testing mostly just RAM and CPU.

@mvz

This comment has been minimized.

Copy link
Contributor

mvz commented Oct 19, 2015

Rubygems is for distributing, not for development. You wouldn't expect C/C++ unit tests to be included with every library (especially when preparing a distro with no compiler by default).

I think that is where the problem lies: Rubygems lacks a uniform method of distributing the source separately from the library as used by the end user. RPMs have this in the form of SRPMs, for example. As a work-around, either more stuff is put in the gem than needed, or an online repo is referenced. The experiences with rubyforge and google code should teach us that that is at least dangerous. A related ticket is here: rubygems/rubygems#735

Anyway, thanks for responding. I think I get it: this expectation of including tests is really just meant to catch broken abstractions of portability (when compiled extensions/FFI, bundled versions of interpreters, etc. are used).

In a sense, yes.

I don't think RuboCop is close enough to a point of failure for this to make sense.

You're probably right there.

@e2

This comment has been minimized.

Copy link
Contributor

e2 commented Oct 19, 2015

For anyone interested in resolving the "gems with tests" debates, I suggest this: rubygems/rubygems#735

@mvz - thanks for the reply, because I'm ignorant about what it's like to be a package manager of Ruby gems and what to watch out for. You've cleared things up for me.

I think there are "layers" with the same problem:

  1. The gem does what it promises (unit tests)
  2. The gem works on one OS with a set of given libraries (unit tests, acceptance tests)
  3. The gem works with multiple interpreters (RVM, Travis, acceptance tests across interpreters - we're stuck at this step, because few gems have JRuby tests passing flawlessly, and constantly green Rubinius tests can be rare)
  4. The gem works on multiple OS's (Travis handling OSX and online build/testing services with VMs - not really available yet)
  5. The gem works on multiple distros (we're not there yet ... and relying on RubyGems for sources and tests doesn't really cut it as a solution)
  6. The gem works on arbitrary setups, different kernels, window managers, unexpected user configs and patches (future ... Docker?)

I think it makes sense for package maintainers to work a bit like app stores - if the gem provides the right tests (what gem authors usually have no idea how to design) and passes distro lints and checkers, it can be available (and promoted - as a reward) in the distro.

In short I think developers should have "reasons" to include distro package building and testing tools - so whenever they release, they "sign off" on the distro tests as having passed (like people do regarding multiple Ruby versions on Travis).

Testing should happen close to the point of failure - so there shouldn't even be a merge unless "distro tests" pass. A gem release itself is not a canvas to work with, but "ready for use". Expecting gems to include tests is "too late".

E.g. currently gem authors can release a gem requiring libraries not yet officially available in distros, but they have not "Travis-like" tool framework to test with - so they can't release a "distro-tested" version of the sources in a gem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment