Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
minor: remove spec files from gem #1555
TL;DR - shipping specs is the wrong thing to do, no matter how you look at it. Lots of people are doing it wrong, unfortunately.
I've spent hours trying to figure out what "development" means for Ruby gems... and I finally "got it".
(I should write a blog about it ... haven't go to it yet).
In a nutshell,
And if you have 'bundler' as a development dependency, it makes sense to put all the dev deps in the 'Gemfile'.
So, the 'pendantic' thing to do would be: move dev deps (except 'bundler') from the
Next, since contributing to RuboCop requires a pull request, you need the source anyway (unless you expect people to somehow ... craft ... PRs ... from ... a ... diff ??).
And, even if tests fail for a released gem, the response is almost always "do they work on master?".
That's because people are trying to do the right thing by copying other people instead of understanding what's going on (which is actually pretty tough given all the legacy stuff involved here).
The result is lots of wasted bandwidth, disk space (in terms of allocated space), access times (find/grep take ages), wasted developer time (trying to work out the right setup for Travis using
(While their ancestors used to install gems using
TL;DR - even including docs in gems used to be a good idea, but now it's more of a problem
@swatosh - that should be in the README or in a
Specs are actually better for documenting behavior (e.g. doc formatting in RSpec) or testing edge cases - so they rarely make good usage guidelines. (Especially with isolated testing).
Acceptance tests are better - to the point that they are much prettier online (e.g. for relisapp.com). Regardless, reading updated documentation online is better anyway - especially since good docs coverage is often really heavy.
For documentation and examples,
Not to mention: navigating to a installed gem location is troublesome.
I now think of gems as a 'versioned snapshot of code intended for apps, not humans'.
RubyGems wanted to be "more" a long time ago (doc, tests, dependencies, rubyforge), but that didn't work out well in the end.
If a gem's specs have to be read to get a working example - then that project seriously needs a PR for the Readme ;)
As a counter-argument, Debian uses the tests/specs from the gem to run the tests when creating a .deb package.
As an aside, @e2, why do you conclude that
TL;DR - why unit tests don't make sense when packaging debs + why dev deps are obsolete
@mvz - I can discuss the issue in depth, so feel free to ask me to explain myself more
Tests/specs basically don't make sense from a maintainer's point of view because:
So in this case, it makes no sense to run a gem's unit tests from a released gem. For a CI - it's better to clone the repo by tag anyway (since additional dev tools or steps may be needed for testing anyway).
"released" means "ready for use", which implies all the tests pass on every environment. Otherwise, you'd be using rubocop tests to test the environment, not the gem itself. (Instead it would make more sense to run the rubygems and bundler gem tests - which would be more thorough).
The point is - development workflows changed much faster than Rubygems can deal with (because it has to be backward-compatible with even the most ancient gems out there) - which is why
And with bundler as a dev dependency anyway, it just makes more sense to put things like
As an analogy - you don't get a full development kit and manufacturing tests when you buy a smartphone. You get just the smartphone. Even if you're a re-seller.
As a side note,
My point - learning by "copying others" doesn't work well where there's too much legacy baggage to deal with.
P.S.2 - I know people "hack" on code in installed gems to debug issues. A better approach is to immediately use Bundler with
@e2 your comment was indeed TL, so I DR it until now ;-).
Most of your arguments are based on an ideal world where gem authors test their gems on all platforms and versions of Ruby (1.9.3 on little-endian ARM, perhaps?), and where there is a perfect separation of unit and feature/integration tests. This world, however, does not exist.
In reality, tests are a very useful tool to find out hidden bugs and dependencies before they hit the end-user of a package. If Debian were to make CRuby 2.2 their default Ruby implementation, how else are they going to find out if the existing packages at their existing versions will still work?
Actually, if I may drop my $0.02 here:
@bbatsov: Bundler will generate a tests stub, if you so choose it. We don't force it onto you, the same way we don't force a license or the code of conduct anymore. You choose how you want it.
As for the whole To Include or not To Include [specs] argument, I'm in favour of including them. There's a lot to be learned from a gem's specs (even if you don't run them), and as an avid user of
As for moving away from
I hope this makes you reconsider your stance on shipping the specs.
TL;DR - "conveniently" having tests in a gem is the wrong solution for distro packaging problems
@mvz - in a nutshell, testing a gem "works" (as opposed to "builds") on ARM seems like "dumb testing". I would make as much sense as testing a gem on every possible version of ruby (including patchsets - you never know).
The focus should be on preventing errors, not integration testing for them. Otherwise, it would take sometimes even days to test a single gem. A "convenient quick test" is not the same as a thorough test.
Also, most gems come with unit tests - not feature tests or integration tests. And some thing just have to be tested manually, otherwise they would require complex CI setups with Virtual Machines (and thus eating up all the RAM on any CI machine).
Most gems are platform agnostic. Those that aren't could have separate packages for "sanity" testing (dependencies mostly). Also, most gems can be used in various ways - it's impossible to set up a "perfect distro". That's why every Rails project has it's own locked set of gems.
Summary: using a gem's unit tests for complex integration tests (that even the owner may not know about) is just fooling oneself. It's the wrong tool for the job. Distro management is a tough job - that's why it needs better solutions than just "gem unit tests", which may find problems, but don't provide ANY certainty about whether even the basic functionality will work.
TL;DR - 1) the situation with using specs as docs has become ridiculous (explained below) and 2) it's the Rakefile's job to call Bundler/setup to e.g. install/use proper development dependencies in non-Rails projects (proper solution below)
@andremedeiros - first, I cannot opt-out of having test files installed with a gem (unlike
Most users don't need test files, so why should they take up precious time/space/bandwidth on deployment environments? Sure disk space may be cheap, but you're polluting the FS cache and thus eating the most precious hosting asset: RAM.
Sure, developer time is expensive - but the balance shifts when you scale (e.g. millions of users wasting time and resources to help just one developer avoid checking out the source).
Wrong tool, for the wrong job.
It's ridiculous, because nowadays developers turn off
... developers are using a Bundler (!) command to find the installation directory of a gem to hack the source and/or read the specs - instead of e.g. using
(If you need to read the specs, something is wrong already - and a PR is the best fix to help others).
It's no surprise that even casual users are often terrified (or discouraged) to do some debugging/hacking themselves.
Actually, bundler always (AFAIK) always resolves all dependencies and skips the sets provided (e.g. with the
So there are 2 arguments for keeping the specs, and both seem flawed:
Summary: gems are for production/usage, source is for development - mixing the two for "convenience" is acceptable locally, but forcing that upon millions of end-users and production setups - doesn't make sense.
@bbatsov - specifically for RuboCop (as opposed to "general" practices described in the comments) I don't see how including specs makes sense for a computation-heavy gem like RuboCop (testing Cop edge cases seems more like just a lame way to test Ruby's base classes, and those specs with edge cases probably aren't good docs for RuboCop itself).
@e2, you misunderstood me. It is indeed dumb for gem developers to test on all platform combinations. This is why it is better to delegate such testing to parties that care about this, such as people packaging gems for a specific distribution.
I still do not see why you keep going on about unit vs integration tests. As far as I can tell, RuboCop's integration tests are located in the
Throwing out (some) automated tests because you can't test everything automatically doesn't seem like a reasonable course of action.
TL;DR - RuboCop's unit tests actually do nothing to test if a distro/package is valid (because that's not their job or scope) - at best they just create a false sense of certainty
@mvz - sorry, I wasn't clear about what I meant by "integration tests" - those generally don't exist for most gems (with the exception of gems such as Bundler, Cucumber, etc.).
unit tests != distro testing (The two are completely unrelated - there's really no overlap.)
Also the test environment can be very different from the distro environment (stubs/mocks, special gem versions, temporary directories, custom environment variables and overrides).
RuboCop (and it's dependencies) basically use just pure Ruby. By running RuboCop's unit tests, you're pretty much just testing minor parts of Ruby internals (for no reason - assuming you ran Ruby's unit tests already and they pass) or you're testing RubyGems and the environment (custom GEM_HOME paths, etc.).
Running unit tests to test a distro is meaningless - if something does come up, the issue is elsewhere - either
(If you're packaging for JRuby, you may not even be able to install the test/dev deps anyway - and even if they fail, it may not mean anything - e.g. many Travis projects ignore failures for Rubinius and JRuby, but that's often just because the test frameworks often have issues, while the gem itself works fine).
RuboCop doesn't have "integration tests" (from a distro point of view), because it doesn't test that RuboCop fails (or works) with a given version of Parser or Ast or Rainbow in a specific setup. The specs don't check if
(Case in point - there's no spec checking if
Sure, there are runtime version specifications in the gemspec, but those are not covered by any "distro-testing integration tests".
Not to mention - the dependencies for running RuboCop (to be included in the distro) may conflict with the gems needed to test RuboCop (meaning - the environment you end up with for testing may not match what users will actually have in their distro!).
So the best way to test RuboCop on a new distro is to ... run it manually - or - with a project that depends on RuboCop and is expected to pass/fail in a certain way (checking exit code, output, etc.).
From a distro point of view - those tests aren't testing "everything", they're actually testing ... nothing. And there's nothing "thrown out", because you can download the source and test by tag anytime you want.
The argument is about trying to include unit tests into a gem - just to create a false sense of security by "pretending" those tests actually do test something worthwhile. Those tests may not even detect that
Conclusion: nothing will break if those files are removed, while the source is easy to download. There is no (reasonable) upside to keeping them - maybe just so 5 people on the planet can avoid downloading the source?
Debian's Ruby package maintainers do not agree with this assertion.
This is extra work that would be avoided by including the tests in the gems.
TL;DR - none of the 2 arguments (package maintainers disagreeing and how downloading is extra work) seem to make sense for RuboCop specifically, and generally I see communication issues (between packagers and gem owners) for which keeping tests seems like the wrong solution.
What exactly are they finding by running RuboCop specs? Bugs in RSpec? In MRI? In Bundler? RuboCop handles internal (Cop) crashes pretty well.
If gem owner are not properly supporting the needs of Ruby package maintainers, it's a communication problem, not a "lack of unit tests attached to releases" problem.
I don't see how "package maintainers do not agree" is a valid technical argument.
As opposed to time spent waiting for mostly meaningless unit tests to finish? (most RuboCop tests are in no way affected by distro specifics any more than Ruby is).
(And how is
Package maintainers have different needs than gem developers and contributors have, so it makes sense to have separate "distro" tests - though this makes little sense for RuboCop itself.
Proper "distro" tests would test a gem from the "outside" - from within the whole distro, not within a "isolated test environment".
P.S. Package maintainers do some of the most unappreciated work in software development - it's important to voice and clarify with specific examples instead of unexplained "general workarounds" which confuse everyone else. I get that unit tests can sometimes find distro-specific bugs, but they are not designed for that. As a dev, I'd rather work on solving the root problem instead of arguing about keeping some testing "side-effect" that is somehow assumed to "help" with some obscure, unclear (to everyone else) challenges package maintainers have.
None of the above.
They mainly find incompatibilities between RuboCop (or any other gem) and the versions of its dependencies packaged in Debian. See mvz/gir_ffi#25 for an example of the kind of issues that might be found.
That specific issue is related to FFI, which is beyond the scope of Rubocop's responsibility. Further more, the bug is about distro differences. If you need RuboCop specs to find such errors, there are obviously tests lacking on an entirely different level.
I don't see how a parser working in a terminal is responsible for detecting distro-specific binary/FFI issues.
I get that it can save some time for overworked and unappreciated package maintainers during some obscure scenarios - but is the price the rest of the world is paying (bandwidth, install time, disk space, disk cache) - given RuboCop's growing popularity - really worth it?
Besides, having a full checkout allows checking various tags/versions and master - many more potential issues discovered (even alongside expected failures), without needing RuboCop to release with test files.
How about package managers provide guidelines on how to make gems more testable and convenient for package managers? I'm guessing there are tests that no one else needs EXCEPT package maintainers. I'd gladly support a "compliant" rake task for running tests in multiple Docker images (in multiple Ubuntu/Debian/Redhat/CentOS versions) before even releasing - I just don't what to test for in gems that don't use FFI or compiled extensions (like RuboCop).
(A failing Docker image would make it easier for gem authors to fix those distro-specific issues themselves).
This was just an example from another gem, of the kind of issues that may be found. In the case of RuboCop, it might be a sudden unintended incompatibility of parser.
That is a slightly different matter. I was merely addressing the usefulness of running any gem's specs as part of the package building process.
As I understand it, they find it convenient to automatically run gems' tests and specs as part of the package building process. If those tests and specs are missing from the gems' distributions, they use the git repository or other source if available.
Given the slower speed of packaging it's probably more effective to have the packagers detect them first.
Why not do the reverse? Download the repo (at given version) and use the gem only as a last measure (if there's e.g. no source and/or proprietary binaries)? Surely it should be possible to automate this to the point where there's no difference.
(This order makes even more sense when patching - since it's easier to quilt/apply and manage not-yet-released upstream patches).
Rubygems is for distributing, not for development. You wouldn't expect C/C++ unit tests to be included with every library (especially when preparing a distro with no compiler by default). If a gem uses something like a headless webkit browser for testing, it's only adding more unnecessary version constraints for maintainers to deal with (and it's testing a development/packaging environment - not the environment the user is going to have, where only "slimed-down"/non-dev versions of libraries may be available).
To me this seems like expecting RedHat package maintainers to include original tests in every .rpm, simply because Debian package maintainers find it more convenient to convert .rpm to .deb packages using e.g.
This reminds me of an old analogy about opening a can of paint with a screwdriver ... and then using that screwdriver to mix the paint - just because it's "conveniently at hand" (even though you end up with a messed up screwdriver when you need it for something else).
What I meant by using Docker is that it helps gem authors reproduce issues - and prevent related issues without asking maintainers "does it work now?"
Anyway, thanks for responding. I think I get it: this expectation of including tests is really just meant to catch broken abstractions of portability (when compiled extensions/FFI, bundled versions of interpreters, etc. are used). I don't think RuboCop is close enough to a point of failure for this to make sense.
If RuboCop is somehow an exception (because of how crappy tests of other gems are), then using it's source to get tests shouldn't be an issue at all. If you need tests for a minimalistic busybox-like environment (where even shell/math operations will fail), you may not even have the possibility to actually run all the tests at all (due to Ruby features disabled, missing libraries, etc.) - and you'll need custom tests anyway (and likely already packaged to even be able to install them).
Special Packaging/Integration Tests would be the answer, not unit tests testing mostly just RAM and CPU.
I think that is where the problem lies: Rubygems lacks a uniform method of distributing the source separately from the library as used by the end user. RPMs have this in the form of SRPMs, for example. As a work-around, either more stuff is put in the gem than needed, or an online repo is referenced. The experiences with rubyforge and google code should teach us that that is at least dangerous. A related ticket is here: rubygems/rubygems#735
In a sense, yes.
You're probably right there.
For anyone interested in resolving the "gems with tests" debates, I suggest this: rubygems/rubygems#735
@mvz - thanks for the reply, because I'm ignorant about what it's like to be a package manager of Ruby gems and what to watch out for. You've cleared things up for me.
I think there are "layers" with the same problem:
I think it makes sense for package maintainers to work a bit like app stores - if the gem provides the right tests (what gem authors usually have no idea how to design) and passes distro lints and checkers, it can be available (and promoted - as a reward) in the distro.
In short I think developers should have "reasons" to include distro package building and testing tools - so whenever they release, they "sign off" on the distro tests as having passed (like people do regarding multiple Ruby versions on Travis).
Testing should happen close to the point of failure - so there shouldn't even be a merge unless "distro tests" pass. A gem release itself is not a canvas to work with, but "ready for use". Expecting gems to include tests is "too late".
E.g. currently gem authors can release a gem requiring libraries not yet officially available in distros, but they have not "Travis-like" tool framework to test with - so they can't release a "distro-tested" version of the sources in a gem.