Draft: Generic scraper for RubyGems

[casaper]

If you’re adding a new scraper, please ensure that you have:

• Tested the scraper on a local copy of DevDocs
• Ensured that the docs are styled similarly to other docs on DevDocs

• Added these files to the public/icons/your_scraper_name/ directory:
  • 16.png: a 16×16 pixel icon for the doc
  • 16@2x.png: a 32×32 pixel icon for the doc
  • SOURCE: A text file containing the URL to the page the image can be found on or the URL of the original image itself

If you're updating an existing documentation to it's latest version, please ensure that you have:

• Updated the versions and releases in the scraper file
• Ensured the license is up-to-date and that the documentation's entry in the array in about_tmpl.coffee matches it's data in self.attribution
• Ensured the icons and the SOURCE file in public/icons/your_scraper_name/ are up-to-date if the documentation has a custom icon
• Ensured self.links contains up-to-date urls if self.links is defined
• Tested the changes locally to ensure:
  • The scraper still works without errors
  • The scraped documentation still looks consistent with the rest of DevDocs
  • The categorization of entries is still good

[simon04]

Woah, this is a huge pull request.

Never in the history of devdocs.io have we added 220 scrapers at once. Until now, devdocs has a nice balance of docs of programming languages and language specific libraries. Adding 220 Ruby libraries at once definitively affects this ratio. Can you share a screenshot of the sidebar when all Ruby libraries have been scraped?

I'm somewhat worried about the long-term maintenance of those 220 scrapers and docs. Not being a Ruby guy myself, I definitively do not want to spend the majority of my devdocs time updating/fixing Ruby docs.

I'd like to get assessments/comments from other active/former devdocs maintainers. Please prepare yourself that reviving this PR will take its time. Thanks.

---

Btw, running generate_all_ruby_gems fails on my system:

Details

$ bundle exec thor docs:generate_all_ruby_gems
Generating docs with 220 RubyGem scrapers.


================================================================================
=
= gem_scraper: rspec_expectations (1/220)
=
================================================================================

Docs allready exist: /Users/simon/src/devdocs/tmp/gem_docs/rspec-expectations-bundler: failed to load command: rdoc (/usr/bin/rdoc)
Gem::Exception: can't find executable rdoc for gem rdoc. rdoc is not currently included in the bundle, perhaps you meant to add it to your Gemfile?
  /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/bundler/rubygems_integration.rb:462:in `block in replace_bin_path'
  /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/bundler/rubygems_integration.rb:482:in `block in replace_bin_path'
  /usr/bin/rdoc:23:in `<top (required)>'
[                                                                                 ] [0/1] [  0.00%] [00:00] [00:00] [0.00/s]ERROR: The gem: rspec-expectations scraper requires the original documentation files to be stored in the "/Users/simon/src/devdocs/docs/gem--rspec-expectations" directory.
Failed! (try running with --debug for more information)

[j-f1]

I’m 👎 on merging this for the reasons @simon04 specified. However, it would be cool if @casaper was willing to host this as a separate app — throwing out all the DevDocs scrapers wholesale and just hosting gem docs shouldn’t be too difficult to maintain in a fork.

[casaper]

@j-f1

The concept of having generated docs for libraries like Gems and Java packages was inspired by Dash...

But as it obviously seems this seems to be seen as kind of some weird competition between different programming languages or so. Not my cup of tea, and as it seems devdocs seems to be not my cup of tea either then.

If I ever decide to do my own thing, I certainly do not intend to fork from devdocs code base, because, no offence, hacking around the concept to do this was not so much fun for doing this draft, so if I go the way, I'll intend to go it with no coffe script and Sinatra 😉

[j-f1]

this seems to be seen as kind of some weird competition between different programming languages or so

Not at all! My concern at least was that people who don’t use Ruby would have trouble finding the docs they want if a significant fraction are gems. I’m also concerned that adding the docs without individual review would decrease the quality of the docs (although it’s possible the gem docs are consistently formatted enough for this to not be an issue).

[ghost]

Maybe the gems could just be another main category and when you collapse it, you see all the individual gems and can select what you want to add to your preferenced documentations. In this way it would not distract other people not using ruby gems.

[mnyon-grandkru]

Gems are libraries. There are docsets for individual libraries in JavaScript. Is Ruby not possible, or not preferred? I'm a Ruby user who'd really love this addition.