Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Improved search feature (elasticsearch based, demo available) #455

Closed
wants to merge 22 commits into from
@karmi

This pull request contains a proposed search feature overhaul for Rubygems.org, implemented with the elasticsearch search engine, via the Tire library.

Objectives

The main objective of the effort is to allow searching in more gem properties then just their names, notably in summaries, descriptions and authors – technically speaking, to increase both precision and recall of search at Rubygems.org.

Using a search engine — as opposed to a LIKE %term% database query — allows not only for better, faster searches, but also for advanced features such as a rich search query language, faceted navigation, and more.

Changelog

All the steps required for implementing the feature are commited on the search-steps branch, with extensive commit messages documenting the process. The important steps are:

  • 86240f2 and 3a17730 implement the most simple search with elasticsearch, adding model integration and using Tire in the controller

  • fa7d1cd adds complex mapping definition for the Rubygem model, allowing to search in gem summaries/descriptions, authors, dependencies, and more.

  • 8c6bee8 adds a sliding panel to the search results page which contains examples of searches with Lucene search query syntax.

Additional Cucumber scenarios were added to document the new search features. Some additional tweaks were required to run the test suite successfuly at Travis CI.

Please review the branch compare page to see the full picture.

Demo Application

A demo application is available at http://rubygems-with-elasticsearch.herokuapp.com.

(UPDATE, **new demo server** here: ****

UPDATE: test servers terminated.

Try out simple searches such as rack or searching in authors: author:john and dependencies: uses:rack. More tips are available as in-page help.

The database contains only a limited subset of gems. The application is running on a free Heroku plan. The elasticsearch service is running on a Amazon EC2 t1.micro instance. Keep in mind, that the application runs in a tweaked development mode (due to issues with assets etc.), so the demo application performance does not reflect the performance in the real production environment.

If a dump of the Rubygems production database would be available, I'd like to import it into the demo application database.

Further Development

If the proposed search implementation is considered desirable, a number of further developments is possible, eg.:

  • more fine-grained score computation based on number of downloads, not straight sorting,
  • allow sorting the results by number of downloads, alphabetically, by created or updated time,
  • highlighting the relevant matched snippets from gem properties,
  • adding faceted search on authors,
  • linking to a specific matched version from search results,
  • displaying aggregated statistics such as authors with most gems, authors with most downloaded gems, etc.,
  • adding Tire's NewRelic instrumentation to track performance.

Installation Instructions

To check out the search feature locally, assuming you had cloned the Rubygems.org repository, set it up according to instructions first:

./script/setup

To import your local gems into the database, run:

bundle exec rake gemcutter:import:process

Then, install elasticsearch using your preffered method. On Mac OS X, the easiest way is to use Homebrew:

brew install elasticsearch

To import the gems from the database into elasticsearch, run:

bundle exec rake environment tire:import CLASS='Rubygem' FORCE=1
@travisbot

This pull request fails (merged 8c6bee8 into 7ac6d16).

@karmi

As noted, @travisbot requires some additional tweaks to run the test suite. (Still, Travis is very unreliable when running the full test suite, see http://travis-ci.org/#!/karmi/rubygems.org/builds)

[Edit] example of a successfull test run: http://travis-ci.org/#!/karmi/rubygems.org/jobs/2286458

@evanphx
Owner

Looks great! I don't have any experience with elasticsearch, does it require running another service? If so, there isn't any details on how to get that service running and we'll need that.

@nz
nz commented

Looks great, @karmi!

@evanphx: I'd consider it a privilege to sponsor the search hosting on http://bonsai.io/

@karmi

@evanphx Thanks! Right now, the elasticsearch service for the demo application runs at EC2 instance, provisioned with Chef. I think there would be no problem getting someone to sponsor the box, possibly including elasticsearch.com company.

As @nz points out, Bonsai is available to host the search service as well -- though issues with the Tire library and Bonsai would have to be sorted out...

@evanphx
Owner

I'm wary of having rubygems.org depend on an external service like bonsai.io. There is A LOT of traffic and I worry the site would depend too heavily on the reliability of something we don't have control over.

@karmi

I understand the concern, Evan.

However, if we want to make the search at Rubygems.org radically better, there's no way around it then depend on some external factor in one way or another — all the major search engines are external processes/services (except TSearch).

It's similar, in fact, to dependency on Redis (for tracking downloads) in the current codebase.

elasticsearch itself is open source and free, based on Lucene, written in Java, and can be run trivially on any real or virtual server. It's particularly suited to run in cloud environments such as Amazon AWS (ie. with little latency to Heroku), but is in no way tied or affiliated with Amazon.

(Regarding the work needed to set up, configure and maintain an elasticsearch server, I can handle such duties just fine.)

If we want to move forward with the proposed search functionality, I think we need to work on these points:

  1. Is the proposed feature something we want to use, eventually, at Rubygems.org? If so, let's discuss what steps are required to merge it into master and roll it into production.

  2. Can a full dump of the Rubygems production database be provided? If so, let me load it up to the demo application so everybody can try various kinds of searches and kick the tires on the feature.

  3. If everybody's happy with the proposed search feature, let's work on polishing it further -- the first thing is using Rubygem#downloads as a score boosting factor, not as a straightforward sorting criterion.

@evanphx
Owner

I'm sorry, I wasn't clear. I don't have an issue running the elasticsearch service on the rubygems.org servers. I am worried about using a hosted elasticsearch service because then usage of it is dependent on a lot more (network conditions, cloud health, etc).

As for the running of it, thats very kind of you to offer to setup, configure, and maintain it but that likely won't work because then you'd need to be effectively on-call all the time. We don't have an issue maintaining it, but I would like some guidance into how it should be configured, how much disk/memory it will use, etc.

@nz
nz commented

@evanphx: ElasticSearch has a pretty good guide for self-hosting on Amazon: http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html

FWIW, we made the same offer to host the search at websolr when Solr was on the table a while back. I did some digging with qrush at one point into the question of traffic volume, and was completely comfortable with the numbers. Besides that, we literally are on call all the time :-)

That said I get the value of self-hosting for you here, and am happy to be available to talk tech when it comes to hosting ES. I'll idle in #gemcutter today (nz) if you want to talk more about capacity planning, which is almost always an experimental process.

@nz
nz commented

Er, make that #rubygems :)

@karmi

Thanks for the clarification, Evan!

I don't have an issue running the elasticsearch service on the rubygems.org servers.
We don't have an issue maintaining it, but I would like some guidance into
how it should be configured, how much disk/memory it will use, etc.

Perfect! elasticsearch is pretty easy to install and operate; in terms of required resources, for the Rubygems.org use case a modest machine will be more then enough.

I can certainly help with the installation and configuration of elasticsearch on your servers — just ask the specifics! The easiest way is to use the Chef cookbook. Please see the tutorial at the elasticsearch.org site.

For the Rubygems.org use case, one elasticsearch node should be enough, though for proper failover and scalability, two nodes would be desirable. In terms of resources needed, elasticsearch needs mostly RAM. Any modest machine comparable to EC2 small to large would be enough, assuming it has couple of gigabytes of memory to spare. (Note, that the demo application uses the micro instance and happily purrs along with just 613MB of RAM.)

Provided the database dump from Rubygems.org is available, I can do some capacity testing with the full set of data against EC2 instances.

@cmeiklejohn
Collaborator

@karmi

Looks like I'm getting some errors on the console when running the test suite, but the tests aren't failing. Is this something to be concerned with?

# Running tests:

..................................................................................................................................................................................................................................................................................................................................................................................................[REQUEST FAILED] curl -X GET "http://localhost:9200/test_rubygems/rubygem/_search?load%5Binclude%5D=versions&page=&per_page=30&size=30&pretty=true" -d '{"query":{"bool":{"should":[{"text":{"name":{"query":"bang!","type":"phrase_prefix","operator":"and","boost":100}}},{"query_string":{"query":"bang!","default_operator":"and"}}]}},"sort":[{"downloads":"desc"},{"name.raw":"asc"}],"filter":{"term":{"indexed":true}},"size":30}'
.[REQUEST FAILED] curl -X GET "http://localhost:9200/test_rubygems/rubygem/_search?load%5Binclude%5D=versions&page=&per_page=30&size=30&pretty=true" -d '{"query":{"bool":{"should":[{"text":{"name":{"query":"bang!","type":"phrase_prefix","operator":"and","boost":100}}},{"query_string":{"query":"bang!","default_operator":"and"}}]}},"sort":[{"downloads":"desc"},{"name.raw":"asc"}],"filter":{"term":{"indexed":true}},"size":30}'
.[REQUEST FAILED] curl -X GET "http://localhost:9200/test_rubygems/rubygem/_search?load%5Binclude%5D=versions&page=&per_page=30&size=30&pretty=true" -d '{"query":{"bool":{"should":[{"text":{"name":{"query":"bang!","type":"phrase_prefix","operator":"and","boost":100}}},{"query_string":{"query":"bang!","default_operator":"and"}}]}},"sort":[{"downloads":"desc"},{"name.raw":"asc"}],"filter":{"term":{"indexed":true}},"size":30}'
.[REQUEST FAILED] curl -X GET "http://localhost:9200/test_rubygems/rubygem/_search?load%5Binclude%5D=versions&page=&per_page=30&size=30&pretty=true" -d '{"query":{"bool":{"should":[{"text":{"name":{"query":"bang!","type":"phrase_prefix","operator":"and","boost":100}}},{"query_string":{"query":"bang!","default_operator":"and"}}]}},"sort":[{"downloads":"desc"},{"name.raw":"asc"}],"filter":{"term":{"indexed":true}},"size":30}'
........................................................................

Finished tests in 115.507678s, 3.9911 tests/s, 6.4411 assertions/s.

461 tests, 744 assertions, 0 failures, 0 errors, 0 skips
@cmeiklejohn cmeiklejohn closed this
@cmeiklejohn
Collaborator

Ugh, whoops for the close. Github UI failure.

@cmeiklejohn cmeiklejohn reopened this
@karmi

@cmeiklejohn Yes, that is intentional -- it's the Tire's STDERR output, coming from tests for "user enters invalid Lucene query", bang! in this case. See https://github.com/karmi/rubygems.org/blob/search-steps/test/functional/searches_controller_test.rb#L57-66 and https://github.com/karmi/rubygems.org/blob/search-steps/features/search.feature#L67-70

@adkron
Collaborator

:+1: I went to the test site and loved the functionality that it provides. What can we do to get this brough back up?

app/models/rubygem.rb
@@ -12,7 +15,48 @@ class Rubygem < ActiveRecord::Base
validate :ensure_name_format
validates :name, :presence => true, :uniqueness => true
- after_create :update_unresolved
+ after_create :update_unresolved, :update_elasticsearch_index
+ after_touch :update_elasticsearch_index
+
+ tire do
@qrush Owner
qrush added a note

All of this is perfect for a Concern module, something like Searchable.

class Rubygem < ActiveRecord::Base
  include Searchable

And that module has all of the necessary includes, methods, etc. Any thoughts about that approach?

@karmi
karmi added a note

Nothing against such approach -- normally, I like to keep mapping/etc definitions inside the model, and since the after_create :update_unresolved hook was already there, I just followed the convention. Do you want to extract everything related to search to a module?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@qrush qrush commented on the diff
features/support/env.rb
@@ -4,6 +4,9 @@
# instead of editing this one. Cucumber will automatically load all features/**/*.rb
# files.
+require 'webmock/cucumber' # Allow connections to elasticsearch
@qrush Owner
qrush added a note

Does this mean the test suite is dependent on an elasticsearch install? How would this work on Travis, etc?

@karmi
karmi added a note

The Cucumber integration test is indeed dependent on Elasticsearch running, since that's the only way how to end-to-end test the feature? Elasticsearch is available on Travis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
app/controllers/searches_controller.rb
@@ -1,8 +1,31 @@
class SearchesController < ApplicationController
+ # Indicate incorrect query to the user
+ rescue_from Tire::Search::SearchRequestFailed do |error|
@qrush Owner
qrush added a note

Does this cover the case where ES is completely unavailable/disconnected?

@karmi
karmi added a note

No, that would have to be handled by a separate rescue_from clause, displaying an error such as "We're sorry, search is currently not available".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@qrush
Owner

Left a few comments. I think we should get ES setup in rubygems/rubygems-aws soon so we can start playing with it...maybe we can wire up the test heroku app to give it a test first.

Some more feedback:

  • Let's get info for how to get ES setup in CONTRIBUTING.md
  • Does anyone else have experience with maintaining a running cluster? What if there's problems? Who will get alerted, who will debug it, etc? (I'm trying to answer this now instead of when the fire is blazing)
  • What if search goes down, can we fall back to the old search?
@karmi

I think we should get ES setup in rubygems/rubygems-aws soon

Please keep me in the loop, I'm the author of the Chef cookbook.

Let's get info for how to get ES setup in CONTRIBUTING.md

I'll put it in, and force push the commits here.

Does anyone else have experience with maintaining a running cluster?

I'm employed by Elasticsearch.com and have some experience with running Elasticsearch clusters :) Let's talk about the exact process.

What if search goes down, can we fall back to the old search?

I don't think that's a good solution. A running Elasticsearch cluster shouldn't just go down -- we just need to ensure there is proper monitoring on the service itself and EC2 level?

@qrush
Owner

@karmi: This all sounds awesome :) Would you be willing to contribute that into https://github.com/rubygems/rubygems-aws ? I'm very sure our new ops contributors would be more than willing to help get everything set up.

"Shouldn't just go down" is not what I've seen...I'd rather account/test for that now when we're doing the switch and migration instead of when it's on fire.

@karmi

Yes, I'll setup the environment for rubygems-aws and add a pull request for Elasticsearch.

As for going down, all services and servers can go down :) But I think we need to come up with a process for that, instead of falling back on the SQL based search; that just doesn't feel right. There are many aspects here, eg. having nodes properly distributes across AWS zones, having an automated strategy for recovering from backup or reindexing from scratch, etc.

@qrush
Owner

Cool. That would be neat. We don't even have any of that in place for the main app yet (AFAIK)

@karmi karmi referenced this pull request from a commit in karmi/rubygems-aws
@karmi karmi [SEARCH] Added configuration for Elasticsearch nodes
This commit adds support for search nodes running Elasticsearch.

* The "elasticsearch" cookbook [https://github.com/elasticsearch/cookbook-elasticsearch/]
  has been added to the Cheffile

* A Vagrant VM named `search` has been added

* A `search` role has been added

* Node configurations (*.json) for Vagrant and EC2 have been added

* The Capistrano tasks have been updated to reflect the changes

To deploy in EC2:

    # Update packages
    #
    RUBYGEMS_EC2_SEARCH=abc-123.compute-1.amazonaws.com \
    DEPLOY_USER=ubuntu \
    DEPLOY_SSH_KEY=~/.ssh/mykey.pem \
      cap rubygems.org invoke COMMAND='sudo apt-get update' SUDO=true

    # Install Chef
    #
    RUBYGEMS_EC2_SEARCH=abc-123.compute-1.amazonaws.com \
    DEPLOY_USER=ubuntu \
    DEPLOY_SSH_KEY=~/.ssh/mykey.pem \
      cap rubygems.org invoke COMMAND='curl -# -L http://www.opscode.com/chef/install.sh | sudo bash -s --' SUDO=true

    # Run Chef
    #
    time \
    RUBYGEMS_EC2_SEARCH=abc-123.compute-1.amazonaws.com \
    DEPLOY_USER=ubuntu \
    DEPLOY_SSH_KEY=~/.ssh/mykey.pem \
      cap rubygems.org chef:search

Related: rubygems/rubygems.org#455
aaa86e5
@karmi karmi referenced this pull request from a commit in karmi/rubygems-aws
@karmi karmi [SEARCH] Added a template for Elasticsearch application initializer
The `elasticsearch_url` variable is set in the "secret/rubygems" data bag,
similar to setting PostgreSQL host, etc.

Alternatively, an environment variable `ELASTICSEARCH_URL` could be used.

Related: rubygems/rubygems.org#455
0f9c517
@karmi karmi referenced this pull request in rubygems/rubygems-aws
Merged

Added Elasticsearch integration #122

@karmi karmi referenced this pull request from a commit in karmi/rubygems-aws
@karmi karmi [SEARCH] Added configuration for Elasticsearch nodes
This commit adds support for search nodes running Elasticsearch.

* The "elasticsearch" cookbook [https://github.com/elasticsearch/cookbook-elasticsearch/]
  has been added to the Cheffile

* A Vagrant VM named `search` has been added

* A `search` role has been added

* Node configurations (*.json) for Vagrant and EC2 have been added

* The Capistrano tasks have been updated to reflect the changes

To deploy in EC2:

    # Update packages
    #
    RUBYGEMS_EC2_SEARCH=abc-123.compute-1.amazonaws.com \
    DEPLOY_USER=ubuntu \
    DEPLOY_SSH_KEY=~/.ssh/mykey.pem \
      cap rubygems.org invoke COMMAND='sudo apt-get update' SUDO=true

    # Install Chef
    #
    RUBYGEMS_EC2_SEARCH=abc-123.compute-1.amazonaws.com \
    DEPLOY_USER=ubuntu \
    DEPLOY_SSH_KEY=~/.ssh/mykey.pem \
      cap rubygems.org invoke COMMAND='curl -# -L http://www.opscode.com/chef/install.sh | sudo bash -s --' SUDO=true

    # Run Chef
    #
    time \
    RUBYGEMS_EC2_SEARCH=abc-123.compute-1.amazonaws.com \
    DEPLOY_USER=ubuntu \
    DEPLOY_SSH_KEY=~/.ssh/mykey.pem \
      cap rubygems.org chef:search

Related: rubygems/rubygems.org#455
02d8518
@karmi karmi referenced this pull request from a commit in karmi/rubygems-aws
@karmi karmi [SEARCH] Added a template for Elasticsearch application initializer
The `elasticsearch_url` variable is set in the "secret/rubygems" data bag,
similar to setting PostgreSQL host, etc.

Alternatively, an environment variable `ELASTICSEARCH_URL` could be used.

Related: rubygems/rubygems.org#455
8e0e3dd
@karmi

Hi all, rebased the branch against current master and added some commits. There's a new test server available here:

http://54.235.152.92:3000/search?utf8=✓&query=name%3Arack

which has been created as part of the rubygems/rubygems-aws#122 pull request.

features/step_definitions/gem_steps.rb
((6 lines not shown))
+ table.hashes.each do |row|
+ # p 'GOT TABLE ROW:', row, '-'*80
+ if row['downloads']
+ rubygem = FactoryGirl.create :rubygem_with_downloads, :name => row['name'], :downloads => row['downloads']
+ else
+ rubygem = FactoryGirl.create :rubygem, :name => row['name']
+ end
+
+ FactoryGirl.create(:version, :rubygem => rubygem) do |version|
+ version.number = row['version']
+ version.authors = row['authors'].split(/\s*,\s*/)
+ version.summary = row['summary']
+ version.description = row['description']
+
+ version.save
+ # p "CREATED RUBYGEM:", version.rubygem, version, '-'*80

this p could be removed now

@karmi
karmi added a note

Both removed in karmi/rubygems.org@dcb887a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
features/step_definitions/gem_steps.rb
@@ -65,3 +70,24 @@
rubygem.ownerships.create :user => user
end
end
+
+Given /^gems with these properties exist:$/ do |table|
+ table.hashes.each do |row|
+ # p 'GOT TABLE ROW:', row, '-'*80

ditto

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@karmi

@vipulnsward Commented out debug statements for Cucumber removed in karmi/rubygems.org@dcb887a.

@cicloid

Just saw the pull request, is there something in need of doing or testing, in order to move this forward?

@karmi

karmi opened this pull request a year ago

Guys, we just passed an anniversary with this pull request. What should be done with it? Should I close it?

/cc @qrush @evanphx @skottler

@vipulnsward

:cry: I hope not.

@knappe

Can we get some more traction on this? This is a very intriguing feature set.

/cc @qrush @evanphx @skottler

@skottler
Owner

@karmi can you please rebase?

@knappe the best way to help move this forward is to do a thorough code review.

karmi added some commits
@karmi karmi [SEARCH] Added "tire" dependency for searching Rubygems.org with elas…
…ticsearch

elasticsearch is an open source search engine based on Lucene, with a RESTful HTTP interface and advanced distributed features.

Tire is a Ruby API/DSL for elasticsearch, with an out-of-the box ActiveRecord/ActiveModel integration.

See:

* "Tire": https://github.com/karmi/tire
* "elasticsearch": http://elasticsearch.org
e18cfb2
@karmi karmi [SEARCH] Allow connections to elasticsearch [localhost:9200] in tests…
… and Cucumber

NOTE: The `disable_net_connect!` call has to come *before* we load the application,
      because Tire checks for index existence on application boot, and shoots
      the entire test suite down.

      See <karmi/retire#136> for more information.
6632bc2
@karmi karmi [SEARCH] Added elementary Tire integration into the Rubygem model
* Added, that the Version model propagates touches to Rubygem [See: http://stackoverflow.com/a/11711477/95696]

* Added Tire ActiveRecord callbacks [See: https://github.com/karmi/tire#activemodel-integration]

* Added a simple mapping definition for Rubygem

* Added a simple `to_indexed_json` serialization for Tire

* Fixed incorrect test case in WebHookTest ("include an Authorization header"):
  1) use the `build`, not the `create` FactoryGirl strategy (to skip Tire indexing), and,
  2) use the _last_ HTTP request from WebMock registry (to skip Tire checking if the index exists)

* Fixed failing "Web Hooks" feature, using the _last_ HTTP request from WebMock registry (see above)

Import your current database with the following Rake task:

    $ bundle exec rake environment tire:import CLASS='Rubygem' FORCE=1

Check the index in your browser:

    <http://localhost:9200/development_rubygems/_search>
2e56fcb
@karmi karmi [SEARCH] Added the simplest possible search with elasticsearch
* Added simple query string search into SearchesController

* Recreate elasticsearch index in the SearchesController functional test setup and in the Cucumber `Before('@search')` callback

* Trigger index update in the FactoryGirl `after(:create)` callback

* Be more defensive in ApplicationHelper#short_info (in test, some gems don't have versions?)

Note: The "beer laser" => "beer_laser" Cucumber scenario fails,
      due to incorrect analysis of Gem names with underscore.
611774a
@karmi karmi [SEARCH] Added, that factories trigger `touch` callbacks after create
NOTE: We need to be absolutely sure the `Rubygem` instance is touched, because
      we rely on it being indexed in elasticsearch in integration tests.
70f0f1f
@karmi karmi [SEARCH] Added proper analyzer for Rubygem names
With the original (standard) analyzer, a Gem name like "url_mount" would be analyzed as "url_mount",
making searches for "url mount" (without the underscore) fail.

With the new analyzer, *tokens* are split by "special characters" defined by the `Patterns` module (.-_).

Try it out yourself:

  <http://localhost:9200/development_rubygems/_analyze?text=url_mount&field=name>

This change makes the "beer laser" => "beer_laser" Cucumber scenario pass.
2389438
@karmi karmi [SEARCH] Changed the search definition to a DSL-based syntax, added s…
…orting by downloads

* Used the DSL notation for defining the search: using a `match` prefix query on the "name" field,
  basically replicating the simple query string search with wildcards with a more performant version,
  and using a filter on the `indexed` property

* Added sorting of the results by downloads (descending)

* Added a Cucumber scenario for showing the more downloaded gems higher in search results

* Added supporting Cucumber code: a "I have a gem with downloads" and "I see these search results" step definitions
bfd2aa7
@karmi karmi [SEARCH] Changed, that search results are ordered first by downloads,…
… then alphabetically

* Changed the `name` property to multi-field, using the "keyword" analyzer on `name.raw` for searching
* Added the sort block with multiple sort fields
* Added a Cucumber scenario

NOTE: Now we should really stop and think twice about how to make the results more relevant.
      We should be able to get better search _precision_ by using the `Rubygem#downloads`
      counter as a factor affecting score, not just plainly sort on its value.
db6e1d3
@karmi karmi [SEARCH] Added a more complex mapping definition and serialization fo…
…r the Rubygem model

* Previously, only the `name`, `downloads` and `indexed` attributes were indexed,
  replicating the functionality of the current search feature.

* The `to_indexed_json` method was removed, relying on Tire's JSON serialization routines
  based on the model mapping definition.

* The `summary`, `description` and `author` gem properties were added, allowing much better
  search results _recall_, ie. allowing search in these fields as well and widening the search “net”.

* A gem which mentions "sinatra" in it's summary/description will now be matched (with a lower score):
  <http://localhost:3000/search?query=sinatra>.

* A gem written by Florian Hanke will now be matched: http://localhost:3000/search?query=florian+hanke

* The `version` gem property was added, allowing searches based on gem versions, for instance looking
  for Sinatra 1.3.2: <http://localhost:3000/search?query=name:sinatra+version:1.3.2>. For improved
  usability, the link from the result listing _should_ lead to the relevant version page,
  ie. http://localhost:3000/gems/sinatra/versions/1.3.2, not the last version.

* The `depends` and `uses` gem properties were added, which index runtime gem dependencies and all
  gem dependencies, respectively. It allows searches such as <http://localhost:3000/search?query=depends:rack>
  (for gems with depend on rack) or http://localhost:3000/search?query=uses:rack (for gems which use rack in
  one way or other).

* The `created_at` and `updated_at` gem properties were added, which allow to search gems updated in a specific
  period, for instance on August, 26th: <http://localhost:3000/search?query=updated_at:[2012-08-26+TO+2012-08-27]>

* The `author`, `created_at` and `updated_at` also allow for a _faceted navigation_ in the future, ie. searching
  for certain gem while restricting the result to certain author or time.

* These properties also allow for computing statistics on the Rubygem collection, such as displaying authors
  with most gems, or authors of the most downloaded gems, etc.

You have to reindex the elasticsearch index, to pick up the new mapping and index records properly:

    $ bundle exec rake environment tire:import CLASS='Rubygem' FORCE=1

See the following resources for information on previous efforts to implement a better Rubygems.org search:

* https://groups.google.com/forum/#!topic/gemcutter/xIzyTmFdXVo/discussion
* http://florianhanke.com/blog/2011/02/13/a-better-rubygems-search.html
* http://blog.websolr.com/post/3505941785/rubygems-search-upgrade-2
* http://blog.websolr.com/post/3505969969/rubygems-search-upgrade-3
aa63c84
@karmi karmi [SEARCH] Mock HTTP responses to Elasticsearch in unit tests 79b6857
@karmi karmi [SEARCH] Added a more complex search query in the SearchesController#…
…show method

Previously, we have been searching gems based on their names only.

With the new, more complex mapping defined in the preceding commit, we can add a more complex search query as well.

We're using a boolean query, keeping the original match prefix query and adding the `query_string` query,
which uses the Lucene query syntax (field specifation, boolean operators, wildcards,
fuzzy search, range and proximity searches, grouping, etc).

See:

* http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html
* http://lucene.apache.org/core/3_6_1/queryparsersyntax.html
415ba28
@karmi karmi [SEARCH] Added a `rescue_from` failed search requests due to incorrec…
…t query syntax

While we exposed the most powerful way of searching to the user (the Lucene query syntax),
it can quite easily lead to application errors when users enter incorrect queries, such as `bang!` or `foo[]`.

Since this is an error on the user's part, and not the application part, we should display a friendly
error explanation and give the user a chance to correct the query.
f0d20d4
@karmi karmi [SEARCH] Added a "user enters a search query with incorrect syntax" C…
…ucumber scenario

Since the application uses Cucumber scenarios for validating its proper operation,
a scenario with user entering an incorrect search query ("bang!") has been added.
5c13726
@karmi karmi [SEARCH] Added the "Search Advanced" Cucumber feature
With the complex queries now available to users of the application, we should add
acceptance tests for the common scenarios.

We'll start with searching in summaries and descriptions (thanks to the `_all` field
automatically generated by elasticsearch).

Use this command to run all search features:

    $ bundle exec cucumber --tag @search

Use this command to run the "advanced search" feature:

    $ bundle exec cucumber features/search_advanced.feature
ac96085
@karmi karmi [SEARCH] Added a Cucumber scenario for searching in gem authors
    Given we now have a more complex search available
    When we search in the `author` field
    We should get some relevant results

* Added a "Searching in authors" scenario
* Added a step for creating more complex Rubygem records into the `gem_steps.rb` definition file

Use this command to run the scenario:

    $ bundle exec cucumber --name "Searching in authors" features/search_advanced.feature
0346d62
@karmi karmi [SEARCH] Refactored the search steps to a higher-level nested step "I…
… search for ..."

Instead of repeating the low-level steps:

    When I go to the homepage
    And I fill in "query" with "<query>"
    And I press "Search"

over and over in our scenarios, we will abstract these steps to a single step:

    When I search for "<query>"

The obvious benefit is less code duplication and more readable steps.
4eb11a8
@karmi karmi [SEARCH] Added "search tips" sliding panel at the search results page
* Added a second form with `query` input, to duplicate the query for easier correction/change
  at the results page

* Added a HTML partial with concrete, interactive examples of queries possible with Lucene,
  hidden by default

* Added a link and JavaScript code to toggle the sliding panel with search examples

* Added CSS styling for the new elements, added a "help.png" icon from the FamFamFam suite
d389af0
@karmi karmi [SEARCH] Added starting of "elasticsearch" in the Travis CI configura…
…tion
01f479b
@karmi karmi [SEARCH] Prevent indexing errors on Rubygem records without a version cf5beec
@karmi karmi [SEARCH] Added information about installing Elasticsearch into "Contr…
…ibution Guidelines"
382b3a6
@karmi karmi [SEARCH] Handle search engine being not available in user-friendly way c1f99aa
@karmi karmi [SEARCH] Changed, that errors when indexing to Elasticsearch are rescued
Previously, when an error occurred while saving the model into the Elasticsearch index,
the whole operation failed and an Exception has been raised.

This patch adds a `rescue` clause which logs the exception and swallows it.
06f2626
@karmi

@skottler Rebased, fixed problems with Webmock stubbing, force pushed.

@karmi

I have terminated the EC2 instances for the demo application.

@jimmycuadra

For a project I'm working on, I'd like to be able to search gems based on gem specification metadata (the metadata hash attribute available from RubyGems 2.0 and up). I was investigating how gem searching is implemented currently, and after seeing that it was such a simple SQL query, figured someone had to be working on an ES-based search feature, and sure enough, here it is in this pull request.

Long story short, I'm very interested in seeing this rolled out and would like to help, since it's been sitting idle for quite some time. Is a code review still the blocker here?

@jimmycuadra

It also looks like the tire gem has been deprecated in favor of multiple gems hosted at elasticsearch/elasticsearch-ruby. This PR should be updated to use the new goods.

@karmi karmi closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 11, 2013
  1. @karmi

    [SEARCH] Added "tire" dependency for searching Rubygems.org with elas…

    karmi authored
    …ticsearch
    
    elasticsearch is an open source search engine based on Lucene, with a RESTful HTTP interface and advanced distributed features.
    
    Tire is a Ruby API/DSL for elasticsearch, with an out-of-the box ActiveRecord/ActiveModel integration.
    
    See:
    
    * "Tire": https://github.com/karmi/tire
    * "elasticsearch": http://elasticsearch.org
  2. @karmi

    [SEARCH] Allow connections to elasticsearch [localhost:9200] in tests…

    karmi authored
    … and Cucumber
    
    NOTE: The `disable_net_connect!` call has to come *before* we load the application,
          because Tire checks for index existence on application boot, and shoots
          the entire test suite down.
    
          See <karmi/retire#136> for more information.
  3. @karmi

    [SEARCH] Added elementary Tire integration into the Rubygem model

    karmi authored
    * Added, that the Version model propagates touches to Rubygem [See: http://stackoverflow.com/a/11711477/95696]
    
    * Added Tire ActiveRecord callbacks [See: https://github.com/karmi/tire#activemodel-integration]
    
    * Added a simple mapping definition for Rubygem
    
    * Added a simple `to_indexed_json` serialization for Tire
    
    * Fixed incorrect test case in WebHookTest ("include an Authorization header"):
      1) use the `build`, not the `create` FactoryGirl strategy (to skip Tire indexing), and,
      2) use the _last_ HTTP request from WebMock registry (to skip Tire checking if the index exists)
    
    * Fixed failing "Web Hooks" feature, using the _last_ HTTP request from WebMock registry (see above)
    
    Import your current database with the following Rake task:
    
        $ bundle exec rake environment tire:import CLASS='Rubygem' FORCE=1
    
    Check the index in your browser:
    
        <http://localhost:9200/development_rubygems/_search>
  4. @karmi

    [SEARCH] Added the simplest possible search with elasticsearch

    karmi authored
    * Added simple query string search into SearchesController
    
    * Recreate elasticsearch index in the SearchesController functional test setup and in the Cucumber `Before('@search')` callback
    
    * Trigger index update in the FactoryGirl `after(:create)` callback
    
    * Be more defensive in ApplicationHelper#short_info (in test, some gems don't have versions?)
    
    Note: The "beer laser" => "beer_laser" Cucumber scenario fails,
          due to incorrect analysis of Gem names with underscore.
  5. @karmi

    [SEARCH] Added, that factories trigger `touch` callbacks after create

    karmi authored
    NOTE: We need to be absolutely sure the `Rubygem` instance is touched, because
          we rely on it being indexed in elasticsearch in integration tests.
  6. @karmi

    [SEARCH] Added proper analyzer for Rubygem names

    karmi authored
    With the original (standard) analyzer, a Gem name like "url_mount" would be analyzed as "url_mount",
    making searches for "url mount" (without the underscore) fail.
    
    With the new analyzer, *tokens* are split by "special characters" defined by the `Patterns` module (.-_).
    
    Try it out yourself:
    
      <http://localhost:9200/development_rubygems/_analyze?text=url_mount&field=name>
    
    This change makes the "beer laser" => "beer_laser" Cucumber scenario pass.
  7. @karmi

    [SEARCH] Changed the search definition to a DSL-based syntax, added s…

    karmi authored
    …orting by downloads
    
    * Used the DSL notation for defining the search: using a `match` prefix query on the "name" field,
      basically replicating the simple query string search with wildcards with a more performant version,
      and using a filter on the `indexed` property
    
    * Added sorting of the results by downloads (descending)
    
    * Added a Cucumber scenario for showing the more downloaded gems higher in search results
    
    * Added supporting Cucumber code: a "I have a gem with downloads" and "I see these search results" step definitions
  8. @karmi

    [SEARCH] Changed, that search results are ordered first by downloads,…

    karmi authored
    … then alphabetically
    
    * Changed the `name` property to multi-field, using the "keyword" analyzer on `name.raw` for searching
    * Added the sort block with multiple sort fields
    * Added a Cucumber scenario
    
    NOTE: Now we should really stop and think twice about how to make the results more relevant.
          We should be able to get better search _precision_ by using the `Rubygem#downloads`
          counter as a factor affecting score, not just plainly sort on its value.
  9. @karmi

    [SEARCH] Added a more complex mapping definition and serialization fo…

    karmi authored
    …r the Rubygem model
    
    * Previously, only the `name`, `downloads` and `indexed` attributes were indexed,
      replicating the functionality of the current search feature.
    
    * The `to_indexed_json` method was removed, relying on Tire's JSON serialization routines
      based on the model mapping definition.
    
    * The `summary`, `description` and `author` gem properties were added, allowing much better
      search results _recall_, ie. allowing search in these fields as well and widening the search “net”.
    
    * A gem which mentions "sinatra" in it's summary/description will now be matched (with a lower score):
      <http://localhost:3000/search?query=sinatra>.
    
    * A gem written by Florian Hanke will now be matched: http://localhost:3000/search?query=florian+hanke
    
    * The `version` gem property was added, allowing searches based on gem versions, for instance looking
      for Sinatra 1.3.2: <http://localhost:3000/search?query=name:sinatra+version:1.3.2>. For improved
      usability, the link from the result listing _should_ lead to the relevant version page,
      ie. http://localhost:3000/gems/sinatra/versions/1.3.2, not the last version.
    
    * The `depends` and `uses` gem properties were added, which index runtime gem dependencies and all
      gem dependencies, respectively. It allows searches such as <http://localhost:3000/search?query=depends:rack>
      (for gems with depend on rack) or http://localhost:3000/search?query=uses:rack (for gems which use rack in
      one way or other).
    
    * The `created_at` and `updated_at` gem properties were added, which allow to search gems updated in a specific
      period, for instance on August, 26th: <http://localhost:3000/search?query=updated_at:[2012-08-26+TO+2012-08-27]>
    
    * The `author`, `created_at` and `updated_at` also allow for a _faceted navigation_ in the future, ie. searching
      for certain gem while restricting the result to certain author or time.
    
    * These properties also allow for computing statistics on the Rubygem collection, such as displaying authors
      with most gems, or authors of the most downloaded gems, etc.
    
    You have to reindex the elasticsearch index, to pick up the new mapping and index records properly:
    
        $ bundle exec rake environment tire:import CLASS='Rubygem' FORCE=1
    
    See the following resources for information on previous efforts to implement a better Rubygems.org search:
    
    * https://groups.google.com/forum/#!topic/gemcutter/xIzyTmFdXVo/discussion
    * http://florianhanke.com/blog/2011/02/13/a-better-rubygems-search.html
    * http://blog.websolr.com/post/3505941785/rubygems-search-upgrade-2
    * http://blog.websolr.com/post/3505969969/rubygems-search-upgrade-3
  10. @karmi
  11. @karmi

    [SEARCH] Added a more complex search query in the SearchesController#…

    karmi authored
    …show method
    
    Previously, we have been searching gems based on their names only.
    
    With the new, more complex mapping defined in the preceding commit, we can add a more complex search query as well.
    
    We're using a boolean query, keeping the original match prefix query and adding the `query_string` query,
    which uses the Lucene query syntax (field specifation, boolean operators, wildcards,
    fuzzy search, range and proximity searches, grouping, etc).
    
    See:
    
    * http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html
    * http://lucene.apache.org/core/3_6_1/queryparsersyntax.html
  12. @karmi

    [SEARCH] Added a `rescue_from` failed search requests due to incorrec…

    karmi authored
    …t query syntax
    
    While we exposed the most powerful way of searching to the user (the Lucene query syntax),
    it can quite easily lead to application errors when users enter incorrect queries, such as `bang!` or `foo[]`.
    
    Since this is an error on the user's part, and not the application part, we should display a friendly
    error explanation and give the user a chance to correct the query.
  13. @karmi

    [SEARCH] Added a "user enters a search query with incorrect syntax" C…

    karmi authored
    …ucumber scenario
    
    Since the application uses Cucumber scenarios for validating its proper operation,
    a scenario with user entering an incorrect search query ("bang!") has been added.
  14. @karmi

    [SEARCH] Added the "Search Advanced" Cucumber feature

    karmi authored
    With the complex queries now available to users of the application, we should add
    acceptance tests for the common scenarios.
    
    We'll start with searching in summaries and descriptions (thanks to the `_all` field
    automatically generated by elasticsearch).
    
    Use this command to run all search features:
    
        $ bundle exec cucumber --tag @search
    
    Use this command to run the "advanced search" feature:
    
        $ bundle exec cucumber features/search_advanced.feature
  15. @karmi

    [SEARCH] Added a Cucumber scenario for searching in gem authors

    karmi authored
        Given we now have a more complex search available
        When we search in the `author` field
        We should get some relevant results
    
    * Added a "Searching in authors" scenario
    * Added a step for creating more complex Rubygem records into the `gem_steps.rb` definition file
    
    Use this command to run the scenario:
    
        $ bundle exec cucumber --name "Searching in authors" features/search_advanced.feature
  16. @karmi

    [SEARCH] Refactored the search steps to a higher-level nested step "I…

    karmi authored
    … search for ..."
    
    Instead of repeating the low-level steps:
    
        When I go to the homepage
        And I fill in "query" with "<query>"
        And I press "Search"
    
    over and over in our scenarios, we will abstract these steps to a single step:
    
        When I search for "<query>"
    
    The obvious benefit is less code duplication and more readable steps.
  17. @karmi

    [SEARCH] Added "search tips" sliding panel at the search results page

    karmi authored
    * Added a second form with `query` input, to duplicate the query for easier correction/change
      at the results page
    
    * Added a HTML partial with concrete, interactive examples of queries possible with Lucene,
      hidden by default
    
    * Added a link and JavaScript code to toggle the sliding panel with search examples
    
    * Added CSS styling for the new elements, added a "help.png" icon from the FamFamFam suite
  18. @karmi
  19. @karmi
  20. @karmi
  21. @karmi
  22. @karmi

    [SEARCH] Changed, that errors when indexing to Elasticsearch are rescued

    karmi authored
    Previously, when an error occurred while saving the model into the Elasticsearch index,
    the whole operation failed and an Exception has been raised.
    
    This patch adds a `rescue` clause which logs the exception and swallows it.
This page is out of date. Refresh to see the latest.
Showing with 491 additions and 41 deletions.
  1. +2 −0  .travis.yml
  2. +11 −7 CONTRIBUTING.md
  3. +1 −0  Gemfile
  4. +11 −0 Gemfile.lock
  5. +41 −3 app/controllers/searches_controller.rb
  6. +1 −1  app/helpers/application_helper.rb
  7. +7 −0 app/helpers/searches_helper.rb
  8. +2 −2 app/models/dependency.rb
  9. +1 −1  app/models/linkset.rb
  10. +1 −1  app/models/ownership.rb
  11. +53 −0 app/models/rubygem.rb
  12. +1 −1  app/models/subscription.rb
  13. +1 −1  app/models/version.rb
  14. +43 −0 app/views/searches/_search_tips.en.html.erb
  15. +9 −1 app/views/searches/show.html.erb
  16. +3 −0  config/environments/test.rb
  17. +2 −0  config/locales/en.yml
  18. +31 −15 features/search.feature
  19. +29 −0 features/search_advanced.feature
  20. +24 −0 features/step_definitions/gem_steps.rb
  21. +15 −0 features/step_definitions/search_steps.rb
  22. +1 −1  features/step_definitions/webhook_steps.rb
  23. +3 −0  features/support/env.rb
  24. +6 −3 features/support/gemcutter.rb
  25. BIN  public/images/help.png
  26. +13 −0 public/javascripts/application.js
  27. +78 −0 public/stylesheets/screen.css
  28. +10 −0 test/factories.rb
  29. +16 −0 test/functional/searches_controller_test.rb
  30. +4 −0 test/test_helper.rb
  31. +5 −0 test/unit/dependencies_middleware_test.rb
  32. +5 −0 test/unit/dependency_test.rb
  33. +5 −0 test/unit/download_test.rb
  34. +5 −0 test/unit/helpers/rubygems_helper_test.rb
  35. +5 −0 test/unit/hostess_test.rb
  36. +5 −0 test/unit/ownership_test.rb
  37. +5 −0 test/unit/pusher_test.rb
  38. +12 −0 test/unit/rubygem_test.rb
  39. +5 −0 test/unit/subscription_test.rb
  40. +5 −0 test/unit/user_test.rb
  41. +5 −0 test/unit/version_test.rb
  42. +9 −4 test/unit/web_hook_test.rb
  43. BIN  vendor/cache/ansi-1.4.3.gem
  44. BIN  vendor/cache/hashr-0.0.22.gem
  45. BIN  vendor/cache/tire-0.6.0.gem
View
2  .travis.yml
@@ -9,3 +9,5 @@ language: ruby
rvm:
- 1.9.3
script: bundle exec rake default
+services:
+ - elasticsearch
View
18 CONTRIBUTING.md
@@ -43,7 +43,7 @@ git remote set-url origin git@github.com:rubygems/rubygems.org.git
Otherwise, you can continue to hack away in your own fork.
-If you’re looking for things to hack on, please check
+If you’re looking for things to hack on, please check
[GitHub Issues](http://github.com/rubygems/rubygems.org/issues). If you’ve
found bugs or have feature ideas don’t be afraid to pipe up and ask the
[mailing list](http://groups.google.com/group/gemcutter) or IRC channel
@@ -127,19 +127,23 @@ running:
**version 2.0 or higher**. If you have homebrew,
do `brew install redis -H`, if you use macports,
do `sudo port install redis`.
+* Install [Elasticsearch](http://www.elasticsearch.org).
+ You can do it with `brew install elasticsearch`,
+ or just download, unzip and run
+ a [release](http://www.elasticsearch.org/download/).
* Rubygems is configured to use PostgreSQL (>= 8.4.x),
for MySQL see below. Install with: `brew install postgres`
**Get the code:**
* Clone the repo: `git clone git://github.com/rubygems/rubygems.org`
-* Move into your cloned rubygems directory if you haven’t already:
+* Move into your cloned rubygems directory if you haven’t already:
`cd rubygems.org`
-
+
**Setup the database:**
* Get set up: `./script/setup`
-* Run the database rake tasks if needed:
+* Run the database rake tasks if needed:
`rake db:create:all db:drop:all db:setup db:test:prepare --trace`
**Running tests:**
@@ -151,7 +155,7 @@ running:
* Set the REDISTOGO_URL environment variable. For example:
`REDISTOGO_URL="redis://localhost:6379"`
-* Import gems if you want to seed the database.
+* Import gems if you want to seed the database.
`rake gemcutter:import:process PATHTO_GEMS/cache`
* _To import a small set of gems you can point the import process to any
gems cache directory, like a very small `rvm` gemset for instance._
@@ -188,8 +192,8 @@ running:
> **Warning:** Gem names are case sensitive (eg. `BlueCloth` vs.
> `bluecloth` 2). MySQL has a `utf8_bin` collation, but it appears
-> that you still need to do `BINARY name = ?` for searching.
-> It is recommended that you stick to PostgreSQL >= 8.4.x
+> that you still need to do `BINARY name = ?` for searching.
+> It is recommended that you stick to PostgreSQL >= 8.4.x
> for development. Some tests will also fail if you use MySQL
> because some queries use SQL functions which don't exist in MySQL..
View
1  Gemfile
@@ -31,6 +31,7 @@ gem 'validates_formatting_of'
gem 'will_paginate'
gem 'xml-simple'
gem 'yajl-ruby', :require => 'yajl'
+gem 'tire'
# enable if on heroku, make sure to toss this into an initializer:
# Rails.application.config.middleware.use HerokuAssetCacher
View
11 Gemfile.lock
@@ -30,6 +30,7 @@ GEM
multi_json (~> 1.0)
addressable (2.3.5)
aggregate (0.2.2)
+ ansi (1.4.3)
arel (3.0.2)
bcrypt-ruby (3.1.2)
bluepill (0.0.66)
@@ -101,6 +102,7 @@ GEM
gherkin (2.12.1)
multi_json (~> 1.3)
gravtastic (3.2.6)
+ hashr (0.0.22)
high_voltage (1.2.4)
highline (1.6.19)
hike (1.2.3)
@@ -212,6 +214,14 @@ GEM
thor (0.18.1)
tilt (1.4.1)
timecop (0.6.3)
+ tire (0.6.0)
+ activemodel (>= 3.0)
+ activesupport
+ ansi
+ hashr (~> 0.0.19)
+ multi_json (~> 1.3)
+ rake
+ rest-client (~> 1.6)
treetop (1.4.15)
polyglot
polyglot (>= 0.3.1)
@@ -279,6 +289,7 @@ DEPENDENCIES
shoulda
sinatra
timecop
+ tire
unicorn
validates_formatting_of
webmock
View
44 app/controllers/searches_controller.rb
@@ -1,11 +1,49 @@
class SearchesController < ApplicationController
+ # Handle search engine not being available
+ #
+ rescue_from Errno::EHOSTUNREACH, Errno::ECONNREFUSED, SocketError do |error|
+ flash.now[:failure] = "Sorry, search is not available at the moment." if params[:query]
+ render :show, :status => :internal_server_error
+ end
+
+ # Indicate incorrect query to the user
+ #
+ rescue_from Tire::Search::SearchRequestFailed do |error|
+ flash.now[:failure] = "Sorry, your query is incorrect." if error.message =~ /SearchParseException/ && params[:query]
+ render :show, :status => :internal_server_error
+ end
+
def show
- if params[:query]
- @gems = Rubygem.search(params[:query]).with_versions.paginate(:page => params[:page])
+ if params[:query].present?
+ @gems = Rubygem.tire.search :page => params[:page],
+ :per_page => Rubygem.per_page,
+ :load => {:include => 'versions'} do |search|
+
+ search.query do |s|
+ s.filtered do |f|
+ f.query do |q|
+ q.boolean do |it|
+ it.should { |q| q.match 'name.raw', params[:query], :boost => 500 }
+ it.should { |q| q.match :name, params[:query], :type => 'phrase_prefix', :operator => 'and', :boost => 100 }
+ it.should { |q| q.string params[:query], :default_operator => 'and' }
+ end
+ end
+ f.filter :term, :indexed => true
+ end
+ end
+
+ search.sort do
+ by 'downloads', :desc
+ by 'name.raw', :asc
+ end
+
+ # STDOUT.puts search.to_curl if Rails.env.development?
+ end
+
@exact_match = Rubygem.name_is(params[:query]).with_versions.first
- redirect_to rubygem_path(@exact_match) if @gems == [@exact_match]
+ redirect_to rubygem_path(@exact_match) if @exact_match && @gems.size == 1 && @gems.first.id == @exact_match.id
end
end
View
2  app/helpers/application_helper.rb
@@ -16,7 +16,7 @@ def atom_feed_link(title, url)
end
def short_info(version)
- truncate(version.info, :length => 100)
+ version ? truncate(version.info, :length => 100) : ''
end
def gravatar(size, id = "gravatar", user = current_user)
View
7 app/helpers/searches_helper.rb
@@ -0,0 +1,7 @@
+module SearchesHelper
+
+ def link_to_example_search(query)
+ link_to query, search_url( :query => query, :anchor => 'tips' )
+ end
+
+end
View
4 app/models/dependency.rb
@@ -1,8 +1,8 @@
class Dependency < ActiveRecord::Base
LIMIT = 250
- belongs_to :rubygem
- belongs_to :version
+ belongs_to :rubygem, :touch => true
+ belongs_to :version, :touch => true
before_validation :use_gem_dependency,
:use_existing_rubygem,
View
2  app/models/linkset.rb
@@ -1,5 +1,5 @@
class Linkset < ActiveRecord::Base
- belongs_to :rubygem
+ belongs_to :rubygem, :touch => true
attr_protected :rubygem_id
LINKS = %w(home wiki docs mail code bugs).freeze
View
2  app/models/ownership.rb
@@ -1,5 +1,5 @@
class Ownership < ActiveRecord::Base
- belongs_to :rubygem
+ belongs_to :rubygem, :touch => true
belongs_to :user
validates :user_id, :uniqueness => {:scope => :rubygem_id}
View
53 app/models/rubygem.rb
@@ -1,6 +1,8 @@
class Rubygem < ActiveRecord::Base
include Patterns
+ include Tire::Model::Search
+
has_many :owners, :through => :ownerships, :source => :user
has_many :ownerships, :dependent => :destroy
has_many :subscribers, :through => :subscriptions, :source => :user
@@ -15,6 +17,50 @@ class Rubygem < ActiveRecord::Base
after_create :update_unresolved
before_destroy :mark_unresolved
+ after_create :update_elasticsearch_index_with_rescue
+ after_destroy :update_elasticsearch_index_with_rescue
+ after_touch :update_elasticsearch_index_with_rescue
+
+ tire do
+ index_prefix Rails.env
+
+ settings :number_of_shards => 1,
+ :number_of_replicas => 1,
+ :analysis => {
+ :analyzer => {
+ :rubygem => {
+ :type => 'pattern',
+ :pattern => "[\s#{Regexp.escape(SPECIAL_CHARACTERS)}]+"
+ }
+ }
+ } do
+ mapping do
+ indexes :name, :type => 'multi_field',
+ :fields => {
+ :name => { :type => 'string', :analyzer => 'rubygem', :boost => 10.0 },
+ :raw => { :type => 'string', :analyzer => 'keyword', :boost => 10.0 }
+ }
+ indexes :indexed, :type => 'boolean', :include_in_all => false, :as => proc { versions.any?(&:indexed?) }
+ indexes :downloads, :type => 'integer', :include_in_all => false
+
+ indexes :summary, :analyzer => 'english', :as => proc { versions.most_recent.try(:summary) }
+ indexes :description, :analyzer => 'english', :as => proc { versions.most_recent.try(:description) }
+ indexes :author, :as => proc { versions.most_recent.try(:authors).try(:split, /\s*,\s*/) }
+
+ indexes :version, :analyzer => 'keyword', :as => proc { versions.map(&:number) },
+ :include_in_all => false
+
+ indexes :uses, :as => proc { versions.most_recent.dependencies.map(&:name) if versions.most_recent rescue nil },
+ :include_in_all => false
+ indexes :depends, :as => proc { versions.most_recent.dependencies.runtime.map(&:name) if versions.most_recent rescue nil },
+ :include_in_all => false
+
+ indexes :created_at, :type => 'date', :include_in_all => false
+ indexes :updated_at, :type => 'date', :include_in_all => false
+ end
+ end
+ end
+
def self.with_versions
where("rubygems.id IN (SELECT rubygem_id FROM versions where versions.indexed IS true)")
end
@@ -268,6 +314,13 @@ def gittip_enabled?
owners.where('gittip_username is not null').count > 0
end
+ def update_elasticsearch_index_with_rescue
+ update_elasticsearch_index
+ rescue Exception => e
+ Rails.logger.error "Error when updating Elasticsearch. Original exception: #{e.inspect}"
+ return true
+ end
+
private
def ensure_name_format
View
2  app/models/subscription.rb
@@ -1,5 +1,5 @@
class Subscription < ActiveRecord::Base
- belongs_to :rubygem
+ belongs_to :rubygem, :touch => true
belongs_to :user
validates :rubygem_id, :uniqueness => {:scope => :user_id}
View
2  app/models/version.rb
@@ -1,5 +1,5 @@
class Version < ActiveRecord::Base
- belongs_to :rubygem
+ belongs_to :rubygem, :touch => true
has_many :dependencies, :order => 'rubygems.name ASC', :include => :rubygem, :dependent => :destroy
before_save :update_prerelease
View
43 app/views/searches/_search_tips.en.html.erb
@@ -0,0 +1,43 @@
+<div id="search-tips">
+<div>
+ <p>
+ When looking for gems, you can use a wide variety of search queries
+ in the <a href="http://lucene.apache.org/core/3_6_1/queryparsersyntax.html" class="external">Lucene syntax</a>.
+ </p>
+
+ <p>
+ Quite simply, you can search in gem names, summaries and descriptions with queries like
+ <code><%= link_to_example_search 'rack' %></code> or
+ <code><%= link_to_example_search 'imap' %></code>
+ </p>
+
+ <p>You can, of course, restrict the search to gem names only:</p>
+ <p><code><%= link_to_example_search 'name:rack' %></code></p>
+
+ <p>To broaden your search, you can use wildcards:</p>
+ <p>
+ <code><%= link_to_example_search 'name:ra*' %></code> or
+ <code><%= link_to_example_search 'web*' %></code>
+ </p>
+
+ <p>You can search for specific gem authors:</p>
+ <p><code><%= link_to_example_search 'author:john' %></code></p>
+
+ <p>Of course, you can combine these queries into complex ones:</p>
+ <p>
+ <code><%= link_to_example_search 'name:ra* AND author:john' %></code> or
+ <code><%= link_to_example_search 'name:ra* AND version:1*' %></code>
+ </p>
+
+ <p>To discover more gems, you can search by their depencies in runtime:</p>
+ <p><code><%= link_to_example_search 'depends:rack' %></code></p>
+ <p>or in development:</p>
+ <p><code><%= link_to_example_search 'uses:rack' %></code></p>
+
+ <p>Lastly, you can restrict your search to gems created or updated in certain timeframe:</p>
+ <p><code><%= link_to_example_search "name:rack AND updated_at:[#{Time.now.to_date.beginning_of_month.to_s(:db)} TO #{Time.now.to_date.end_of_month.to_s(:db)}]" %></code></p>
+
+ <p class="legend">The searchable fields are <em>name</em>, <em>summary</em>, <em>description</em>, <em>author</em>, <em>version</em>, <em>uses</em>, <em>depends</em>, <em>created_at</em>, <em>updated_at</em> and <em>downloads</em>.</p>
+
+</div>
+</div>
View
10 app/views/searches/show.html.erb
@@ -1,6 +1,14 @@
<% @title = "search" %>
+
+<% @subtitle = t('.subtitle', :query => nil) if params[:query].present? %>
+<%= form_tag search_url, :id => "in-page-search", :method => :get do %>
+ <%= text_field_tag :query, params[:query] if params[:query].present? %>
+ <a href="#" id="search-tips-toggle" title="<%= t '.tips_tooltip' %>"><%= t '.tips' %></a>
+<% end %>
+
+<%= render :partial => 'search_tips' %>
+
<% if @gems %>
- <% @subtitle = t('.subtitle', :query => content_tag(:em, h(params[:query]))) %>
<% if @exact_match %>
<p><%= t '.exact_match' %></p>
<div class="gems border">
View
3  config/environments/test.rb
@@ -1,3 +1,6 @@
+require 'webmock' # Allow connections to elasticsearch
+WebMock.disable_net_connect!(:allow => /localhost\:9200/)
+
Gemcutter::Application.configure do
config.cache_classes = true
config.whiny_nils = true
View
2  config/locales/en.yml
@@ -166,6 +166,8 @@ en:
show:
subtitle: "for %{query}"
exact_match: Exact match
+ tips: Tips
+ tips_tooltip: "Show search tips"
sessions:
new:
View
46 features/search.feature
@@ -1,3 +1,5 @@
+@search
+
Feature: Search
In order to find a gem I want
As a ruby developer
@@ -11,11 +13,8 @@ Feature: Search
| name: twitter | social junk |
| name: twitter-cli | command line |
| name: beer_laser | amazing beer |
- When I go to the homepage
- And I fill in "query" with "<query>"
- And I press "Search"
- Then I should see "search for <query>"
- And I should see "<result>"
+ When I search for "<query>"
+ Then I should see "<result>"
Examples:
| query | result |
@@ -29,9 +28,7 @@ Feature: Search
Given the following version exists:
| rubygem | description |
| name: foos-paperclip | paperclip |
- When I go to the homepage
- And I fill in "query" with "paperclip"
- And I press "Search"
+ When I search for "paperclip"
Then I should not see "Exact match"
But I should see "foos-paperclip"
@@ -48,9 +45,7 @@ Feature: Search
Given the following version exists:
| rubygem | number | indexed |
| name: RGem | 1.0.0 | false |
- When I go to the homepage
- And I fill in "query" with "RGem"
- And I press "Search"
+ When I search for "RGem"
Then I should not see "RGem (1.0.0)"
Scenario: The most recent version of a gem is yanked
@@ -59,8 +54,29 @@ Feature: Search
| name: RGem | 1.2.1 | true |
| name: RGem | 1.2.2 | false |
| name: RGem2 | 2.0.0 | true |
- When I go to the homepage
- And I fill in "query" with "RGem"
- And I press "Search"
- And I should see "RGem (1.2.1)"
+ When I search for "RGem"
+ Then I should see "RGem (1.2.1)"
And I should not see "RGem (1.2.2)"
+
+ Scenario: The most downloaded gem is listed first
+ Given a rubygem "Cereal-Bowl" exists with version "0.0.1" and 500 downloads
+ And a rubygem "Cereal" exists with version "0.0.9" and 5 downloads
+ When I search for "cereal"
+ Then I should see these search results:
+ | Cereal-Bowl (0.0.1) |
+ | Cereal (0.0.9) |
+
+ Scenario: The most downloaded gem is listed first and the rest of results is ordered alphabetically
+ Given a rubygem "Straight-F" exists with version "0.0.1" and 10 downloads
+ And a rubygem "Straight-B" exists with version "0.0.1" and 0 downloads
+ And a rubygem "Straight-A" exists with version "0.0.1" and 0 downloads
+ When I search for "straight"
+ Then I should see these search results:
+ | Straight-F (0.0.1) |
+ | Straight-A (0.0.1) |
+ | Straight-B (0.0.1) |
+
+ Scenario: The user enters a search query with incorrect syntax
+ When I search for "bang!"
+ Then I should not see /Displaying.*Rubygem/
+ But I should see "Sorry, your query is incorrect."
View
29 features/search_advanced.feature
@@ -0,0 +1,29 @@
+@search
+
+Feature: Search Advanced
+ In order to discover more gems
+ As a Ruby developer
+ I should be able to use advanced search on gemcutter
+
+ Scenario: Search in summaries and descriptions
+ Given the following versions exist:
+ | rubygem | number | summary | description |
+ | name: sinatra | 0.0.1 | Sinatra is a DSL ... | |
+ | name: vegas | 0.0.1 | executable versions ... Sinatra/Rack apps | |
+ | name: capybara | 0.0.1 | | ... testing Sinatra ... |
+ When I search for "sinatra"
+ Then I should see these search results:
+ | capybara (0.0.1) |
+ | sinatra (0.0.1) |
+ | vegas (0.0.1) |
+
+Scenario: Searching in authors
+ Given gems with these properties exist:
+ | name | version | authors | downloads |
+ | sinatra | 0.0.1 | Blake Mizerany, Ryan Tomayko | 500 |
+ | beefcake | 0.0.1 | Blake Mizerany | 50 |
+ | vegas | 0.0.1 | Aaron Quint | 5 |
+ When I search for "author:blake"
+ Then I should see these search results:
+ | sinatra (0.0.1) |
+ | beefcake (0.0.1) |
View
24 features/step_definitions/gem_steps.rb
@@ -19,6 +19,11 @@
create(:version, :rubygem => rubygem, :number => version_number)
end
+Given /^a rubygem "([^\"]*)" exists with version "([^\"]*)" and (\d+) downloads$/ do |name, version, downloads|
+ rubygem = create(:rubygem_with_downloads, :name => name, downloads: downloads)
+ create(:version, :rubygem => rubygem, :number => version)
+end
+
Given /^I have a gem "([^\"]*)" with version "([^\"]*)" and homepage "([^\"]*)"$/ do |name, version, homepage|
gemspec = new_gemspec(name, version, "Gemcutter", "ruby")
gemspec.homepage = homepage
@@ -65,3 +70,22 @@
rubygem.ownerships.create :user => user
end
end
+
+Given /^gems with these properties exist:$/ do |table|
+ table.hashes.each do |row|
+ if row['downloads']
+ rubygem = FactoryGirl.create :rubygem_with_downloads, :name => row['name'], :downloads => row['downloads']
+ else
+ rubygem = FactoryGirl.create :rubygem, :name => row['name']
+ end
+
+ FactoryGirl.create(:version, :rubygem => rubygem) do |version|
+ version.number = row['version']
+ version.authors = row['authors'].split(/\s*,\s*/)
+ version.summary = row['summary']
+ version.description = row['description']
+
+ version.save
+ end
+ end
+end
View
15 features/step_definitions/search_steps.rb
@@ -0,0 +1,15 @@
+When /^I search for "([^"]*)"$/ do |query|
+ steps %{
+ When I go to the homepage
+ And I fill in "query" with "#{query}"
+ And I press "Search"
+ }
+end
+
+Then /^I should see these search results:$/ do |expected_table|
+ # TODO: Make less brittle with an explicit CSS class in the view
+ results = page.all(".gems:last-child li a strong").collect(&:text)
+
+ assert_not_empty results
+ expected_table.diff! Cucumber::Ast::Table.new( results.map { |r| Array(r) } )
+end
View
2  features/step_definitions/webhook_steps.rb
@@ -18,7 +18,7 @@
Then /^the webhook "([^\"]*)" should receive a POST with gem "([^\"]*)" at version "([^\"]*)"$/ do |web_hook_url, gem_name, version_number|
WebMock.assert_requested(:post, web_hook_url, :times => 1)
- request = WebMock::RequestRegistry.instance.requested_signatures.hash.keys.first
+ request = WebMock::RequestRegistry.instance.requested_signatures.hash.keys.last
json = MultiJson.load(request.body)
assert_equal gem_name, json["name"]
View
3  features/support/env.rb
@@ -4,6 +4,9 @@
# instead of editing this one. Cucumber will automatically load all features/**/*.rb
# files.
+require 'webmock/cucumber' # Allow connections to elasticsearch
@qrush Owner
qrush added a note

Does this mean the test suite is dependent on an elasticsearch install? How would this work on Travis, etc?

@karmi
karmi added a note

The Cucumber integration test is indeed dependent on Elasticsearch running, since that's the only way how to end-to-end test the feature? Elasticsearch is available on Travis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+WebMock.disable_net_connect!(:allow => /localhost\:9200/)
+
require 'cucumber/rails'
# Capybara defaults to XPath selectors rather than Webrat's default of CSS3. In
View
9 features/support/gemcutter.rb
@@ -1,6 +1,3 @@
-require 'webmock'
-WebMock.disable_net_connect!
-
Hostess.local = true
Capybara.app_host = "https://gemcutter.local"
@@ -19,3 +16,9 @@
FileUtils.rm_rf(TEST_DIR)
$redis.flushdb
end
+
+Before('@search') do |s|
+ Rails.logger.debug "[TIRE] Recreating the elasticsearch index"
+ Rubygem.tire.index.delete
+ Rubygem.tire.create_elasticsearch_index
+end
View
BIN  public/images/help.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
13 public/javascripts/application.js
@@ -2,4 +2,17 @@ $(document).ready(function() {
$('#version_for_stats').change(function() {
window.location.href = $(this).val();
});
+
+ if (window.location.hash != '#tips') { $('#search-tips').hide(); }
+ $('#search-tips-toggle').click(function(e) {
+ e.preventDefault();
+ var o = $('#search-tips');
+ if ( o.is(':visible') ) {
+ o.hide('fast');
+ window.location.hash = '';
+ } else {
+ o.show('fast');
+ window.location.hash = '#tips';
+ }
+ });
});
View
78 public/stylesheets/screen.css
@@ -1731,3 +1731,81 @@ h5#downloads {
font-weight: bold;
padding-left: 24px;
}
+
+/* In page search */
+#in-page-search {
+ margin: 0 0 0 1em;
+ padding: 0;
+ display: inline-block;
+}
+
+#in-page-search input[type="text"] {
+ padding: 0.5em 0.5em 0.75em 0.5em;
+ border: 1px solid rgba(0,0,0,0.15);
+ background: rgba(255, 255, 255, 0.6);
+ border-radius: 0.5em;
+ width: 41em;
+ bottom: 0.2em;
+ position: relative;
+}
+
+#in-page-search input[type="text"]:focus {
+ background: rgba(255, 255, 255, 0.9);
+}
+
+#search-tips-toggle {
+ color: #6b604a;
+ background: url('/images/help.png') 0 0px no-repeat;
+ padding: 4px 0 0 20px;
+ margin: 0 0 0 0.25em;
+ height: 20px;
+ display: inline-block;
+ bottom: 0.25em;
+ position: relative;
+}
+#search-tips-toggle:hover {
+ color: #300000;
+ text-decoration: underline;
+}
+
+#search-tips {
+ font-size: 80%;
+ background: rgba(255, 255, 255, 0.4);
+ border-top: 1px solid rgba(0,0,0,0.1);
+ border-bottom: 1px solid rgba(0,0,0,0.1);
+ margin: 2em 0 0 0;
+ padding: 1.75em 0 1em 0;
+}
+
+#search-tips p {
+ margin-top: 0;
+ margin-bottom: 0.5em;
+}
+
+#search-tips a.external {
+ text-decoration: underline;
+}
+
+#search-tips code {
+ color: #5E543E;
+ background: rgba(255, 255, 0, 0.4);
+ padding: 0.25em 0.75em 0.25em 0.75em;
+}
+#search-tips code:hover {
+ color: #fff;
+ background: rgba(0, 0, 0, 0.6);
+}
+
+#search-tips code a {
+ color: #5E543E;
+}
+#search-tips code:hover a {
+ color: #fff;
+}
+
+#search-tips .legend {
+ color: #85775b;
+ border-top: 1px solid rgba(0,0,0,0.05);
+ margin-top: 1.5em;
+ padding-top: 1em;
+}
View
10 test/factories.rb
@@ -62,6 +62,11 @@
linkset
name
+ after(:create) do |this|
+ this.touch
+ Rubygem.tire.index.refresh
+ end
+
factory :rubygem_with_downloads do
after(:create) do |r|
$redis[Download.key(r)] = r['downloads']
@@ -84,6 +89,11 @@
requirements "Opencv"
rubygem
size 1024
+
+ after(:create) do |this|
+ this.rubygem.touch
+ Rubygem.tire.index.refresh
+ end
end
factory :version_history do
View
16 test/functional/searches_controller_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class SearchesControllerTest < ActionController::TestCase
+ def setup
+ super
+ Rubygem.tire.index.delete
+ Rubygem.tire.create_elasticsearch_index
+ end
context 'on GET to show with no search parameters' do
setup { get :show }
@@ -60,4 +65,15 @@ class SearchesControllerTest < ActionController::TestCase
should respond_with :redirect
should redirect_to('the gem') { rubygem_path(@sinatra) }
end
+
+ context 'on GET to show with bad search query' do
+ setup { get :show, :query => 'bang!' }
+
+ should respond_with :internal_server_error
+ should render_template :show
+ should set_the_flash.now[:failure].to /query is incorrect/
+ should "see no results" do
+ assert ! page.has_content?("Results")
+ end
+ end
end
View
4 test/test_helper.rb
@@ -1,4 +1,8 @@
ENV["RAILS_ENV"] = "test"
+
+require 'webmock/test_unit' # Allow connections to elasticsearch
+WebMock.disable_net_connect!(:allow => /localhost\:9200/)
+
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
View
5 test/unit/dependencies_middleware_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class DependenciesMiddlewareTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
def app
V1MarshaledDepedencies.new
end
View
5 test/unit/dependency_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class DependencyTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should belong_to :rubygem
should belong_to :version
View
5 test/unit/download_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class DownloadTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should "load up all downloads with just raw strings and process them" do
rubygem = create(:rubygem, :name => "some-stupid13-gem42-9000")
version = create(:version, :rubygem => rubygem)
View
5 test/unit/helpers/rubygems_helper_test.rb
@@ -4,6 +4,11 @@ class RubygemsHelperTest < ActionView::TestCase
include Rails.application.routes.url_helpers
include ApplicationHelper
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should "create the directory" do
directory = link_to_directory
("A".."Z").each do |letter|
View
5 test/unit/hostess_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class HostessTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
def app
Hostess.new
end
View
5 test/unit/ownership_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class OwnershipTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should "be valid with factory" do
assert build(:ownership).valid?
end
View
5 test/unit/pusher_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class PusherTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
context "getting the server path" do
should "return just the root server path with no args" do
assert_equal "#{Rails.root}/server", Pusher.server_path
View
12 test/unit/rubygem_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class RubygemTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
context "with a saved rubygem" do
setup do
@rubygem = create(:rubygem, :name => "SomeGem")
@@ -283,6 +288,13 @@ class RubygemTest < ActiveSupport::TestCase
assert !@rubygem.owned_by?(nil)
assert @rubygem.unowned?
end
+
+ should "not fail when cannot connect to Elasticsearch on save" do
+ WebMock::API.stub_request(:any, /.*localhost:9200.*/).to_raise(Errno::ECONNREFUSED)
+ assert_nothing_raised do
+ @rubygem.save # Calls @rubygem.update_elasticsearch_index_with_rescue
+ end
+ end
end
context "with subscribed users" do
View
5 test/unit/subscription_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class SubscriptionTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should belong_to :rubygem
should belong_to :user
View
5 test/unit/user_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class UserTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should have_many(:ownerships)
should have_many(:rubygems).through(:ownerships)
should have_many(:subscribed_gems).through(:subscriptions)
View
5 test/unit/version_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class VersionTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should belong_to :rubygem
should have_many :dependencies
View
13 test/unit/web_hook_test.rb
@@ -1,6 +1,11 @@
require 'test_helper'
class WebHookTest < ActiveSupport::TestCase
+ def setup
+ super
+ WebMock.stub_request(:any, /.*localhost:9200.*/).to_return(:body => '{}', :status => 200)
+ end
+
should belong_to :user
should belong_to :rubygem
@@ -171,9 +176,9 @@ class WebHookTest < ActiveSupport::TestCase
context "with a non-global hook job" do
setup do
@url = 'http://example.com/gemcutter'
- @rubygem = create(:rubygem)
- @version = create(:version, :rubygem => @rubygem)
- @hook = create(:web_hook,
+ @rubygem = build(:rubygem)
+ @version = build(:version, :rubygem => @rubygem)
+ @hook = build(:web_hook,
:rubygem => @rubygem,
:url => @url)
stub_request(:post, @url)
@@ -182,7 +187,7 @@ class WebHookTest < ActiveSupport::TestCase
end
should "include an Authorization header" do
- request = WebMock::RequestRegistry.instance.requested_signatures.hash.keys.first
+ request = WebMock::RequestRegistry.instance.requested_signatures.hash.keys.last
authorization = Digest::SHA2.hexdigest(@rubygem.name + @version.number + @hook.user.api_key)
assert_equal authorization, request.headers['Authorization']
View
BIN  vendor/cache/ansi-1.4.3.gem
Binary file not shown
View
BIN  vendor/cache/hashr-0.0.22.gem
Binary file not shown
View
BIN  vendor/cache/tire-0.6.0.gem
Binary file not shown
Something went wrong with that request. Please try again.