New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improved search feature (elasticsearch based, demo available) #455
Changes from all commits
e18cfb2
6632bc2
79b6857
70f0f1f
2e56fcb
611774a
2389438
bfd2aa7
db6e1d3
aa63c84
415ba28
f0d20d4
5c13726
ac96085
0346d62
4eb11a8
d389af0
cf5beec
01f479b
382b3a6
c1f99aa
06f2626
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,3 +9,5 @@ language: ruby | |
rvm: | ||
- 1.9.3 | ||
script: bundle exec rake default | ||
services: | ||
- elasticsearch |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
module SearchesHelper | ||
|
||
def link_to_example_search(query) | ||
link_to query, search_url( :query => query, :anchor => 'tips' ) | ||
end | ||
|
||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,8 @@ | ||
class Rubygem < ActiveRecord::Base | ||
include Patterns | ||
|
||
include Tire::Model::Search | ||
|
||
has_many :owners, :through => :ownerships, :source => :user | ||
has_many :ownerships, :dependent => :destroy | ||
has_many :subscribers, :through => :subscriptions, :source => :user | ||
|
@@ -15,6 +17,50 @@ class Rubygem < ActiveRecord::Base | |
after_create :update_unresolved | ||
before_destroy :mark_unresolved | ||
|
||
after_create :update_elasticsearch_index_with_rescue | ||
after_destroy :update_elasticsearch_index_with_rescue | ||
after_touch :update_elasticsearch_index_with_rescue | ||
|
||
tire do | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All of this is perfect for a Concern module, something like class Rubygem < ActiveRecord::Base
include Searchable And that module has all of the necessary includes, methods, etc. Any thoughts about that approach? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nothing against such approach -- normally, I like to keep mapping/etc definitions inside the model, and since the |
||
index_prefix Rails.env | ||
|
||
settings :number_of_shards => 1, | ||
:number_of_replicas => 1, | ||
:analysis => { | ||
:analyzer => { | ||
:rubygem => { | ||
:type => 'pattern', | ||
:pattern => "[\s#{Regexp.escape(SPECIAL_CHARACTERS)}]+" | ||
} | ||
} | ||
} do | ||
mapping do | ||
indexes :name, :type => 'multi_field', | ||
:fields => { | ||
:name => { :type => 'string', :analyzer => 'rubygem', :boost => 10.0 }, | ||
:raw => { :type => 'string', :analyzer => 'keyword', :boost => 10.0 } | ||
} | ||
indexes :indexed, :type => 'boolean', :include_in_all => false, :as => proc { versions.any?(&:indexed?) } | ||
indexes :downloads, :type => 'integer', :include_in_all => false | ||
|
||
indexes :summary, :analyzer => 'english', :as => proc { versions.most_recent.try(:summary) } | ||
indexes :description, :analyzer => 'english', :as => proc { versions.most_recent.try(:description) } | ||
indexes :author, :as => proc { versions.most_recent.try(:authors).try(:split, /\s*,\s*/) } | ||
|
||
indexes :version, :analyzer => 'keyword', :as => proc { versions.map(&:number) }, | ||
:include_in_all => false | ||
|
||
indexes :uses, :as => proc { versions.most_recent.dependencies.map(&:name) if versions.most_recent rescue nil }, | ||
:include_in_all => false | ||
indexes :depends, :as => proc { versions.most_recent.dependencies.runtime.map(&:name) if versions.most_recent rescue nil }, | ||
:include_in_all => false | ||
|
||
indexes :created_at, :type => 'date', :include_in_all => false | ||
indexes :updated_at, :type => 'date', :include_in_all => false | ||
end | ||
end | ||
end | ||
|
||
def self.with_versions | ||
where("rubygems.id IN (SELECT rubygem_id FROM versions where versions.indexed IS true)") | ||
end | ||
|
@@ -268,6 +314,13 @@ def gittip_enabled? | |
owners.where('gittip_username is not null').count > 0 | ||
end | ||
|
||
def update_elasticsearch_index_with_rescue | ||
update_elasticsearch_index | ||
rescue Exception => e | ||
Rails.logger.error "Error when updating Elasticsearch. Original exception: #{e.inspect}" | ||
return true | ||
end | ||
|
||
private | ||
|
||
def ensure_name_format | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
<div id="search-tips"> | ||
<div> | ||
<p> | ||
When looking for gems, you can use a wide variety of search queries | ||
in the <a href="http://lucene.apache.org/core/3_6_1/queryparsersyntax.html" class="external">Lucene syntax</a>. | ||
</p> | ||
|
||
<p> | ||
Quite simply, you can search in gem names, summaries and descriptions with queries like | ||
<code><%= link_to_example_search 'rack' %></code> or | ||
<code><%= link_to_example_search 'imap' %></code> | ||
</p> | ||
|
||
<p>You can, of course, restrict the search to gem names only:</p> | ||
<p><code><%= link_to_example_search 'name:rack' %></code></p> | ||
|
||
<p>To broaden your search, you can use wildcards:</p> | ||
<p> | ||
<code><%= link_to_example_search 'name:ra*' %></code> or | ||
<code><%= link_to_example_search 'web*' %></code> | ||
</p> | ||
|
||
<p>You can search for specific gem authors:</p> | ||
<p><code><%= link_to_example_search 'author:john' %></code></p> | ||
|
||
<p>Of course, you can combine these queries into complex ones:</p> | ||
<p> | ||
<code><%= link_to_example_search 'name:ra* AND author:john' %></code> or | ||
<code><%= link_to_example_search 'name:ra* AND version:1*' %></code> | ||
</p> | ||
|
||
<p>To discover more gems, you can search by their depencies in runtime:</p> | ||
<p><code><%= link_to_example_search 'depends:rack' %></code></p> | ||
<p>or in development:</p> | ||
<p><code><%= link_to_example_search 'uses:rack' %></code></p> | ||
|
||
<p>Lastly, you can restrict your search to gems created or updated in certain timeframe:</p> | ||
<p><code><%= link_to_example_search "name:rack AND updated_at:[#{Time.now.to_date.beginning_of_month.to_s(:db)} TO #{Time.now.to_date.end_of_month.to_s(:db)}]" %></code></p> | ||
|
||
<p class="legend">The searchable fields are <em>name</em>, <em>summary</em>, <em>description</em>, <em>author</em>, <em>version</em>, <em>uses</em>, <em>depends</em>, <em>created_at</em>, <em>updated_at</em> and <em>downloads</em>.</p> | ||
|
||
</div> | ||
</div> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this cover the case where ES is completely unavailable/disconnected?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, that would have to be handled by a separate
rescue_from
clause, displaying an error such as "We're sorry, search is currently not available".