test/integration/delta_test.rb @@ -1,4 +1,6 @@ +v1.9. Delta indexing. ERb now supported in .base files. Allow setting the searched indexes at runtime. + v1.8.1. Use multifind/multiget for record loading; avoid using HashWithIndifferentAccess internally for speed; other minor performance improvements. v1.8. Update client for compatibility with Sphinx 0.9.8 r1112. This is a breaking release! You need to update Sphinx along with Ultrasphinx. Float range bugfix; text faceting on association_include bugfix. Postgres users, please note that the return type of CRC32() has changed to bigint. CHANGELOG @@ -15,12 +15,16 @@ You will <b>not</b> want to keep the generated <tt>development.conf</tt> in the It's easy to keep the search daemon and the indexer running in a production environment. Cronjobs are the best way: - 0,30 * * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \ - rake ultrasphinx:index >> /log/ultrasphinx-index.log 2>&1' + */6 * * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \ + rake ultrasphinx:index:delta >> /log/ultrasphinx-index.log 2>&1' + 1 4 * * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \ + rake ultrasphinx:index:main >> /log/ultrasphinx-index.log 2>&1' */3 * * * * bash -c 'cd /path/to/production/current/; RAILS_ENV=production \ rake ultrasphinx:daemon:start >> /log/ultrasphinx-daemon.log 2>&1' -The first line runs the indexer every thirty minutes. The second line will try to restart the search daemon every three minutes. If it's already running, nothing happens. +The first line reindexes the delta index every 10 minutes. The second line reindexes the main index once a day at 4am. The third line will try to restart the search daemon every three minutes. If it's already running, nothing happens. + +Of course if you don't have any models with deltas, don't include the <tt>ultrasphinx:index:delta</tt> task. If you are under severe memory limitations you might want to manage the daemon with Monit instead, so you can keep a closer eye on it. The search daemon is extremely reliable, so don't bother with fancy monitoring infrastructure unless you're sure you need it. DEPLOYMENT_NOTES @@ -4,7 +4,9 @@ These Rake tasks are made available to your Rails app: <tt>ultrasphinx:configure</tt>:: Rebuild the configuration file for this particular environment. -<tt>ultrasphinx:index</tt>:: Reindex the database and send an update signal to the search daemon. +<tt>ultrasphinx:index</tt>:: Reindex and rotate all indexes. +<tt>ultrasphinx:index:delta</tt>:: Reindex and rotate the delta index. +<tt>ultrasphinx:index:main</tt>:: Reindex and rotate the main index. <tt>ultrasphinx:daemon:restart</tt>:: Restart the search daemon. <tt>ultrasphinx:daemon:start</tt>:: Start the search daemon. <tt>ultrasphinx:daemon:stop</tt>:: Stop the search daemon. @@ -12,4 +14,4 @@ These Rake tasks are made available to your Rails app: <tt>ultrasphinx:spelling:build</tt>:: Rebuild the custom spelling dictionary. You may need to use <tt>sudo</tt> if your Aspell folder is not writable by the app user. <tt>ultrasphinx:bootstrap</tt>:: Bootstrap a full Sphinx environment by running configure, index, then daemon:start. -All tasks have shortcuts. Use <tt>us:conf</tt>, <tt>us:in</tt>, <tt>us:restart</tt>, <tt>us:start</tt>, <tt>us:stop</tt>, <tt>us:stat</tt>, <tt>us:spell</tt>, and <tt>us:boot</tt>. \ No newline at end of file +All tasks have shortcuts. Use <tt>us:conf</tt>, <tt>us:index</tt>, <tt>us:main</tt>, <tt>us:delta</tt>, <tt>us:restart</tt>, <tt>us:start</tt>, <tt>us:stop</tt>, <tt>us:stat</tt>, <tt>us:spell</tt>, and <tt>us:boot</tt>. \ No newline at end of file RAKE_TASKS @@ -14,7 +14,7 @@ If you use this software, please {make a donation}[http://blog.evanweaver.com/do == Requirements * MySQL 5.0, or PostgreSQL 8.2 -* Sphinx 0.9.8-dev r1112 +* Sphinx 0.9.8-rc1 * Rails 2.0.2 More recent versions than listed are usually ok. @@ -26,14 +26,15 @@ Sphinx/Ultrasphinx is the fastest and most stable Rails fulltext search solution Features include: * searching and ranking across orthogonal models +* delta index support * excerpt highlighting * Google-style query parser * spellcheck * faceting on text, date, and numeric fields * field weighting, merging, and aliases * <tt>belongs_to</tt> and <tt>has_many</tt> includes -* drop-in compatibility with Interlock[http://blog.evanweaver.com/files/doc/fauna/interlock/] * drop-in compatibility with will_paginate[http://err.lighthouseapp.com/projects/466/home] +* drop-in compatibility with Interlock[http://blog.evanweaver.com/files/doc/fauna/interlock/] * multiple deployment environments * comprehensive Rake tasks @@ -53,7 +54,7 @@ Then, install the plugin: Next, copy the <tt>examples/default.base</tt> file to <tt>RAILS_ROOT/config/ultrasphinx/default.base</tt>. This file sets up the Sphinx daemon options such as port, host, and index location. -If you need per-environment configuration, you can use <tt>RAILS_ROOT/config/ultrasphinx/development.base</tt>, etc. +If you need per-environment configuration, you can use <tt>RAILS_ROOT/config/ultrasphinx/development.base</tt>, etc. Note that ERb is also allowed within the <tt>.base</tt> files, and can be an alternative way to DRY up multiple configurations. Now, in your models, use the <tt>is_indexed</tt> method to configure a model as searchable. For example: @@ -93,6 +94,23 @@ Once the <tt>@search</tt> object has been <tt>run</tt>, it is directly compatibl == Spell checking See Ultrasphinx::Spell. + +== Delta indexing + +Delta indexing speeds up your updates by not reindexing the entire dataset every time. + +First, in your <tt>.base</tt> file, set the indexer option <tt>delta</tt> to your maximum interval between full reindexes. A day or a week is good, depending. Add a little bit to account for the time it takes the actual index to run: + + delta = <%= 1.day + 30.minutes %> + +Now, configure your models for delta indexing in the <tt>is_indexed</tt> call: + + is_indexed :fields => ['created_at', 'title', 'body'], + :delta => true + +Now you can run <tt>rake ultrasphinx:index:delta</tt> frequently, and only records that were changed within 1 day will be reindexed. You will need to run <tt>rake ultrasphinx:index:main</tt> once a day to move the delta contents into the main index. + +See ActiveRecord::Base .is_indexed and DEPLOYMENT_NOTES[link:files/DEPLOYMENT_NOTES.html] for more. == Available Rake tasks README @@ -8,6 +8,5 @@ Planned: Not sure: -* Delta indexing * Use Pat Allan's association configurator, possibly with an API change to avoid the message-passing DSL * Use Treetop for the query parser instead of regexes TODO @@ -3,8 +3,8 @@ # Sphinx/Ultrasphinx user-configurable options. # # Copy this file to RAILS_ROOT/config/ultrasphinx. You can use individual -# namespaces if you want (e.g. development.base, production.base, -# test.base). +# namespaces if you want (e.g. development.base, production.base, +# test.base). Note that ERb is also allowed. # # This file should not be handed directly to Sphinx. Use the rake task # @@ -15,56 +15,61 @@ # to Sphinx. # # It is safe to edit .base files by hand. It is not safe to edit the generated -# .conf files. Do not symlink the .conf file to the .base file! I don't know why -# people think they need to do that. It's wrong. +# .conf files. Do not symlink the .conf file to the .base file; it's wrong. # +<% path = "/opt/local/var/db/sphinx/" %> + +# Indexing options indexer -{ - # Indexer running options +{ mem_limit = 256M + + # Ultrasphinx-specific key + delta = <%= 1.day + 30.minutes %> } +# Daemon options searchd -{ - # Daemon options +{ # What interface the search daemon should listen on and where to store its logs address = 0.0.0.0 port = 3312 seamless_rotate = 1 - log = /opt/local/var/db/sphinx/log/searchd.log - query_log = /opt/local/var/db/sphinx/log/query.log + log = <%= path %>log/searchd.log + query_log = <%= path %>log/query.log read_timeout = 5 max_children = 300 - pid_file = /opt/local/var/db/sphinx/log/searchd.pid + pid_file = <%= path %>log/searchd.pid max_matches = 100000 } +# Client options client { - # Client options # Name of the Aspell dictionary (two letters max) dictionary_name = ap + # How your application connects to the search daemon (not necessarily the same as above) server_host = localhost server_port = 3312 } +# Individual SQL source options source -{ - # Individual SQL source options +{ sql_ranged_throttle = 0 sql_range_step = 5000 sql_query_post = } +# Index building options index -{ - # Index building options - path = /opt/local/var/db/sphinx/ - docinfo = extern # just leave this alone +{ + path = <%= path %> + docinfo = extern # Just leave this alone morphology = stem_en - stopwords = # /path/to/stopwords.txt + stopwords = # <%= path %>/ap-stopwords.txt min_word_len = 1 # HTML-specific options examples/default.base @@ -17,7 +17,7 @@ require 'ultrasphinx/associations' require 'ultrasphinx/core_extensions' require 'ultrasphinx/is_indexed' -if (ActiveRecord::Base.connection rescue nil) # XXX Forget why this needed to be wrapped +if (ActiveRecord::Base.connection rescue nil) # XXX Not sure why this needed to be wrapped. require 'ultrasphinx/configure' require 'ultrasphinx/autoload' require 'ultrasphinx/fields' lib/ultrasphinx.rb @@ -41,21 +41,33 @@ module Ultrasphinx say "rebuilding configurations for #{RAILS_ENV} environment" say "available models are #{MODEL_CONFIGURATION.keys.to_sentence}" - File.open(CONF_PATH, "w") do |conf| - + File.open(CONF_PATH, "w") do |conf| conf.puts global_header - sources = [] - - say "generating SQL" - cached_groups = Fields.instance.groups.join("\n") - MODEL_CONFIGURATION.each_with_index do |model_options, class_id| - model, options = model_options - klass, source = model.constantize, model.tableize.gsub('/', '__') - sources << source - conf.puts build_source(Fields.instance, model, options, class_id, klass, source, cached_groups) + say "generating SQL" + + INDEXES.each do |index| + sources = [] + cached_groups = Fields.instance.groups.join("\n") + + MODEL_CONFIGURATION.each_with_index do |model_and_options, class_id| + # This relies on hash sort order being deterministic per-machine + model, options = model_and_options + klass = model.constantize + source = "#{model.tableize.gsub('/', '__')}_#{index}" + + if index != DELTA_INDEX or options['delta'] + # If we are building the delta, we only want to include the models that requested it + conf.puts build_source(index, Fields.instance, model, options, class_id, klass, source, cached_groups) + sources << source + end + end + + if sources.any? + # Don't generate a delta index if there are no delta tables + conf.puts build_index(index, sources) + end + end - - conf.puts build_index(sources) end end @@ -68,7 +80,8 @@ module Ultrasphinx ["\n# Auto-generated at #{Time.now}.", "# Hand modifications will be overwritten.", "# #{BASE_PATH}\n", - INDEXER_SETTINGS._to_conf_string('indexer'), + INDEXER_SETTINGS.except('delta')._to_conf_string('indexer'), + "", DAEMON_SETTINGS._to_conf_string("searchd")] end @@ -86,9 +99,31 @@ module Ultrasphinx end conf.sort.join("\n") end + + + def build_delta_condition(index, klass, options) + if index == DELTA_INDEX and options['delta'] + # Add delta condition if necessary + table, field = klass.table_name, options['delta']['field'] + source_string = "#{table}.#{field}" + delta_column = klass.columns_hash[field] + + if delta_column + raise ConfigurationError, "#{source_string} is not a :datetime" unless delta_column.type == :datetime + if (options['fields'] + options['concatenate'] + options['include']).detect { |entry| entry['sortable'] } + # Warning about the sortable problem + # XXX Kind of in an odd place, but I want to happen at index time + Ultrasphinx.say "warning; text sortable columns on #{klass.name} will return wrong results with partial delta indexing" + end + string = "#{source_string} > #{SQL_FUNCTIONS[ADAPTER]['delta']._interpolate(INDEXER_SETTINGS['delta'])}"; + else + Ultrasphinx.say "warning; #{klass.name} will reindex the entire table during delta indexing" + end + end + end - def setup_source_arrays(klass, fields, class_id, conditions) + def setup_source_arrays(index, klass, fields, class_id, conditions) condition_strings = Array(conditions).map do |condition| "(#{condition})" end @@ -101,12 +136,13 @@ module Ultrasphinx end - def range_select_string(klass) + def range_select_string(klass, delta_condition) ["sql_query_range = SELECT", SQL_FUNCTIONS[ADAPTER]['range_cast']._interpolate("MIN(#{klass.primary_key})"), - ", ", + ",", SQL_FUNCTIONS[ADAPTER]['range_cast']._interpolate("MAX(#{klass.primary_key})"), - "FROM #{klass.table_name}" + "FROM #{klass.table_name}", + ("WHERE #{delta_condition}" if delta_condition), ].join(" ") end @@ -116,11 +152,16 @@ module Ultrasphinx end - def build_source(fields, model, options, class_id, klass, source, groups) + def build_source(index, fields, model, options, class_id, klass, source, groups) column_strings, join_strings, condition_strings, group_bys, use_distinct, remaining_columns = setup_source_arrays( - klass, fields, class_id, options['conditions']) + index, klass, fields, class_id, options['conditions']) + + delta_condition = + build_delta_condition( + index, klass, options) + condition_strings << delta_condition if delta_condition column_strings, join_strings, group_bys, remaining_columns = build_regular_fields( @@ -140,7 +181,7 @@ module Ultrasphinx "source #{source}\n{", SOURCE_SETTINGS._to_conf_string, setup_source_database(klass), - range_select_string(klass), + range_select_string(klass, delta_condition), build_query(klass, column_strings, join_strings, condition_strings, use_distinct, group_bys), "\n" + groups, query_info_string(klass, class_id), @@ -271,13 +312,13 @@ module Ultrasphinx end - def build_index(sources) + def build_index(index, sources) ["\n# Index configuration\n\n", - "index #{UNIFIED_INDEX_NAME}\n{", + "index #{index}\n{", sources.sort.map do |source| " source = #{source}" end.join("\n"), - INDEX_SETTINGS.merge('path' => INDEX_SETTINGS['path'] + "/sphinx_index_#{UNIFIED_INDEX_NAME}")._to_conf_string, + INDEX_SETTINGS.merge('path' => INDEX_SETTINGS['path'] + "/sphinx_index_#{index}")._to_conf_string, "}\n\n"] end lib/ultrasphinx/configure.rb @@ -58,7 +58,7 @@ The keys <tt>:facet</tt>, <tt>:sortable</tt>, <tt>:class_name</tt>, <tt>:associa == Concatenating several fields within one record -Use the <tt>:concatenate</tt> key (MySQL only). +Use the <tt>:concatenate</tt> key. Accepts an array of option hashes. @@ -88,6 +88,26 @@ Also, If you want to include a model that you don't have an actual ActiveRecord Ultrasphinx is not an object-relational mapper, and the association generation is intended to stay minimal--don't be afraid of <tt>:association_sql</tt>. +== Enabling delta indexing + +Use the <tt>:delta</tt> key. + +Accepts either <tt>true</tt>, or a hash with a <tt>:field</tt> key. + +If you pass <tt>true</tt>, the <tt>updated_at</tt> column will be used for choosing the delta records, if it exists. If it doesn't exist, the entire table will be reindexed at every delta. Example: + + :delta => true + +If you need to use a non-default column name, use a hash: + + :delta => {:field => 'created_at'} + +Note that the column type must be time-comparable in the DB. Also note that faceting may return higher counts than actually exist on delta-indexed tables, and that sorting by string columns will not work well. These are both limitations of Sphinx's index merge scheme. You can perhaps mitigate the issues by only searching the main index for facets or sorts: + + Ultrasphinx::Search.new(:query => "query", :indexes => Ultrasphinx::MAIN_INDEX) + +The date range of the delta include is set in the <tt>.base</tt> file. + = Examples == Complex configuration @@ -110,6 +130,7 @@ Here's an example configuration using most of the options, taken from production {:association_name => 'comments', :field => 'body', :as => 'comments', :conditions => "comments.item_type = '#{base_class}'"} ], + :delta => {:field => 'published_at'}, :conditions => self.live_condition_string end @@ -137,16 +158,38 @@ If the associations weren't just <tt>has_many</tt> and <tt>belongs_to</tt>, you def self.is_indexed opts = {} opts = HashWithIndifferentAccess.new(opts) - opts.assert_valid_keys ['fields', 'concatenate', 'conditions', 'include'] + opts.assert_valid_keys ['fields', 'concatenate', 'conditions', 'include', 'delta'] + + # Single options - Array(opts['fields']).each do |entry| + if opts['conditions'] + # Do nothing + end + + if opts['delta'] + if opts['delta'] == true + opts['delta'] = {'field' => 'updated_at'} + elsif opts['delta'].is_a? String + opts['delta'] = {'field' => opts['delta']} + end + opts['delta'].stringify_keys! + opts['delta'].assert_valid_keys ['field'] + end + + # Enumerable options + + opts['fields'] = Array(opts['fields']) + opts['concatenate'] = Array(opts['concatenate']) + opts['include'] = Array(opts['include']) + + opts['fields'].each do |entry| if entry.is_a? Hash entry.stringify_keys! entry.assert_valid_keys ['field', 'as', 'facet', 'function_sql', 'sortable'] end end - Array(opts['concatenate']).each do |entry| + opts['concatenate'].each do |entry| entry.stringify_keys! entry.assert_valid_keys ['class_name', 'association_name', 'conditions', 'field', 'as', 'fields', 'association_sql', 'facet', 'function_sql', 'sortable'] raise Ultrasphinx::ConfigurationError, "You can't mix regular concat and group concats" if entry['fields'] and (entry['field'] or entry['class_name'] or entry['association_name']) @@ -155,11 +198,11 @@ If the associations weren't just <tt>has_many</tt> and <tt>belongs_to</tt>, you raise Ultrasphinx::ConfigurationError, "Regular concatenations should have multiple fields" if entry['fields'] and !entry['fields'].is_a?(Array) end - Array(opts['include']).each do |entry| + opts['include'].each do |entry| entry.stringify_keys! entry.assert_valid_keys ['class_name', 'association_name', 'field', 'as', 'association_sql', 'facet', 'function_sql', 'sortable'] end - + Ultrasphinx::MODEL_CONFIGURATION[self.name] = opts end end lib/ultrasphinx/is_indexed.rb @@ -42,6 +42,7 @@ The hash lets you customize internal aspects of the search. <tt>:weights</tt>:: A hash. Text-field names and associated query weighting. The default weight for every field is 1.0. Example: <tt>:weights => {'title' => 2.0}</tt> <tt>:filters</tt>:: A hash. Names of numeric or date fields and associated values. You can use a single value, an array of values, or a range. (See the bottom of the ActiveRecord::Base page for an example.) <tt>:facets</tt>:: An array of fields for grouping/faceting. You can access the returned facet values and their result counts with the <tt>facets</tt> method. +<tt>:indexes</tt>:: An array of indexes to search. Currently only <tt>Ultrasphinx::MAIN_INDEX</tt> and <tt>Ultrasphinx::DELTA_INDEX</tt> are available. Defaults to both; changing this is rarely needed. Note that you can set up your own query defaults in <tt>environment.rb</tt>: @@ -100,6 +101,10 @@ Note that your database is never changed by anything Ultrasphinx does. :per_page => 20, :sort_by => nil, :sort_mode => 'relevance', + :indexes => [ + MAIN_INDEX, + (DELTA_INDEX if Ultrasphinx.delta_index_present?) + ].compact, :weights => {}, :class_names => [], :filters => {}, @@ -125,7 +130,8 @@ Note that your database is never changed by anything Ultrasphinx does. :max_missing_records => 5, :max_retries => 4, :retry_sleep_time => 0.5, - :max_facets => 100, + :max_facets => 1000, + :max_matches_offset => 1000, # Whether to add an accessor to each returned result that specifies its global rank in # the search. :with_global_rank => false, @@ -149,45 +155,12 @@ Note that your database is never changed by anything Ultrasphinx does. INTERNAL_KEYS = ['parsed_query'] #:nodoc: - def self.get_models_to_class_ids #:nodoc: - # Reading the conf file makes sure that we are in sync with the actual Sphinx index, - # not whatever you happened to change your models to most recently - unless File.exist? CONF_PATH - Ultrasphinx.say "configuration file not found for #{RAILS_ENV.inspect} environment" - Ultrasphinx.say "please run 'rake ultrasphinx:configure'" - else - begin - lines = open(CONF_PATH).readlines - - sources = lines.select do |line| - line =~ /^source \w/ - end.map do |line| - line[/source ([\w\d_-]*)/, 1].gsub('__', '/').classify - end - - ids = lines.select do |line| - line =~ /^sql_query / - end.map do |line| - line[/(\d*) AS class_id/, 1].to_i - end - - raise unless sources.size == ids.size - Hash[*sources.zip(ids).flatten] - - rescue - Ultrasphinx.say "#{CONF_PATH} file is corrupted" - Ultrasphinx.say "please run 'rake ultrasphinx:configure'" - end - - end - end - - MODELS_TO_IDS = get_models_to_class_ids || {} + MODELS_TO_IDS = Ultrasphinx.get_models_to_class_ids || {} IDS_TO_MODELS = MODELS_TO_IDS.invert #:nodoc: - - MAX_MATCHES = DAEMON_SETTINGS["max_matches"].to_i + MAX_MATCHES = DAEMON_SETTINGS["max_matches"].to_i + FACET_CACHE = {} #:nodoc: # Returns the options hash. @@ -296,6 +269,7 @@ Note that your database is never changed by anything Ultrasphinx does. @options['query'] = @options['query'].to_s @options['class_names'] = Array(@options['class_names']) @options['facets'] = Array(@options['facets']) + @options['indexes'] = Array(@options['indexes']).join(" ") raise UsageError, "Weights must be a Hash" unless @options['weights'].is_a? Hash raise UsageError, "Filters must be a Hash" unless @options['filters'].is_a? Hash @@ -308,18 +282,26 @@ Note that your database is never changed by anything Ultrasphinx does. say "discarded invalid keys: #{extra_keys * ', '}" if extra_keys.any? and RAILS_ENV != "test" end - # Run the search, filling results with an array of ActiveRecord objects. Set the parameter to false if you only want the ids returned. + # Run the search, filling results with an array of ActiveRecord objects. Set the parameter to false + # if you only want the ids returned. def run(reify = true) @request = build_request_with_options(@options) say "searching for #{@options.inspect}" perform_action_with_retries do - @response = @request.query(parsed_query, UNIFIED_INDEX_NAME) + @response = @request.query(parsed_query, @options['indexes']) say "search returned #{total_entries}/#{response[:total_found].to_i} in #{time.to_f} seconds." if self.class.client_options['with_subtotals'] @subtotals = get_subtotals(@request, parsed_query) + + # If the original query has a filter on this class, we will use its more accurate total rather the facet's + # less accurate total. + if @options['class_names'].size == 1 + @subtotals[@options['class_names'].first] = response[:total_found] + end + end Array(@options['facets']).each do |facet| @@ -363,8 +345,8 @@ Note that your database is never changed by anything Ultrasphinx does. end.flatten excerpting_options = { - :docs => docs, - :index => UNIFIED_INDEX_NAME, + :docs => docs, + :index => MAIN_INDEX, # http://www.sphinxsearch.com/forum/view.html?id=100 :words => strip_query_commands(parsed_query) } self.class.excerpting_options.except('content_methods').each do |key, value| lib/ultrasphinx/search.rb @@ -17,7 +17,7 @@ module Ultrasphinx @match_mode = :extended # Force extended query mode @offset = opts['per_page'] * (opts['page'] - 1) @limit = opts['per_page'] - @max_matches = [@offset + @limit, MAX_MATCHES].min + @max_matches = [@offset + @limit + Ultrasphinx::Search.client_options['max_matches_offset'], MAX_MATCHES].min end # Sorting @@ -130,12 +130,12 @@ module Ultrasphinx @group_clauses = '@count desc' @offset = 0 @limit = Ultrasphinx::Search.client_options['max_facets'] - @max_matches = [@limit, MAX_MATCHES].min + @max_matches = [@limit + Ultrasphinx::Search.client_options['max_matches_offset'], MAX_MATCHES].min end # Run the query begin - matches = request.query(query, UNIFIED_INDEX_NAME)[:matches] + matches = request.query(query, options['indexes'])[:matches] rescue DaemonError raise ConfigurationError, "Index seems out of date. Run 'rake ultrasphinx:index'" end @@ -175,7 +175,7 @@ module Ultrasphinx # Concatenates might not work well type, configuration = nil, nil - MODEL_CONFIGURATION[klass.name].except('conditions').each do |_type, values| + MODEL_CONFIGURATION[klass.name].except('conditions', 'delta').each do |_type, values| type = _type configuration = values.detect { |this_field| this_field['as'] == facet } break if configuration lib/ultrasphinx/search/internals.rb @@ -32,12 +32,16 @@ module Ultrasphinx MAX_INT = 2**32-1 - MAX_WORDS = 2**16 # maximum number of stopwords built + MAX_WORDS = 2**16 # The maximum number of stopwords built - UNIFIED_INDEX_NAME = "complete" + MAIN_INDEX = "main" + + DELTA_INDEX = "delta" + + INDEXES = [MAIN_INDEX, DELTA_INDEX] CONFIG_MAP = { - # These must be symbols for key mapping against Rails itself + # These must be symbols for key mapping against Rails itself. :username => 'sql_user', :password => 'sql_pass', :host => 'sql_host', @@ -59,12 +63,14 @@ module Ultrasphinx SQL_FUNCTIONS = { 'mysql' => { 'group_concat' => "CAST(GROUP_CONCAT(DISTINCT ? SEPARATOR ' ') AS CHAR)", + 'delta' => "DATE_SUB(NOW(), INTERVAL ? SECOND)", 'hash' => "CAST(CRC32(?) AS unsigned)", 'range_cast' => "?", 'stored_procedures' => {} }, 'postgresql' => { 'group_concat' => "GROUP_CONCAT(?)", + 'delta' => "(NOW() - '? SECOND'::interval)", 'range_cast' => "cast(coalesce(?,1) AS integer)", 'hash' => "CRC32(?)", 'stored_procedures' => Hash[*( @@ -90,7 +96,8 @@ sql_query_pre = ) + SQL_FUNCTIONS['postgresql']['stored_procedures'].values.join ADAPTER = ActiveRecord::Base.connection.instance_variable_get("@config")[:adapter] rescue 'mysql' - # Install the stored procedures + # Install the stored procedures. + # XXX This shouldn't be done at every index, say the Postgres people. SQL_FUNCTIONS[ADAPTER]['stored_procedures'].each do |key, value| ActiveRecord::Base.connection.execute(value) end @@ -106,35 +113,71 @@ sql_query_pre = ) + SQL_FUNCTIONS['postgresql']['stored_procedures'].values.join else STDERR.puts msg end - end - nil + end + nil # Explicitly return nil end # Configuration file parser. def self.options_for(heading, path) - section = open(path).read[/^#{heading.gsub('/', '__')}\s*?\{(.*?)\}/m, 1] + # Evaluate ERB + template = ERB.new(File.open(path) {|f| f.read}) + contents = template.result(binding) - unless section - Ultrasphinx.say "warning; heading #{heading} not found in #{path}; it may be corrupted. " - {} - else + # Find the correct heading. + section = contents[/^#{heading.gsub('/', '__')}\s*?\{(.*?)\}/m, 1] + + if section + # Convert to a hash options = section.split("\n").map do |line| line =~ /\s*(.*?)\s*=\s*([^\#]*)/ $1 ? [$1, $2.strip] : [] end Hash[*options.flatten] - end - + else + # XXX Is it safe to raise here? + Ultrasphinx.say "warning; heading #{heading} not found in #{path}; it may be corrupted. " + {} + end end + + def self.get_models_to_class_ids #:nodoc: + # Reading the conf file makes sure that we are in sync with the actual Sphinx index, not + # whatever you happened to change your models to most recently. + if File.exist? CONF_PATH + lines, hash = open(CONF_PATH).readlines, {} + msg = "#{CONF_PATH} file is corrupted. Please run 'rake ultrasphinx:configure'." + + lines.each_with_index do |line, index| + # Find the main sources + if line =~ /^source ([\w\d_-]*)_#{MAIN_INDEX}/ + # Derive the model name + model = $1.gsub('__', '/').classify - # Introspect on the existing generated conf files + # Get the id modulus out of the adjacent sql_query + query = lines[index..-1].detect do |query_line| + query_line =~ /^sql_query / + end + raise ConfigurationError, msg unless query + hash[model] = query[/(\d*) AS class_id/, 1].to_i + end + end + raise ConfigurationError, msg unless hash.values.size == hash.values.uniq.size + hash + else + # We can't raise here because you may be generating the configuration for the first time + Ultrasphinx.say "configuration file not found for #{RAILS_ENV.inspect} environment" + Ultrasphinx.say "please run 'rake ultrasphinx:configure'" + end + end + + # Introspect on the existing generated conf files. INDEXER_SETTINGS = options_for('indexer', BASE_PATH) CLIENT_SETTINGS = options_for('client', BASE_PATH) DAEMON_SETTINGS = options_for('searchd', BASE_PATH) SOURCE_SETTINGS = options_for('source', BASE_PATH) INDEX_SETTINGS = options_for('index', BASE_PATH) - # Make sure there's a trailing slash + # Make sure there's a trailing slash. INDEX_SETTINGS['path'] = INDEX_SETTINGS['path'].chomp("/") + "/" DICTIONARY = CLIENT_SETTINGS['dictionary_name'] || 'ap' @@ -149,7 +192,7 @@ sql_query_pre = ) + SQL_FUNCTIONS['postgresql']['stored_procedures'].values.join if File.exist? CONF_PATH begin if options_for( - "source #{MODEL_CONFIGURATION.keys.first.tableize}", + "source #{MODEL_CONFIGURATION.keys.first.tableize}_#{MAIN_INDEX}", CONF_PATH )['sql_db'] != ActiveRecord::Base.connection.instance_variable_get("@config")[:database] say "warning; configured database name is out-of-date" @@ -159,5 +202,14 @@ sql_query_pre = ) + SQL_FUNCTIONS['postgresql']['stored_procedures'].values.join end end end - + + # See if a delta index was defined. + def self.delta_index_present? + if File.exist?(CONF_PATH) + File.open(CONF_PATH).readlines.detect do |line| + line =~ /^index delta/ + end + end + end + end lib/ultrasphinx/ultrasphinx.rb @@ -12,7 +12,7 @@ namespace :ultrasphinx do desc "Bootstrap a full Sphinx environment" task :bootstrap => [:_environment, :configure, :index, :"daemon:restart"] do say "done" - say "please restart Mongrel" + say "please restart your application containers" end desc "Rebuild the configuration file for this particular environment." @@ -20,32 +20,23 @@ namespace :ultrasphinx do Ultrasphinx::Configure.run end - desc "Reindex the database and send an update signal to the search daemon." - task :index => [:_environment] do - rotate = ultrasphinx_daemon_running? - index_path = Ultrasphinx::INDEX_SETTINGS['path'] - mkdir_p index_path unless File.directory? index_path - - cmd = "indexer --config '#{Ultrasphinx::CONF_PATH}'" - cmd << " #{ENV['OPTS']} " if ENV['OPTS'] - cmd << " --rotate" if rotate - cmd << " #{Ultrasphinx::UNIFIED_INDEX_NAME}" - - say cmd - system cmd - - if rotate - sleep(4) - failed = Dir[index_path + "/*.new.*"] - if failed.any? - say "warning; index failed to rotate! Deleting new indexes" - failed.each {|f| File.delete f } - else - say "index rotated ok" - end + namespace :index do + desc "Reindex and rotate the main index." + task :main => [:_environment] do + ultrasphinx_index(Ultrasphinx::MAIN_INDEX) end + + desc "Reindex and rotate the delta index." + task :delta => [:_environment] do + ultrasphinx_index(Ultrasphinx::DELTA_INDEX) + end + + end + + desc "Reindex and rotate all indexes." + task :index => [:_environment] do + ultrasphinx_index("--all") end - namespace :daemon do desc "Start the search daemon" @@ -126,30 +117,56 @@ namespace :us do task :restart => ["ultrasphinx:daemon:restart"] task :stop => ["ultrasphinx:daemon:stop"] task :stat => ["ultrasphinx:daemon:status"] + task :index => ["ultrasphinx:index"] task :in => ["ultrasphinx:index"] + task :main => ["ultrasphinx:index:main"] + task :delta => ["ultrasphinx:index:delta"] task :spell => ["ultrasphinx:spelling:build"] task :conf => ["ultrasphinx:configure"] task :boot => ["ultrasphinx:bootstrap"] end -# support methods +# Support methods def ultrasphinx_daemon_pid - open(open(Ultrasphinx::BASE_PATH).readlines.map do |line| - line[/^\s*pid_file\s*=\s*([^\s\#]*)/, 1] - end.compact.first).readline.chomp rescue nil # XXX ridiculous + open(Ultrasphinx::DAEMON_SETTINGS['pid_file']).readline.chomp rescue nil end def ultrasphinx_daemon_running? if ultrasphinx_daemon_pid and `ps #{ultrasphinx_daemon_pid} | wc`.to_i > 1 true else - # remove bogus lockfiles + # Remove bogus lockfiles. Dir[Ultrasphinx::INDEX_SETTINGS["path"] + "*spl"].each {|file| File.delete(file)} false end end +def ultrasphinx_index(index) + rotate = ultrasphinx_daemon_running? + index_path = Ultrasphinx::INDEX_SETTINGS['path'] + mkdir_p index_path unless File.directory? index_path + + cmd = "indexer --config '#{Ultrasphinx::CONF_PATH}'" + cmd << " #{ENV['OPTS']} " if ENV['OPTS'] + cmd << " --rotate" if rotate + cmd << " #{index}" + + say "$ #{cmd}" + system cmd + + if rotate + sleep(4) + failed = Dir[index_path + "/*.new.*"] + if failed.any? + say "warning; index failed to rotate! Deleting new indexes" + failed.each {|f| File.delete f } + else + say "index rotated ok" + end + end +end + def say msg Ultrasphinx.say msg end tasks/ultrasphinx.rake @@ -4,5 +4,6 @@ class Geo::Address < ActiveRecord::Base is_indexed 'fields' => ['name'], 'concatenate' => [{'fields' => ['line_1', 'line_2', 'city', 'province_region', 'zip_postal_code'], 'as' => 'content'}], - 'include' => [{'association_name' => 'state', 'field' => 'name', 'as' => 'state'}] + 'include' => [{'association_name' => 'state', 'field' => 'name', 'as' => 'state'}], + 'delta' => true end test/integration/app/app/models/geo/address.rb @@ -5,7 +5,8 @@ class User < ActiveRecord::Base is_indexed :fields => ['login', 'email', 'deleted'], :include => [{:association_name => 'specific_seller', :field => 'company_name', :as => 'company', :facet => true}, {:class_name => 'Seller', :field => 'sellers_two.company_name', :as => 'company_two', :facet => true, 'association_sql' => 'LEFT OUTER JOIN sellers AS sellers_two ON users.id = sellers_two.user_id', 'function_sql' => "REPLACE(?, '6', ' replacement ')"}], - :conditions => "deleted = '0'" + :conditions => "deleted = '0'", + :delta => {:field => 'created_at'} def self.find_all_by_id(*args) raise "Wrong finder" test/integration/app/app/models/person/user.rb @@ -8,7 +8,8 @@ class Seller < ActiveRecord::Base 'created_at', 'capitalization', 'user_id' - ] + ], + :delta => true def name company_name test/integration/app/app/models/seller.rb @@ -19,32 +19,40 @@ # people think they need to do that. It's wrong. # +<% tmp = "/tmp/sphinx/" %> + indexer { # Indexer running options mem_limit = 256M + + # Ultrasphinx-specific key + delta = <%= 1.day + 30.minutes %> } searchd { # Daemon options - # What interface the search daemon should listen on and where to store its logs + + # What interface the search daemon should listen on address = 0.0.0.0 port = 3312 seamless_rotate = 1 - log = /tmp/sphinx/searchd.log - query_log = /tmp/sphinx/query.log + + # Where to to store the logs + log = <%= tmp %>searchd.log + query_log = <%= tmp %>query.log read_timeout = 5 max_children = 300 - pid_file = /tmp/sphinx/searchd.pid + pid_file = <%= tmp %>searchd.pid max_matches = 100000 } client { - # Client options + # Ultrasphinx-specific client options # Name of the Aspell dictionary (two letters max) - dictionary_name = ap + dictionary_name = <%= dictionary = 'ap' %> # How your application connects to the search daemon (not necessarily the same as above) server_host = localhost server_port = 3312 @@ -61,10 +69,10 @@ source index { # Index building options - path = /tmp/sphinx + path = <%= tmp %> docinfo = extern # just leave this alone morphology = stem_en - stopwords = # /path/to/stopwords.txt + stopwords = # <%= tmp %>/<%= dictionary %>-stopwords.txt min_word_len = 1 # HTML-specific options test/integration/app/config/ultrasphinx/default.base @@ -1,10 +1,11 @@ -# Auto-generated at Wed Feb 13 15:31:03 -1000 2008. +# Auto-generated at Sat Mar 08 03:07:27 -0500 2008. # Hand modifications will be overwritten. -# /Users/eweaver/Desktop/projects/chow/vendor/plugins/ultrasphinx/test/integration/app/config/ultrasphinx/default.base +# /Users/eweaver/Desktop/projects/ub/vendor/plugins/ultrasphinx/test/integration/app/config/ultrasphinx/default.base indexer { mem_limit = 256M } + searchd { read_timeout = 5 max_children = 300 @@ -19,7 +20,7 @@ searchd { # Source configuration -source geo__states +source geo__states_main { sql_range_step = 5000 sql_query_post = @@ -33,7 +34,7 @@ sql_db = ultrasphinx_development sql_host = localhost sql_pass = sql_user = root -sql_query_range = SELECT MIN(id) , MAX(id) FROM states +sql_query_range = SELECT MIN(id) , MAX(id) FROM states sql_query = SELECT (states.id * 5 + 0) AS id, CAST(GROUP_CONCAT(DISTINCT addresses.name SEPARATOR ' ') AS CHAR) AS address_name, 0 AS capitalization, 'Geo::State' AS class, 0 AS class_id, '' AS company, 0 AS company_facet, '' AS company_name, 0 AS company_name_facet, '' AS company_two, 0 AS company_two_facet, '' AS content, 18000 AS created_at, 0 AS deleted, '' AS email, '' AS login, '' AS mission_statement, '' AS name, '' AS state, 0 AS user_id FROM states LEFT OUTER JOIN addresses AS addresses ON states.id = addresses.state_id WHERE states.id >= $start AND states.id <= $end GROUP BY states.id sql_attr_float = capitalization @@ -51,7 +52,7 @@ sql_query_info = SELECT * FROM states WHERE states.id = (($id - 0) / 5) # Source configuration -source sellers +source sellers_main { sql_range_step = 5000 sql_query_post = @@ -65,7 +66,7 @@ sql_db = ultrasphinx_development sql_host = localhost sql_pass = sql_user = root -sql_query_range = SELECT MIN(id) , MAX(id) FROM sellers +sql_query_range = SELECT MIN(id) , MAX(id) FROM sellers sql_query = SELECT (sellers.id * 5 + 1) AS id, '' AS address_name, sellers.capitalization AS capitalization, 'Seller' AS class, 1 AS class_id, '' AS company, 0 AS company_facet, sellers.company_name AS company_name, CAST(CRC32(sellers.company_name) AS unsigned) AS company_name_facet, '' AS company_two, 0 AS company_two_facet, '' AS content, UNIX_TIMESTAMP(sellers.created_at) AS created_at, 0 AS deleted, '' AS email, '' AS login, sellers.mission_statement AS mission_statement, '' AS name, '' AS state, sellers.user_id AS user_id FROM sellers WHERE sellers.id >= $start AND sellers.id <= $end GROUP BY sellers.id sql_attr_float = capitalization @@ -83,7 +84,7 @@ sql_query_info = SELECT * FROM sellers WHERE sellers.id = (($id - 1) / 5) # Source configuration -source geo__addresses +source geo__addresses_main { sql_range_step = 5000 sql_query_post = @@ -97,7 +98,7 @@ sql_db = ultrasphinx_development sql_host = localhost sql_pass = sql_user = root -sql_query_range = SELECT MIN(id) , MAX(id) FROM addresses +sql_query_range = SELECT MIN(id) , MAX(id) FROM addresses sql_query = SELECT (addresses.id * 5 + 2) AS id, '' AS address_name, 0 AS capitalization, 'Geo::Address' AS class, 2 AS class_id, '' AS company, 0 AS company_facet, '' AS company_name, 0 AS company_name_facet, '' AS company_two, 0 AS company_two_facet, CONCAT_WS(' ', addresses.line_1, addresses.line_2, addresses.city, addresses.province_region, addresses.zip_postal_code) AS content, 18000 AS created_at, 0 AS deleted, '' AS email, '' AS login, '' AS mission_statement, addresses.name AS name, state.name AS state, 0 AS user_id FROM addresses LEFT OUTER JOIN states AS state ON state.id = addresses.state_id WHERE addresses.id >= $start AND addresses.id <= $end GROUP BY addresses.id sql_attr_float = capitalization @@ -115,7 +116,7 @@ sql_query_info = SELECT * FROM addresses WHERE addresses.id = (($id - 2) / 5) # Source configuration -source users +source users_main { sql_range_step = 5000 sql_query_post = @@ -129,7 +130,7 @@ sql_db = ultrasphinx_development sql_host = localhost sql_pass = sql_user = root -sql_query_range = SELECT MIN(id) , MAX(id) FROM users +sql_query_range = SELECT MIN(id) , MAX(id) FROM users sql_query = SELECT (users.id * 5 + 3) AS id, '' AS address_name, 0 AS capitalization, 'User' AS class, 3 AS class_id, specific_seller.company_name AS company, CAST(CRC32(specific_seller.company_name) AS unsigned) AS company_facet, '' AS company_name, 0 AS company_name_facet, REPLACE(sellers_two.company_name, '6', ' replacement ') AS company_two, CAST(CRC32(REPLACE(sellers_two.company_name, '6', ' replacement ')) AS unsigned) AS company_two_facet, '' AS content, 18000 AS created_at, users.deleted AS deleted, users.email AS email, users.login AS login, '' AS mission_statement, '' AS name, '' AS state, 0 AS user_id FROM users LEFT OUTER JOIN sellers AS specific_seller ON users.id = specific_seller.user_id LEFT OUTER JOIN sellers AS sellers_two ON users.id = sellers_two.user_id WHERE users.id >= $start AND users.id <= $end AND (deleted = '0') GROUP BY users.id sql_attr_float = capitalization @@ -147,7 +148,7 @@ sql_query_info = SELECT * FROM users WHERE users.id = (($id - 3) / 5) # Source configuration -source geo__countries +source geo__countries_main { sql_range_step = 5000 sql_query_post = @@ -161,7 +162,7 @@ sql_db = ultrasphinx_development sql_host = localhost sql_pass = sql_user = root -sql_query_range = SELECT MIN(id) , MAX(id) FROM countries +sql_query_range = SELECT MIN(id) , MAX(id) FROM countries sql_query = SELECT (countries.id * 5 + 4) AS id, '' AS address_name, 0 AS capitalization, 'Geo::Country' AS class, 4 AS class_id, '' AS company, 0 AS company_facet, '' AS company_name, 0 AS company_name_facet, '' AS company_two, 0 AS company_two_facet, '' AS content, 18000 AS created_at, 0 AS deleted, '' AS email, '' AS login, '' AS mission_statement, countries.name AS name, '' AS state, 0 AS user_id FROM countries WHERE countries.id >= $start AND countries.id <= $end GROUP BY countries.id sql_attr_float = capitalization @@ -179,13 +180,130 @@ sql_query_info = SELECT * FROM countries WHERE countries.id = (($id - 4) / 5) # Index configuration -index complete +index main +{ + source = geo__addresses_main + source = geo__countries_main + source = geo__states_main + source = sellers_main + source = users_main + charset_type = utf-8 + html_index_attrs = + charset_table = 0..9, A..Z->a..z, -, _, ., &, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F,U+C5->U+E5, U+E5, U+C4->U+E4, U+E4, U+D6->U+F6, U+F6, U+16B, U+0c1->a, U+0c4->a, U+0c9->e, U+0cd->i, U+0d3->o, U+0d4->o, U+0da->u, U+0dd->y, U+0e1->a, U+0e4->a, U+0e9->e, U+0ed->i, U+0f3->o, U+0f4->o, U+0fa->u, U+0fd->y, U+104->U+105, U+105, U+106->U+107, U+10c->c, U+10d->c, U+10e->d, U+10f->d, U+116->U+117, U+117, U+118->U+119, U+11a->e, U+11b->e, U+12E->U+12F, U+12F, U+139->l, U+13a->l, U+13d->l, U+13e->l, U+141->U+142, U+142, U+143->U+144, U+144,U+147->n, U+148->n, U+154->r, U+155->r, U+158->r, U+159->r, U+15A->U+15B, U+15B, U+160->s, U+160->U+161, U+161->s, U+164->t, U+165->t, U+16A->U+16B, U+16B, U+16e->u, U+16f->u, U+172->U+173, U+173, U+179->U+17A, U+17A, U+17B->U+17C, U+17C, U+17d->z, U+17e->z, + min_word_len = 1 + # enable_star = 1 + stopwords = + html_strip = 0 + path = /tmp/sphinx//sphinx_index_main + docinfo = extern + morphology = stem_en + # min_infix_len = 1 +} + + +# Source configuration + +source sellers_delta +{ + sql_range_step = 5000 + sql_query_post = + sql_ranged_throttle = 0 + +type = mysql +sql_query_pre = SET SESSION group_concat_max_len = 65535 +sql_query_pre = SET NAMES utf8 + +sql_db = ultrasphinx_development +sql_host = localhost +sql_pass = +sql_user = root +sql_query_range = SELECT MIN(id) , MAX(id) FROM sellers WHERE sellers.updated_at > DATE_SUB(NOW(), INTERVAL 88200 SECOND) +sql_query = SELECT (sellers.id * 5 + 1) AS id, '' AS address_name, sellers.capitalization AS capitalization, 'Seller' AS class, 1 AS class_id, '' AS company, 0 AS company_facet, sellers.company_name AS company_name, CAST(CRC32(sellers.company_name) AS unsigned) AS company_name_facet, '' AS company_two, 0 AS company_two_facet, '' AS content, UNIX_TIMESTAMP(sellers.created_at) AS created_at, 0 AS deleted, '' AS email, '' AS login, sellers.mission_statement AS mission_statement, '' AS name, '' AS state, sellers.user_id AS user_id FROM sellers WHERE sellers.id >= $start AND sellers.id <= $end AND sellers.updated_at > DATE_SUB(NOW(), INTERVAL 88200 SECOND) GROUP BY sellers.id + +sql_attr_float = capitalization +sql_attr_uint = class_id +sql_attr_uint = company_facet +sql_attr_uint = company_name_facet +sql_attr_uint = company_two_facet +sql_attr_timestamp = created_at +sql_attr_bool = deleted +sql_attr_str2ordinal = mission_statement +sql_attr_uint = user_id +sql_query_info = SELECT * FROM sellers WHERE sellers.id = (($id - 1) / 5) +} + + +# Source configuration + +source geo__addresses_delta +{ + sql_range_step = 5000 + sql_query_post = + sql_ranged_throttle = 0 + +type = mysql +sql_query_pre = SET SESSION group_concat_max_len = 65535 +sql_query_pre = SET NAMES utf8 + +sql_db = ultrasphinx_development +sql_host = localhost +sql_pass = +sql_user = root +sql_query_range = SELECT MIN(id) , MAX(id) FROM addresses +sql_query = SELECT (addresses.id * 5 + 2) AS id, '' AS address_name, 0 AS capitalization, 'Geo::Address' AS class, 2 AS class_id, '' AS company, 0 AS company_facet, '' AS company_name, 0 AS company_name_facet, '' AS company_two, 0 AS company_two_facet, CONCAT_WS(' ', addresses.line_1, addresses.line_2, addresses.city, addresses.province_region, addresses.zip_postal_code) AS content, 18000 AS created_at, 0 AS deleted, '' AS email, '' AS login, '' AS mission_statement, addresses.name AS name, state.name AS state, 0 AS user_id FROM addresses LEFT OUTER JOIN states AS state ON state.id = addresses.state_id WHERE addresses.id >= $start AND addresses.id <= $end GROUP BY addresses.id + +sql_attr_float = capitalization +sql_attr_uint = class_id +sql_attr_uint = company_facet +sql_attr_uint = company_name_facet +sql_attr_uint = company_two_facet +sql_attr_timestamp = created_at +sql_attr_bool = deleted +sql_attr_str2ordinal = mission_statement +sql_attr_uint = user_id +sql_query_info = SELECT * FROM addresses WHERE addresses.id = (($id - 2) / 5) +} + + +# Source configuration + +source users_delta +{ + sql_range_step = 5000 + sql_query_post = + sql_ranged_throttle = 0 + +type = mysql +sql_query_pre = SET SESSION group_concat_max_len = 65535 +sql_query_pre = SET NAMES utf8 + +sql_db = ultrasphinx_development +sql_host = localhost +sql_pass = +sql_user = root +sql_query_range = SELECT MIN(id) , MAX(id) FROM users WHERE users.created_at > DATE_SUB(NOW(), INTERVAL 88200 SECOND) +sql_query = SELECT (users.id * 5 + 3) AS id, '' AS address_name, 0 AS capitalization, 'User' AS class, 3 AS class_id, specific_seller.company_name AS company, CAST(CRC32(specific_seller.company_name) AS unsigned) AS company_facet, '' AS company_name, 0 AS company_name_facet, REPLACE(sellers_two.company_name, '6', ' replacement ') AS company_two, CAST(CRC32(REPLACE(sellers_two.company_name, '6', ' replacement ')) AS unsigned) AS company_two_facet, '' AS content, 18000 AS created_at, users.deleted AS deleted, users.email AS email, users.login AS login, '' AS mission_statement, '' AS name, '' AS state, 0 AS user_id FROM users LEFT OUTER JOIN sellers AS specific_seller ON users.id = specific_seller.user_id LEFT OUTER JOIN sellers AS sellers_two ON users.id = sellers_two.user_id WHERE users.id >= $start AND users.id <= $end AND (deleted = '0') AND users.created_at > DATE_SUB(NOW(), INTERVAL 88200 SECOND) GROUP BY users.id + +sql_attr_float = capitalization +sql_attr_uint = class_id +sql_attr_uint = company_facet +sql_attr_uint = company_name_facet +sql_attr_uint = company_two_facet +sql_attr_timestamp = created_at +sql_attr_bool = deleted +sql_attr_str2ordinal = mission_statement +sql_attr_uint = user_id +sql_query_info = SELECT * FROM users WHERE users.id = (($id - 3) / 5) +} + + +# Index configuration + +index delta { - source = geo__addresses - source = geo__countries - source = geo__states - source = sellers - source = users + source = geo__addresses_delta + source = sellers_delta + source = users_delta charset_type = utf-8 html_index_attrs = charset_table = 0..9, A..Z->a..z, -, _, ., &, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F,U+C5->U+E5, U+E5, U+C4->U+E4, U+E4, U+D6->U+F6, U+F6, U+16B, U+0c1->a, U+0c4->a, U+0c9->e, U+0cd->i, U+0d3->o, U+0d4->o, U+0da->u, U+0dd->y, U+0e1->a, U+0e4->a, U+0e9->e, U+0ed->i, U+0f3->o, U+0f4->o, U+0fa->u, U+0fd->y, U+104->U+105, U+105, U+106->U+107, U+10c->c, U+10d->c, U+10e->d, U+10f->d, U+116->U+117, U+117, U+118->U+119, U+11a->e, U+11b->e, U+12E->U+12F, U+12F, U+139->l, U+13a->l, U+13d->l, U+13e->l, U+141->U+142, U+142, U+143->U+144, U+144,U+147->n, U+148->n, U+154->r, U+155->r, U+158->r, U+159->r, U+15A->U+15B, U+15B, U+160->s, U+160->U+161, U+161->s, U+164->t, U+165->t, U+16A->U+16B, U+16B, U+16e->u, U+16f->u, U+172->U+173, U+173, U+179->U+17A, U+17A, U+17B->U+17C, U+17C, U+17d->z, U+17e->z, @@ -193,8 +311,9 @@ index complete # enable_star = 1 stopwords = html_strip = 0 - path = /tmp/sphinx//sphinx_index_complete + path = /tmp/sphinx//sphinx_index_delta docinfo = extern morphology = stem_en # min_infix_len = 1 } + test/integration/app/config/ultrasphinx/development.conf.canonical @@ -5,7 +5,7 @@ company_name: <%= "seller#{num}" %> capitalization: <%= num * 1.548 %> mission_statement: <%= %w(Add value through developing superior products.).sort_by(&:rand).join(" ") %> - updated_at: <%= ((time = Time.parse("Tue Oct 23 04:28:11")) - num.days).to_s(:db) %> + updated_at: <%= ((time = Time.parse("Tue Oct 23 04:28:11 2007")) - num.days).to_s(:db) %> created_at: <%= (time - num.weeks).to_s(:db) %> <% end %> test/integration/app/test/fixtures/sellers.yml @@ -5,7 +5,7 @@ crypted_password: "2fdefe5c83d80a03a828dd65e90cfff65f0fb42d043a254ca2cad6af968d0e15" #password email: <%= "user#{num}@test.com" %> salt: "1000" - updated_at: <%= (time = Time.parse("Tue Oct 23 04:28:11")).to_s(:db) %> + updated_at: <%= (time = Time.parse("Tue Oct 23 04:28:11 2007")).to_s(:db) %> created_at: <%= (time - 180).to_s(:db) %> deleted: <%= num == 41 ? true : false %> <% end %> test/integration/app/test/fixtures/users.yml @@ -16,8 +16,20 @@ class SearchTest < Test::Unit::TestCase def test_with_subtotals_option S.client_options['with_subtotals'] = true - @s = S.new.run + + # No delta index; accurate subtotals sum + @s = S.new(:indexes => Ultrasphinx::MAIN_INDEX).run assert_equal @s.total_entries, @s.subtotals.values._sum + + # With delta; subtotals sum not less than total sum + @s = S.new.run + assert @s.subtotals.values._sum >= @s.total_entries + + # With delta and filter; request class gets accurate count regardless + @s = S.new(:class_names => 'Seller').run + assert_equal @s.total_entries, @s.subtotals['Seller'] + assert @s.subtotals.values._sum >= @s.total_entries + S.client_options['with_subtotals'] = false end @@ -84,6 +96,34 @@ class SearchTest < Test::Unit::TestCase ) end + def test_individual_totals_with_pagination + Ultrasphinx::MODEL_CONFIGURATION.keys.each do |class_name| + if class_name == "User" + assert_equal User.count(:conditions => {:deleted => false }), + S.new(:class_names => class_name, :page => 2).total_entries + else + assert_equal class_name.constantize.count, + S.new(:class_names => class_name, :page => 2).total_entries + end + end + end + + def test_individual_totals_without_pagination + Ultrasphinx::MODEL_CONFIGURATION.keys.each do |class_name| + begin + if class_name == "User" + assert_equal User.count(:conditions => {:deleted => false }), + S.new(:class_names => class_name).total_entries + else + assert_equal class_name.constantize.count, + S.new(:class_names => class_name).total_entries + end + rescue Object + raise class_name + end + end + end + def test_sort_by_date assert_equal( Seller.find(:all, :limit => 5, :order => 'created_at DESC').map(&:created_at), test/integration/search_test.rb 44b2af416d6d6be266bc441516da359ae8687f99 Evan Weaver evan@cloudbur.st http://github.com/fauna/ultrasphinx/commit/fb625ad4627a19ca0a16fcd0c255686ca3ef2fbd fb625ad4627a19ca0a16fcd0c255686ca3ef2fbd 2008-03-08T12:24:20-08:00 2008-03-08T12:24:20-08:00 Import 1.9 from private branch. Adds delta index support. 1c6dd6ff7b42394d75856fb5e09552a025dc648d Evan Weaver evan@cloudbur.st