"Riddle::ResponseError: No response from searchd" with seperate Sphinx and MySQL servers

[thoughtpunch]

I'm getting at Riddle::ResponseError: No response from searchd when trying to communicate with a seperate remote Sphinx instance and a remote MySQL server using the latest versions thinking-sphinx,

Everything works perfectly when testing in our local and staging environments (which both have searchd and a local instance of MySQL running). It's only the remote configuration that breaks.

What's weird about our setup is that we have Sphinx and MySQL running on a seperate boxes from the web application server. Somehow Sphinx is supposed to use the generated production.sphinx.conf to know how to talk to the Sphinx instance seperately from the MySQL instance, I'm assuming.

GEM VERSIONS + ENV

 #ruby-1.9.3-p194
 thinking-sphinx (3.0.3)
      activerecord (>= 3.1.0)
      builder (>= 2.1.2)
      innertube (>= 1.0.2)
      middleware (>= 0.1.0)
      riddle (>= 1.5.6)
mysql2 (= 0.3.12b6)

STACKTRACE

Riddle::ResponseError: No response from searchd
[GEM_ROOT]/gems/riddle-1.5.3/lib/riddle/client.rb:655:in `request'
[GEM_ROOT]/gems/riddle-1.5.3/lib/riddle/client.rb:226:in `run'
[GEM_ROOT]/gems/riddle-1.5.3/lib/riddle/client.rb:338:in `query'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:429:in `block (2 levels) in populate'
[GEM_ROOT]/gems/activesupport-3.2.13/lib/active_support/notifications.rb:123:in `block in instrument'
[GEM_ROOT]/gems/activesupport-3.2.13/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
[GEM_ROOT]/gems/activesupport-3.2.13/lib/active_support/notifications.rb:123:in `instrument'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:556:in `log'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:565:in `log'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:428:in `block in populate'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:606:in `call'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:606:in `retry_on_stale_index'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:426:in `populate'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search.rb:187:in `method_missing'
[GEM_ROOT]/bundler/gems/thinking-sphinx-6c0e31e9b5fa/lib/thinking_sphinx/search_methods.rb:412:in `search_count'

sphinx.conf on Sphinx instance

indexer
{
  mem_limit = 512M
}

searchd
{
  listen = *SPHINX_INSTANCE*:9312:mysql41
}

source user_core_0
{
  type = mysql
  sql_host = *SQL_SERVER_INSTANCE*
  sql_user = ******
  sql_pass = ******
  sql_db = ****
  sql_query = ******
  sql_attr_uint = sphinx_internal_id
  sql_attr_uint = sphinx_deleted
  sql_attr_uint = state
  sql_attr_timestamp = created_at
  sql_attr_string = sphinx_internal_class
  sql_field_string = full_name
  sql_field_string = nickname
  sql_query_info = SELECT `users`.* FROM `users`.....
}

index user_core
{
  path = /home/../../shared/db/sphinx/user_core
  docinfo = extern
  morphology = stem_en
  charset_type = utf-8
  source = user_core_0
}

Any ideas????

[pat]

Hi Dan

Guessing we can close this issue, given #529 - but also, your stack trace has TS v2, but your gem list says TS v3, so not quite sure what's going on here.

Let me know if this is a separate problem and we can re-open it.

[thoughtpunch]

@pat Thanks for your help with this. You've definitely given me enough info to track the issue down. 🍔

[bmishkin]

been seeing the same issue at seemingly random times in our production environment only (separate app / sphinx / mysql servers) for the last few weeks.

[Riddle::ResponseError] Riddle::ResponseError: No response from searchd (status: 23380, version: 30053)

Backtrace
[GEM_ROOT]/gems/riddle-1.5.6/lib/riddle/client.rb:298 in rescue in run
[GEM_ROOT]/gems/riddle-1.5.6/lib/riddle/client.rb:228 in run
[GEM_ROOT]/gems/riddle-1.5.6/lib/riddle/client.rb:344 in query
[GEM_ROOT]/gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:429 in block (2 levels) in populate

I was able to run a rails console in our production environment when this started occuring, and noted that it happened for some but not all indexes.. so Model1.search() failed as above, Model2.search() succeeded, and ThinkingSphinx.search() failed. Then suddenly all was well again.

Any suggestions or ideas (besides upgrading to TS 3)?

[pat]

I'd recommend upgrading to 2.1.0 - this has connection pooling which will reset connections and retry searches a few times when these errors crop up.

On 30/05/2013, at 1:28 AM, Ben Mishkin wrote:

been seeing the same issue at seemingly random times in our production environment only (separate app / sphinx / mysql servers) for the last few weeks.

[Riddle::ResponseError] Riddle::ResponseError: No response from searchd (status: 23380, version: 30053)

Backtrace
[GEM_ROOT]/gems/riddle-1.5.6/lib/riddle/client.rb:298 in rescue in run
[GEM_ROOT]/gems/riddle-1.5.6/lib/riddle/client.rb:228 in run
[GEM_ROOT]/gems/riddle-1.5.6/lib/riddle/client.rb:344 in query
[GEM_ROOT]/gems/thinking-sphinx-2.0.14/lib/thinking_sphinx/search.rb:429 in block (2 levels) in populate

I was able to run a rails console in our production environment when this started occuring, and noted that it happened for some but not all indexes.. so Model1.search() failed as above, Model2.search() succeeded, and ThinkingSphinx.search() failed. Then suddenly all was well again.

Any suggestions or ideas (besides upgrading to TS 3)?

—
Reply to this email directly or view it on GitHub.