Skip to content

Loading…

AR 3.1 / pgbouncer / PostgreSQL issue with prepared statements #1627

Closed
underley opened this Issue · 73 comments
@underley

Using exec_cache with pgbouncer in transaction mode, as the connections pool can cause random problems.

Scenario:

  • the application performs a query through ActiveRecord:

Something.find 2
Something Load (0.9ms) SELECT "somethings" .* FROM "somethings" WHERE "somethings." Id "= $ 1 LIMIT 1 [[" id ", 2]]

  • pgbouncer connects to the backend (lets call it PG-1)
  • ActiveRecord creates a prepared statement
  • pgbouncer prepares for the backend PG-1 prepared statement called a3:

2011-06-10 9:34:04 EDT LOG: duration: 0.505 ms parse a3: SELECT "somethings" .* FROM "somethings" WHERE "somethings." Id "= $ 1 LIMIT 1

  • ActiveRecord execute prepared statement
  • pgbouncer performs pg backend EXECUTE a3 (...) on PG-1

2011-06-10 9:34:04 EDT LOG: duration: 0.048 ms bind a3: SELECT "somethings" .* FROM "somethings" WHERE "somethings." Id "= $ 1 LIMIT 1
2011-06-10 9:34:04 GMT DETAIL: parameters: $ 1 = '2 '
2011-06-10 9:34:04 EDT LOG: duration: 0.045 ms execute a3: SELECT "somethings" .* FROM "somethings" WHERE "somethings." Id "= $ 1 LIMIT 1
2011-06-10 9:34:04 GMT DETAIL: parameters: $ 1 = '2 '

  • the application receives the result

Something Id: 2, symbol: "sss2", description: nil, created_at: "2011-06-10 07:19:34", updated_at: "2011-06-10 07:19:34"

  • at this time a different application process connects to pgbouncer, goes to the backend PG-1 and takes it (for example - with begin transaction)
  • the first application process once again executes the same query, with other parameters

Something.find 3

  • pgbouncer connects to the backend (but this time it goes to the PG-2, because the PG-1 is busy)
  • ActiveRecord has prepared statement in exec_cache (called a3)
  • ActiveRecord execute prepared statemnet a3

Something Load (1.2ms) SELECT "somethings" .* FROM "somethings" WHERE "somethings." Id "= $ 1 LIMIT 1 [[" id ", 3]]

  • pgbouncer performs EXECUTE a3 (...) on backend PG-2
  • Fail - the PG-2 has no such prepared statement

PGError: ERROR: Prepared statement "a3" does not exist

Solutions:

  • switching pgbouncer mode to session - bad, because it increases the consumption of resources
  • wrapping request in transaction - wrong - also increases consumption of resources, with additional side effec - there may be long-lasting transactions that will cause deadlocks
  • add option to not use ActiveRecord exec_cache - fix all issues with pgbouncer and additionaly another described here
@tenderlove tenderlove was assigned
@jake3030 jake3030 pushed a commit to jake3030/rails that referenced this issue
@josh josh Allow multiple conditions for callbacks [#1627 state:resolved] 1e45818
@joevandyk

It would be nice to have an option for disabling prepared statements.

The issues I've ran into with prepared statements:

  • All Rails processes have to be shutdown during a database schema migration, otherwise the prepared statement won't match.
  • Using prepared statements prevents the right index from being picked in certain cases.
@tenderlove
Ruby on Rails member

The schema migrations issue should be fixed in 3.1.2 with 818d285

Which cases prevent correct indexes from being used?

@tenderlove
Ruby on Rails member

Has this actually happened to you yet? I don't think it's actually possible to hit a case like this unless you're using the adapter directly and writing your own statements. But regardless of that, we can specify hints as to the param type and eliminate this case.

parameter data types can optionally be specified

@joevandyk

No, hasn't happened to me yet. :)

But I don't see a reason why ActiveRecord wouldn't generate a query like:

PREPARE test2 (bool) AS SELECT * FROM users WHERE is_active = $1 ORDER BY username ASC limit 50;
EXECUTE test2(true);
@patshaughnessy

FYI more interest in adding a "disable prepared statement" option; see the discussion in the comments at: http://patshaughnessy.net/2011/10/22/show-some-love-for-prepared-statements-in-rails-3-1

@kwi

Definitely agree, we need to be able to disable prepared statement :/
We are running behing pgbouncer and can't upgrade to rails 3.1 until we can disable them.
Please :)

@elight

I'm running into this problem while using Resque. Despite suggestions I've read, setting a Resque.before_fork to open a new connection isn't working for me.

@ericschiller

Elight, did you try restarting the Resque workers? That was my problem. Then the before_fork fix worked perfectly on rails 3.2 + Resque.

@inossidabile

Definitely agree, we need to be able to disable prepared statement :/

+1. There are billions of different PG setups. And sometimes it makes great sense to disable this feature. We've just found our one (using pgpool) as very troublesome. Random exceptions and all that stuff :(

@yaroslav

+1 here, having problems with PgBouncer as well. Just having an easy option to disable that would solve it all.

@elight

ericschiller: turns out it was another gem that monkey patched run_hool that was preventing the before_fork from occurring.

Monkey patching is EEEEEVILLLLLLLL.

@Phill

I have had nothing but problems with prepared statement. First it was Unicorn and now it's Resque. The before_fork doesn't seem to help for me (and yes I'm restarting the queues). I wonder if I have a Gem that's monkey patching something.

@rafaelfranca
Ruby on Rails member

@tenderlove can we close this one?

@elight
@Phill

Yes, I found that the iPhone notification gem I was using was monkey patching Resque. I switched to something less good, but works.

@kwi

Pgbouncer still doesn't allow to work with prepared statement, so we can't move to Rails 3.1+ until we can disable them.
We have an extensive rails cluster with 2000+ rails instances connected to the same postgres, pgbouncer is essential for us.

So not sure we can close this issue :)

@jonleighton
Ruby on Rails member

@tenderlove I just ran into this too.

I think what happens is something like this:

  • worker process is created. AR connection established. prepared statement counter is at 0.
  • worker forks child to do work. AR connection is copied to child.
  • child creates some prepared statements. counter is at X
  • child exits
  • worker forks second child
  • counter is at 0, but there are actually X prepared statements already allocated in the connection.
  • second child tries to create prepared statement => error

Inside the postgres statement cache, the @cache is keyed based on the pid, but the pid does not affect @counter or #next_key in any way, hence this issue.

I think we should:

  • Make #next_key include the pid as well as the @counter
  • deallocate all of a given process' prepared statements in its at_exit hook

Doing these things would prevent this bug, I think. But I am not sure if there would be a noticeable performance cost of deallocating at_exit. Thoughts?

@jonleighton
Ruby on Rails member

I should note that my scenario described above related to resque workers, which is where I encountered the problem.

@fingermark

I'm not using resque, but unicorn + pg. I just got the same error. I have no idea how to fix it before we deploy either, and it's kind of hard to reproduce.

@fingermark

Also is this related to #5872?

@steveklabnik
Ruby on Rails member

@fingermark what version of everything are you using? (also, if you mention issues by typing #5872 github will link them)

@fingermark

rails 3.2.6
pg 0.14.0
unicorn 4.3.1 (with 2 workers)

staging:
  adapter: postgresql
  encoding: unicode
  database: myproj_development
  username: postgres
  password: kittensfromhell

And more info on the error:

  SQL (2.0ms)  DELETE FROM "categories" WHERE "categories"."id" = $1  [["id", 4]]
PG::Error: ERROR:  prepared statement "a3" already exists
: DELETE FROM "categories" WHERE "categories"."id" = $1
   (0.5ms)  ROLLBACK
Error setting category: PG::Error: ERROR:  prepared statement "a3" already exists
: DELETE FROM "categories" WHERE "categories"."id" = $1:
  /var/www/myproj/shared/bundle/ruby/1.9.1/gems/activerecord-3.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:1192:in `prepare'

The only other odd thing about my setup is that model has the following:

  establish_connection "myproj_#{Rails.env}"

So, to reproduce the issue I just rapidly click on the submit form and would eventually hit that error. By adding prepared_statement: false to my database.yml, I have yet to hit that error while rapidly submitting the form.

@serbrech

Hitting the exact same probleme using resque, but on earlier version of rails.
Setting prepared_statement: false seemed to resolve it for me too.

rails 3.2.2
pg 0.13.2
resque 1.21

@joevandyk

@fingermark is prepared_statements really an option in database.yml?

@joevandyk

It looks like you can set prepared_statements: true in config/database.yml and it seems to disable prepared statements in postgresql.

Product.find(1) executes this sql: SELECT "products".* FROM "products" WHERE "products"."id" = 1 LIMIT 1

So, I think this ticket can be closed? @tenderlove ?

@agnoster

On activerecord-3.2.8 we're having the same issue. In particular, the calls to #table_exists? (and a few other methods on the Postgresql adapter) always executes a prepared statement (via #exec_cache), regardless of whether prepared_statements: false is set.

Our use case, like (it seems) others, is to be able to use a connection pooler (in this case, pgbouncer) - unfortunately, the reliance on prepared statements and the query cache seems to make this unworkable, which is incredibly unfortunate.

Are there any plans to fix this? Or is the official answer "don't use a connection pooler or don't use activerecord >=3.1"?

@agnoster

For anyone else who finds this thread: it seems like ar-3.2.9 might fix the issue. I'll try it out and report back ;-)

@ajsharp

@agnoster Just tried applying this and we're still seeing the error :(

@agnoster

@ajsharp Are you also setting prepared_statements: false in your database.yml? What's the specific error you're getting?

@ajsharp

@agnoster We don't have prepared_statements in our db config at all, so, I'm not sure what the default value is...

The error we were getting was the same "prepared statement already exists" that's documented above. This was happening in resque workers, and adding a reconnect block fixed it: #5872 (comment).

@agnoster
@bobek

I am hitting the same wall with a bit different set up. My setup is goliath with em-synchrony and em-postgresql-adapter. When

  prepared_statements: true

then running ab -c 100 -n 100 leads to 1-10 failed requests with

PG::Error: ERROR:  prepared statement "a1" does not exis

interestingly setting

  prepared_statements: false

leads to 97 failed requests with typical error being

PG::Error: ERROR:  bind message supplies 1 parameters, but prepared statement "a1" requires 0
@rafaelfranca
Ruby on Rails member

Is this still present on 4.0+?

@toreriklinnerud

We're hitting the same issue in production with unicorn and Rails 4. We have hooks defined to disconnect and connect, but it still happens every couple of days.

@aripollak

Sorry if this is not relevant, but I am also seeing a similar issue occasionally on Rails 3.2.15 under Unicorn with multiple workers when using memcachier/dalli as a memcache client. Database queries start failing with a "prepared statement already exists" error, and it seems to be correlated with the memcache server going down for some period of time; the prepared statement problems don't go away until Unicorn is completely restarted.

@toreriklinnerud

Interesting, we are also using memcachier/dalli in the app that is seeing the problem.

@HenleyChiu

@toreriklinnerud You are still seeing this problem, even with prepared_statements disabled in your database.yml?

@toreriklinnerud

We run on heroku so we can't touch database.yml and haven't tried disabling it. It hasn't happen for a while now (a few weeks?), so possibly there is an additional component required for it to happen that we haven't identified yet.

@aripollak

I was finally able to locally reproduce the behavior I was seeing, though it's still pretty rare. It seems to be because we're using Ruby's Timeout.timeout (something similar is also used in rack-timeout). If the time limit is reached and timeout happens to raise an exception in an inopportune place in the main thread (like right here, which is not inside the begin/ensure), the main process gets into a weird state and can't be cleanly recovered.
Do the Rails developers want patches to fix these situations, or is this an unsupported use case and I should just kill the process entirely to avoid any risk of corruption?

@dougcole

It seems like this should be safe to close, I don't see any reports in here of bugs after prepared_statements: false is added to the database.yml file.

@senny senny added the PostgreSQL label
@oelmekki

Regardless of if the specific problem @underley faced has been
resolved, is it now possible to disable prepared statements on specific
queries, as mentioned in oldest comments ?

I do unusual AR queries manipulation and would love to be able to
decide when not to use prepared statements.

EDIT : oh nervermind, just saw AbstractAdapter#unprepared_statement

@matthewd
Ruby on Rails member

@oelmekki can you elaborate on your use case for AbstractAdapter#unprepared_statement? It was introduced as a work-around for an issue with to_sql, and I had hoped we could deprecate it once that was fixed. (Largely because it currently does two different things, and only one of those actually involves the use of PREPARE/EXECUTE.)

To be clear, before anyone panics: I'm only talking about the block method, not the prepared_statements connection option.

@oelmekki

@matthewd I didn't used it, finally.

I wanted it because on one of my projects, I manipulate relations to OR them. The idea is to extract relevant values (bind_values, join_values, etc), then use arel #or method on their constraints to compute a new query, attaching bound values back.

In that context, prepared statement would fail, because if you join arel constraints on something like foo = $1 and bar = $2 and fooz = $1 and baz = $2, it will produce (foo = $1 and baz = $2) or (fooz = $1 and baz = $2).

So, I wanted to use #unprepared_statement to avoid that, but it turned out statement where already prepared before even reaching my methods, making #unprepared_statement unusable.

TL;DR : feel free to remove it :)

@matthewd
Ruby on Rails member

@oelmekki Awesome! Thanks for letting me know.

@rafaelfranca
Ruby on Rails member

Closing base on #1627 (comment)

@tenderlove tenderlove was unassigned by rafaelfranca
@aripollak

I'm still seeing this on Rails 4.1.4 when combined with timeout. Is this just not a supported configuration? It seems pretty common, and is something Heroku recommends using.

@bobbus

Also happen to us and let one of the unicorn worker in a corrupted state.

I'm wondering if we should expect Rails to keep the database in a clean state wherever an exception can happen ?

If yes, there is probably something to do to fix the problem with prepared statement, we encounter the problem on rails 4.1.1, exception was raised here

If no, we probably have to restart the unicorn worker if a timeout happen. This is actually what does unicorn timeout mechanism but we can't get details and backtrace with this..

Maybe a new issue would be better to discuss this ?

cc @rafaelfranca , @tenderlove

@rafaelfranca
Ruby on Rails member

No new issues please. If we want to continue the discussion I can reopen the issue, but seems disabling prepared statement fix the issue.

@bobbus

@rafaelfranca , I suggest new issue because the original one mention AR 3.1 in the title.

As it may be possible to deactivate prepared statement to fix the issue, won't it degrade performance in production ?

I'm wondering if we should expect Rails to keep the database in a clean state wherever an exception can happen ?

I would love to know what is the core team position about this.
It seems for me a real issue to leave a database connection open but unusable, isn't it ?

@rafaelfranca
Ruby on Rails member

How is the behaviour at current master?

@bobbus

@rafaelfranca , this is hardly reproducible as it happen when a timeout exception occur when the execution is at the "right" place.

I have no idea how a test for can be written, do you ? I may try to reproduce it locally.

@rafaelfranca rafaelfranca reopened this
@bobbus

@rafaelfranca , thanks for reopening.
I would love to help if I can have some answer or directive.

@rafaelfranca
Ruby on Rails member

I'm never had this issue but I'd say we should keep the database in a consistent state. @matthewd could help more than I.

@matthewd
Ruby on Rails member

we should expect Rails to keep the database in a clean state wherever an exception can happen

Yes, we should. But what we're talking about here is different: we're talking about some other thread injecting an exception at any possible point in the code. I'm not convinced that's a realistic problem to solve.

Thread#kill. :neutral_face:

Sufficiently palatable PRs to fix this are welcome... but it feels like we'd likely be chasing it forever.

@matthewd
Ruby on Rails member

However, if this is becoming a discussion about AR getting confused by inopportune firing of a ruby timeout, then yes, we should move to a new issue: it might have vaguely similar symptoms, but I think it's actually quite unrelated to what everyone else here was talking about 1-3 years ago.

@abezzub

This happened to the app I am working on. Rails 4.1, unicorn, we use timeout, deployed on Heroku.
From what I see errors started from a timeout

@abezzub

This happened again in our app after timeout. Is there any way I can help to debug this?
Are there any workarounds for this issue? Heroku support suggests to turn off prepared statements, but I would rather not do it

@rud

@abezzub: trying the suggested fix and reporting back would be most helpful. This seems hard to reproduce over a span of years, so any extra details you can capture would be great. You are in a unique position as this seems to happen regularly to you.

@abezzub

I looked at the logs/code again and I think I might know what is going on.

Heroku suggests using their version of timeout gem that raises an error in main thread if it takes too long:
https://github.com/heroku/rack-timeout/blob/master/lib/rack/timeout.rb#L91-L108

The "prepared statement already exists" error happened in following method in postgres adapter: https://github.com/rails/rails/blob/4-1-stable/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb#L864-L878

As I understand Heroku's gem created timeout exception just at the right time. Let's assume timeout exception happens right after @connection.prepare nextkey, sql finishes successfully:

        def prepare_statement(sql)
          sql_key = sql_key(sql)
          unless @statements.key? sql_key
            nextkey = @statements.next_key
            begin
              @connection.prepare nextkey, sql
            rescue => e
              raise translate_exception_class(e, sql)
            end

           # timeout happens here!

            # Clear the queue
            @connection.get_last_result
            @statements[sql_key] = nextkey # next key is "reserved" here
          end
          @statements[sql_key]
        end

then Postgres has prepared statement, but @statements was not updated and does not know that nextkey was already used. So the next time when prepare_statement is going to be called it is guaranteed to fail because @statements.next_key will return the same nextkey that was already used in PG. Current version of code that determines what next_key is:

        def next_key
          "a#{@counter + 1}"
        end

        def []=(sql, key)
          while @max <= cache.size
            dealloc(cache.shift.last)
          end
          @counter += 1
          cache[sql] = key
        end

I think if nextkey is reserved before @connection.prepare nextkey, sql is started then the issue will be fixed. If this is the case I would be happy/interested to work on a fix.

cc: @matthewd

@abezzub

I created the PR. I think moving counter increment to next_key method should be enough to fix it.

@rud

@abezzub: excellent digging, looks absolutely right to me.

A simple fix might be to just the increment @counter whenever next_key is called, would that not fix the issue? It would mean the potential for "holes" in the sequence of prepared statement names, but I don't think that would be much of an issue. Thoughts?

@abezzub

@rafaelfranca, is it possible to include this fix in 4.2?

@abezzub

I can confirm that my assumption was correct. I digged in the logs a bit more and I see that timeout exception that triggered "prepared statement" errors was raised before key is saved in the cache:

https://github.com/rails/rails/blob/4-1-stable/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb#L512

Here is stack trace from my app's log:

Rack::Timeout::RequestTimeoutError (Request ran for longer than 25 seconds.): 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:537:in `dealloc' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:512:in `[]=' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:875:in `prepare_statement' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:826:in `exec_cache' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:138:in `exec_query' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/postgresql_adapter.rb:954:in `select' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/database_statements.rb:24:in `select_all' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `block in select_all' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:83:in `cache_sql' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `select_all' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/relation/calculations.rb:265:in `execute_simple_calculation' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/relation/calculations.rb:227:in `perform_calculation' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/relation/calculations.rb:119:in `calculate' 
vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.6/lib/active_record/relation/calculations.rb:75:in `sum' 
@dwbutler

We just experienced this exact problem in production. Glad to see we're not alone. I vote for incrementing the counter in next_key.

@kapso

+1 just experienced it in production

gem 'rails', '4.2.0'
gem 'puma', '2.11.0'
gem 'pg', '0.18.1'
gem 'postgres_ext', '2.4.0'
vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:637:in `prepare': PG::DuplicatePstatement: ERROR:  prepared statement "a3" already exists
: SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1 (ActiveRecord::StatementInvalid)
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:637:in `prepare_statement'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:596:in `exec_cache'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in `execute_and_clear'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:336:in `select'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `block in select_all'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/query_cache.rb:83:in `cache_sql'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `select_all'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/querying.rb:39:in `find_by_sql'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/statement_cache.rb:107:in `execute'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/core.rb:186:in `find_by'
    from vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/dynamic_matchers.rb:70:in `find_by_id'
    from vendor/bundle/ruby/2.2.0/gems/sorcery-0.9.0/lib/sorcery/adapters/active_record_adapter.rb:85:in `find_by_id'
@yellowaj

+1 running into this error in production on Heroku

gem 'rails', '4.2.0'
gem 'puma', '2.11.0'
gem 'pg', '0.18.1'
gem 'rack-timeout', '0.2.0'
ActiveRecord::StatementInvalid: PG::DuplicatePstatement: ERROR: prepared statement "a665" already exists
: SELECT "posts".* FROM "posts" WHERE "posts"."user_id" = $1 ORDER BY "posts"."created_at" ASC, "posts"."created_at" DESC
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb" line 637 in prepare
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb" line 637 in prepare_statement
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb" line 596 in exec_cache
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql_adapter.rb" line 585 in execute_and_clear
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/postgresql/database_statements.rb" line 160 in exec_query
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb" line 336 in select
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb" line 32 in select_all
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/query_cache.rb" line 68 in block in select_all
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/query_cache.rb" line 83 in cache_sql
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/connection_adapters/abstract/query_cache.rb" line 68 in select_all
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/querying.rb" line 39 in find_by_sql
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/relation.rb" line 638 in exec_queries
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/association_relation.rb" line 19 in exec_queries
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/relation.rb" line 514 in load
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/relation.rb" line 243 in to_a
File "/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.0/lib/active_record/relation/delegation.rb" in to_ary
@rud

@yellowaj this ticket is probably never going to be fixed as the issue is correctly blamed on rack-timeout.

The suggested fix is to remove rack-timeout from your Gemfile (it raises exceptions on the main thread and is completely unsafe) and then fix any long-running database requests as best you can.

Hope this helps you forward.

@yellowaj

@rud thanks for the suggestion!

@dwbutler

We ended up turning off prepared statements. It didn't make any noticeable performance difference.

@rafatower rafatower referenced this issue in CartoDB/cartodb
Closed

ActiveRecord breaks on prepared statement #3128

@rafatower rafatower added a commit to CartoDB/cartodb that referenced this issue
@rafatower rafatower Fix issue with prepared statements #3128
See rails/rails#1627 and
rails/rails#5872 for details.
568bf68
@nesQuick nesQuick referenced this issue in puma/puma
Merged

Add thread reaping to thread pool #702

@rails-bot

This issue has been automatically marked as stale because it has not been commented on for at least
three months.

The resources of the Rails team are limited, and so we are asking for your help.

If you can still reproduce this error on the 4-2-stable, 4-1-stable branches or on master,
please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions.

@rails-bot rails-bot added the stale label
@jdurand

I hit this when switching from unicorn to puma. Removing rack-timeout "fixed" it for me.

@rails-bot rails-bot closed this
@rails-bot

This issue has been automatically closed because of inactivity.

If you can still reproduce this error on the 4-2-stable, 4-1-stable branches or on master,
please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.