Resque.enqueue failing on second run #1339
Comments
|
Looks like it might be a bug with Resque or Postgres gem than Rails 3.1 bug. P.D: Please post the backtrace here, some people wouldn't go to StackOverflow to see the error :P |
|
Bundle install says: Using pg (0.11.0), Using resque (1.16.1). Are you on ruby 1.9.2? The backtrace as requested: |
|
Guilleiguaran, can you also show me what you have in: lib/tasks/resque.rake In your module for one of your jobs, are you using active record to look up a record? For instance. Something like: Its interesting to know how you got this to work in Rails 3.1rc. |
|
I'm using Ruby 1.9.2 and I'm querying with ActiveRecord in my jobs. I'm using pg 0.11.0 and resque 1.16.1 config/initializers/resque.rb: require 'resque/job_with_status'
require 'resque/failure/multiple'
require 'resque/failure/hoptoad'
require 'resque/failure/redis'
require 'resque/server'
require 'resque/status_server'
Resque::Server.use Rack::Auth::Basic do |username, password|
password == "test-password"
end
redis_config = YAML.load_file(File.join(Rails.root, 'config', 'redis.yml'))[Rails.env]
redis_url = ENV["REDISTOGO_URL"] || redis_config['redis_url']
uri = URI.parse(redis_url)
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque::Status.expire_in = (24 * 60 * 60)lib/tasks/resque.rake: |
|
Do your jobs make any queries to any records? Second, I did a fresh install of Rails 3.1rc on a new RVM gemset running 1.9.2. I don't think this should be a problem. I'm hoping I didn't miss out on any configurations. |
|
Yes, I make a lot of queries on all my jobs. |
|
Guilleiguaran, is your app on Heroku running on a dedicated server (pgsql 9) or shared (pgsql 8)? Are you on OSX? Still can't seem to pinpoint the problem. If its running fine on your 3.1rc environment, I wonder why it isn't on mine. |
|
It seems I have just run into the same problem. Backtrace: This error starts to appear in my Rails application after upgrade from 2.3.11 to 3.1rc (I didn't try any 3.0.* version). The backtrace is from apache log file, there is no record in rails log file. About 1 of 20 requests results in Internal server error and the
|
|
@krzkrzkrz I use OSX locally and shared database in Heroku |
|
Maybe I found the reason why thinks go wrong. Passenger spawner creates the first thread which works fine. As soon as some any threads is needed, it forks the first thread, try to reestabilish database connections since it's not allowed to use a database connection after fork (see http://www.postgresql.org/docs/9.0/static/libpq-connect.html). The problem part is clearing database connections - all prepared statements are deallocated by sending query to the database. Since the connection isn't valid because of forking it results in undefined behavior. In previous versions of Rails it works because there weren't any prepared statements and clearing database connection doesn't perform any query. I'm not sure who is responsible for this issue. It seems active record doesn't allow to clear invalid database connection, but both passenger and resque do it. |
|
The temporary fix is to use |
|
As pointed out by carhartl: https://github.com/defunkt/resque/issues/306#issuecomment-1421034 Using While after_fork for me always caused the "closed connection" error, using before_fork fixed both that one as well as the "prepared statement already exists" error. Not yet sure about the implications of it. ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-darwin10.6.0] |
|
Having this same issue with: ruby 1.8.7 I get the error: ActiveRecord::StatementInvalid: PGError: server closed the connection unexpectedly Adding the before_fork lambda seemingly fixed this. |
|
Same problem here. ruby 1.8.7 We fixed it in Resque by using the before_fork block, but it occurs as well in rails + passenger. |
|
Same problem. Was ok until I upgraded from Rails 3.0.x to rails 3.1.0rc5. Before_fork didn't fix it. Only temporary fix so far is PassengerSpawnMethod conservative as mentioned above. |
|
I'm experiencing this issue as well, Ruby 1.9.0 (p290), Rails 3.2.0-beta, Passenger 3.0.8. @drogus - Thanks for the workaround using |
|
Is it possible to rename this issue to better describe the actual nature and impact of it? Prepared statements in ActiveRecord 3.1 + Postgres + Passenger basically blow up big time, and require the conservative spawn method which is undesirable. |
|
i am facing similar problem. although setting It seems to end up premature closing my rails thread, it seems.
might be a separate issue. |
|
@rsanheim, I don't think that would be a suitable title for the issue. When I was experiencing this problem, I was not using Passenger. Although a lot of devs are coming across this issue with Passenger (so it seems), its not solely Passenger related. |
|
Just to inform you guys that I move my DB to MySQL and the problem is solved. |
|
I can vouch for this too. I moved databases from PgSQL to MySql, and the problem seems to have gone away. |
|
Problem is that on connection reset PgSQL AR adapter is trying to deallocate all prepared statements. But for smart forks in passenger it makes no sense, because these statements are not valid for forked process. I use following monkey patch to make smart spawn methods work with PostgreSQL. # Monkey pactch to prevent postgres prepared statements deallocation on fork
# Original file https://github.com/FooBarWidget/passenger/blob/master/lib/phusion_passenger/utils.rb
if defined?(PhusionPassenger)
module PhusionPassenger
module Utils
protected
def before_handling_requests_with_pg(forked, options)
if forked && defined?(::ActiveRecord::Base)
begin
if ::ActiveRecord::Base.respond_to?(:connection)
if ::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::ADAPTER_NAME == ::ActiveRecord::Base.connection.adapter_name
# Reset all prepared statements, they will be handled properly in donor thread
::ActiveRecord::Base.connection.instance_variable_set(:@statements, {})
end
end
rescue Exception => e
print_exception(self.class.to_s, e)
end
end
before_handling_requests_without_pg(forked, options)
end
alias_method_chain :before_handling_requests, :pg
end
end
end
|
|
Mr. MasterLambaster, you are a star! Just in time, mate! Until a solution makes it into the postgres adapter, will give your monkey patch a shot. |
|
@jiannis Problem with the patch is that it's need to be made both in passenger and AR PostgreSQL adapter. |
Idea Boat: Rack+Rails Expose Fork AspectsPerhaps Rails could expose Rack could potentially expose these hooks too (so rack-compatible servers could find them), and a Rails config.ru could forward the Rack hooks to the Rails hooks. Passenger and Unicorn could then call the appropriate Rack hooks before, after, and around forking. Note that in this example Rails would not actually implement the forking. Passenger and Unicorn would merely be able to discover and call the hooks when Passenger and Unicorn implement the forking. Then you wouldn't need complicated cross-library patches. The ActiveRecord PosgreSql Adapter would, in the ActiveRecord railtie's initializiation, set up the hooks. Done. This idea doesn't help the current state of affairs, where these hooks don't exist and where cross-library patches remain necessary. But this may help the future state of affairs. |
|
For those suggesting moving from MySQL to PostgreSQL, you are correct that the problem is only when using PostgreSQL. However, "just switching databases" is definitely not an acceptable solution by any means. |
|
I've threatened @tenderlove with bodily harm unless this is fixed in 3.1.1. He claims he's going to partition by PID in the cache, but he's historically untrustworthy. |
|
@MasterLambaster : Is this patch is only for rails 3.1.X ? thanks |
|
Yes, that's for 3.1.x, later versions have separate class for statements cache - StatementPool |
|
I have 3.0.10, I thought it could be a solution for us :) |
|
3.0 doesn't use a statement cache. It tends that you have other problem that this particular one |
|
Just wondering why this issue has been closed. Has it been deemed resolved? I saw @tenderlove made a recent commit. Does this highlight / fix the issue at hand? |
|
for rails 3.1, it should be fixed. For rails 3.0, the truth is out there... |
|
@krzkrzkrz Yes, @tenderlove commit is fix for this issue. Prepared statements are stored with the process_id, so after fork the statements cache will be empty because proc id will change. |
|
Awesome. Great job Rails team! Thumbs up! |
|
@MasterLambaster thanks again mate for the monkey patch, @tenderlove thanks for the ingenious fix! Love you both! |
|
Had no luck with the above fix for unicorn. Reducing the number of workers to 1 seems to make the problem go away. |
|
I also had no luck with unicorn until I used the before_fork approach mentioned by @krzkrzkrz. That seemed to do it. |
|
Everything was fine on Monday. Now I'm getting the problem. Rails 3.1.3, Unicorn 4.1.1, PG 0.12.0, Postgresql 9.1 Happened when I did a bundle update on Tuesday to upgrade to Rails 3.1.3. I seen that Unicorn upgraded also. I downgraded everything but I still have the issue. Just wanted to add my 2c here. PS, switched to WEBrick for local development on my Mac to fix the problem. Only change was to remove Unicorn from my Gemfile. PPS, works fine on my Debian server. Only broken on my Mac. PPPS, Instead of using "rails s" to start my app, I used "unicorn" and it works fine. No more PG errors. |
|
I'm having this problem as well. Rails 3.1.3 |
|
I'm having a similar issue, sans Resque/PG, with: Rails 3.1.3 After a few seconds of light load, the Rack processes that Passenger spawns begin to eat up large amounts of CPU/Memory and eventually the box becomes unusable. An strace on one of the errant processes looks like this, over and over again: select(22, [], [20], [], {42, 971988}) = 1 (out [20], left {42, 971985}) The issue is resolved by setting: passenger_spawn_method conservative; |
|
We are seeing a similar issue with:
We are connecting to postgresql using SSL This seems to happen sporadically, I suspect it is after a Passenger forking or , but I have not been able to come up with steps to reliably reproduce the issue.. Relevant stack trace on the error is ActiveRecord::StatementInvalid: PGError: no connection to the server : SELECT "categories".* FROM "categories" LIMIT 1 |
|
@robdimarco it's other issue. The connection had to be reset on fork, but the passenger reset connection on fork. So the problem may be in network connectivity or at some point pg server goes away. |
|
@MasterLambaster thanks for the recommendation, will try. |
|
Quick note to others having the issues: 1: The Devise gem causes problems when running Rails with unicorn-rails (rails server). Clearance works fine. |
Its better if I provide the link to the problem description instead. Link contains error message and config files.
I suspect, this may be an issue with activerecord-3.1.0.rc1, since the error (described in stackoverflow link) is bringing it up. Had no problems running this exactly the same way in Rails 3.0.3.
http://stackoverflow.com/questions/6137570/resque-enqueue-failing-on-second-run
In summary:
In console, I do:
Everything works so far. Resque-web reports no failed jobs. And, I get the two 'puts' from module EncodeSong.
However, running Resque.enqueue(EncodeSong, Song.find(20).id, Song.find(20).unencoded_url) a second time will return the following error in resque-web (described in stackoverflow link). To make the error go away, I would have to close the process thats running: QUEUE=* rake environment resque:work and rerun it in the console window. But the problem comes back after trying to Resque.enqueue() after the first time.
The text was updated successfully, but these errors were encountered: