Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Rails migrations failing randomly without a proper error message #258
So I ran in to this today after a migration failed in our Rails/MySQL application. It passed on the staging environment which is an exact copy of production.
This thread on the forums talks about the issue. https://forums.aws.amazon.com/message.jspa?messageID=587770#587770
The error that is shown makes no sense and gives no clue to what is going wrong.
Further as I wrote in the thread I was able to get around the issue with the following workaround. which make little sense
I've seen this problem too. Let me ask you, how many instances you have in production? :)
The solution is actually quite simple, we do the following deployment strategy, assuming we have 8 app servers:
Both can be done at the same time. This is not ideal for haproxy and in case one fail, but it works for now.
Opsworks could fix it easily as well by changing how the "rake db:migrate" works, all needs to be done is to run the migration "only" on one of the rails app servers, for example, run rake db:migrate only if the server index is zero on the array of instances for the rails layer. i.e. node["opsworks"]["layers"][layer]["instances"]
...should be a few lines fix.
I actually agree with K0nserv... this error also happens when the migrations fail due do some other reason. Deploying without migrations and then running the migrations manually will show you the error.
So I tried running the migration manually and noticed the error...
So I added following into Gemfile and ran "/usr/local/bin/bundle update"
Which added following gems:
After that the error did not appear in next Deployment with migration... so it seems that this error & Chef exception are quite misleading!
Also this did not occur during 'development' environment. Perhaps because the I was using the 'sqllite3' in Dev but 'mysql' in Prod.
@fred If I remember correctly Amazon should be picking a single instance for the migration instead of running on all of them according to their docs anyway. And it is doing that in our case.
@n1t1nv3rma I don't think you should have to do add
I'm trying the AWS Opsworks, where I follow this workshop of Rails conferes 2015, for learn more about this service and how works, so on the way working every step of the workshop I got a error trying to deploy the Rails App - here is the log
[2015-10-25T18:00:44+00:00] ERROR: Running exception handlers [2015-10-25T18:00:44+00:00] ERROR: Exception handlers complete [2015-10-25T18:00:44+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out [2015-10-25T18:00:44+00:00] ERROR: deploy[/srv/www/todoapp] (deploy::rails line 65) had an error: Chef::Exceptions::Exec: if [ -f Gemfile ]; then echo 'OpsWorks: Gemfile found - running migration with bundle exec' && /usr/local/bin/bundle exec /usr/local/bin/rake db:migrate; else echo 'OpsWorks: no Gemfile - running plain migrations' && /usr/local/bin/rake db:migrate; fi returned 1, expected 0 [2015-10-25T18:00:44+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
So i'm looking but i couldn't find anything that help, any idea how can i solve this issue.
Nota: I deployed one time but without the migration, that mean my app doesn't work at all.
I have experienced the same problem using Postgresql and the problem happen both in production (where I have 3 running instances) and in staging (just one instance).
Sometimes seems that the deploy with migrations fails with this error:
but the migrations were performed! Because if I log via ssh inside my instance and I check the database I can see the changes.
Has someone any news?
Hi all, maybe I found the problem, at least it worked for me.
The reason why the migration fail can depend for many reason and the OpsWorks logs can't help us.
Login via ssh and then:
For me the error was:
And I fixed it just updating to Rails 4.2.6 (before was 4.2.3).
Hope it helps.
Turns out the app wouldn't boot. When I ran