backup with mongodump gets stuck

[sradu]

First I want to say that I'm a huge fan of backup and you've done an amazing job.

We migrated our db to a new AWS instance and for some reason backup gets stuck at the end of the mongodump process. Running mongodump manually, the exact same command backup executes, ends correctly. Using mongo 3.0.6.

This is after I killed it after 6 hours. It usually takes 15 minutes.

[2015/09/16 17:54:01][error] 2015-09-16T11:06:15.830+0000 [####################....] production.tests 116941/135916 (86.0%)
[2015/09/16 17:54:01][error] Killed
[2015/09/16 17:54:01][error]
[2015/09/16 17:54:01][error] Backtrace:
[2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/utilities.rb:199:in run' [2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/utilities.rb:217:inblock (2 levels) in module:Helpers'
[2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/database/mongodb.rb:78:in dump!' [2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/database/mongodb.rb:61:inperform!'
[2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/model.rb:269:in each' [2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/model.rb:269:inblock in perform!'
[2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/model.rb:268:in each' [2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/model.rb:268:inperform!'
[2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/lib/backup/cli.rb:156:in perform' [2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/thor-0.18.1/lib/thor/command.rb:27:inrun'
[2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/thor-0.18.1/lib/thor/invocation.rb:120:in invoke_command' [2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/thor-0.18.1/lib/thor.rb:363:indispatch'
[2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/thor-0.18.1/lib/thor/base.rb:439:in start' [2015/09/16 17:54:01][error] /var/lib/gems/2.1.0/gems/backup-4.1.12/bin/backup:5:in<top (required)>'
[2015/09/16 17:54:01][error] /usr/local/bin/backup:23:in `load'

this is ps aux before the kill -9.
admin 2700 0.0 0.0 4328 752 ? Ss 11:00 0:00 /bin/sh -c /usr/local/bin/backup perform --trigger standard
admin 2701 0.0 0.2 129936 48664 ? Sl 11:00 0:01 /usr/bin/ruby2.1 /usr/local/bin/backup perform --trigger standard
admin 2707 0.0 0.0 4328 752 ? S 11:00 0:00 sh -c /usr/bin/mongodump --db='production' --username='user'....
admin 2710 1.5 0.0 831792 12456 ? Sl 11:00 6:27 /usr/bin/mongodump --db=production --username=user...

Not sure what other information to provide, it's definitely weird.

[tombruijn]

Hi @sradu sounds like a problem yes.

Does the mongodump process still show up in the process list or has it exited already while backup is still running?

It sounds a bit like another problem that was reported. #617
If this problem is caused by the same problem I'm afraid we won't be able to fix it. The problem lay is in open4 that if there was too much output in a command it would hang. Try configuring mongodb that it doesn't provide any output in STDOUT.
Eventually the reporter worked around it by executing a command in a before block.

[sradu]

@tombruijn that's a great idea. Is there a simple way to add --quiet to the mongodump command via Backup?

[sradu]

--additional_options?

[tombruijn]

@sradu that would be the place to put it

mongodb.additional_options = ["--quiet"]

[sradu]

--quiet did the trick, thanks @tombruijn!

[tombruijn]

Good to hear it's working @sradu !

[beeva-robertoandradas]

good trick, it works for me too.