Can't repair database after unclean shutdown

[jjpeet]

After an unclean shutdown, I've been unable to restart Mongo. I've tried restoring a number of backups which I'm sure are fine, but it still won't start.

The startup logging is as follows;

2018-08-31T14:36:10.174+1000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=d59b5517b17c
2018-08-31T14:36:10.174+1000 I CONTROL  [initandlisten] db version v3.6.5
2018-08-31T14:36:10.174+1000 I CONTROL  [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618
2018-08-31T14:36:10.174+1000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2018-08-31T14:36:10.175+1000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-08-31T14:36:10.175+1000 I CONTROL  [initandlisten] modules: none
2018-08-31T14:36:10.175+1000 I CONTROL  [initandlisten] build environment:
2018-08-31T14:36:10.175+1000 I CONTROL  [initandlisten]     distmod: debian81
2018-08-31T14:36:10.175+1000 I CONTROL  [initandlisten]     distarch: x86_64
2018-08-31T14:36:10.175+1000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-08-31T14:36:10.176+1000 I CONTROL  [initandlisten] options: { net: { bindIpAll: true } }
2018-08-31T14:36:10.185+1000 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-08-31T14:36:10.190+1000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=487M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-08-31T14:36:11.004+1000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1535690171:4430][1:0x7fda37f9ea00], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2018-08-31T14:36:11.004+1000 E STORAGE  [initandlisten] WiredTiger error (0) [1535690171:4575][1:0x7fda37f9ea00], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2018-08-31T14:36:11.009+1000 E STORAGE  [initandlisten] WiredTiger error (0) [1535690171:9213][1:0x7fda37f9ea00], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2018-08-31T14:36:11.009+1000 E STORAGE  [initandlisten] WiredTiger error (0) [1535690171:9287][1:0x7fda37f9ea00], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2018-08-31T14:36:11.018+1000 E -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 416
2018-08-31T14:36:11.019+1000 I STORAGE  [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2018-08-31T14:36:11.020+1000 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-08-31T14:36:11.020+1000 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-08-31T14:36:11.020+1000 I CONTROL  [initandlisten] now exiting
2018-08-31T14:36:11.020+1000 I CONTROL  [initandlisten] shutting down with code:100

What's the best way to repair the current database or get a backup up and running?

I tried running up a new mongo docker and adding a shell to it to run mongod --repair, but it won't work too. Are there command line parameters that I should be using for this manual startup of Mongod with --repair?

In future, should I be doing a shutdown of Mongo to back it up?

Thx
James

[wglambert]

The issue could be permissions, otherwise an improper backup/restore:
file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt

The recommended way to backup a database is outlined here

$ docker exec some-mongo sh -c 'exec mongodump -d <database_name> --archive' > /some/path/on/your/host/all-collections.archive

To restore https://docs.mongodb.com/manual/tutorial/backup-and-restore-tools/#basic-mongorestore-operations

If you have further questions you should ask the Docker Community Forums, the Docker Community Slack, or Stack Overflow. As these repositories are for issues with the image and not necessarily for questions of usability

[jjpeet]

What command line options are used within the Mongo docker to start mongo?
JP

[wglambert]

That would be in https://github.com/docker-library/mongo/blob/master/4.1/docker-entrypoint.sh#L228

And dependent upon environment variables given and if it's creating an initial database or not