mongodump: illegal argument combination: cannot specify --username and --uri

[dumbasPL]

Summary

Mongo db backup broke

Steps to reproduce

1. run the docker-compose from #Environment
2. observe error

error parsing command line options: illegal argument combination: cannot specify --username and --uri
try 'mongodump --help' for more information

What is the expected correct behavior?

DB Backup of 'mongo_test_mongo_20220929-114619.archive.gz' completed successfully

Relevant logs and/or screenshots

tmpnvdkfyvuf8-mongo-backup-1  |                                                                        ,---.
tmpnvdkfyvuf8-mongo-backup-1  | ,--------.,--.                 ,--.            ,---.    ,--. ,--------.|   |
tmpnvdkfyvuf8-mongo-backup-1  | '--.  .--'`--',--.--. ,---.  ,-|  |     ,---. /  .-'    |  | '--.  .--'|  .'
tmpnvdkfyvuf8-mongo-backup-1  |    |  |   ,--.|  .--'| .-. :' .-. |    | .-. ||  `-,    |  |    |  |   |  |
tmpnvdkfyvuf8-mongo-backup-1  |    |  |   |  ||  |   \   --.\ `-' |    ' '-' '|  .-'    |  |.--.|  |   `--'
tmpnvdkfyvuf8-mongo-backup-1  |    `--'   `--'`--'    `----' `---'      `---' `--'      `--''--'`--'   .--.
tmpnvdkfyvuf8-mongo-backup-1  |                                                                        '--'
tmpnvdkfyvuf8-mongo-backup-1  | Image:  tiredofit/db-backup | Version  3.4.0 Type 'image_changelog' for details
tmpnvdkfyvuf8-mongo-backup-1  | Repository/Issues/Support:  https://github.com/tiredofit/docker-db-backup/
tmpnvdkfyvuf8-mongo-backup-1  |            Sponsor me for development and upkeep: https://www.tiredofit.ca/sponsor
tmpnvdkfyvuf8-mongo-backup-1  | 
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:47 [NOTICE] ** [scheduling] Container configured for scheduled tasks with 'cron'
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:47 [STARTING] ** [scheduling] [1] Starting cron
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [INFO] ** [db-backup] Next Backup at 2022-09-29 11:49:52 GMT
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [NOTICE] ** [db-backup] Dumping MongoDB database: 'test' and compressing with gzip
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29T11:49:52.582+0000	error parsing command line options: illegal argument combination: cannot specify --username and --uri
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29T11:49:52.582+0000	try 'mongodump --help' for more information
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [ERROR] ** [db-backup] DB Backup of 'mongo_test_mongo_20220929-114952.archive.gz' reported errors
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [ERROR] ** [db-backup] Skipping Checksum creation because backup did not complete successfully
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [ERROR] ** [db-backup] Skipping moving DB Backup to final location because backup did not complete successfully
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [NOTICE] ** [db-backup] DB Backup for '' time taken: Hours: 0 Minutes: 00 Seconds: 00
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [INFO] ** [db-backup] Backup routines finish time: 2022-09-29 11:49:52 GMT with overall exit code 1
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [NOTICE] ** [db-backup] Backup routines time taken: Hours: 0 Minutes: 00 Seconds: 00
tmpnvdkfyvuf8-mongo-backup-1  | 2022-09-29.11:49:52 [NOTICE] ** [db-backup] Sleeping for another 86400 seconds. Waking up at 2022-09-30 11:49:52 GMT

Environment

• Image version / tag: 3.4.0-3.4.2 (3.3.12 works fine)
• Host OS: Debian 11 and Arch Linux

Any logs | docker-compose.yml

version: '3.8'
services:
  mongo:
    image: mongo:5
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: test
  mongo-backup:
    image: tiredofit/db-backup:3.4.2
    restart: always
    volumes:
      - ./backup:/backup
    environment:
      # - DEBUG_MODE=true
      - CONTAINER_ENABLE_MONITORING=false
      - DB_AUTH=admin
      - DB_TYPE=mongo
      - DB_HOST=mongo
      - DB_NAME=test
      - DB_USER=root
      - DB_PASS=test

versions 3.4.0 and up are affected, below that works as expected

Possible fixes

use the correct mongodump syntax for the new version

[tiredofit]

Can you try switching your image to tiredofit/db-backup:develop and entering the container and typing backup_now and seeing if you get a successful backup? There is work happening in that branch that needs to be signed off by the requestor which is related to Mongo which would resolve.

[dumbasPL]

the develop branch is currently broken as well. There are missing quotes here. After fixing that, it throws the same exact issue that I initially reported.

The problem is that we can use either --host --port --username --password --authenticationDatabase --db or just --uri and include everything in it but not both at the same time.

[tiredofit]

Thanks for the headsup on the quotes.

Looks like I'm going to need you in #149. Once someone puts in a URI it takes the place of the arguments are you saw above. What arguments are you trying to add in? I've just added the ability to use EXTRA_OPTS with the URI function as well. Let me know if that works, otherwise we'll have to further expand.

[dumbasPL]

What arguments are you trying to add in

nothing extra (see docker-compose in my original comment)

the problem that #149 made is that it's not possible to use mongodb+srv without using --uri and using --uri prevents us from using all the other parameters like username, password, host etc...

The only way I see forward would be to forget about individual arguments and construct a full mongo uri from the provided data.

Some important things to watch out for

• username, password, authSource, db must be URL encoded before adding them to the final URI
• port must not be present when using mongodb+srv
• When a custom uri is provided we should ignore all other options except EXTRA_OPTS

[tiredofit]

I think what I'll do is build some routines to use the old way unless the custom URI is set. I have a couple other things on the go right now but will get to this in a few days.

[tiredofit]

Sorry, I had a health issue which limited my ability to work on this. I'd like to close this down. I just pushed the ability to use EXTRA_OPTS with MONGO_CUSTOM_URI and will review how I am assembling the final URI to see we can get back on track.

[tiredofit]

A new version which restores the old functionality of backing up with --host --user etc parameters is now in place. If detected MONGO_CUSTOM_URI - then that will override and use that string along with --uri as a prefix to backup.

EXTRA_OPTS functions in both cases as well. Can you let me know how the new :develop works?

[dumbasPL]

Sorry for the late response. the latest :develop seems to have a quotation problem.

Here is the invalid command it's trying to run. there is an extra ' before --host and at the end.
mongodump --archive=/tmp/backups/mongo_test_mongo_20221008-184622.archive.gz --gzip '--host mongo --port 27017 --username root --password test --authenticationDatabase admin --db test '

Here is a relevant portion of the debug log.

tmpfqchaukyoo-mongo-backup-1  | 2022-10-08.18:46:22 [DEBUG] /etc/services.available/10-db-backup/run ** [db-backup] Backup routines started time: 2022-10-08 18:46:22 GMT
tmpfqchaukyoo-mongo-backup-1  | + case "${dbtype,,}" in
tmpfqchaukyoo-mongo-backup-1  | + check_availability
tmpfqchaukyoo-mongo-backup-1  | + var_false FALSE
tmpfqchaukyoo-mongo-backup-1  | + '[' false = false ']'
tmpfqchaukyoo-mongo-backup-1  | + case "$dbtype" in
tmpfqchaukyoo-mongo-backup-1  | + '[' -n '' ']'
tmpfqchaukyoo-mongo-backup-1  | + counter=0
tmpfqchaukyoo-mongo-backup-1  | + nc -z mongo 27017
tmpfqchaukyoo-mongo-backup-1  | + backup_mongo
tmpfqchaukyoo-mongo-backup-1  | + prepare_dbbackup
tmpfqchaukyoo-mongo-backup-1  | ++ date +%s
tmpfqchaukyoo-mongo-backup-1  | + dbbackup_start_time=1665254782
tmpfqchaukyoo-mongo-backup-1  | ++ date +%Y%m%d-%H%M%S
tmpfqchaukyoo-mongo-backup-1  | + now=20221008-184622
tmpfqchaukyoo-mongo-backup-1  | ++ date +%H:%M:%S
tmpfqchaukyoo-mongo-backup-1  | + now_time=18:46:22
tmpfqchaukyoo-mongo-backup-1  | ++ date +%Y-%m-%d
tmpfqchaukyoo-mongo-backup-1  | + now_date=2022-10-08
tmpfqchaukyoo-mongo-backup-1  | + target=mongo_test_mongo_20221008-184622.sql
tmpfqchaukyoo-mongo-backup-1  | + '[' '' = none ']'
tmpfqchaukyoo-mongo-backup-1  | + '[' '' = false ']'
tmpfqchaukyoo-mongo-backup-1  | + target=mongo_test_mongo_20221008-184622.archive.gz
tmpfqchaukyoo-mongo-backup-1  | + mongo_compression=--gzip
tmpfqchaukyoo-mongo-backup-1  | + compression_string='and compressing with gzip'
tmpfqchaukyoo-mongo-backup-1  | + '[' -n '' ']'
tmpfqchaukyoo-mongo-backup-1  | + mongo_backup_parameter='--host mongo --port 27017  --username root --password test --authenticationDatabase admin --db test '
tmpfqchaukyoo-mongo-backup-1  | + pre_dbbackup test
tmpfqchaukyoo-mongo-backup-1  | + '[' -n '' ']'
tmpfqchaukyoo-mongo-backup-1  | + '[' -d /assets/custom-scripts/pre ']'
tmpfqchaukyoo-mongo-backup-1  | + '[' -d /assets/scripts/pre/ ']'
tmpfqchaukyoo-mongo-backup-1  | + print_notice 'Dumping MongoDB database: '\''test'\'' and compressing with gzip'
tmpfqchaukyoo-mongo-backup-1  | + output_off
tmpfqchaukyoo-mongo-backup-1  | + '[' true = true ']'
tmpfqchaukyoo-mongo-backup-1  | + set +x
tmpfqchaukyoo-mongo-backup-1  | 2022-10-08.18:46:22 [NOTICE] /etc/services.available/10-db-backup/run ** [db-backup] Dumping MongoDB database: 'test' and compressing with gzip
tmpfqchaukyoo-mongo-backup-1  | + silent mongodump --archive=/tmp/backups/mongo_test_mongo_20221008-184622.archive.gz --gzip '--host mongo --port 27017  --username root --password test --authenticationDatabase admin --db test '
tmpfqchaukyoo-mongo-backup-1  | + '[' true = true ']'
tmpfqchaukyoo-mongo-backup-1  | + mongodump --archive=/tmp/backups/mongo_test_mongo_20221008-184622.archive.gz --gzip '--host mongo --port 27017  --username root --password test --authenticationDatabase admin --db test '
tmpfqchaukyoo-mongo-backup-1  | 2022-10-08T18:46:22.999+0000	error parsing command line options: unknown option "host mongo --port 27017  --username root --password test --authenticationDatabase admin --db test "
tmpfqchaukyoo-mongo-backup-1  | 2022-10-08T18:46:22.999+0000	try 'mongodump --help' for more information
tmpfqchaukyoo-mongo-backup-1  | + exit_code=1
tmpfqchaukyoo-mongo-backup-1  | + check_exit_code mongo_test_mongo_20221008-184622.archive.gz
tmpfqchaukyoo-mongo-backup-1  | + print_debug 'DB Backup Exit Code is 1'
tmpfqchaukyoo-mongo-backup-1  | + output_off
tmpfqchaukyoo-mongo-backup-1  | + '[' true = true ']'
tmpfqchaukyoo-mongo-backup-1  | + set +x
tmpfqchaukyoo-mongo-backup-1  | 2022-10-08.18:46:23 [DEBUG] /etc/services.available/10-db-backup/run ** [db-backup] DB Backup Exit Code is 1
tmpfqchaukyoo-mongo-backup-1  | + case "${exit_code}" in
tmpfqchaukyoo-mongo-backup-1  | + print_error 'DB Backup of '\''mongo_test_mongo_20221008-184622.archive.gz'\'' reported errors'
tmpfqchaukyoo-mongo-backup-1  | + output_off
tmpfqchaukyoo-mongo-backup-1  | + '[' true = true ']'
tmpfqchaukyoo-mongo-backup-1  | + set +x
tmpfqchaukyoo-mongo-backup-1  | 2022-10-08.18:46:23 [ERROR] /etc/services.available/10-db-backup/run ** [db-backup] DB Backup of 'mongo_test_mongo_20221008-184622.archive.gz' reported errors

[tiredofit]

Thanks let me fix this immediately and have a new develop in place momentarily.
Edit : In place.

[dumbasPL]

Wasn't expecting an answer this quick. Well, rip my PR I guess.

Yes, It does work correctly with the quotes removed

[tiredofit]

Great. Are you able to test the MONGO_CUSTOM_URI variable for me as well against your server? I see you are using the legacy function before I started breaking things spectacularly.

[dumbasPL]

specifying just MONGO_CUSTOM_URI gives
[ERROR] ** [db-backup] No 'Database Host' Entered! - Set '$DB_HOST'

Other than that everything works fine when using the URI (also tested the MongoDB atlas mongodb+srv URI and it works fine too).

The host is technically not needed since it's already in the URI but I've seen it's used in other places like the backup file name so I'm not sure how to feel about this.

We could keep it as is, and still require the host but just make it clear in the documentation that when using MONGO_CUSTOM_URI the DB_HOST will not be used to perform the backup

[tiredofit]

I actually should be populating DB_HOST to build the filename. Thanks for this. I'm going to need 5 minutes or so and then I can see whats going on here.

[tiredofit]

Oof. I see it now. It would work with a manual backup but not with the container starting. I've shuffled some things around again. One last time if you could for a test? Seems good on my end.

[dumbasPL]

Looks good to me 👍