feature request: option to stepdown after backup

[jamesholcomb]

Background

I have a mongodb@4.4 three member replica set running on k8s and automated backups using mgob. Its memory limit (400M) is set to the max operational amount based on the working set to save $. This works great until the nightly backup is executed. When a backup completes, the primary memory spikes and will eventually do one of two things:

1. terminate with OOMKilled
2. even worse, succumb to disk paging with huge query times until the memory limit is reached

My efforts to figure out how to limit mongodb memory usage have been fruitless...I've tried changing the wired tiger cache limits, increasing memory limits, etc. The cluster is maxed to point where I would have to spin up more nodes ($) to increase memory further.

I'd like to propose a an option to stepdown the primary after successful dump. This is what I end up doing after seeing the primary is getting close to OOMKilled or performance goes to hell.

What do you think?

Thank you for taking on the maintenance of this critical piece of infra!

[maxisam]

Interesting. Could you explain what did you mean by stepdown the primary, like set it as a secondary and restart it? How do you do it in command line? Just trying to make sure I will get it right. Lately I am busy, but I would like to implement this one when I have time and know how to do it exactly.

Welcome to send a PR if you want.

[jamesholcomb]

Yes, that is correct. One way to do it would be rs.stepDown() followed bydb.getSiblingDB("admin").shutdownServer(). If the member is part of a k8s Stateful set (or perhaps hooked into some other process monitor), the instance would get rescheduled and startup normally.

I don't have any experience with go, so probably not a great candidate to submit a PR.

[maxisam]

https://www.mongodb.com/docs/manual/reference/method/db.shutdownServer/#shut-down-a-primary-mongod-with-longer-timeout

according to the document. It seems like I only need to call

db.getSiblingDB("admin").shutdownServer({ "timeoutSecs": 60 })

and it should handle stepdown and shutdown together.

I do wanna know if there is a way to do restart directly from the command

[maxisam]

#43
Could you try dev.114 ? I haven't have time to test it. But I think it should work.

[jamesholcomb]

Thanks for doing this...do you know if it handles the case where mongod was started without auth?

From the docs:

Behavior

For a mongod started with Authentication, you must run db.shutdownServer() over an authenticated connection. See Access Control for more information.

For a mongod started without Authentication, you must run db.shutdownServer() from a client connected to the localhost interface. For example, run a mongo with the --host "127.0.0.1" option on the same host machine as the mongod.

[maxisam]

I don't think so. Neither I realize this part. But I think it is not too hard to set authentication right?

[jamesholcomb]

Perhaps just a note in the README explaining that auth is required would suffice?

[maxisam]

Agree. Does this work anyway?

[jamesholcomb]

Other than my prod cluster, I don't have any other clusters that utilize both replica sets and auth...adding to that, I have yet to switch over from stefanprodan:mgob. I'll report back here once I have been able to test this out.

[jamesholcomb]

Sorry for the delay. i began work on testing this out, but could not find a way to have the db password retrieved from a secret or env variable expansion.

An issue was created but never addressed in the original repo (auto closed once it was archived) to allow passwords stored in secrets: stefanprodan/mgob#58

A workaround was proposed to put the entire config in a secret, but that seems burdensome and overly complicates configuration.

Would you be interested in implementing this feature? I use the bitnami/mongodb chart which automatically creates a k8s secret with name=db-mongodb and key=mongodb-root-password

[maxisam]

Well, I am not sure what is the best way for this one. Welcome to make a PR. I think the easiest way is making the whole config load from the secret. For me I don't have this issue, all my secrets are injected on the fly. So whether it is secret or config doesn't really matter to me.

[jamesholcomb]

How do you inject secrets on the fly? Perhaps I can use that technique.

[jamesholcomb]

A simple hacky approach for env var expansion I found from a quick scan: https://mtyurt.net/post/go-using-environment-variables-in-configuration-files.html

The mgob config would become:

target:
  password: ${MONGODB_ROOT_PASSWORD}

The mgob statefulset:

env:
  -name: MONGODB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: mongodb-root-password
              name: db-mongodb

Or piggyback on the above approach, but utilize https://github.com/spf13/viper along with templates spf13/viper#315 (comment) which would enable mgob to remove all sensitive info (passwords/secrets/keys) from the config.

[maxisam]

what if someone has multiple mongodbs? It might work by using a init container to load the secret and update the config before running the main part. However, I don't have time do work on this right now. Welcome to make a PR tho.

[maxisam]

@jamesholcomb I think I got what you want. you can use env variable like PLAN-ID_AZURE_CONNECTIONSTRING.

Give it a try docker pull maxisam/mgob:dev.225

[jamesholcomb]

Sure, will give this a try. I see you added viper to read the config in #80. Is this using env var expansion or templates or ...? Please show an example if you can.

[maxisam]

mgob/.github/workflows/build.yml

Line 102 in 81b39b0

-e MONGO-TEST_AZURE_CONNECTIONSTRING="$AZURE_CONNECTIONSTRING" \

it just use environment variable so far.

env name format is like <plan-id>_<property name>. All upper case and use _ for different level

ex PLAN-ID_AZURE_CONNECTIONSTRING.

[jamesholcomb]

How would I would inject a mongodb password from a k8s secret with this technique?

[maxisam]

https://medium.com/@milindawelagedara/pass-database-username-and-password-as-environment-variables-to-kubernetes-deployment-using-392b27934f60