Consider MongoDB replacement

[andresmgot]

Right now we are using MongoDB to store the information about the available chart of the existing chart repositories. MongoDB is a bit slow to start which causes failures at startup time (#434) apart from other issues (already resolved: #594, #479, #472). We can consider using a more light-weight solution like Redis or etcd since we don't need a lot of functionality.

Having said that, we may want to keep mongodb around if we plan to use it for any future feature. WDYT @migmartri @prydonius @arapulido ?

[prydonius]

Thanks @andresmgot. Given the way we currently use MongoDB, it might make sense to switch to something more lightweight like Redis. etcd would actually be overkill IMO, as we don't need the consistency and high availability for this cache store, and writes are slower than MongoDB or Redis.

Switching to a key-value store might mean we can't really query the way we currently do, e.g. something like: https://github.com/kubeapps/kubeapps/blob/master/cmd/chart-repo/utils.go#L244-L248, but I haven't thought to deeply about how we would do that with Redis.

we may want to keep mongodb around if we plan to use it for any future feature

Indeed, if we ever need to start storing more complex data and need to do more interesting queries, MongoDB would be useful.

[migmartri]

I guess that the question is. Are we planning on using MongoDB for more than just caching charts? If that's the case, Redis might be not a good fit, although I do not have experience with Redis apart of simple caching and content allocated storage nor I am familiar with the go clients.

We should also take into account that Monocular will rely on whatever chartsvc backend has, and in the case of kubeapps hub we will need to maybe maintain both (since we store users?)

I personally think that unless there is demand for it, or a big technical reason (apart of startup, which is a one time thing and can be diminished by using a 3rd party managed service), I'd not change it for now.

[andresmgot]

Thanks for the input! I will close this issue for the moment since we are not going to take actions unless we discover new blockers.

[malejpavouk]

My two cents: we consider MongoDB to be an issue to us, because its simply not part of our installation. And adding new DB technology is always a pain (you need backups, verify restore periodically, HA installation, additonal RAM consumption, training of support...).

I would suggest to go with some more proliferated technology such as SQL (with some ORM wrapper to cover most of the DB dialects) - huge majority of projects already has some SQL database with all the processes in place.

The other option may be etcd. I agree that it may sound as a on overkill, on the other hand Kubeapps is an app for Kubernetes. And Kubernetes itself relies on etcd - hence it already there. While it is not probably no the best choice to run K8s and apps on the same etcd instance, for small installations this may just work.

---

Regarding the arguments about HA - Kubeapps needs HA of the DB, if its meant to run in HA mode (and that is quite common requirement for any part of the infrastructure).
Speed of reads, writes: I am not expert in schema of Kubeapps, but I would guess that the DB is not really a bottleneck.

[migmartri]

Thanks @malejpavouk, so it stays in context I am adding to this discussion the link you provided in Slack https://dba.stackexchange.com/questions/191513/posgtresql-gin-btree-ordering as well as https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres

@andresmgot @prydonius I am reopening this issue but tagging it with needs-more-evidence so we can discuss it further?

[fabianschwarzfritz]

Hi @andresmgot ,

thanks for this overall great tooling and keeping the threats together in this repository.

I see from this issue and the PullRequest #1333, that the shift for a SQL based storage technology was decided and the first implementation work for connecting KubeApps with SQL based technology (PostgreSQL) is being done.

Is there a timeline if and when the support of MongoDB will be dropped? Is there a roadmap document describing these technology decisions?

Thanks and best regards,
Fabian

[andresmgot]

Hi @fabianschwarzfritz,

Yes, you can find the discussion document here: #1313. Feel free to add any comment or thought you may have!

We don't plan to drop support for MongoDB at least until we release a new major version (which may happen in the following months). In the meantime, both MongoDB and PostgreSQL will be supported.

[andresmgot]

This can be considered finished. It's documented and it will be available in the next version.