S3 Connector -- update metadata backend to mongodb

[LaurenSpiegel]

update motion docker compose to use mongo metadata backend and include mongo docker.

[xinaxu]

@masih do you know what is this about?

[LaurenSpiegel]

currently the docker compose is using just a file system backend for the metadata zenko stores. This is not meant for production. There is a mongo db option that should be used instead.

[LaurenSpiegel]

@elijaharita , this seems like a P0 for actually guiding people to use motion with s3. otherwise, it's just in a demo local storage mode. cc @davidd8

[elijaharita]

@LaurenSpiegel which Dockerfile is this configured in?

[LaurenSpiegel]

it should be here -- https://github.com/filecoin-project/motion-s3-connector/blob/main/docker-compose.yaml#L123

you need to set cloudserver to use mongo and make sure you are including a mongo instance

[masih]

@elijaharita this is the branch I started and have not got a chance to complete. Please feel free to take over it. thank you!

[Angelo-gh3990]

Maybe use yugabyte if it needs high availability and scalability?

[masih]

Maybe use yugabyte if it needs high availability and scalability?

This stack lives entirely on the ISV side, not SP.
My sense would be to avoid involving yugabyte here because: 1) there is no evidence yet to suggest that mongodb doesn't cut it for ISVs who want S3, and 2) I do not believe Yugabyte as a backing store is supported by Zenko cloudserver, and I am hesitent to spend time implementing this backing store support in Zenko cloud server/ Zenko arsenal.

[Angelo-gh3990]

I agree . Maybe re-use the mongdb from singularly... extending it with a few tables or something

[gammazero]

@elijaharita See also, filecoin-project/motion-s3-connector#11

[xinaxu]

See above link for PR and documentation of how to use mongodb as metadata backend in Production environment.