Database Connectors 2022

[Licenser]

This ticket is for the CNCF spring mentorship 2022

Describe the problem you are trying to solve

Connectors are tremors interface to the outside world, they allow us to integrate with third-party systems. Currently, tremor only has a limited set of connectors for databases, we support s3 and google cloud storage for object stores, and have a k/v connector that offers a simple integrated key-value store. While this is a good starting point interfacing with more databases will make tremor easier to use for our end users.

Describe the solution you'd like

For each database, we need a connector that can both write data to the database as well as query it. We need a mapping of requests (query and write requests) to the tremor data representation, this should be considered as the foundation for not only one but all database connectors to give them a unified look and feel when using where possible. In addition, a mapping between database types and tremor types will be required. All of this requires both testing and documentation.

Primary goal:

• clickhouse connector to read and write from Yandex clickhouse
• A data mapping from databases types to tremor types
• A command mapping for tremors interface with databases

Stretch goals:

• Any additional relational database like Postgres, MySQL, SQLite, etc.
• Any non-relational database like Cassandra, Redis, Riak, etc.
• A mapping for database control (creating, destroying tables, etc.)

Notes

[tharun208]

Hello, @Licenser I have been trying out and understanding various projects in the CNCF ecosystem and I am particularly interested in those related to data processing so I would like to get involved with this project and work on this issue. I would like to contribute to this project and increase my knowledge. I would be familiarising myself with terminology and get to know better about Tremor in the coming days. Any help appreciated.

[Licenser]

this mentorship is concluded