Rethinking the storage of API

[Schnitzel]

I think we could rethink how the API stores it's data. Currently we are required to store data in the Hiera YAML format as our v3 infrastructure also uses this api in order to provision servers, etc.
The idea initially was that the v4 infrastructure uses the same exact same api and then in the future when everything is migrated we can remove the hiera YAML format and move to another storage system - I would call this process the parallel migration process.

I think though we should rethink this idea and have the following suggestion:

• We leave the Client, Sitegroup, Sites Storage within Hiera YAML and the API like it is right now (Read Only, no mutations)
• For the new v4 infra (openshift) we implement a new storage system (couch? neo4j? mongo? tbd), maybe even with new terminology
• If needed I'm also happy to fork the current api into a new version which handles this new storage system, in order to not need to query two storage systems at once (as it will actually never happen that we need in one single request data from the new and old storage at the same time)
• we would also need to point the cli to the new system only, the old system does not need a cli (clients would therefore need to migrate first to the v4 infra before they can use the cli, but that's okay)
• If we like we could also come up with new terminology. We currently have the problem that our clients don't know what a "SiteGroup" really is, and rather describe it as a git repo or a "project". also "Site" is not really understood, it's more described as an "environment"

[fubhy]

If we like we could also come up with new terminology. We currently have the problem that our clients don't know what a "SiteGroup" really is, and rather describe it as a git repo or a "project". also "Site" is not really understood, it's more described as an "environment"

I like "Project" and "Environment"

If needed I'm also happy to fork the current api into a new version which handles this new storage system, in order to not need to query two storage systems at once (as it will actually never happen that we need in one single request data from the new and old storage at the same time)

We might be able to pull this off without forking the API because we actually already put some effort in decoupling the storage from the rest of the architecture.

[Schnitzel]

As discussed after a call with @ryyppy, we decided to do the following:

• Keep the storage for the objects Client, Sitegroup and Site within YAML, do not implement mutations for these objects, we will just support Reading (with that we save ourselves a huge amount of pain for parallel writing into git, merge conflicts, etc.
• Create 3 new Objects Customer, Project, Environment in the API that will be saved within an PostgreSQL database and will support full mutations.
• These new Objects will contain only key and values specifically required for Lagoon and nothing for the old Hiera architecture
• We will rebuild all Lagoon Services to use the new Objects