Spec Representation of Catalog Entries (Apps) within the Control Plane

[puja108]

Mon, Mar 23, '20 status in this story: #95
Wed, Apr 8, '20 Spec: Representation of apps in the CP
Mon, Apr 20, '20 update: When spec is done, team needs a call to spec out implementation.

User Story

This is an internal story to solve for several internal technical needs and requirements of user facing stories.

This is the basis. The user stories depending on this are:

• Listing of Apps
• Updating of Apps (update events/notifications)
• Defaulting of App CR fields (user just says I want prometheus and we default to latest version)
• Validation of App CRs (specifically validate if app name and version are exisiting in said catalog)

Latter 2 stories are concerned with havng functionality baked into our Control Plane and its Kubernetes API so that we are not dependent on the to-be-deprecated GS API.

Future stories depending on this might be:

• Dependency management between apps and other apps
• Dependency management between apps and GS versions
• Making required config fields explicit
• Other metadata to apps (Screenshots, long descriptions, example configs)

Current State

Currently, an App Catalog is represented solely by its appcatalog CR in the CP. There's no information about its contents in the CP.

Happa talks directly to the storage of the Helm repository depicted in the appcatalog CR (i.e. in our cases Github) to get the index.yaml, parse it, and use it for some defaulting and representation cases.

The functionality to list apps is not present in our API. Thus, UX is being able to see the SSoT, i.e. Github, and following it to check out which apps are present.

Requirements

What we need is a way to get the SSoT, i.e. a list of apps, their versions, as well as metadata, from a definite place inside the Control Plane.

Options

As we are extending the Kubernetes API of the CP there's basically 2 ways that are native:

1. CRD + Controller (i.e. Operator Pattern)
2. Microservice that offers an aggregated API

While 1 has been our goto solution for most issues, we often have mitigated the impact of such an asynchronous, event-driven API towards the user by adding a microservice or API logic of our own. Latter patterns we want to get rid of as our API and the microservices that "translate" towards CRDs are something we want to deprecate mid-term.

For 2 we do not have any production experience. It is used upstream for the metrics API and Team Ludacris is considering using it for #13 (Health of TCs). There's a PoC that they have built in the Hackathon last week.

Team Batman has extensively evaluated Option 1 (see also https://docs.google.com/document/d/1LCd4XeLh8fYKILjTfyvBILrWZc07Gn0zdrbpC-MZUOg/edit), so we will describe this and also a lower fidelity spec of Option 2 in the following.

Keep in mind that no matter what option we go with the SSoT stays within the Helm Repository, no matter if that repository is blob storage with an index.yaml like right now, or might have a better API that can offer such information as in chart-museum and Harbor. Currently, I do not see a way around having a local representation somehow (more on this in the harbor case below). Correct me if this assumption is wrong.

1) AppCatalogEntry CRD

The idea is to have a cluster-scoped AppCatalogEntry CR per App in a Catalog. This CR is to be created by a new controller in app-operator based on the SSoT that is the appCatalog storage, i.e. github or other helm repo and its index.yaml

Sample CR (details can be discussed in apiextensions PR once we have a better plan here)

metadata:
  name: kong-incubator
  labels:
   application.giantswarm.io/catalog: giantswarm-incubator
spec:
  name: kong-app
  description: The Cloud-Native Ingress and Service Mesh for APIs and Microservices
  engine: gotpl
  home: https://KongHQ.com/
  Icon: https://s3.amazonaws.com/downloads.kong/universe/assets/icon-kong-inc-large.png
  sources:
    - https://github.com/Kong/kong
  urls:
    - https://giantswarm.github.com/giantswarm-incubator-catalog/kong-app-0.2.0.tgz
versions:
  - version: 0.2.1
    appVersion: 1.2.3
    created: "2019-09-28T13:18:25.002458784Z"
  - version: 0.2.0
    appVersion: 1.2.2
    created: "2019-08-28T13:18:22.004458784Z"

This CRD can then be used for all possible use cases mentioned above. It can be extended to include additional metadata with time, but the idea would be to start small.

2) appcatalog-service

A microservice serving an aggregated API that does one of the 2 following things:

1. In case of blob storage for Helm Repository, it downloads the index.yaml, keeps a purged representation of that in its "cache" and answers requests for apps, versions, and metadata.
2. In case of chart-museum/harbor it proxies (and maybe caches) requests to said chart registry.

The reason for a proxy in case 2 is that it abstracts away from the actual representation and storage in case those change. Another reason is that for a better UX a single place to get the information about app and version availability and install would make more sense. Having the user switch to a Chart Registry frontend to explore which apps they want and then come back and install and configure them in clusters sounds icky. Again correct me if I'm wrong.

[puja108]

I'd like to get some input on this from @giantswarm/team-batman first before I open it up to many people. Also pinging @kopiczko as you already started commenting on such issues in the apiextension PR and I do think your point of the CRD feeling weird is valid.

[kopiczko]

Re AppCatalogEntry CRD

General usage of AppCatalogEntry makes sense to me with this explanation thanks. The important bit here is that nothing but a single controller in app-operator accesses the underlying app storage with the approach. It sounds like an optimisation though.

Re appcatalog-service

First of all we don't need to use the frontend of chartmuseum. We could only use the API. I'm not sure if it has any frontend at all. The main problems here would be where to run it (we have operations cluster now - I don't know if this is a good fit) and what storage backend to use (I'm not sure if Github is available).

One problem that this would solve is simultaneous writes to the catalog we have right now. https://github.com/giantswarm/giantswarm/issues/6692

---

As I wrote in the PR I'm not against the CR. Now with better understanding I'm convinced even more. I still have mixed feelings if this is better than using upstream solution which seems to fit the bill quite well possibly even already having features that we will need to implement in the future. Not to mention that it would make us closer to the community. Having expressed my concerns I leave the final decision to your team. I believe you have better insights to the problem than I do.

[puja108]

I also talked to @teemow about related things and he kind of supported the CRD approach and made me see some benefits with having things like status. And yes, it is a local cache, and should not be replacing our need for sth like chart-museum, but maybe make us more independent there. Maybe chart-museum would even make this easier to implement (e.g. as then we would have an API instead of needing to parse the index.yaml ourselves)

Would also be interesting to talk with @giantswarm/sig-architecture about this as there might be general arch patterns to derive from issues like this.

[piontec]

I'm not aware of any particular pattern that fits here.
Some thoughts about the topic from my side:

• Currently, there's no notion of application and application versions in upstream API. So, to me, it looks like we're starting something that we would like - ultimately - to be the standard, a promoted by us way to discover applications possible to install in kuberentes cluster. Sure that we do that for our product and we need to focus on that, but it would be nice if it becomes the standard way later. As such, we should probably also have a look if anybody else already does something like that and how.
• as for native way, I also see these 2 ways. And yes, we should abstract actual backend providing the applications from the client and leave us the option o switch backends without impacting interface we have for clients. This is a good idea IMHO, despite the fact it creates another alyer of abstraction and a piece of software that needs maintenance.
• as for CRD vs aggregated API, I think the final solution in both cases will be very similar to each other: we have to poll/pull application info backend (helm and such) and transform that data into standardized way and put it in API CR objects or some cache that the service for aggregated API can access. So what differs is just how we present the transformed data. To me, aggregated API seems a little bit simpler and more natural fit here, but if we have good experience only with controllers+CR,, this should work as well.

[cokiengchiara]

@puja108 by "SoT" you mean State of the Art? FYI I changed all instances of SoT to SOTA to reflect our internal lingo / dictionary: https://github.com/giantswarm/giantswarm/blob/master/sigs/operator/sota_operator.md

[cokiengchiara]

The next step would be to

• Assign to someone to create tech specs and break down into issues.

Maybe do this with the team collaboratively?

[cokiengchiara]

@puja108 and @piontec will bring the spec to markdown and ask people to review. @tomahawk28 will work on this.

Maybe this cycle, we'll aim to get the CRD merged. We have to think about the design of the controller.

[cokiengchiara]

@puja108 and @piontec will bring the spec to markdown and ask people to review. @tomahawk28 will work on this.

Maybe this cycle, we'll aim to get the CRD merged. We have to think about the design of the controller.

hi @puja108 @piontec , I guess we forgot (?) to work on this last cycle...? first step was to bring the spec to markdown?

[piontec]

Yes, on my plate :| As mentioned today in the testing meeting with Halo, I'll try to work on this soon (next 1.5 week)

[piontec]

Related issue about keeping metadata in Halo team: #95

[cokiengchiara]

@oponder I had this on my todo list ""Test versions" need to be in the meta-data thing w lukasz"

Just to confirm that is for this task in this issue https://github.com/giantswarm/giantswarm/issues/8021 right?

• Happa allows selecting between stable and stable+resting quality apps

[piontec]

Let's be careful with "testing" and "pre-release".
"Testing" is what we agreed to go to "*-testing-catalog" catalogs and is never visible to customers; these are dev-time builds.
"Pre-release" are "-alpha*", "-beta*", "-rc*" builds that are to go to the normal catalog, but are meant to show the quality is pre-release. I believe this is what we want to show in happa.

[piontec]

This will be a shared work between batman and halo. Please have a look at the first draft (CR naming to be adjusted): https://www.dropbox.com/scl/fi/vo6a6x89k5vuediy91q43/Application-metadata-for-app-catalog-entries.paper?dl=0&rlkey=n1t51gtt2w4ae2exqzix2ddni.
Please comment in the doc, @giantswarm/team-batman

[piontec]

Update 2020-03-20:

• I created a spec from scratch, while there was one already created; merged them, shared with both teams, waiting for feedback

[cokiengchiara]

Spec is ready. Move to implementation.

[cokiengchiara]

@piontec do you have a ticket for implementing this story? Or should I create one?

[cokiengchiara]

@piontec will commit spec. Then create issues for implementation.