Applying the configuration through the Airy controller

[ljupcovangelski]

We need to be able to do airy config apply or use some other flag to apply the configuration through the airy-controller and not directly to Kubernetes.

If the controller exposes an endpoint to create/read/update/delete configuration, then we can also apply configuration for the sources and other components through the UI.

Depends on #2755 .

[ljupcovangelski]

After internal discussions, there are two ways how we can approach this issue:

1. Build a full API service as part of the airy-controller that accepts create, update, delete and list of components. Having that, we would be able to manage every component independently.
2. Build a simple endpoint that accepts the whole airy.yaml, it un-marsals it and creates the resources in order to manage the components.

As we see the first option as a more long-term option, for now we will do the second option in order to remove the dependence of the Airy CLI on Kubernetes. The controller will have no state, the airy.yaml will be saved in a configMap, from where it will also be retrieved.

We would need to be able to send the following request, to send the airy.yaml file to the controller:

curl -X POST -H "Content-Type: application/json" -d @airy.yaml http://airy.core/controller/config

Then we need to impelment this in the Airy CLI when running airy config apply

This endpoint should be made functional once the dependent authentication issue is resolved.

We will need an option to also retrieve the current airy.yaml configuration, as well as manage concurrency requests to update the configuration. We will create separate issues for these items.

[chrismatix]

To get the API proposal in the format of our docs I would propose something like this:

POST /cluster.applyState

{
    "data": "{stringified airy yaml}"
}

[ljupcovangelski]

Currently the airy.yaml file holds both the values for helm and the configuration of the Airy components. The configuration of the Airy components is under:

security:
  ...
components:
  component-type:
    component-name:
      ...

On the long run, I see the airy.yaml file holding only the helm values, as the configuration will be applied mostly from the UI, or by posting JSON from the CLI. I advise not to write the configurations of the components back to the airy.yaml file, as those are mostly security credentials.

As for the values in the airy.yaml file, if it relates only to helm, they can be easily be retrieved if one has access to the Kubernetes cluster, using:

helm get values airy-release

In that sense was thinking to make it possible to now apply the configuration using a binary transfer of a YAML file for now and at the same time through JSON. The YAML should work only temporary while we maintain the legacy config apply in the CLI.

Eventually, configuration should be applied only through JSON this will be handled

For example we would need to post this:

{
  "security": {
    "allowedOrigins": "*"
  },
  "components": {
    "sources": {
      "facebook": {
        "token1": "token",
        "token2": "token"
      },
      "google": {
        "token1": "token",
        "token2": "token"
      }
    }
  }
}

as equivalent to this:

security:
  allowedOrigins: "*"
components:
  sources:
    facebook:
      token1: token
      token2: token
    google:
      token1: token
      token2: token

As how the applying of the configuration works at the moment, it doesn't matter if we apply only one component at a time, or all at once. That is good I think for the frontend, as users would configure one component/connector at a time.

To apply only one component, the controller can be made to accept only:

{
  "components": {
    "sources": {
      "facebook": {
        "token1": "token",
        "token2": "token"
      }
    }
  }
}

and apply the configuration only for that component, without removing the configuration of the others.

I propose that we do it like this adding one more endpoint for DELETE-ing a configuration for a component, which will result in deleting the configmap and deactivating the component.

[juan-sebastian]

I would suggest to just send the yaml and set the HTTP header to application/x-yaml. We don't have any reason to go with JSON in this case. So why complicate things with stringify or binary marshal

[ljupcovangelski]

I think we are using the yaml format only because of helm. Now as this would be separated, I see the yaml only as a temporary solution for the configuration.

[chrismatix]

Then I'd suggest that we go straight for the JSON API so that we have to maintain fewer endpoints and follow our API conventions. Adding the data subfield is merely so that we can later easily pass options to that requests such as failOnOutOfDate or something like that.

[ljupcovangelski]

Already done.