Design packaging API for configurable UI components

[absoludity]

Description:

Moving discussion from Antonio's comment on 4346. Since inception, we've planned to provide, eventually, a way for packaging plugins to provide custom UI fields for forms such as the create/update installed package, or create/update app repository.

Antonio said:

Mid-term, we should consider the plugin to provide those options to the frontend somehow. I'm kind of against providing UI components (like the "custom component" logic we already have) since it would be coupled to our current client's tech stack. I would rather go with a simple json schema manifest, declaring the properties a plugin would accept (including whether they are required or not), plus a mapping to which API request body it refers to.

Like:

// carvel plugin schema
{
  "type": "object",
  "properties": {
    "serviceAccountName": { "type": "string", binding: "v1alpha1GetServiceAccountNamesResponse/serviceaccountNames/[0]"},
    "versionConstraint": { "type": "string", "enum": ["default", "patch", "minor", "major"] },
  },
  "required": ["serviceAccount"]
}

I think everyone agrees on providing an implementation-agnostic json schema like this (as opposed to embedding react components or something that may work for Kubeapps now, but isn't helpful to other potential clients as Antonio mentions). We just need to ensure features such as dependentRequired meet any needs we have for conditional form fields, as well as support in react-jsonschema for our Kubeapps client etc.

[castelblanque]

I've been giving this a thought, that I'm dumping here. Maybe it is material to discuss in this document? 😃

TLDR

Each plugin can expose a new endpoint named GetSchemaMetadata with all schemas that have custom fields for a certain operation.
Schema is basically a JSONSchema with minor additions, as Antonio pointed out here.
React invokes that endpoint per plugin, and renders custom form controls where needed, based on the schema requirements.

Ubiquitous language

• Operation: Name of the plugin endpoint being invoked (e.g. KappControllerPackagesService.CreateInstalledPackage)
• Schema property: Each of the fields of an object that require special rules or restrictions.
• Context: Data available to be used by any schema property (e.g. context.NamespaceServiceAccounts
• Binding: Data in the context that can be used for a schema field (e.g. all service accounts in a ns eligible for the reconciliation)

Features

• Leaves current APIs untouched. We are adding a metadata endpoint, which means that the current APIs can continue functioning without it.
• It is not strictly related to UI, it only specifies a strict schema for API objects.
• If other clients make use of any API posting data with incorrect schema (e.g. missing serviceAccountName), regular backend validation should continue handling it.
• Fully leverages JSONSchema

Drawbacks

• No layout information possible, as it would be related to UX and not strictly a schema.

Let's see a detailed example step by step.

Creating Carvel package install

1. Kapp plugin will implement a new endpoint called GetSchemaMetadata, being full fixed path:

   kubeappsapis.plugins.kapp_controller.packages.v1alpha1.MetadataService.GetSchemaMetadata

2. This endpoint will return something like:

   {
       "operations": {
           "KappControllerPackagesService.CreateInstalledPackage": {
               "type": "object",
   
               "properties": {
                   "reconciliationOptions.serviceAccountName": {
                       "type": "string",
                       "description": "Service account",
                       "binding": "context.NamespaceServiceAccounts"
                   },
                   "versionConstraint": {
                       "type": "string",
                       "description": "Version constraint",
                       "enum": ["default", "patch", "minor", "major"]
                   }
               },
   
               "required": ["reconciliationOptions.serviceAccountName"]
           },
           "KappControllerRepositoriesService.AddPackageRepository": {
               ...
           }
       }
   }

3. Before rendering the "Deploy package" form for the user, UI will:

   • Invoke first (or maybe have it previously in store?) plugin's GetSchemaMetadata to get the schema.
   • Fill in the context with the available data, like list of service accounts.

4. React form controls will be rendered for each item under operations.properties. In this case, two controls (reconciliationOptions.serviceAccountName and versionConstraint), one of them making use of the context.

5. If a schema item has the property binding, the content will be evaluated to get the possible values. In this case, context.NamespaceServiceAccounts will be retrieved from the context object, and its values will fill the dropdown. For this purpose, I considered projects like Binding Schema too complex.

6. The key of each property item represents the target field in the request object that will be submitted by the form.
   So if React rendered a dropdown identified as reconciliationOptions.serviceAccountName, when submitting the form and prior to invoking CreateInstalledPackage, the value of that form control will be set in the object reconciliationOptions of the request, under the field serviceAccountName. This is something straight forward to do in Javascript.

7. Backend endpoint should receive the data meeting the required schema.

Another example for Helm's AddPackageRepository:

{
    "operations": {
        "HelmRepositoriesService.AddPackageRepository": {
            "type": "object",

            "properties": {
                "customDetail.filterRule": {
                    "type": "object",
                    "description": "Filter rule",
                    "properties": {
                        "jq": {
                            "type": "string",
                            "description": "Whether there is view state information to use"
                        },
                        "variables": {
                            "type": "object",
                            "additionalProperties": {"type": "string"}
                        }
                    }
                },
                "customDetail.ociRepositories": {
                    "type": "array",
                    "description": "OCI repositories",
                    "items": {
                        "type": "string"
                    }
                },
                "customDetail.dockerRegistrySecrets": {
                    "type": "array",
                    "description": "Docker registry secrets",
                    "items": {
                        "type": "string"
                    },
                    "binding": "context.NamespaceSecrets"
                }
            }
        }
    }
}

I haven't added too much information about the "why" of some stuff here, but this is just a proposal, food for discussion.

[absoludity]

Thanks for taking a look at this Rafa! It'll be great to finally start working on a solution to this problem.

A couple of questions:

1. (perhaps it's an implementation detail) if we already have the above information (data name, type, description etc.) in the schema or proto, wouldn't we need less info to define which fields to be used or bound? (maybe not to the client - easiest if it gets everything required to render, but perhaps wherever in the backend the schema is defined)
2. It looks like you're planning that these endpoints return only the extra fields that aren't part of the current default UX? That might work well if we consistently display the extra fields after the standard ones, in the client. But if the endpoint wants to provide client schema info including where these fields should sit related to the default fields, it could be trickier. Not sure how we could cater for that. Maybe, if a schema is populated for the plugin/rpc, then it should include all the fields?

I've been giving this a thought, that I'm dumping here. Maybe it is material to discuss in this document? smiley

Personally, I'd create a separate document: I know this is related to the new API, but that existing doc is hard to navigate now - it may be best to leave that doc as the original packaging/repositories/resources APIs, and suggest new work in separate docs. See what you think, fine either way.

[ppbaena]

+1 to create a separate document to discuss it.

[castelblanque]

Thanks for the answer Michael!

if we already have the above information (data name, type, description etc.) in the schema or proto...

Please notice that in my examples I don't mention where the definitions come from, just the end result returned by the endpoint. There are three pieces of data (per field) that we don't have in the .proto file.

1. Target field in the posted request
2. Short label for UI (i.e. description)
3. Bounded data for source (optional)

Maybe developers of a certain plugin should only define somewhere those three things and getting the rest from the proto definition. Whether that "somewhere" is a custom piece of code, or proto custom options? That is more for a deeper spec phase, no?

if the endpoint wants to provide client schema info including where these fields should sit related to the default fields

I've worked with many CMSs, and been before in scenarios where just some customization wants to be made at certain points of the layout. In those cases, there are many approaches possible, but I see two as the more feasible:

• Define specific slots in the layout where plugins can add custom fields. E.g. one slot at the top, one in the middle somewhere and one at the bottom. Not very flexible IMHO, as maybe a plugin wants a specific field to be right after another field that isn't before one of the slots.
• UI assigns weights to all fields. E.g. all fields for CreateInstalledPackage are assigned weight numbers in order like 100, 200, 300, etc. Then plugins can define their fields with a specific weight like 201, so that it can be inserted by the UI when rendering right after the field with weight 200. Only thing we would need is an integer when defining the custom field, which is pretty generic.

Otherwise we might need to return the specification for all fields (including the default ones), and in case of repeating (or diverting from default if a field is modified) it might be error prone?

Anyway, I will add the proposal to a new design document as starting point for discussion with all stakeholders.

[absoludity]

Sounds great. Thanks Rafa!

[dlaloue-vmware]

I have been involved in several frameworks for forms and UIs in previous projects, and a static schema-based approach never fits the bill and is consistently a work in progress.

My preference would be to just open the content to plugins to do whatever they need (hopefully following best practices regarding styling etc...).

if we go with a schema approach, there are a few common issues: conditional fields, cross-field dependencies, bindings.
The Replicated.com product has a schema-driven UI for helm values, and they use go-templates as a way to provide some control.

A solution that we have implemented in a previous project with Greg is similar to schema-driven but dynamically. That is instead of returning a schema just once, the UI interacts with a backend.
The backend will be called to return the initial list of fields (could be in the form of a schema as well), and those fields may have "triggers" that indicates that the fields must be refreshed (e.g. a dropdown value changed).
On a trigger, the UI will call the backend to refresh the schema, and thus render the new schema.

For simple custom UIs, this solution is basically the same as the schema-driven solution proposed (schema is fetched only once).
For more complex custom UIs, there is more work on the plugin's side, but it gives more flexibility to plugins without having to invent and define a very complex schema.

[dlaloue-vmware]

otherwise, have you looked at this: https://jsonforms.io/ ?

[absoludity]

otherwise, have you looked at this: https://jsonforms.io/ ?

Thanks for sharing your previous experience Dimitri, really helpful to hear from someone who's been through that process a few times. Definitely looks like a field that people have investigated pretty thoroughly - I immediately like the separate schema and ui-schema of jsonforms. Look forward to hearing what you find with your investigation Rafa.

[antgamdia]

FTR: we did an initial implementation of our basic form using jsonforms: #3526 (comment)

[castelblanque]

I have been involved in several frameworks for forms and UIs in previous projects, and a static schema-based approach never fits the bill and is consistently a work in progress.

We don't need a fully fledged CMS or UI framework. Initially it would suffice with customizing existing forms by changing, adding and hiding fields. This is, customizing the core functionality of Kubeapps. In a later phase we could tackle extending the UI with more bells and whistles.
I have worked and developed stuff for CMSs like Adobe AEM, Bloomreach CMS, SAP Commerce WCMS or Drupal, and we just need a very little fraction of what they do, specially headless CMSs, but we can mimic some approaches like separating the definition ("what" is going to be shown) from the visualization ("how" is going to be shown).
In our case, backend plugins would need to provide both.
There is no silver bullet, best solution is the one that fits the needs 😃

there are a few common issues: conditional fields, cross-field dependencies, bindings.

JSON schema seems a good fit for our case in the sense that could cover many of those situations.

That is instead of returning a schema just once, the UI interacts with a backend.

I'm not a big fan of having backend plugins tightly coupled with the UI. For now we only need some changes to existing forms, and depending data will not be that dynamic (fixed enums, service accounts, secrets?). It should be enough with a plugin exposing the incremental schema at one point, and the UI details on another one.
For example: Adding a new repo.

1. UI gets the schemas for all three plugins (Helm, Kapp, Flux)
2. User chooses Helm option
3. UI overlays the base form schema (core part) with the plugin incremental schema
4. UI renders the resulting form

The above is just an example to show the idea, but it could be that we have an initial, complete huge form with conditions e.g. depending on the field "type" (Helm, Kapp or Flux).

otherwise, have you looked at this: https://jsonforms.io/ ?

Yep, I've been investigating it together with react-jsonschema-form.

[castelblanque]

Design document available here.
Please let's discuss and reshape as needed.

[absoludity]

Design document available here. Please let's discuss and reshape as needed.

I've requested access :P (My vmware google account apparently doesn't have access)

[castelblanque]

I've requested access

Document has now public permissions for commenting 😄

[ppbaena]

Anything pending to close this issue?

[castelblanque]

Yes, saw the issue still opened this morning.
We discussed in a meeting to add to the repo an .md file with an overview of the ideas in the design. It would link to the discussion document in GDrive too.
I was waiting to include that file in order to close the issue. Or I can just add it independently if you want.