devfile components does not show up on odo catalog list components with -o json flag

[elsony]

/kind bug

What versions of software are you using?

1.2.1

Operating System:
mac

Output of odo version:
odo v1.2.1 (b7b7c2f)

How did you run odo exactly?

Running the command 'odo catalog list components -o json' with experimental flag set to true

Actual behavior

eyuentp2:devfile2-registry eyuen$ odo catalog list components -o json
{
    "metadata": {
        "creationTimestamp": null
    },
    "items": null
}

It returns an empty list.

However, if I run the same command without -o json, the entries shows up correctly:

eyuentp2:devfile2-registry eyuen$ odo catalog list components 
Odo Devfile Components:
NAME                 DESCRIPTION                           REGISTRY                   SUPPORTED
maven                Upstream Maven and OpenJDK 11         DefaultDevfileRegistry     YES
nodejs               Stack with NodeJS 10                  DefaultDevfileRegistry     YES
openLiberty          Open Liberty microservice in Java     DefaultDevfileRegistry     YES
java-spring-boot     Spring Boot® using Java               DefaultDevfileRegistry     YES

Expected behavior

It should return the list of devfile components when -o json is specified.

Any logs, error output, etc?

n/a

/area devfile

[yangcao77]

Should be a quick fix. I will create a PR for this issue.
/assign

[yangcao77]

The json output format has been changed by this issue. The List will contain separate arrays for s2i items and devfile items.
This change would need IDE change to properly display the information.

The detailed example list is shown as below.

{
    "kind": "List",
    "apiVersion": "odo.dev/v1alpha1",
    "metadata": {
        "creationTimestamp": null
    },
    "s2iItems": [
        {
            "kind": "ComponentType",
            "apiVersion": "odo.dev/v1alpha1",
            "metadata": {
                "name": "dotnet",
                "namespace": "openshift",
                "creationTimestamp": null
            },
            "spec": {
                "allTags": [
                    "2.0",
                    "latest"
                ],
                "nonHiddenTags": [
                    "2.0",
                    "latest"
                ],
                "supportedTags": []
            }
        },
    ]
    "devfileItems": [
        {
            "Name": "openLiberty",
            "DisplayName": "Open Liberty",
            "Description": "Open Liberty microservice in Java",
            "Link": "/devfiles/openLiberty/devfile.yaml",
            "Support": true,
            "Registry": {
                "Name": "DefaultDevfileRegistry",
                "URL": "https://raw.githubusercontent.com/elsony/devfile-registry/master"
            }
        },
    ]
}

[dgolovin]

@yangcao77 This approach will require to fix all IDE's after new Component Type is introduced.
New component types should IMO fit into old json format, so IDE workflow would keep working with any new odo release without changing IDE's code.

Look at example below. This implementation would work with all IDEs and 'New Component' workflow would not be affected, because both "dotnet" and "openLiberty" have json structure IDEs understand already. It means if experimental flag is set to true, IDEs will start showing devfile components in 'New Component' workflow and that does not require any code modification to support new Component Types.

I would say that description, link and registry should not be required in this json output, because all what IDEs need is component_name:version used as part of odo create component_name:version command. If additional details required for component type, IMO that should be part for describe command.

{
    "kind": "List",
    "apiVersion": "odo.dev/v1alpha1",
    "metadata": {
        "creationTimestamp": null
    },
    "items": [
        {
            "kind": "ComponentType",
            "apiVersion": "odo.dev/v1alpha1",
            "metadata": {
                "type": "S2I",
                "name": "dotnet",
                "namespace": "openshift",
                "creationTimestamp": null
            },
            "spec": {
                "allTags": [
                    "2.0",
                    "latest"
                ],
                "nonHiddenTags": [
                    "2.0",
                    "latest"
                ],
                "supportedTags": []
            }
        },
        {
            "kind": "ComponentType",
            "apiVersion": "odo.dev/v1alpha1",
            "metadata": {
                "type": "devfile",
                "name": "openLiberty",
                "displayName": "Open Liberty",
                "description": "Open Liberty microservice in Java",
                "link": "/devfiles/openLiberty/devfile.yaml",
                "Registry": {
                   "Name": "DefaultDevfileRegistry",
                   "URL": "https://raw.githubusercontent.com/elsony/devfile-registry/master"
                },
                "namespace": null,
                "creationTimestamp": null
            },
            "spec": {
                "allTags": [
                    "latest"
                ],
                "nonHiddenTags": [
                    "latest"
                ],
                "supportedTags": [
                    "latest"
                ]
            }
        },
    ]
}

[dgolovin]

this issue is duplicate of #3091.

[yangcao77]

@dgolovin s2iItems items and devfileItems cannot be simply merged and put into items list. They are with two different struct type.
Also if user run odo catalog list component without json format, the output is separated into two lists. -o json is only to output the two list into json format, should not change the behavior(i.e. merge the two lists into a single one as your example)

#3264 (comment)
We had a discussion in the PR for this issue, and had an agreement with @dharmit and @kadel to show the json format as described above.

[dgolovin]

@yangcao77 @dharmit @kadel

s2iItems items and devfileItems cannot be simply merged and put into items list. They are with two different struct type.

I just did in comment above. The -o json was added for external tool to consume output, nobody is consuming it except IDE extensions/plugins. What are you about to do is going to break all of them: Eclipse plugin, Intelji plugin and VSCode Extension.
The same happened to services, in 1.2.2 release and all IDE's 'New Service' commands got broken. This should not happen for patch releases.

IDE's relay on agreement that odo patch or service releases should stay backward compatible.

[yangcao77]

I just did in comment above. The -o json was added for external tool to consume output, nobody is consuming it except IDE extensions/plugins. What are you about to do is going to break all of them: Eclipse plugin, Intelji plugin and VSCode Extension.
The same happened to services, in 1.2.2 release and all IDE's 'New Service' commands got broken. This should not happen for patch releases.

IDE's relay on agreement that odo patch or service releases should stay backward compatible.

@dgolovin
I understand your concern of the change, and we understand the change requires IDE change before making the change.
However, the metadata of s2i items and devfile items are different, IDE needs to deal with them separately any ways.
I saw in your example, you have type in the metadata to distinguish s2i items from devfile items. Does IDE already have the code in place accordingly? If so, I'm fine with your structure.
@kadel @GeekArthur @dharmit Any thoughts?

[dgolovin]

@yangcao77
CC: @jeffmaury
We don't have code yet, but having the same structure for all components will make it work for S21 and Devfile components for the first iteration. Because for example to create local component for s2i and devfile we wold ask user to select component type and then just call odo create ${selectedType} componentName. Then if we can make changes in workflow to deal with specific component types or discuss other way to deal with changes in cli options without hardcoding it on IDE side (service catalog CLI might be a good alternative to hardcoded workflow).

[yangcao77]

@dgolovin Had a discussion with @elsony and @GeekArthur . We still think having two separate lists make more sense in this case.
However, we understand that due to the structure change, s2i items cannot be properly displayed even experimental mode is disabled. We want to propose that having the new structure hiding behind experimental mode, i.e. if experimental mode is turned off, the structure of the list would still be the same as before. Is that acceptable to you?

[jeffmaury]

The question is not about experimental mode or not but rather to the evolution of odo. For IDEs and userd, odo s there in order to hide the complexity of the underlying infrastructure (Openshift vs Kube vs Docker).
So from our point of view;

• whenever there is a new cluster type (vanillay kube), it should be as much transparent as possible
• whenverver there is a new component type, same rule applies.

So Denis concern is that having Devfile components added should not cause the catalog list components to be changed.
Please note that such changes breaks compatibility between odo versions and force us to link our tooling to a specific version of odo and goes against the CLI CRD that is being pushed where the tools would be downloaded from the cluster when the tooling connects to that cluster

[elsony]

The problem is that the content of the information for the two different types of components (s2i vs devfile) are different. When it comes to IDE integration, if we try to ignore the fact that they are different and trying to handle them in the same way, we are going to provide a half-baked solution in the IDE. There are some fundamental differences between two models.

Take registry support on devfile as an example: the registry information is part of the devfile model when returned from the catalog list but there is no such notion on the s2i model. If we try to create a component and just using the component type name as-is via the IDE and disregard the registry information, you may ending up creating the wrong component that the user wants, e.g. when component with the same name are coming from two different registries, and the user wants to create the component that is not comes with the default registry (high on the priority list), you'll need to pass in the registry information to complete the task properly; therefore, the notion of registry needs to be surface it up on the IDE side to the user for devfile component.

Another example is the support of creating sample application for which the list of starter project is going to be returned as part of the catalog list. That does not exists on s2i type of component at all.

Note: Keep in mind that the devfile change is a major change in concepts for odo, not just another type of component that we support. It is different enough that when devfile function is moving out of experimental mode, most likely odo is going to have a major version change.

[dgolovin]

@elsony thank you for explanation and examples. Format above would let us discriminate components by type and provide workflow specific for component type, so we can ask registry url and any additional information required. Or we can simply show 'Not supported' message for unknown component types.

Do you plan to support both kinds (s2i and devfile) of component types when devfile support goes out of experimental?

[dgolovin]

@elsony @jeffmaury Supporting tools downloaded with CLI CRD is going to be challenging. We have to find a solution to provide IDE features for different tool versions that are not backward compatible. We are going to have multiple implementations for different versions and load them on demand based on tool version detected.

[kadel]

IDE's relay on agreement that odo patch or service releases should stay backward compatible.

Yes, patch release should stay compatible and we are trying to do that.
But what we are discussing here will get to odo 2.0.0.
We have to introduce some changes like this into the odo v2, otherwise, we won't be able to make Devfiles work properly.