Integration tests fail after change in devfile registry

[feloy]

/kind bug

Context

This PR changes metadata in Devfiles.
#6137 (comment)

These changes break the integration tests

Acceptance criteria

See #6138 (comment)

• Spin up our own Devfile registry, that we control. Odo tests should test odo features, not that the Devfile stacks are valid/usable with odo, which should be the scope of Devfile registry tests (see: Get DEVFILE_REGISTRY value for tests, to use local Registry #6705)
• Have one or just a few tests that validate that that odo can communicate with the default Devfile registry. They should not test other odo features, but could just check that the right Devfile has been downloaded. (see: Integration tests fail after change in devfile registry  #6138 (comment))
• Remove the NGINX server used for caching Devfiles (cf. Install and use a reverse proxy for staging devfile registry #5922)
• We should not use starter projects because we might have issues there as well. Ensure that we use projects in our local code base.

See #6138 (comment) for details on ACs completion.

[rm3l]

#6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

[kadel]

There is also another thing. This change only broke our tests, and not odo itself. But next time there could be a change that breaks odo.
We should consider running our tests against https://registry.stage.devfile.io/ every change in devfile registry gets first deployed into staging registry before going to production, we should try to catch as much as we can before changes are in production. Using staging could help with that.

[rm3l]

There is also another thing. This change only broke our tests, and not odo itself. But next time there could be a change that breaks odo.

That's a good point indeed.

We should consider running our tests against registry.stage.devfile.io every change in devfile registry gets first deployed into staging registry before going to production, we should try to catch as much as we can before changes are in production. Using staging could help with that.

I think we are already using the staging registry in our tests: https://github.com/redhat-developer/odo/blob/main/tests/helper/helper_generic.go#L326-L334
This is why we caught this issue today (devfile/registry#123 was merged yesterday); I think this change has not been deployed yet to the production registry.

[kadel]

I think we are already using the staging registry in our tests: https://github.com/redhat-developer/odo/blob/main/tests/helper/helper_generic.go#L326-L334
This is why we caught this issue today (devfile/registry#123 was merged yesterday); I think this change has not been deployed yet to the production registry.

I forgot about this. That is good. So that is good that it broke our tests, it worked as expected :-D, it just came in the worst possible time it could.

[valaparthvi]

#6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

I remember we discussed some robust solution to this, but I can't recall what. Can you help me recall it? @rm3l @feloy

[feloy]

#6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

I remember we discussed some robust solution to this, but I can't recall what. Can you help me recall it? @rm3l @feloy

We discussed two possibilities:

• pass a devfile version when running odo init (are different versions stored in registry?)
• during integration tests, read the content of the devfile to get the values to assert, instead of hardcoding them in the test source code

[valaparthvi]

pass a devfile version when running odo init (are different versions stored in registry?)

I don't think this would work. There is only one version at a time. I say this by checking odo registry, if that is what you are pointing at.

odo registry output

$ odo registry
 NAME                          REGISTRY         DESCRIPTION                                 
 dotnet50                      StagingRegistry  Stack with .NET 5.0                         
 dotnet60                      StagingRegistry  Stack with .NET 6.0                         
 dotnetcore31                  StagingRegistry  Stack with .NET Core 3.1                    
 go                            StagingRegistry  Go is an open source programming languag... 
 java-maven                    StagingRegistry  Upstream Maven and OpenJDK 11               
 java-openliberty              StagingRegistry  Java application Maven-built stack using... 
 java-openliberty-gradle       StagingRegistry  Java application Gradle-built stack usin... 
 java-quarkus                  StagingRegistry  Quarkus with Java                           
 java-springboot               StagingRegistry  Spring Boot® using Java                     
 java-vertx                    StagingRegistry  Upstream Vert.x using Java                  
 java-websphereliberty         StagingRegistry  Java application Maven-built stack using... 
 java-websphereliberty-gradle  StagingRegistry  Java application Gradle-built stack usin... 
 java-wildfly                  StagingRegistry  Upstream WildFly                            
 java-wildfly-bootable-jar     StagingRegistry  Java stack with WildFly in bootable Jar ... 
 nodejs                        StagingRegistry  Stack with Node.js 16                       
 nodejs-angular                StagingRegistry  Angular is a development platform, built... 
 nodejs-nextjs                 StagingRegistry  Next.js gives you the best developer exp... 
 nodejs-nuxtjs                 StagingRegistry  Nuxt is the backbone of your Vue.js proj... 
 nodejs-react                  StagingRegistry  React is a free and open-source front-en... 
 nodejs-svelte                 StagingRegistry  Svelte is a radical new approach to buil... 
 nodejs-vue                    StagingRegistry  Vue is a JavaScript framework for buildi... 
 php-laravel                   StagingRegistry  Laravel is an open-source PHP framework,... 
 python                        StagingRegistry  Flask is a web framework, it’s a Pytho...   
 python-django                 StagingRegistry  Django is a high-level Python web framew... 

during integration tests, read the content of the devfile to get the values to assert, instead of hardcoding them in the test source code

I am inclined towards this solution, but then again, this cannot be done for all the test cases. We have been using local devfiles for most of our cases, but some tests in odo init and odo registry still call the command with devfile name to fetch it.