Tests for devfiles in odo devfile registry

[kadel]

We should have tests for each devfile in https://github.com/odo-devfiles/registry that will verify that they are valid and usable.

There should a common test flow that will be executed using each devfile in the registry.

• odo create <devfile>
• odo url create
• odo push
• verify that application is accessible in a given url
• odo push --debug
• odo debug port-forward
• verify that it is possible to make a connection to debugger
• odo delete

The tests should be executed on https://github.com/odo-devfiles/registry repository for every PR

We can start by running this on Travis

/area registry
/kind test
/priority medium

[openshift-ci-robot]

@kadel: The label(s) area/ cannot be applied, because the repository doesn't have them

In response to this:

We should have tests for each devfile in https://github.com/odo-devfiles/registry that will verify that they are valid and usable.

There should a common test flow that will be executed using each devfile in the registry.

• odo create <devfile>
• odo url create
• odo push
• verify that application is accessible in a given url
• odo push --debug
• odo debug port-forward
• verify that it is possible to make a connection to debugger
• odo delete

/area registry
/kind test
/priority medium

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kadel]

/remove-priority medium
/priority high

[girishramnani]

@amitkrout please observe that the tests are to be added in the odo-devfiles/registry

[amitkrout]

@kadel @girishramnani i started few scripting steps, then thought let me check how you see the requirement and the solution.

1. We are planning to run each devfile registry test in our each odo pr run, for example when odo pr triggers we need to clone/download https://github.com/odo-devfiles/registry and verifies it through our test script with the flow mentioned in the description.

OR

2. Something like to another CI setup like we did for https://github.com/openshift/odo-init-image.

NOTE: If we are following the type 1 then we can not guaranty that https://github.com/odo-devfiles/registry master is not broken. In type two we can atleast make sure that odo-devfiles/registry master is not broken.

Can you please confirm ?

[kadel]

This should be another CI setup independent from openshift/odo. Those tests should run for PRs in odo-devfiles/registry

The tests should use latest released odo binary.

[amitkrout]

This should be another CI setup independent from openshift/odo. Those tests should run for PRs in odo-devfiles/registry

The tests should use latest released odo binary.

@kadel For CI job configuration i need admin access to this repo https://github.com/odo-devfiles/registry. Can you please provide me the required access

[kadel]

@kadel For CI job configuration i need admin access to this repo https://github.com/odo-devfiles/registry. Can you please provide me the required access

done

[amitkrout]

Hit the issue #3476 while i was applying the test flow through the script.

[amitkrout]

Hit another issue while accessing the route - #3477

[dharmit]

Focus on Travis for now and get tests working on it. Focus on Prow later.

[mohammedzee1000]

After discussion with @amitkrout Here is what I propose:

Every devfile directory will contain a file test.sh as entry point to testing containing the odo commands to run as part of testing and a directory example, containing an example app to be used for testing in the test.sh script for eg

❯ tree
.
└── devfiles
    ├── devfile1
    │   ├── devfile.yaml
    │   ├── example
    │   └── test.sh
    └── devfile2
        ├── devfile.yaml
        ├── example
        └── test.sh

In travis script, we will

1. Setup openshift cluster
2. Fetch latest odo binary from https://mirror.openshift.com/pub/openshift-v4/clients/odo/latest/ and put it into PATH
3. for each devfile directory, pushd into it, run test.sh if it exists or fail, popd out of it

Note: to list the directories in a directory shell script, you can either

❯ cd /path/to/devfiles
❯ ls -d *
devfile1  devfile2

or

❯ pwd
some/random/location
❯ find /path/to/devfiles -maxdepth 1 -mindepth 1 -type d -printf '%f\n'
devfile2
devfile1

cc @kadel

[amitkrout]

@mohammedzee1000 Actually i was doing over engineering. This looks cool to me 👍

[amitkrout]

As per the last discussion with @kadel, steps to be followed

• Setup travis env
• Run oc cluster up
• Download latest release odo binary
• Create a shell script which will iterate over the devfile registry - https://github.com/odo-devfiles/registry/tree/master/devfiles with the mentioned e2e flow

[amitkrout]

Validation steps remaining

odo push --debug
odo debug port-forward
verify that it is possible to make a connection to debugger

Ping @anandrkskd

[anandrkskd]

• Test need to be run for specific devfile ie. java-maven, java-quarkus, nodejs.
• To check if the connection to debugger is possible or not can be done by replicating the integration test for devfile_debug_test in bash script.