Start testing Minishift with Minikube/CentOS ISO as part nightly job

[agajdosi]

This is follow-up issue for #1532.

After the addition of support for Minikube iso into integration tests, we should start testing with the Minikube iso on regular basis. For beginning I suggest to start with nightly job, which will test current master with integration tests on Minikube iso. We can later progress more if results of Minikube will be good enough.

What do you think? Will be happy for any comments and ideas around this.

[agajdosi]

As agreed on today sprint planning, we will start testing of Minikube iso on PR test runs, not on nightly test runs.

[agajdosi]

As mentioned above, on last sprint planning it was said we should start testing minikube as part of PR test and not wait for nightly being set up. There are 2 possible ways of doing so:

• change the basic.feature from being run with b2d.iso to being run with minikube.iso, this will keep the execution time nearly same as it is now, we will loose the coverage of basic.feature on b2d.iso, but as most of its content is covered also in all other features, it basically should not matter
• add one extra run of basic.feature with minikube.iso, in this case we will need to remove some other feature to keep the execution time the same

I think the 1st plan is better, WDYT? @minishift/minishift-dev

[praveenkumar]

FYI kubernetes/minikube#2177 , we will soon have, testing from minikube side.

[LalatenduMohanty]

change the basic.feature from being run with b2d.iso to being run with minikube.iso, this will keep the execution time nearly same as it is now, we will loose the coverage of basic.feature on b2d.iso, but as most of its content is covered also in all other features, it basically should not matter

We can only go in this route when we make Minikube ISO as the default ISO.

add one extra run of basic.feature with minikube.iso, in this case we will need to remove some other feature to keep the execution time the same

I think we should do this for few weeks. It is a temporary arrangement, so increase in execution test time should not hurt us badly. @minishift/minishift-dev WDYT?

[hferentschik]

So why can we not just create another job? One which is configured against Minikube. Keep everything in our setup as is until we officially make Minikube the default.

Am I missing something here?

[LalatenduMohanty]

So why can we not just create another job? One which is configured against Minikube. Keep everything in our setup as is until we officially make Minikube the default.

+1

[agajdosi]

@hferentschik @LalatenduMohanty Ok, so it seems like it is heading towards creation of parallel job for Minikube.iso.

I am adding @budhrg to conversation as this starts to be more centos-ci oriented.

[agajdosi]

Hi @budhrg, I hopefully I have progressed with this issue, if you have some time, your feedback would really helpful. There are 2 branches for this issue which can be checked: agajdosi/minishift 1568-2 and agajdosi/minishift-ci-jobs add-minikube-nightly.

In minishift-ci-jobs I have appended an ISO name to the name of nightly test job:

job-template:
  name: '{git_repo}-nightly-{iso_url}'
  description: |
         'Minishift nighly job! Managed by Jenkins Job Builder, do not edit manually! Update via https://github.com/minishift/minishift-ci-jobs.'
[...]

and then defined 2 jobs from this template:

      # Nightly build running @ midnight daily
      - '{git_repo}-nightly-{iso_url}':
          git_repo: minishift
          ci_project: '{name}'
          ci_cmd: '/bin/bash centos_ci.sh'
          iso_url: b2d
          timeout: '120m'
      - '{git_repo}-nightly-{iso_url}':
          git_repo: minishift
          ci_project: '{name}'
          ci_cmd: '/bin/bash centos_ci.sh'
          iso_url: minikube
          timeout: '120m'

For minishift repo I have added elif statement for both nightly jobs:

elif [[ "$JOB_NAME" = "minishift-nightly-b2d" ]]; then
  perform_nightly "b2d";
elif [[ "$JOB_NAME" = "minishift-nightly-minikube" ]]; then
  perform_nightly "minikube";

and updated the perform_nightly() function to accept names of ISOs as argument:

MINISHIFT_ISO_URL=$1 MINISHIFT_VM_DRIVER=kvm make integration_all

Does this seem fine to you? Would be glad for any feedback.