Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor bootstrap and run commands #109

Merged
merged 1 commit into from
May 9, 2019

Conversation

sathnaga
Copy link
Collaborator

  1. Let's avoid bootstrapping kvm tests in powervm lpar
    and if asked explicilty.

  2. Adds a wrapper for running commands and exit incase
    of failure.

Signed-off-by: Satheesh Rajendran sathnaga@linux.vnet.ibm.com

@sathnaga
Copy link
Collaborator Author

  1. PowerNV platform with disable-kvm
# python avocado-setup.py --bootstrap --disable-kvm
07:22:29 INFO    : Check for environment
07:22:38 INFO    : install packages via pip interface
07:22:41 INFO    : Creating Avocado Config
07:22:44 INFO    : Bootstrapping Avocado
07:22:49 INFO    : Installing repo: /home/sath/wrapper/avocado
07:23:06 INFO    : Updating the repo: avocado-misc-tests in /home/sath/wrapper/tests/avocado-misc-tests
07:23:08 WARNING : Avocado varianter_yaml_to_mux plugin not installed
07:23:08 INFO    : Installing optional plugin: varianter_yaml_to_mux
07:23:09 INFO    : Removing temporary mux dir
  1. PowerNV platform with disable_kvm and want to run guest tests, so it overrides disable_kvm
python avocado-setup.py --bootstrap --disable-kvm --run-suite guest_cpu
07:23:46 WARNING : Overriding user setting and enabling kvm bootstrap as guest tests are requested
07:23:46 INFO    : Check for environment
07:23:46 INFO    : Creating temporary mux dir
07:23:55 INFO    : install packages via pip interface
07:23:58 INFO    : Creating Avocado Config
07:24:02 INFO    : Bootstrapping Avocado
07:24:02 INFO    : Updating the repo: avocado in /home/sath/wrapper/avocado
07:24:02 INFO    : Installing repo: /home/sath/wrapper/avocado
07:24:20 INFO    : Installing repo: /home/sath/wrapper/avocado-vt
07:24:26 INFO    : Bootstrapping vt libvirt
07:24:52 INFO    : Bootstrapping vt qemu
07:24:57 INFO    : Updating the repo: avocado-misc-tests in /home/sath/wrapper/tests/avocado-misc-tests
07:24:58 INFO    : Downloading the guest os image
07:25:51 WARNING : Avocado varianter_yaml_to_mux plugin not installed
07:25:51 INFO    : Installing optional plugin: varianter_yaml_to_mux
07:25:51 INFO    : Summary of test results can be found below:
TestSuite                                                                   TestRun    Summary             
 
guest_cpu                                                                   Cant_Run   Config file not present

@sathnaga
Copy link
Collaborator Author

  1. Auto detecting need for bootstrap while required packages are not installed
# pip uninstall avocado_plugins_vt
Uninstalling avocado-plugins-vt-69.0:
  /usr/lib/python2.7/site-packages/avocado_plugins_vt-69.0-py2.7.egg
Proceed (y/n)? y
  Successfully uninstalled avocado-plugins-vt-69.0
[root@9 wrapper]# pip uninstall avocado_plugins_vt
Cannot uninstall requirement avocado-plugins-vt, not installed
[root@9 wrapper]# python avocado-setup.py --disable-kvm --run-suite guest_cpu
07:31:27 WARNING : Overriding user setting and enabling kvm bootstrap as guest tests are requested
07:31:27 INFO    : Check for environment
07:31:27 INFO    : Creating temporary mux dir
07:31:35 INFO    : install packages via pip interface
07:31:39 WARNING : Avocado vt plugin not installed
07:31:39 WARNING : Avocado vt-list plugin not installed
07:31:40 WARNING : Avocado vt-bootstrap plugin not installed
07:31:40 INFO    : Creating Avocado Config
07:31:44 INFO    : Bootstrapping Avocado
07:31:44 INFO    : Updating the repo: avocado in /home/sath/wrapper/avocado
07:31:44 INFO    : Installing repo: /home/sath/wrapper/avocado
07:31:57 INFO    : Updating the repo: avocado-vt in /home/sath/wrapper/avocado-vt
07:31:57 INFO    : Installing repo: /home/sath/wrapper/avocado-vt
07:32:04 INFO    : Bootstrapping vt libvirt
07:32:08 INFO    : Bootstrapping vt qemu
07:32:12 INFO    : Updating the repo: avocado-misc-tests in /home/sath/wrapper/tests/avocado-misc-tests
07:32:13 INFO    : Downloading the guest os image
07:32:38 WARNING : Avocado varianter_yaml_to_mux plugin not installed
07:32:38 INFO    : Installing optional plugin: varianter_yaml_to_mux
07:32:39 INFO    : Summary of test results can be found below:
TestSuite                                                                   TestRun    Summary             
 
guest_cpu                                                                   Cant_Run   Config file not present

07:32:39 INFO    : Removing temporary mux dir

@harish-24
Copy link
Collaborator

Needs a args.run_suite check before "guest_" in args.run_suite:

Traceback (most recent call last):
  File "avocado-setup.py", line 630, in <module>
    if "guest_" in args.run_suite:
TypeError: argument of type 'NoneType' is not iterable

@harish-24
Copy link
Collaborator

00:31:33 ERROR   : avocado command not installed or not found in path which: no avocado in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)

@harish-24
Copy link
Collaborator

Tested after removing existing avocado, works well.

@narasimhan-v
Copy link
Collaborator

PowerVM:

# python avocado-setup.py --bootstrap --disable-kvm
16:23:00 INFO    : Check for environment
16:23:00 INFO    : Creating temporary mux dir
16:23:06 INFO    : install packages via pip interface
16:23:08 INFO    : Creating Avocado Config
16:23:10 INFO    : Bootstrapping Avocado
16:23:10 INFO    : Updating the repo: avocado in /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
16:23:10 INFO    : Installing repo: /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
16:23:19 INFO    : Updating the repo: avocado-misc-tests in /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/tests/avocado-misc-tests
16:23:20 WARNING : Avocado varianter_yaml_to_mux plugin not installed
16:23:20 INFO    : Installing optional plugin: varianter_yaml_to_mux
16:23:20 INFO    : Removing temporary mux dir
# python avocado-setup.py --bootstrap
16:17:58 INFO    : Check for environment
16:18:05 INFO    : install packages via pip interface
16:18:06 INFO    : Creating Avocado Config
16:18:08 INFO    : Bootstrapping Avocado
16:18:11 INFO    : Installing repo: /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
16:18:23 WARNING : Avocado varianter_yaml_to_mux plugin not installed
16:18:23 INFO    : Installing optional plugin: varianter_yaml_to_mux
16:18:23 INFO    : Removing temporary mux dir

@narasimhan-v
Copy link
Collaborator

PowerNV:

# python avocado-setup.py --bootstrap --disable-kvm
16:20:41 INFO    : Check for environment
16:20:41 INFO    : Creating temporary mux dir
16:20:49 ERROR   : Please install following dependancy packages qemu-kvm
# python avocado-setup.py --bootstrap
16:24:39 INFO    : Check for environment
16:24:48 ERROR   : Please install following dependancy packages qemu-kvm

@narasimhan-v
Copy link
Collaborator

narasimhan-v commented May 8, 2019

@sathnaga PowerNV package list needs handling when --disable-kvm is opted for

import sys
import argparse
import ConfigParser
import binascii
from shutil import copyfile
from lib.logger import logger_init
from lib.helper import runcmd
from lib import helper
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can import helper and use the same to invoke as helper.runcmd()

lib/helper.py Outdated
:return: Status and output of the command
"""
if info_str:
logger.info("%s", info_str)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logger.info(info_str)

lib/helper.py Outdated
if info_str:
logger.info("%s", info_str)
if debug_str:
logger.debug("%s", debug_str)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

lib/helper.py Outdated
if err_str:
logger.error("%s %s", err_str, output)
sys.exit(1)
logger.debug("%s", output)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

lib/helper.py Outdated
return dist


def get_dist_ver():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_dist() and get_dist_ver() can be merged to a single method.

@sathnaga sathnaga force-pushed the kvm_bootstrap branch 2 times, most recently from cca6a69 to 1b0e428 Compare May 8, 2019 14:22
@sathnaga
Copy link
Collaborator Author

sathnaga commented May 8, 2019

@harish-24 @balamuruhans @narasimhan-v @PraveenPenguin needs your quick check and ack...lets get this in and do further improvements...

@sathnaga
Copy link
Collaborator Author

sathnaga commented May 8, 2019

@sles15 and rhel8.0 lpar would be more helpful, rest other envs aswell welcome...

@sathnaga
Copy link
Collaborator Author

sathnaga commented May 8, 2019

tested on fedora powernv
note: use local repos in env.conf

# python avocado-setup.py --bootstrap --disable-kvm
10:34:16 INFO    : Check for environment
10:34:16 INFO    : Creating temporary mux dir
10:34:20 INFO    : install packages via pip interface
10:34:23 INFO    : Creating Avocado Config
10:34:27 INFO    : Bootstrapping Avocado
10:34:31 INFO    : Installing repo: /home/sath/kvm_bootstrap/avocado
10:34:50 WARNING : Avocado html plugin not installed
10:34:50 INFO    : Installing optional plugin: html
10:34:51 WARNING : Avocado varianter_yaml_to_mux plugin not installed
10:34:51 INFO    : Installing optional plugin: varianter_yaml_to_mux
10:34:52 INFO    : Removing temporary mux dir
python avocado-setup.py --bootstrap
10:38:28 INFO    : Check for environment
10:38:37 INFO    : install packages via pip interface
10:38:40 INFO    : Creating Avocado Config
10:38:43 INFO    : Bootstrapping Avocado
10:38:47 INFO    : Installing repo: /home/sath/kvm_bootstrap/avocado
10:39:34 INFO    : Installing repo: /home/sath/kvm_bootstrap/avocado-vt
10:39:46 INFO    : Bootstrapping vt libvirt
10:40:15 INFO    : Bootstrapping vt qemu
10:40:21 INFO    : Updating the repo: avocado-misc-tests in /home/sath/kvm_bootstrap/tests/avocado-misc-tests
10:40:22 INFO    : Downloading the guest os image
10:41:15 WARNING : Avocado varianter_yaml_to_mux plugin not installed
10:41:15 INFO    : Installing optional plugin: varianter_yaml_to_mux
10:41:16 INFO    : Removing temporary mux dir

unit testing

Type "help", "copyright", "credits" or "license" for more information.
>>> from lib import helper
>>> helper.get_machine_type()
'NV'
>>> helper.get_dist()
('fedora', '28')
>>> 

Copy link
Contributor

@balamuruhans balamuruhans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harish-24
Copy link
Collaborator

harish-24 commented May 8, 2019

SLES15SP1:

# python avocado-setup.py 
21:45:31 INFO    : Check for environment
21:45:31 INFO    : Creating temporary mux dir
21:45:39 INFO    : install packages via pip interface
21:45:43 INFO    : Creating Avocado Config
21:45:46 INFO    : Bootstrapping Avocado
21:45:46 INFO    : Updating the repo: avocado in /home/harish/tests/avocado
21:45:47 INFO    : Installing repo: /home/harish/tests/avocado
21:45:54 WARNING : Avocado html plugin not installed
21:45:54 INFO    : Installing optional plugin: html
21:45:56 WARNING : Avocado varianter_yaml_to_mux plugin not installed
21:45:56 INFO    : Installing optional plugin: varianter_yaml_to_mux
21:45:57 INFO    : Removing temporary mux dir

RHEL8

# python avocado-setup.py 
11:00:13 INFO    : Check for environment
11:00:17 INFO    : install packages via pip interface
11:02:01 INFO    : Creating Avocado Config
11:02:01 INFO    : Bootstrapping Avocado
11:04:31 INFO    : Installing repo: /root/tests/avocado
11:04:48 WARNING : Avocado html plugin not installed
11:04:48 INFO    : Installing optional plugin: html
11:04:53 WARNING : Avocado varianter_yaml_to_mux plugin not installed
11:04:53 INFO    : Installing optional plugin: varianter_yaml_to_mux
11:04:55 INFO    : Removing temporary mux dir

Both on PowerVM

@narasimhan-v
Copy link
Collaborator

SLES 12 SP3 on PowerVM

# python avocado-setup.py --bootstrap
01:04:57 INFO    : Check for environment
01:04:57 INFO    : Creating temporary mux dir
01:05:02 INFO    : install packages via pip interface
01:05:03 INFO    : Creating Avocado Config
01:05:05 INFO    : Bootstrapping Avocado
01:05:05 INFO    : Updating the repo: avocado in /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
01:05:05 INFO    : Installing repo: /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
01:05:14 INFO    : Updating the repo: avocado-misc-tests in /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/tests/avocado-misc-tests
01:05:15 WARNING : Avocado varianter_yaml_to_mux plugin not installed
01:05:15 INFO    : Installing optional plugin: varianter_yaml_to_mux
01:05:15 INFO    : Removing temporary mux dir
# python avocado-setup.py --bootstrap --disable-kvm
01:05:25 INFO    : Check for environment
01:05:25 INFO    : Creating temporary mux dir
01:05:29 INFO    : install packages via pip interface
01:05:31 INFO    : Creating Avocado Config
01:05:33 INFO    : Bootstrapping Avocado
01:05:33 INFO    : Updating the repo: avocado in /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
01:05:33 INFO    : Installing repo: /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
01:05:42 INFO    : Updating the repo: avocado-misc-tests in /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/tests/avocado-misc-tests
01:05:43 WARNING : Avocado varianter_yaml_to_mux plugin not installed
01:05:43 INFO    : Installing optional plugin: varianter_yaml_to_mux
01:05:43 INFO    : Removing temporary mux dir
linux-u7ta:~/avocado-fvt-wrapper_test/avocado-fvt-wrapper #

@narasimhan-v
Copy link
Collaborator

RHEL 8 PowerNV:

# python avocado-setup.py --bootstrap
01:07:00 INFO    : Check for environment
01:07:09 ERROR   : Please install following dependancy packages qemu-kvm
# python avocado-setup.py --bootstrap --disable-kvm
01:07:18 INFO    : Check for environment
01:07:21 INFO    : install packages via pip interface
01:07:26 INFO    : Creating Avocado Config
01:07:26 INFO    : Bootstrapping Avocado
01:07:28 INFO    : Installing repo: /root/avocado-fvt-wrapper_test/avocado-fvt-wrapper/avocado
01:08:26 WARNING : Avocado html plugin not installed
01:08:26 INFO    : Installing optional plugin: html
01:09:40 WARNING : Avocado varianter_yaml_to_mux plugin not installed
01:09:40 INFO    : Installing optional plugin: varianter_yaml_to_mux
01:09:40 INFO    : Removing temporary mux dir

avocado-setup.py Outdated
logger.error("%s %s", err_str, error)
sys.exit(1)
helper.runcmd(cmd,
debug_str="Cloning the repo: %s" % repo_name,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sathnaga why is this debug_str and not info_str ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it could be Cloning the repo: %s in %s % (repo_name, repo_path), similar to the updating section.

Copy link
Collaborator Author

@sathnaga sathnaga May 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@narasimhan-v makes sense making this change,

git diff
diff --git a/avocado-setup.py b/avocado-setup.py
index 33d21ee..aa887f0 100644
--- a/avocado-setup.py
+++ b/avocado-setup.py
@@ -204,7 +204,7 @@ def install_repo(path, name):
    :param name: name of the repository
    """
    cmd = "cd %s;make requirements;make requirements-selftests;python setup.py install" % path
-    helper.runcmd(cmd, info_str="Installing repo: %s" % path,
+    helper.runcmd(cmd, info_str="Installing %s from %s" % (name, path),
                  err_str="Failed to install %s repository:" % name)


@@ -225,7 +225,7 @@ def get_repo(repo, basepath, install=False):
    else:
        cmd = "cd %s;git clone %s %s" % (basepath, repo, repo_name)
        helper.runcmd(cmd,
-                      debug_str="Cloning the repo: %s" % repo_name,
+                      info_str="Cloning the repo: %s in %s" % (repo_name, repo_path),
                      err_str="Failed to clone %s repository:" % repo_name)
    if install:
        install_repo(repo_path, repo_name)

1. Let's avoid bootstrapping kvm tests in powervm lpar
and if asked explicilty.

2. Adds a wrapper for running commands and exit incase
of failure.

3. Added independant package list based on environment,
that way we can avoid unwanted package install.

Signed-off-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>
@sathnaga
Copy link
Collaborator Author

sathnaga commented May 9, 2019

Thanks all, merging now...

@sathnaga sathnaga merged commit 6519379 into lop-devops:master May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants