test_positive_end_to_end_bulk_update (#6956)

[swadeley]

• Test the Update All Packages button on Update Packages dialogue
• Use the Host Collection view to get to it.

Edit: This test depends on the change in SatelliteQE/airgun#332

[codecov]

Codecov Report

Merging #6974 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6974   +/-   ##
=======================================
  Coverage   69.76%   69.76%           
=======================================
  Files          29       29           
  Lines        3493     3493           
=======================================
  Hits         2437     2437           
  Misses       1056     1056

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 197b77b...d6f23cc. Read the comment docs.

[mirekdlugosz]

Test itself looks OK to me, I left couple of comments for things around it.

Also, I think it depends on Airgun PR that is not yet merged? Could you add reference to said PR in initial PR message?

[swadeley]

Test itself looks OK to me, I left couple of comments for things around it.

Also, I think it depends on Airgun PR that is not yet merged? Could you add reference to said PR in initial PR message?
OK, added that now. (I did look for some kind of depends-on function like in Bugzilla but settled for the Do Not merge flag).

[swadeley]

Fixed indentation in docstring

[swadeley]

Hello

I installed Satellite 6.4 and upgraded it to Satellite 6.5. I then entered this command:

[root@cloud-qe-21 ~]# rpm -q tfm-rubygem-katello tfm-rubygem-katello-3.10.0.46-1.el7sat.noarch

I then ran test which failed as expected because packages do not get updated due to BZ#1712069:

robottelo]$ pytest -v tests/foreman/ui_airgun/test_contenthost.py -k test_positive_end_to_end_bulk_update
============================ test session starts ==================================
platform linux -- Python 3.6.6, pytest-4.0.2, py-1.8.0, pluggy-0.12.0 -- .virtualenvs/robottelo-master/bin/python3
cachedir: .pytest_cache
shared_function enabled - OFF - scope: - storage: file
rootdir: /sat6/robottelo, inifile:
plugins: services-1.3.1, forked-1.0.2, xdist-1.28.0, cov-2.7.1, mock-1.10.0
collecting ... 2019-06-04 07:36:40 - conftest - DEBUG - BZ deselect is disabled in settings

collected 17 items / 16 deselected

tests/foreman/ui_airgun/test_contenthost.py::test_positive_end_to_end_bulk_update FAILED [100%]

============================== FAILURES =============================
______________________ test_positive_end_to_end_bulk_update ________________________

session = <airgun.session.Session object at 0x7f8b0c3ecb70>
vm = <robottelo.vm.VirtualMachine object at 0x7f8b0c3ecb38>

@upgrade
@tier3
def test_positive_end_to_end_bulk_update(session, vm):
    """Create VM, set up VM as host, register it as a content host,
    read content host details, install a package ( e.g. walrus-0.71) and
    use bulk action (Update All Packages) to update the package by name
    to a later version.

    :id: d460ba30-82c7-11e9-9af5-54ee754f2151

    :expectedresults: package installation and update to a later version
    are successful.

    :BZ: 1712069

    :CaseLevel: System
    """
    hc_name = gen_string('alpha')
    description = gen_string('alpha')
    result = vm.run('yum -y install {0}'.format(FAKE_1_CUSTOM_PACKAGE))
    assert result.return_code == 0
    with session:
        # Ensure content host is searchable
        assert session.contenthost.search(vm.hostname)[0]['Name'] == vm.hostname
        # Update package using bulk action
        # use the Host Collection view to access Update Packages dialogue
        session.hostcollection.create({
            'name': hc_name,
            'unlimited_hosts': False,
            'max_hosts': 2,
            'description': description
        })
        session.hostcollection.associate_host(hc_name, vm.hostname)
        # Update the package by name
        session.hostcollection.manage_packages(
                    hc_name, content_type='Package',
                    packages=FAKE_1_CUSTOM_PACKAGE_NAME,
                    action='update_all'
        )
        # Ensure package updated to a later version
        packages = session.contenthost.search_package(vm.hostname, FAKE_2_CUSTOM_PACKAGE_NAME)

      assert packages[0]['Installed Package'] == FAKE_2_CUSTOM_PACKAGE

E AssertionError: assert 'walrus-0.71-1.noarch' == 'walrus-5.21-1.noarch'
E - walrus-0.71-1.noarch
E ? ^ ^
E + walrus-5.21-1.noarch
E ? ^ ^

tests/foreman/ui_airgun/test_contenthost.py:257: AssertionError

I then upgraded Sat6.5 to latest snap of 6.6 and entered this command:
[root@cloud-qe-21 ~]# rpm -q tfm-rubygem-katello tfm-rubygem-katello-3.12.0.rc2-11.master.el7sat.noarch

I then ran the test again:

robottelo]$ pytest --show-capture=no tests/foreman/ui_airgun/test_contenthost.py -k test_positive_end_to_end_bulk_update
============================ test session starts ==============================
platform linux -- Python 3.6.6, pytest-4.0.2, py-1.8.0, pluggy-0.12.0
shared_function enabled - OFF - scope: - storage: file
rootdir: /sat6/robottelo, inifile:
plugins: services-1.3.1, forked-1.0.2, xdist-1.28.0, cov-2.7.1, mock-1.10.0
collecting ... 2019-06-04 08:45:01 - conftest - DEBUG - BZ deselect is disabled in settings

collected 17 items / 16 deselected

tests/foreman/ui_airgun/test_contenthost.py . [100%]

================= 1 passed, 16 deselected, 2 warnings in 807.97 seconds =================

[mirekdlugosz]

All my comments are addressed. Looks good 👍

@swadeley can you confirm that these upgrades are just part of your testing environment, not actually required by the test? I mean - if you fresh-install 6.5, we can expect it to fail (due to bug), and if you fresh-install 6.6, we can expect it to pass (because bug is already fixed). Is that correct?

Because if you have to run upgrades as part of test scenario, we probably should put it somewhere else. Scenarios in tests/foreman are also run for fresh install of each new snap.

[swadeley]

@Mirzal The issue opened by @ntkathole says "Add upgrade test" but I am struggling to find evidence it is upgrade related. I will run the test against a fresh install of 6.5 and also ask SME to confirm bug is upgrade related (he was a away so could not ask before).

But in any case, the upgrade is not part of the test, that was only to reproduce the bug so I could check the test finds it. But if we find its not upgrade related, then we could remove the tag @upgrade

More news soon

[mirekdlugosz]

But if we find its not upgrade related, then we could remove the tag @upgrade

It's up to you. @upgrade means "run this test on Satellite that used to be newest GA, but was upgraded to latest snap". We kind of assume that upgraded Satellite is the same as freshly-installed Satellite, but to check this claim, we run subset of our automation on upgraded instance. @upgrade means your test will be in that subset.

[swadeley]

Hello

I have decided to add an extra step, to make sure package event can take place if there is slower system or network traffic. So blocking the merge for now.

[swadeley]

But if we find its not upgrade related, then we could remove the tag @upgrade

It's up to you. @upgrade means "run this test on Satellite that used to be newest GA, but was upgraded to latest snap". We kind of assume that upgraded Satellite is the same as freshly-installed Satellite, but to check this claim, we run subset of our automation on upgraded instance. @upgrade means your test will be in that subset.

Thanks for that.

I tracked it down, the source of the word "upgrade" is from an e-mail with subject "Unable to apply package update after upgrade from 6.4 to 6.5". But the SME, the developer, thinks the issue is not related to the fact that the Satellite was upgraded. To confirm I installed a new 6.5 system late yesterday and the fault was present, so I will remove that tag.

[ntkathole]

I tracked it down, the source of the word "upgrade" is from an e-mail with subject "Unable to apply package update after upgrade from 6.4 to 6.5". But the SME, the developer, thinks the issue is not related to the fact that the Satellite was upgraded. To confirm I installed a new 6.5 system late yesterday and the fault was present, so I will remove that tag.

Hey @swadeley please keep upgrade decorator. This will cover both the scenarios of fresh 6.5 install and upgraded 6.5.

[swadeley]

@ntkathole OK, keep upgrade decorator

[swadeley]

Hello

I added wait_for_tasks with 10 sec poll time because my testing suggests we need to allow at least a minute for the client to complete the installation and do the upload profile. I also do not want to overload the server with searches that will not find anything.

I can squash commits later or the merger can do it. New test results coming.

Thank you

[swadeley]

robottelo]$ pytest --show-capture=no tests/foreman/ui_airgun/test_contenthost.py::test_positive_end_to_end_bulk_update

========================== test session starts ======================================
platform linux -- Python 3.6.6, pytest-4.0.2, py-1.8.0, pluggy-0.12.0
shared_function enabled - OFF - scope: - storage: file
rootdir: /sat6/robottelo, inifile:
plugins: services-1.3.1, forked-1.0.2, xdist-1.28.0, cov-2.7.1, mock-1.10.0
collecting ... 2019-06-07 09:37:20 - conftest - DEBUG - BZ deselect is disabled in settings

collected 1 item

tests/foreman/ui_airgun/test_contenthost.py . [100%]

==================== 1 passed, 2 warnings in 704.79 seconds =====================

[JacobCallahan]

Just the one change. Everything else looks good to me!

[swadeley]

robottelo]$ pytest --show-capture=no tests/foreman/ui_airgun/test_contenthost.py::test_positive_end_to_end_bulk_update
========================== test session starts ===========================
platform linux -- Python 3.6.6, pytest-4.0.2, py-1.8.0, pluggy-0.12.0
shared_function enabled - OFF - scope: - storage: file
rootdir: /sat6/robottelo, inifile:
plugins: services-1.3.1, forked-1.0.2, xdist-1.28.0, cov-2.7.1, mock-1.10.4
collecting ... 2019-06-12 20:47:10 - conftest - DEBUG - BZ deselect is disabled in settings

collected 1 item

tests/foreman/ui_airgun/test_contenthost.py . [100%]

=================== 1 passed, 5 warnings in 745.71 seconds ====================

[swadeley]

Hello, due to merge conflicts, I did git pull upstream master --rebase and then a force push.

[JacobCallahan]

ACK