Arange tests in tiers

[mkluson]

Rearanging the test plans to follow tiers structure like this:

• tier0 = tests running on source system
• tier1 = test running on target system
• tier2 = test of other components in convert workflow

[fernflower]

I have a couple of questions\remarks:

1. How do I run single tests in your approach? Like I want to run just a NetworkManager test, how do I do it?
2. With tests from tierN running on 1 guest this causes the need for us to track what is actually happening in an SST test, if there are any changes from the last version and if they are destructive (meaning the test can't be run in a tierN anymore). Is the speedup really worth the management complexity it brings? I really loved the "I don't care what's inside" approach to be honest :(
3. How do we mark destructive tests (the one that should be run 1 test per vm)? By not adding them to tiers?

[kokesak]

Just few things:

• those "env_from" files are there for some reason?
• I don't like the specification of those tiers, I would swap tier1 and tier0 as those these tests in tier1 are more "basic" and should be run more often. Also some tests in tier0 already do the full conversion (eg. "changed_yum_conf"). However that's just my opinion.

Otherwise I like splitting tests into tiers))

[kokesak]

We need to add the preparation role that adds custom repos on the target machine

[kokesak]

Now this plans is only for Centos8, we should not run this one as it's oracle specific

[kokesak]

Bear in mind that these tests:

• "resolve-dependency" from "rhsm" conversion folder
• "one-kernel-scenarion" from "custom-repos" conversion folder

They need some extra preparation steps. Maybe we could move them to the tier0?

[packit-as-a-service]

Based on your Packit configuration the settings of the @oamg/convert2rhel Copr project would need to be updated as follows:

field	old value	new value
chroots	['epel-6-x86_64', 'epel-7-x86_64', 'epel-8-x86_64', 'oraclelinux-7-x86_64', 'oraclelinux-8-x86_64']	['epel-6-x86_64', 'epel-8-x86_64', 'epel-7-x86_64']

Packit was unable to update the settings above as it is missing admin permissions on the @oamg/convert2rhel Copr project.

To fix this you can do one of the following:

• Grant Packit admin permissions on the @oamg/convert2rhel Copr project on the permissions page.
• Change the above Copr project settings manually on the settings page to match the Packit configuration.
• Update the Packit configuration to match the Copr project settings.

Please retrigger the build, once the issue above is fixed.

[packit-as-a-service]

Based on your Packit configuration the settings of the @oamg/convert2rhel Copr project would need to be updated as follows:

field	old value	new value
chroots	['epel-6-x86_64', 'epel-7-x86_64', 'epel-8-x86_64', 'oraclelinux-7-x86_64', 'oraclelinux-8-x86_64']	['epel-6-x86_64', 'epel-8-x86_64', 'epel-7-x86_64']

Packit was unable to update the settings above as it is missing admin permissions on the @oamg/convert2rhel Copr project.

To fix this you can do one of the following:

• Grant Packit admin permissions on the @oamg/convert2rhel Copr project on the permissions page.
• Change the above Copr project settings manually on the settings page to match the Packit configuration.
• Update the Packit configuration to match the Copr project settings.

Please retrigger the build, once the issue above is fixed.

[bocekm]

The tests are failing on pytest conftest.py:

Warning: Permanently added '10.29.163.132' (ECDSA) to the list of known hosts.

ImportError while loading conftest '/var/ARTIFACTS/work-changed_yum_confqgGEXm/plans/tier0/changed_yum_conf/discover/default/tests/tests/integration/conftest.py'.
../conftest.py:26: in <module>
    logging.basicConfig(level="DEBUG" if env.str("DEBUG") else "INFO", stream=sys.stderr)
/usr/local/lib/python3.6/site-packages/envparse.py:35: in method
    return self.__call__(var, cast=cast, **kwargs)
/usr/local/lib/python3.6/site-packages/envparse.py:99: in __call__
    raise ConfigurationError(error_msg)
E   envparse.ConfigurationError: Environment variable 'DEBUG' not set.

It seems the env.str("DEBUG") raises an Exception when the requested env var is not set, which is not what we need.
We apparently use this library https://github.com/rconradharris/envparse which is not documented (at least I can't find any documentation). I'd say we should use something more standard.

[kokesak]

So the situation is that currently we are able to run only tier0 and tier1 tests on testing farm for Centos7 and Centos8. OracleLinux images have some issues so we decided to disable them until we fix them.
Tier2 tests are disabled as it is not mandatory to have them working now.

[kokesak]

Yes, you are right. I was overthinking about that condition xD

[kokesak]

Broken rollback test is disabled for centos-7 until the OAMG-5455 is done.

[mkluson]

oh, I can not merge it :-( I am technically still an author. @bocekm could you please approve and merge it?