Test for older versions in new Fedora releases. Closes #28 #91

atodorov · 2016-07-06T10:49:38Z

This test loops over the contents of commandMap and dataMap and tries to find handlers (e.g. Fedora versions) for which the listed commands and data attributes are not the latest possible version defined in the corresponding command.py file.

For example if the latest release is Fedora24 and we're testing the handler for Fedora23's "logvol" then we expect the mapping to be F23_LogVol or lover, even if F24_LogVol is present. However if the mapping is F20_LogVol then this is a bug b/c F23_LogVol provides newer implementation which isn't used. Currently the test fails with 51 errors.


ERROR: In `handlers/f23.py` the "fcoe" command maps to "F13_Fcoe" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_Fcoe".


ERROR: In `handlers/f23.py` "FcoeData" maps to "F13_FcoeData" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_FcoeData".


ERROR: In `handlers/f7.py` the "network" command maps to "FC6_Network" while in
`pykickstart.commands.network` there is newer implementation: "RHEL5_Network".


ERROR: In `handlers/f7.py` the "logvol" command maps to "FC4_LogVol" while in
`pykickstart.commands.logvol` there is newer implementation: "RHEL5_LogVol".


ERROR: In `handlers/f7.py` the "bootloader" command maps to "FC4_Bootloader" while in
`pykickstart.commands.bootloader` there is newer implementation: "RHEL5_Bootloader".


ERROR: In `handlers/f7.py` the "partition" command maps to "FC4_Partition" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL5_Partition".


ERROR: In `handlers/f7.py` the "part" command maps to "FC4_Partition" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL5_Partition".


ERROR: In `handlers/f7.py` "PartData" maps to "FC4_PartData" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL5_PartData".


ERROR: In `handlers/f7.py` "LogVolData" maps to "FC4_LogVolData" while in
`pykickstart.commands.logvol` there is newer implementation: "RHEL5_LogVolData".


ERROR: In `handlers/f7.py` "RaidData" maps to "FC5_RaidData" while in
`pykickstart.commands.raid` there is newer implementation: "RHEL5_RaidData".


ERROR: In `handlers/f16.py` the "iscsi" command maps to "F10_Iscsi" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_Iscsi".


ERROR: In `handlers/f16.py` the "iscsi" command maps to "F10_Iscsi" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_Iscsi".


ERROR: In `handlers/f16.py` "IscsiData" maps to "F10_IscsiData" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_IscsiData".


ERROR: In `handlers/f16.py` "IscsiData" maps to "F10_IscsiData" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_IscsiData".


ERROR: In `handlers/f14.py` the "autopart" command maps to "F12_AutoPart" while in
`pykickstart.commands.autopart` there is newer implementation: "RHEL6_AutoPart".


ERROR: In `handlers/f14.py` the "network" command maps to "F9_Network" while in
`pykickstart.commands.network` there is newer implementation: "RHEL6_Network".


ERROR: In `handlers/f14.py` the "iscsi" command maps to "F10_Iscsi" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_Iscsi".


ERROR: In `handlers/f14.py` the "iscsi" command maps to "F10_Iscsi" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_Iscsi".


ERROR: In `handlers/f14.py` "PartData" maps to "F12_PartData" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL6_PartData".


ERROR: In `handlers/f14.py` "LogVolData" maps to "F12_LogVolData" while in
`pykickstart.commands.logvol` there is newer implementation: "RHEL6_LogVolData".


ERROR: In `handlers/f14.py` "IscsiData" maps to "F10_IscsiData" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_IscsiData".


ERROR: In `handlers/f14.py` "IscsiData" maps to "F10_IscsiData" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_IscsiData".


ERROR: In `handlers/f14.py` "NetworkData" maps to "F8_NetworkData" while in
`pykickstart.commands.network` there is newer implementation: "RHEL6_NetworkData".


ERROR: In `handlers/f14.py` "RaidData" maps to "F12_RaidData" while in
`pykickstart.commands.raid` there is newer implementation: "RHEL6_RaidData".


ERROR: In `handlers/devel.py` the "user" command maps to "F19_User" while in
`pykickstart.commands.user` there is newer implementation: "F24_User".


ERROR: In `handlers/devel.py` the "fcoe" command maps to "F13_Fcoe" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_Fcoe".


ERROR: In `handlers/devel.py` "FcoeData" maps to "F13_FcoeData" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_FcoeData".


ERROR: In `handlers/f22.py` the "logvol" command maps to "F20_LogVol" while in
`pykickstart.commands.logvol` there is newer implementation: "F21_LogVol".


ERROR: In `handlers/f22.py` the "logvol" command maps to "F20_LogVol" while in
`pykickstart.commands.logvol` there is newer implementation: "RHEL7_LogVol".


ERROR: In `handlers/f22.py` the "fcoe" command maps to "F13_Fcoe" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_Fcoe".


ERROR: In `handlers/f22.py` "LogVolData" maps to "F20_LogVolData" while in
`pykickstart.commands.logvol` there is newer implementation: "F21_LogVolData".


ERROR: In `handlers/f22.py` "LogVolData" maps to "F20_LogVolData" while in
`pykickstart.commands.logvol` there is newer implementation: "RHEL7_LogVolData".


ERROR: In `handlers/f22.py` "FcoeData" maps to "F13_FcoeData" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_FcoeData".


ERROR: In `handlers/f8.py` the "logvol" command maps to "FC4_LogVol" while in
`pykickstart.commands.logvol` there is newer implementation: "RHEL5_LogVol".


ERROR: In `handlers/f8.py` the "partition" command maps to "FC4_Partition" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL5_Partition".


ERROR: In `handlers/f8.py` the "part" command maps to "FC4_Partition" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL5_Partition".


ERROR: In `handlers/f8.py` "PartData" maps to "FC4_PartData" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL5_PartData".


ERROR: In `handlers/f8.py` "LogVolData" maps to "FC4_LogVolData" while in
`pykickstart.commands.logvol` there is newer implementation: "RHEL5_LogVolData".


ERROR: In `handlers/f8.py` "RaidData" maps to "FC5_RaidData" while in
`pykickstart.commands.raid` there is newer implementation: "RHEL5_RaidData".


ERROR: In `handlers/f15.py` the "autopart" command maps to "F12_AutoPart" while in
`pykickstart.commands.autopart` there is newer implementation: "RHEL6_AutoPart".


ERROR: In `handlers/f15.py` the "network" command maps to "F9_Network" while in
`pykickstart.commands.network` there is newer implementation: "RHEL6_Network".


ERROR: In `handlers/f15.py` the "iscsi" command maps to "F10_Iscsi" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_Iscsi".


ERROR: In `handlers/f15.py` the "iscsi" command maps to "F10_Iscsi" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_Iscsi".


ERROR: In `handlers/f15.py` "PartData" maps to "F12_PartData" while in
`pykickstart.commands.partition` there is newer implementation: "RHEL6_PartData".


ERROR: In `handlers/f15.py` "IscsiData" maps to "F10_IscsiData" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_IscsiData".


ERROR: In `handlers/f15.py` "IscsiData" maps to "F10_IscsiData" while in
`pykickstart.commands.iscsi` there is newer implementation: "RHEL6_IscsiData".


ERROR: In `handlers/f15.py` "NetworkData" maps to "F8_NetworkData" while in
`pykickstart.commands.network` there is newer implementation: "RHEL6_NetworkData".


ERROR: In `handlers/f18.py` the "network" command maps to "F16_Network" while in
`pykickstart.commands.network` there is newer implementation: "F18_Network".


ERROR: In `handlers/f24.py` the "user" command maps to "F19_User" while in
`pykickstart.commands.user` there is newer implementation: "F24_User".


ERROR: In `handlers/f24.py` the "fcoe" command maps to "F13_Fcoe" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_Fcoe".


ERROR: In `handlers/f24.py` "FcoeData" maps to "F13_FcoeData" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_FcoeData".

The Fedora vs. newer RHEL are probably false negatives. Then the test correctly identifies the logvol problems in f22.py and some in devel.py.

I've also made some minor fixes and adjustments in the first several commits so I can get this working. The test itself is in the last commit.

@vpodzime, @clumens please comment on this.

atodorov · 2016-07-06T11:39:56Z

With the latest commit I don't report errors when the command implementation and the handler differ in terms of RHEL vs. Fedora. (Deliberately made as separate commit for easier review, can squash later). The error list is now reduced to 7 errors which seem legitimate:

ERROR: In `handlers/devel.py` the "fcoe" command maps to "F13_Fcoe" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_Fcoe".


ERROR: In `handlers/devel.py` the "user" command maps to "F19_User" while in
`pykickstart.commands.user` there is newer implementation: "F24_User".


ERROR: In `handlers/devel.py` "FcoeData" maps to "F13_FcoeData" while in
`pykickstart.commands.fcoe` there is newer implementation: "RHEL7_FcoeData".


ERROR: In `handlers/f22.py` the "logvol" command maps to "F20_LogVol" while in
`pykickstart.commands.logvol` there is newer implementation: "F21_LogVol".


ERROR: In `handlers/f22.py` "LogVolData" maps to "F20_LogVolData" while in
`pykickstart.commands.logvol` there is newer implementation: "F21_LogVolData".


ERROR: In `handlers/f18.py` the "network" command maps to "F16_Network" while in
`pykickstart.commands.network` there is newer implementation: "F18_Network".


ERROR: In `handlers/f24.py` the "user" command maps to "F19_User" while in
`pykickstart.commands.user` there is newer implementation: "F24_User".

atodorov · 2016-07-06T12:51:37Z

tests/handlers.py

+                            s_impl_version.startswith('RHEL')) or \
+                            (s_handler_version.startswith('RHEL') and \
+                            s_impl_version.startswith('F')):
+                            continue


I don't like this block. It introduces the possibility to silent legitimate errors. For example modify handlers/rhel7.py to contain "autopart': pykickstart.commands.autopart.F20_AutoPart then run the test. The error isn't detected.

Part of the problem is the fact that RHEL7_AutoPart = F21_AutoPart. If we define the class like this

class RHEL7_AutoPart(F21_AutoPart): pass

then the error is detected even with the skip block in place. Maybe I can add another test for those Fxy_Command = Fxx_Command definitions and update the code where necessary. What do you guys think ?

This test loops over the contents of commandMap and dataMap and tries to find handlers (e.g. Fedora versions) for which the listed commands and data attributes are not the latest possible version defined in the corresponding command.py file. For example if the latest release is Fedora24 and we're testing the handler for Fedora23's "logvol" then we expect the mapping to be F23_LogVol or lover, even if F24_LogVol is present. However if the mapping is F20_LogVol then this is a bug b/c F23_LogVol provides newer implementation which isn't used. The test doesn't report errors for RHEL vs Fedora differences and vice versa.

atodorov · 2016-07-08T07:02:56Z

I've rebase to latest and fixed the identified errors in the last commit. I also fixed a few minor issues along the way.

@clumens do you want to merge everything as-is or want me to split out the commits which are not related to this test into a separate PR?

clumens · 2016-07-08T14:05:47Z

I'll just merge everything as-is, thanks.

atodorov force-pushed the handler_test branch from 154044d to 155cf81 Compare July 6, 2016 10:52

atodorov reviewed Jul 6, 2016
View reviewed changes

atodorov mentioned this pull request Jul 6, 2016

Test how command and data classes are defined #92

Merged

atodorov added 6 commits July 8, 2016 09:39

Remove unused import

c1d42e5

KSOptionParser accepts description, not help argument

7fb7afa

Update sys.path in handlers/control.py if not already updated

2a9f2fb

Rename FC16 to F16 so we can find it later in versionMap

1357487

Fix command handler errors identified by previous test

fe2c2ee

atodorov force-pushed the handler_test branch from c820055 to fe2c2ee Compare July 8, 2016 06:54

Fix unused-variable warning

2654f78

clumens merged commit 9f6dfc4 into pykickstart:master Jul 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test for older versions in new Fedora releases. Closes #28 #91

Test for older versions in new Fedora releases. Closes #28 #91

atodorov commented Jul 6, 2016

atodorov commented Jul 6, 2016

atodorov Jul 6, 2016

atodorov commented Jul 8, 2016

clumens commented Jul 8, 2016

Test for older versions in new Fedora releases. Closes #28 #91

Test for older versions in new Fedora releases. Closes #28 #91

Conversation

atodorov commented Jul 6, 2016

atodorov commented Jul 6, 2016

atodorov Jul 6, 2016

Choose a reason for hiding this comment

atodorov commented Jul 8, 2016

clumens commented Jul 8, 2016