New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Remove EnvGroup and EnvProperty models in favor of Tags #484

Open
atodorov opened this Issue Aug 21, 2018 · 7 comments

Comments

Projects
None yet
2 participants
@atodorov
Member

atodorov commented Aug 21, 2018

Environment Groups + Properties and Tags are two different methods designed to add extra properties/information to test execution. At this moment I am in favor of removing EnvGroup and EnvProperty models and leaving only Tag.

Tags appear to be used more often in the source code (did a quick grep) and they also seem to be a bit more flexible.

The numbers from the live database are as follow:

>>> Tag.objects.count()
115
>>> EnvGroup.objects.count()
0
>>> EnvProperty.objects.count()
15

>>> TestPlanTag.objects.count()
2
>>> TestCaseTag.objects.count()
2
>>> TestRunTag.objects.count()
4

although these may not represent real-world use-cases.

I will prepare a PR to remove Properties and Groups after Sept 1st if I don't hear otherwise.

atodorov added a commit that referenced this issue Aug 21, 2018

Remove EnvGroup, EnvProperty and EnvValue models. Closes #484
these are removed in favor of Tag

atodorov added a commit that referenced this issue Aug 21, 2018

Remove EnvGroup, EnvProperty and EnvValue models. Closes #484
these are removed in favor of Tag. Closes #197, #284

atodorov added a commit that referenced this issue Aug 23, 2018

Remove EnvGroup, EnvProperty and EnvValue models. Closes #484
these are removed in favor of Tag. Closes #197, #284, #319

atodorov added a commit that referenced this issue Aug 23, 2018

Convert TestRun edit page to Patternfly
NOTES:

- estimated_time is scheduled for removal so not shown in the form
- product_version is scheduled for removal in favor of
  TR.plan.product_version
- Product & Version can't be editted anymore. Must be set on the
  parent TestPlan instead.
- TestRun still allows specifying builds
- environment is scheduled for deprecation (see #484) and is not
  shown anymore. Fix #445

atodorov added a commit that referenced this issue Aug 23, 2018

Convert TestRun edit page to Patternfly
NOTES:

- estimated_time is scheduled for removal so not shown in the form
- product_version is scheduled for removal in favor of
  TR.plan.product_version
- Product & Version can't be editted anymore. Must be set on the
  parent TestPlan instead.
- TestRun still allows specifying builds
- environment is scheduled for deprecation (see #484) and is not
  shown anymore. Fix #445

atodorov added a commit that referenced this issue Aug 23, 2018

Remove EnvGroup, EnvProperty and EnvValue models. Closes #484
these are removed in favor of Tag. Closes #197, #284, #319
@okainov

This comment has been minimized.

Show comment
Hide comment
@okainov

okainov Aug 24, 2018

Contributor

??? could you provide more details abut background?

Having this Environment was one of the major reasons why we selected Kiwi across other tools. ANd we use it extensively now. Why do you want to remove it and how is the "migration plan"?

Env is much more powerful mechanism than tags IMO. It allows you to create groups and use them as well as filter test runs by environment values. Tags do not provide you predefined set of values.

Contributor

okainov commented Aug 24, 2018

??? could you provide more details abut background?

Having this Environment was one of the major reasons why we selected Kiwi across other tools. ANd we use it extensively now. Why do you want to remove it and how is the "migration plan"?

Env is much more powerful mechanism than tags IMO. It allows you to create groups and use them as well as filter test runs by environment values. Tags do not provide you predefined set of values.

@atodorov

This comment has been minimized.

Show comment
Hide comment
@atodorov

atodorov Aug 24, 2018

Member

Can you describe your use cases in more detail ? On which pages do you use environment properties and what do you use them for ?

Member

atodorov commented Aug 24, 2018

Can you describe your use cases in more detail ? On which pages do you use environment properties and what do you use them for ?

@okainov

This comment has been minimized.

Show comment
Hide comment
@okainov

okainov Aug 24, 2018

Contributor

In each test plan we have environment group including such env properties as:
OS. Values "Windows 10, Ubuntu 16, Ubuntu 18" etc. We don;t want to have possibility to type random text which we won;t be able to filter afterwards.
Testing type: Scheduled or Ad-hoc. We use only Scheduled test runs to show in our statistics page (some our addon on top of Kiwi showing testing summary)
Used component version. This is the most important part for us. Our product under test includes 3 other component and we want to have information about exact component version to be included in test run. Thus we can filter to get information about product builds which had some component version and link it with component test runs.
Example: "Library A" with values "1.2132.2834", "1.4232.5674" etc. And we have separated test plan for Library A where "1.2132.2834", "1.4232.5674" go as "Product version", so we can make a link between them (again, it was implemented in our "statistics" page we created on top of Kiwi)

On which pages do you use environment properties

What do you mean? We assign env group to test plan and it's mandatory to fill env values for each test run we create. So we have it on every test run page.

Here is some screenshot of typical test run we have
image

Contributor

okainov commented Aug 24, 2018

In each test plan we have environment group including such env properties as:
OS. Values "Windows 10, Ubuntu 16, Ubuntu 18" etc. We don;t want to have possibility to type random text which we won;t be able to filter afterwards.
Testing type: Scheduled or Ad-hoc. We use only Scheduled test runs to show in our statistics page (some our addon on top of Kiwi showing testing summary)
Used component version. This is the most important part for us. Our product under test includes 3 other component and we want to have information about exact component version to be included in test run. Thus we can filter to get information about product builds which had some component version and link it with component test runs.
Example: "Library A" with values "1.2132.2834", "1.4232.5674" etc. And we have separated test plan for Library A where "1.2132.2834", "1.4232.5674" go as "Product version", so we can make a link between them (again, it was implemented in our "statistics" page we created on top of Kiwi)

On which pages do you use environment properties

What do you mean? We assign env group to test plan and it's mandatory to fill env values for each test run we create. So we have it on every test run page.

Here is some screenshot of typical test run we have
image

@atodorov

This comment has been minimized.

Show comment
Hide comment
@atodorov

atodorov Aug 27, 2018

Member

@okainov to summarize:

  1. You assign EnvGroup when creating a Test Plan
  2. When creating a TestRun you select individual properties from the environment group
  3. There is custom reporting page built on top of this
  4. You don't want the user to be able to select arbitrary text (or mistype some of the values)

Is this how you select the properties for a new TestRun ?
tr_with_env

From what I can tell tags can be used with the following limitations:

  1. Upon creation of TestRun currently it is not possible to select tags. They can be added after creation.
  2. In some places we have tooltips for tags (e.g. auto complete) but nothing prevents you from misspelling or entering an arbitrary value.
Member

atodorov commented Aug 27, 2018

@okainov to summarize:

  1. You assign EnvGroup when creating a Test Plan
  2. When creating a TestRun you select individual properties from the environment group
  3. There is custom reporting page built on top of this
  4. You don't want the user to be able to select arbitrary text (or mistype some of the values)

Is this how you select the properties for a new TestRun ?
tr_with_env

From what I can tell tags can be used with the following limitations:

  1. Upon creation of TestRun currently it is not possible to select tags. They can be added after creation.
  2. In some places we have tooltips for tags (e.g. auto complete) but nothing prevents you from misspelling or entering an arbitrary value.
@okainov

This comment has been minimized.

Show comment
Hide comment
@okainov

okainov Aug 27, 2018

Contributor

Is this how you select the properties for a new TestRun ?

Yes

In some places we have tooltips for tags (e.g. auto complete) but nothing prevents you from misspelling or entering an arbitrary value.

This is the problem. We (1) rely on set of available values for each property and (2) want to have exactly those environment properties to be set.

Contributor

okainov commented Aug 27, 2018

Is this how you select the properties for a new TestRun ?

Yes

In some places we have tooltips for tags (e.g. auto complete) but nothing prevents you from misspelling or entering an arbitrary value.

This is the problem. We (1) rely on set of available values for each property and (2) want to have exactly those environment properties to be set.

@okainov

This comment has been minimized.

Show comment
Hide comment
@okainov

okainov Aug 29, 2018

Contributor

NOTE: TestRun edit() view doesn't allow for editting properties
which is pre-existing behavior.

This is a problem. For some of env properties we either use two same env properties with the same test run (i.e. Version: Abc AND Version: 321)
And of course sometimes you entered something by mistake so you need to edit it

Contributor

okainov commented Aug 29, 2018

NOTE: TestRun edit() view doesn't allow for editting properties
which is pre-existing behavior.

This is a problem. For some of env properties we either use two same env properties with the same test run (i.e. Version: Abc AND Version: 321)
And of course sometimes you entered something by mistake so you need to edit it

@atodorov

This comment has been minimized.

Show comment
Hide comment
@atodorov

atodorov Sep 17, 2018

Member

A little update. There are few things that need to happen before we can safely remove EnvGroups/EnvProperties:

  1. Show tags in admin panel so that site admins can add/edit them

  2. It looks to me that permissions are not taken into account when tags are added to TestPlan, TestCase and TestRun objects. The individual permissions exist but from what I can tell they are not applied.

  3. Almost everywhere where we work with tags and user input we use get_or_create(). This needs to be modified to work according to permissions. If the user can't create tags they should be forced to use only the ones that already exist.

  4. Everywhere where tags are used we need autocomplete. I don't think this is the case but need to check.

  5. The only difference with environment is that test runs supposedly restrict the user to selecting properties only from the chosen EnvGroup. This is only true when creating a new TestRun. Properties can also be modified after the TR has been created and currently users are free to select any property and any value, even if they are not part of the same EnvGroup assigned to the TestRun. So the current situation is no different than being able to select any tag!

  6. to 4) can be fixed and we have something in the works for them.

For 5) I don't have an opinion at the moment.

Member

atodorov commented Sep 17, 2018

A little update. There are few things that need to happen before we can safely remove EnvGroups/EnvProperties:

  1. Show tags in admin panel so that site admins can add/edit them

  2. It looks to me that permissions are not taken into account when tags are added to TestPlan, TestCase and TestRun objects. The individual permissions exist but from what I can tell they are not applied.

  3. Almost everywhere where we work with tags and user input we use get_or_create(). This needs to be modified to work according to permissions. If the user can't create tags they should be forced to use only the ones that already exist.

  4. Everywhere where tags are used we need autocomplete. I don't think this is the case but need to check.

  5. The only difference with environment is that test runs supposedly restrict the user to selecting properties only from the chosen EnvGroup. This is only true when creating a new TestRun. Properties can also be modified after the TR has been created and currently users are free to select any property and any value, even if they are not part of the same EnvGroup assigned to the TestRun. So the current situation is no different than being able to select any tag!

  6. to 4) can be fixed and we have something in the works for them.

For 5) I don't have an opinion at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment