Support grouping in testgrid #489
Comments
jlewi
pushed a commit
to jlewi/testing
that referenced
this issue
Oct 17, 2019
* We want to support putting junit files in subdirectories e.g.
junit/junit_workflow1/junit_test1.xml
junit/junit_workflow2/junit_test1.xml
* prow_artifacts.py needs to fetch the junit files from GCS in order to
determine if there were any test failures to properly set the prow job
status
* The code however was making an assumption that all results were in
the artifacts dir and not subdirectories
* run_e2e_workflow.py should set a keyword argument junit_class_name that
to the name of the workflow (as set in prow_config.yaml) for each of
the python functions creating argo workflows.
* We want to use the junit attribute classname to group tests coming
from the same workflow. To do that we need to know the name of the workflow.
* Related to kubeflow#489
jlewi
pushed a commit
to jlewi/testing
that referenced
this issue
Oct 17, 2019
* We want to support putting junit files in subdirectories e.g.
junit/junit_workflow1/junit_test1.xml
junit/junit_workflow2/junit_test1.xml
* prow_artifacts.py needs to fetch the junit files from GCS in order to
determine if there were any test failures to properly set the prow job
status
* The code however was making an assumption that all results were in
the artifacts dir and not subdirectories
* run_e2e_workflow.py should set a keyword argument test_target_name
to the name of the workflow (as set in prow_config.yaml) for each of
the python functions creating argo workflows.
* We want to use the junit attribute classname to group tests coming
from the same workflow. To do that we need to know the name of the workflow.
* We also use this as a prefix for the test name to support grouping
hierarchically
* Related to kubeflow#489
* Update kfunittests to Use a junit subdirectory to make sure subdirectories are properly handled.
* Update docs
* update toc.
* Convert test_py_lint to use pytest. Do this because we want to be able to test the use of junit features.
* Add a separate mkdir step.
k8s-ci-robot
pushed a commit
that referenced
this issue
Oct 17, 2019
…ame (#490) * We want to support putting junit files in subdirectories e.g. junit/junit_workflow1/junit_test1.xml junit/junit_workflow2/junit_test1.xml * prow_artifacts.py needs to fetch the junit files from GCS in order to determine if there were any test failures to properly set the prow job status * The code however was making an assumption that all results were in the artifacts dir and not subdirectories * run_e2e_workflow.py should set a keyword argument test_target_name to the name of the workflow (as set in prow_config.yaml) for each of the python functions creating argo workflows. * We want to use the junit attribute classname to group tests coming from the same workflow. To do that we need to know the name of the workflow. * We also use this as a prefix for the test name to support grouping hierarchically * Related to #489 * Update kfunittests to Use a junit subdirectory to make sure subdirectories are properly handled. * Update docs * update toc. * Convert test_py_lint to use pytest. Do this because we want to be able to test the use of junit features. * Add a separate mkdir step.
jlewi
changed the title
jlewi
pushed a commit
to jlewi/kubeflow
that referenced
this issue
Oct 18, 2019
* See kubeflow/testing#489 we want to support having run_e2e_workflow.py pass extra arguments to the workflow. To do that we need to update the test to first accept unknown arguments so we don't break once run_e2e_workflow.py starts adding extra arguments.
jlewi
pushed a commit
to jlewi/testing
that referenced
this issue
Oct 18, 2019
…n test grid by name (kubeflow#490)" (kubeflow#493)" This reverts commit 81326be. * Relates to kubeflow#489 * Roll forward the orginal change now that kfctl_create_e2e_workflow.py has been updated.
k8s-ci-robot
pushed a commit
to kubeflow/kubeflow
that referenced
this issue
Oct 18, 2019
* Support extra arguments in kfctl_e2e_workflow.py * See kubeflow/testing#489 we want to support having run_e2e_workflow.py pass extra arguments to the workflow. To do that we need to update the test to first accept unknown arguments so we don't break once run_e2e_workflow.py starts adding extra arguments. * Fix.
jlewi
pushed a commit
to jlewi/testing
that referenced
this issue
Oct 18, 2019
…n test grid by name (kubeflow#490)" (kubeflow#493)" This reverts commit 81326be. * Relates to kubeflow#489 * Roll forward the orginal change now that kfctl_create_e2e_workflow.py has been updated to handle the extra argument. add a leading /
|
/area testing |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions. |
|
This issue has been closed due to inactivity. |
saffaalvi
pushed a commit
to StatCan/kubeflow
that referenced
this issue
Feb 11, 2021
* Support extra arguments in kfctl_e2e_workflow.py * See kubeflow/testing#489 we want to support having run_e2e_workflow.py pass extra arguments to the workflow. To do that we need to update the test to first accept unknown arguments so we don't break once run_e2e_workflow.py starts adding extra arguments. * Fix.
saffaalvi
pushed a commit
to StatCan/kubeflow
that referenced
this issue
Feb 12, 2021
* Support extra arguments in kfctl_e2e_workflow.py * See kubeflow/testing#489 we want to support having run_e2e_workflow.py pass extra arguments to the workflow. To do that we need to update the test to first accept unknown arguments so we don't break once run_e2e_workflow.py starts adding extra arguments. * Fix.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Our E2E tests run Argo workflows. We would like to group related tests when viewing the results in testgrid. For example we'd like to see all tests corresponding to the "GCP-IAP" config grouped together vs the same tests running in the "GCP-basic auth" config.
Test grid supports organizing by test target which corresponds to classname in the junit xml file.
When generating junit files from pytest we can use record_my_xml_attribute
http://doc.pytest.org/en/latest/usage.html#record-xml-attribute
to override the classname (it defaults to python module name).
The obvious value to use for the class name is the name assigned in the prow_config file
https://github.com/kubeflow/kubeflow/blob/70d3119fc5f075ff9cf715e6bfe0a8ce00e05736/prow_config.yaml#L32
This ensures each set of parameters would correspond to a different grouping.
To support that we should have run_e2e_worfklow.py pass that name as argument "junit_class_name" to all create_worfklow functions.
testing/py/kubeflow/testing/run_e2e_workflow.py
Line 330 in c629925
Then a workflow can set an environment variable to that value and pytest functions can use that environment variable to override the class name.
The text was updated successfully, but these errors were encountered: