New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flaky decorator to mac_system and mac_timezone tests #48583

Merged
merged 2 commits into from Jul 13, 2018

Conversation

Projects
None yet
2 participants
@Ch3LL
Contributor

Ch3LL commented Jul 13, 2018

What does this PR do?

A lot of our mac tests/modules that use this command systemsetup have always provided back flaky test results. I believe I have kind of figured out why and you can see below:

Mac-6:testing root# systemsetup -setlocalsubnetname blah4
setlocalsubnetname: blah4
Mac-6:testing root# systemsetup -getlocalsubnetname
Local Subnet Name: blah3

As you can see above sometimes using this systemsetup command just does not work. Its not a timeing issue thing. I wait 5-10 minutes and go back to get hte localsubnetname and it never changed. But sometimes it does change as shown below:

Mac-6:testing root# systemsetup -setlocalsubnetname blah2
setlocalsubnetname: blah2
Mac-6:testing root# systemsetup -getlocalsubnetname
Local Subnet Name: blah2

I would say for all of the systemsetup commands i have tried 1 out of 10 times it will fail. I really do not know why its failing on mac and I do not see anything in the logs or stderr and the exit code is 0. My google fu is not strong here either and I can't seem to figure out why on mac sometimes systemsetup works and sometimes it does not. So I don't believe we can workaround this in salts code.

Also i have tested this on sierra and highsierra. But i'm pretty certain it would exhibit the same behavior in all mac OSs.

Now with the @flaky decorator it will re-run the tests if it fails so we can get some useful results. Currently the tests for mac_power are using this @flaky decorator.

@rallytime

This comment has been minimized.

Contributor

rallytime commented Jul 13, 2018

@Ch3LL This has a pylint failure: https://jenkinsci.saltstack.com/job/pr-lint/job/PR-48583/1/warnings52Result/new/

The flaky decorator needs to be imported.

@Ch3LL

This comment has been minimized.

Contributor

Ch3LL commented Jul 13, 2018

doh... silly mistake. fixing now

@rallytime rallytime merged commit aa6dcf3 into saltstack:2017.7 Jul 13, 2018

4 of 16 checks passed

jenkins/PR/salt-pr-linode-cent7-py3 Pull Requests » Salt PR - Linode CentOS 7 - PY3 #6456 — FAILURE
Details
jenkins/pr/py3-centos-7 The py3-centos-7 job has failed
Details
continuous-integration/jenkins/pr-merge This commit is being built
Details
default Build started sha1 is merged.
Details
jenkins/PR/salt-pr-linode-ubuntu14-n Pull Requests » Salt PR - Linode Ubuntu14.04 #24384 — RUNNING
Details
jenkins/PR/salt-pr-linode-ubuntu16-py3 Pull Requests » Salt PR - Linode Ubuntu16.04 - PY3 #11426 — RUNNING
Details
jenkins/PR/salt-pr-lint-n Pull Requests » Salt PR - Code Lint #23338 — RUNNING
Details
jenkins/PR/salt-pr-rs-cent7-n Pull Requests » Salt PR - RS CentOS 7 #20509 — RUNNING
Details
jenkins/pr/docs Testing docs...
Details
jenkins/pr/py2-centos-7 running py2-centos-7...
Details
jenkins/pr/py2-ubuntu-1604 running py2-ubuntu-1604...
Details
jenkins/pr/py3-ubuntu-1604 running py3-ubuntu-1604...
Details
WIP ready for review
Details
jenkins/PR/salt-pr-clone Pull Requests » Salt PR - Clone #26672 — SUCCESS
Details
jenkins/PR/salt-pr-docs-n Pull Requests » Salt PR - Docs #18701 — SUCCESS
Details
jenkins/pr/lint The lint job has passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment