Stage 3 hangs in integration testing (possibly due to tuned module)

[smithfarm]

Description of Issue/Question

For no apparent reason, Stage 3 hangs (i.e. enters an endless loop) transiently during integration tests. Examination of the minion logs shows the last thing that happened before the hang was related to tuned.

Upon further examination and discussion, it was concluded that the tuned Salt module is known to be buggy and it's possible that Salt is hanging (looping endlessly) in that module.

To address this, I propose to implement a --tuned option to health-ok.sh which will default to "off", meaning that, by default, the tuned orchestration will be disabled before running Stage 3. That way, once we are ready to investigate this further, we can implement a --tuned=on test case.

[smithfarm]

@mogeb @jschmid1 @rjfd

[jschmid1]

I think the less intrusive option is to add a --tuned=off/on option and add the version with --tuned=off in tier0 whilst adding tuned=on in tier2.

[smithfarm]

@jschmid1 tier2 is run by the deepsea nightly CI test and by the Staging CI test. Having --tuned=on in tier2 would mean those CI runs are exposed to the (presumed) bug in Salt's tuned module. Not sure I like that idea.

I was going to propose that we add a "tuned" test case to tier3 (which is not run by any automated CI, only triggered manually).

Once the Salt tuned module is reported to be fixed, then we could turn tuned back on. What do you think?