tests: fix raciness in stop mode tests

[zyga]

This branch fixes raciness in the stop-mode tests. This test was failing for
the following reasons:

• the registration of the signal handler was racing with the kill
• the shell semantics for trap is error prone and was used incorrectly

The first issue is fixed with a "stamp" like file that we look for in the test.
This ensures that before we attempt to stop the service we wait for it to
initialize fully, that is to register the signal handlers.

The second issue is fixed by re-writing the test in Python 3. This requires
some more explanation. We had established signal handlers using the "trap"
shell builtin and were expecting them to run when a signal (sent by systemd as
a result of stopping the service) arrives. What we can see by running the shell
script with "strace -e %signal" indicates that shell disables trap handlers
before executing any command. A trivial test involved increasing the sleep time
to a large value and sending the expected signal at that time. The signal
handler never runs in that case.

This is fixed by writing the test service in Python and not relying on
additional processes. This has the expected semantics and works well in
practice.

Shell is tricky, more than we though, and we keep learning this the hard way.

Signed-off-by: Zygmunt Krynicki zygmunt.krynicki@canonical.com

[codecov-io]

Codecov Report

Merging #5526 into master will increase coverage by 0.23%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5526      +/-   ##
==========================================
+ Coverage   79.06%   79.29%   +0.23%     
==========================================
  Files         514      514              
  Lines       38949    39492     +543     
==========================================
+ Hits        30795    31316     +521     
- Misses       5681     5688       +7     
- Partials     2473     2488      +15

Impacted Files	Coverage Δ
overlord/ifacestate/implicit.go	96.42% <0%> (-3.58%)	⬇️
overlord/hookstate/hookmgr.go	75.23% <0%> (+0.93%)	⬆️
overlord/snapstate/snapstate.go	85.6% <0%> (+3.55%)	⬆️
overlord/ifacestate/helpers.go	78.79% <0%> (+7.88%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4dd420d...85d675d. Read the comment docs.

[zyga]

As a general comment, the test still suffers some issues but we suspect those are in our general log extracting helpers. We've decided to merge this as is as it fixes real issues. Investigation of the log extractor bug continues in the branch referenced by mvo above.

[mvo5]

Looks good, thank you! Much nicer than the original shell version.