tests: use stop/start over restart for service to improve flaky system test

[reubenmiller]

Proposed changes

Improving flaky system test:

Check PID number in lock file after restarting the services

Suspected root cause:

systemctl restart does not wait for all file descriptors to be flushed before returning which is most likely the reason why some pid information is still being detected via pgrep -f. Using stop/start does not suffer from this problem.

flake-finder was used to run the suite of tests 200 times and all passed:

Overall: PASSED
Results: 200 iterations, 200 passed, 0 failed
Elapsed time: 9:20:38.046294

And a second run of 300 iterations all passed except for 1 unrelated error due to a loss of connectivity during the system test run:

Overall: FAILED
Results: 300 iterations, 299 passed, 1 failed
Elapsed time: 14:02:55.328967
Failed iterations: [227]

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#2087

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s)
• I ran cargo fmt as mentioned in CODING_GUIDELINES
• I used cargo clippy as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
356	0	3	356	100	57m37.942s

[didier-wenzek]

Approved