Add resource leak integration test

[judysng]

Description of changes

• Added integration test for checking that there isn't a file handler leak in computemgtd
  • Previously, there was a file handler leak that was fixed, so we wanted to add a new test to make sure that we catch any leaks
• Added resource leak checking as callable functions. Called the functions in starccm test because the starccm runs for ~20 mins, which is long enough to notice the leak

Tests

• Ran on personal Jenkins
  • Current computemgtd's number of file descriptors doesn't change over time
  • Ran with a modified computemgtd that introduces the above file handler leak -- the test fails and number of open files increases as expected

References

• Fix file handler leak in computemgtd aws-parallelcluster-node#421

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (3c5d0fc) 90.21% compared to head (599feb1) 90.21%.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #6009   +/-   ##
========================================
  Coverage    90.21%   90.21%           
========================================
  Files          180      180           
  Lines        15822    15822           
========================================
  Hits         14274    14274           
  Misses        1548     1548           

Flag	Coverage Δ
unittests	90.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[enrico-usai]

can we avoid having a specific integration test for it? and instead add this logic to check file descriptors and resource leak by default in all the tests. I mean, we don't need to wait 30 minutes in every test, but we can do the check.

[judysng]

I updated the PR to add the logic in functions in utils.py and called it in the starccm test. Unfortunately I don't think we can add it by default to all the tests unless we add the assert in cluster creation/deletion, which I think we'd want to avoid

[dreambeyondorange]

How long does the starccm test run typically?

[judysng]

~20 mins, which is long enough for the resource leak test to fail. I tested running this starccm test with the resource leak issue here and it failed as expected.

[dreambeyondorange]

I'm not sure this function name best describes what this is doing. It seems like this just gets the number of open files.

[judysng]

True, I renamed it to be get_compute_ip_to_no_files

[dreambeyondorange]

nit: I think the term no is less explicit than num to abbreviate the number of files. no has more binary implications or boolean implications