Add proxy script for qstat

[berland]

Issue
Resolves #3541

Approach
This shell script is significantly faster to run compared to the
real qstat ("backend") as it only greps in a file on /tmp.
The file on /tmp acts as a cache, and will be updated whenever
it gets too old (~seconds).

The queue manager in ERT polls the queing system (that is qstat)
once pr. second for every realization. This can replicate a DDoS on
the cluster queue server, which we mitigate by using this proxy.

Blocked by #3537

Pre review checklist

• Added appropriate release note label
• PR title captures the intent of the changes, and is fitting for release notes.
• Commit history is consistent and clean, in line with the contribution guidelines.

Adding labels helps the maintainers when writing release notes. This is the list of release note labels.

[codecov-commenter]

Codecov Report

Merging #3553 (56611b7) into main (0079c8e) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #3553      +/-   ##
==========================================
- Coverage   63.34%   63.28%   -0.06%     
==========================================
  Files         597      595       -2     
  Lines       46243    46198      -45     
  Branches     4144     4156      +12     
==========================================
- Hits        29293    29237      -56     
- Misses      15662    15667       +5     
- Partials     1288     1294       +6     

Impacted Files	Coverage Δ
src/libres/lib/include/ert/res_util/string.hpp	65.78% <0.00%> (-1.96%)	⬇️
src/libres/lib/enkf/enkf_obs.cpp	64.22% <0.00%> (-0.86%)	⬇️
src/libres/lib/enkf/ensemble_config.cpp	42.47% <0.00%> (-0.25%)	⬇️
src/libres/lib/res_util/block_fs.cpp	67.75% <0.00%> (ø)
src/libres/lib/enkf/enkf_main_ensemble.cpp
src/libres/lib/enkf/enkf_main_manage_fs.cpp
src/libres/lib/enkf/ert_template.cpp	15.17% <0.00%> (+0.39%)	⬆️
src/libres/lib/enkf/enkf_main.cpp	37.54% <0.00%> (+4.08%)	⬆️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[berland]

TODO:

• Fails on macos due to diffent syntax for the stat command (solved with stat -f %B on mac)
• Write tests provoking race conditions when qstat backend takes a second to answer
• mv command is atomic (or very close). Use that to move qstat output into the correct place when it is done.
• Approach is not portable to mac as flock does not exist for mac. shlock on mac is an option but it states it does not work on NFS.
• Use cmake to install the proxy script for tests, setup.py for normal usage.
• Verify that an "Ensemble experiment" on the poly-case on an Azure node with 1000 realizations runs smoothly.

[jondequinor]

Why did you choose bash? flock (fcntl), stat, cat should all be supported by python stdlib.

Unless there's a really good reason to choose bash, we should choose the portable python route.

For example, try building a komodo distribution on macos. It will fail mostly because it assumes rename, rmdir to come from coreutils, which on macos they do not.

[berland]

sh was chosen as I speculate it is significantly cheaper/faster to spawn compared to Python. As this is command will be called every second for every realization with the current queue manager, startup time can be critical. The file locking used by flock is in any case not portable to Mac.

I buy the speed argument

[xjules]

2x the same

[xjules]

Just to recap our discussion: add one more test to check that proxyfile gets new content when timeout happens. Otherwise it's a solid piece of work! Nice @berland

[berland]

Rebase pending reviewer @xjules

[xjules]

Nice @berland ! Very good job! 🚀