[WIP] record the maximum memory used by LSF jobs

[drkennetz]

Taking over @mr-c PR #2857 . This will currently log the command and max mem used by the command for an LSF job utilizing bjobs -l:

[2020-11-12T16:08:40-0600] [Thread-3  ] [I] [toil.batchSystems.lsf] [job ID 111913722, Command  _toil_worker CWLJob file:/research/rgs01/home/clusterHome/ dkennetz/toil_test/cwl/TEST kind-CWLJob/instance-shr9enqg --context gANjdG9pbC5iYXRjaFN5c3RlbXMuYWJzdHJhY3RCYXRjaFN5 c3RlbQpXb3JrZXJDbGVhbnVwQ29udGV4dApxACmBcQF9cQIoWBEAAAB3b3 JrZXJDbGVhbnVwSW5mb3EDY3RvaWwuYmF0Y2hTeXN0ZW1zLmFic3RyYWN0 QmF0Y2hTeXN0ZW0KV29ya2VyQ2xlYW51cEluZm8KcQRYPwAAAC9yZXNlYX JjaC9yZ3MwMS9ob21lL2NsdXN0ZXJIb21lL2RrZW5uZXR6L3RvaWxfdGVz dC9jd2wvd29ya2RpcnEFWCQAAAAxNWI5Mzg0Yi02NGJiLTQ0ODYtODFmZi 02YWU5MTI4MWFlYjlxBlgGAAAAYWx3YXlzcQeHcQiBcQlYBQAAAGFyZW5h cQpOdWIu] the maximum memory used was: 1 Gbytes
[2020-11-12T16:08:41-0600] [Thread-3  ] [I] [toil.batchSystems.lsf] [job ID 111913722, Command  _toil_worker CWLJob file:/research/rgs01/home/clusterHome/ dkennetz/toil_test/cwl/TEST kind-CWLJob/instance-shr9enqg --context gANjdG9pbC5iYXRjaFN5c3RlbXMuYWJzdHJhY3RCYXRjaFN5 c3RlbQpXb3JrZXJDbGVhbnVwQ29udGV4dApxACmBcQF9cQIoWBEAAAB3b3 JrZXJDbGVhbnVwSW5mb3EDY3RvaWwuYmF0Y2hTeXN0ZW1zLmFic3RyYWN0 QmF0Y2hTeXN0ZW0KV29ya2VyQ2xlYW51cEluZm8KcQRYPwAAAC9yZXNlYX JjaC9yZ3MwMS9ob21lL2NsdXN0ZXJIb21lL2RrZW5uZXR6L3RvaWxfdGVz dC9jd2wvd29ya2RpcnEFWCQAAAAxNWI5Mzg0Yi02NGJiLTQ0ODYtODFmZi 02YWU5MTI4MWFlYjlxBlgGAAAAYWx3YXlzcQeHcQiBcQlYBQAAAGFyZW5h cQpOdWIu] the maximum memory used was: 5.1 Gbytes
[2020-11-12T16:08:42-0600] [Thread-3  ] [I] [toil.batchSystems.lsf] [job ID 111913722, Command  _toil_worker CWLJob file:/research/rgs01/home/clusterHome/ dkennetz/toil_test/cwl/TEST kind-CWLJob/instance-shr9enqg --context gANjdG9pbC5iYXRjaFN5c3RlbXMuYWJzdHJhY3RCYXRjaFN5 c3RlbQpXb3JrZXJDbGVhbnVwQ29udGV4dApxACmBcQF9cQIoWBEAAAB3b3 JrZXJDbGVhbnVwSW5mb3EDY3RvaWwuYmF0Y2hTeXN0ZW1zLmFic3RyYWN0 QmF0Y2hTeXN0ZW0KV29ya2VyQ2xlYW51cEluZm8KcQRYPwAAAC9yZXNlYX JjaC9yZ3MwMS9ob21lL2NsdXN0ZXJIb21lL2RrZW5uZXR6L3RvaWxfdGVz dC9jd2wvd29ya2RpcnEFWCQAAAAxNWI5Mzg0Yi02NGJiLTQ0ODYtODFmZi 02YWU5MTI4MWFlYjlxBlgGAAAAYWx3YXlzcQeHcQiBcQlYBQAAAGFyZW5h cQpOdWIu] the maximum memory used was: 5.1 Gbytes

[drkennetz]

The random spaces in paths and text were introduced by copying.

[DailyDreaming]

@drkennetz Is this still WIP? I'm approving and would be fine with merging as is.

One comment is below and also the parseMaxMem function doesn't seem to need to return anything, so I would consider removing the return statements.

[drkennetz]

@DailyDreaming I gave it a return because @mr-c said it might be useful to actually return from getJobExitCode in the future for provenance. It serves no purpose at the moment though, except as one less thing to think about if we decide to implement this change in the future.

I am happy to remove the return, and make the logging more verbose as suggested. It is no longer a WIP. I'll wait on your response before committing changes (potentially removing the return and updating logging).

[drkennetz]

Sorry for the delay, I have updated based on the recommendations here.