Update lock_module decorator to handle abandoned locks

[BradleySappington]

Update @lock_module decorator

When any protected module kicks off, it will check if the lock exists.
If it DOES exist, it will extract the PID from the file and do the following
If the PID is currently running
The command will exit because its locked.
If the PID isn’t running
It will delete the erroneous lock file
It will email jwql@stsci.edu to let us know that something happened with the previous run (no
action needed on our end, but will help us answer questions from scientists as to why their data is available)
It will start a fresh run
If there is a bad lock file without a PID in it
We will receive an email with instructions to manually investigate and the process will NOT currently run (this allows us to manually lock something for any reason we want by creating an empty lock file) We will also get emailed every time until the bad lock file is removed so we wont forget about it.

[BradleySappington]

Ready for review

[melanieclarke]

Testing locally, changing the notification address to my email, this looks good. It behaved appropriately for all conditions I checked:

• missing PID sent email
• running after a killed process sent email
• a successfully completed process removed its lock file
• running again while previous process was running did nothing

However, the test in test_protect_module.py needs to be modified. It currently sends a "No PID found" email every time it runs.

[BradleySappington]

@melanieclarke thanks for the test reminder!

[BradleySappington]

@melanieclarke - tests updated

[mfixstsci]

Thanks for reviewing and approving @melanieclarke. @BradleySappington are you okay with me updating the branch and merging? Do you have any other features to add?

[BradleySappington]

@mfixstsci - This is ready for merge