z/OS: Invalid java.io.tmpdir causes joblog spool to fill, requiring IPL

[mattcolegate]

z/OS specific: Semaphores are now based on IPC keys, which require a backing file system to generate unique identifiers. Currently the place to create that backing file system, named .com_ibm_tools_hc, is taken from the value of the Java system property java.io.tmpdir.
If that property is set to an invalid, read-only or non-existent location, the file system is not created and the semaphore is also not created, which causes an error message to be output. Crucially though, the attempts to create a semaphore continue every few seconds, which continually fail and output error messages. If left unattended without checking, these error messages repeat until the entire job log is full, at which point the system grinds to a halt and an IPL is needed to restore the system to working order.
A possible solution is to check whether the directory pointed to by java.io.tmpdir exists and is writable before attempting semaphore creation, falling through to the default /tmp if it does not. A numerical guard should also be placed around semaphore creation, stopping and exiting the agent if semaphores can't be created after a certain amount of time / number of attempts.
Example of message seen in logs:

WARNING: Failed to create directory /<java.io.tmpdir>/.com_ibm_tools_hc; semaphore key not created