Thread safe logging #441
Merged
Thread safe logging #441
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When extracting features in parallel, conflicts may arise when writing to the log-file, corrupting the records. To prevent this from happening, use a QueueHandler in combination with a QueueListener to process log records for the log file. Log messages written to the console are not affected. Additionally, use dictConfig to configure the logging. Finally, do not catch the Keyboard and SystemExit errors.
Capture KeyboardInterrupts and SystemExit errors separately to allow for cancellation of batch jobs. In case of parallel processing, slightly more needs to be done: - Use map_async to start the parallel processing in a non-blocking mode - Use a while loop to check if the process is done and only then get the results. This allows the main process to react to a keyboard interrupt. - When the main process is cancelled, terminate the pool to stop further processing of jobs. - When extracting in serial, propagate the cancellation errors, but just return None in case of parallel extraction.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement thread-safe logging to a file by using a
QueueHandlerandQueueListenerto process the log records intended for the logfile. This ensures only 1 thread is writing to the file, preventing corruption in case of collisions.Additionally, improve the error catching code in the commandline script to allow for cancellation by the user (via
KeyboardInterrupt/SystemExitand in case of parallel processing; by callingpool.terminate())cc @Radiomics/developers