Adding cleaning up of the resources owned by the terminology service. #1391
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…(e.g. connection pool and cache manager) at the JVM shutdown.
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #1391 +/- ##
============================================
- Coverage 85.54% 85.53% -0.02%
Complexity 139 139
============================================
Files 338 339 +1
Lines 7784 7806 +22
Branches 517 519 +2
============================================
+ Hits 6659 6677 +18
- Misses 835 840 +5
+ Partials 290 289 -1
☔ View full report in Codecov by Sentry. |
johngrimes
requested changes
May 3, 2023
There was a problem hiding this comment.
Thanks for your work on this @piotrszul!
I've added come comments, would you mind taking a look?
terminology/src/main/java/au/csiro/pathling/terminology/DefaultTerminologyServiceFactory.java
Outdated
Show resolved
Hide resolved
...src/main/java/au/csiro/pathling/terminology/caching/PersistentCachingTerminologyService.java
Outdated
Show resolved
Hide resolved
| * @author Piotr Szul | ||
| */ | ||
| @Slf4j | ||
| public class ResourceHolder implements Closeable { |
There was a problem hiding this comment.
I think this class should be called something that is a bit more specific and communicates its purpose - perhaps CloseableResourceHolder?
There was a problem hiding this comment.
I think that 'Resource' implies an object, which lifecycle needs to me managed (in particular that needs to be closes).
This is based on the semantics of the try-with-resource Statement (see: https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html)
So I would leave it as is.
There was a problem hiding this comment.
Renamed to ResourceCloser.
| * @param <T> the type of the resource | ||
| * @return the resource | ||
| */ | ||
| protected <T extends Closeable> T registerResource(@Nonnull final T resource) { |
There was a problem hiding this comment.
Perhaps this could be renamed to registerCloseableResource?
| log.debug("Closing {} resources for: {}", resourcesToClose.size(), this); | ||
| synchronized (resourcesToClose) { | ||
| for (Closeable closeable : resourcesToClose) { | ||
| log.debug("Closing resource: {} in: {}", closeable, this); |
There was a problem hiding this comment.
Should I be seeing this logging if I use a Python script with something like the following?
spark = (
SparkSession.builder
.config("spark.jars", find_jar())
.getOrCreate()
)
pc = PathlingContext.create(
spark,
terminology_server_url="https://r4.ontoserver.csiro.au/fhir",
cache_override_expiry=2_628_000,
cache_storage_type="disk",
cache_storage_path=".local/tx-cache"
)
pc.spark.sparkContext.setLogLevel("DEBUG")There was a problem hiding this comment.
Yes, at the end:
23/06/20 14:34:43 INFO ShutdownHookManager: Shutdown hook called
23/06/20 14:34:43 INFO ShutdownHookManager: Deleting directory /private/var/folders/x5/cw9kr5ld00g595qv0gg7lxrw00cjzw/T/spark-17e8a874-d7f2-44cb-bb94-d8f8b83df7c9
23/06/20 14:34:43 INFO ShutdownHookManager: Deleting directory /private/var/folders/x5/cw9kr5ld00g595qv0gg7lxrw00cjzw/T/spark-5c43a9fb-16b2-4554-b414-a79937f7c20e
23/06/20 14:34:43 INFO ShutdownHookManager: Deleting directory /private/var/folders/x5/cw9kr5ld00g595qv0gg7lxrw00cjzw/T/spark-17e8a874-d7f2-44cb-bb94-d8f8b83df7c9/pyspark-a21b3294-186f-45e2-af3f-02f5c0291829
23/06/20 14:34:43 DEBUG ResourceHolder: Closing 2 resources for: au.csiro.pathling.terminology.caching.InMemoryCachingTerminologyService@6aeae867
23/06/20 14:34:43 DEBUG ResourceHolder: Closing resource: org.apache.http.impl.client.InternalHttpClient@6487e431 in: au.csiro.pathling.terminology.caching.InMemoryCachingTerminologyService@6aeae867
23/06/20 14:34:43 DEBUG PoolingHttpClientConnectionManager: Connection manager is shutting down
23/06/20 14:34:43 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: Close connection
23/06/20 14:34:43 DEBUG PoolingHttpClientConnectionManager: Connection manager shut down
23/06/20 14:34:43 DEBUG ResourceHolder: Closing resource: DefaultCacheManager DefaultCacheManager in: au.csiro.pathling.terminology.caching.InMemoryCachingTerminologyService@6aeae867
23/06/20 14:34:43 DEBUG DefaultCacheManager: Stopping cache manager DefaultCacheManager
23/06/20 14:34:43 DEBUG CacheImpl: Stopping cache lookup on DefaultCacheManager
23/06/20 14:34:43 DEBUG CacheImpl: Stopping cache org.infinispan.CONFIG on DefaultCacheManager
23/06/20 14:34:43 DEBUG CacheImpl: Stopping cache validate-code on DefaultCacheManager
23/06/20 14:34:43 DEBUG CacheImpl: Stopping cache translate on DefaultCacheManager
23/06/20 14:34:43 DEBUG CacheImpl: Stopping cache subsumes on DefaultCacheManager
23/06/20 14:34:43 DEBUG DefaultCacheManager: Stopped cache manager DefaultCacheManager
23/06/20 14:34:43 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:529)); Key: (szu004 (auth:SIMPLE))@file://; URI: file:///; Object Identity Hash: 597778cb
23/06/20 14:34:43 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.RawLocalFileSystem.close(RawLocalFileSystem.java:759)); Key: null; URI: file:///; Object Identity Hash: 77576ebb
23/06/20 14:34:43 DEBUG ShutdownHookManager: Completed shutdown in 0.122 seconds; Timeouts: 0
23/06/20 14:34:43 DEBUG ShutdownHookManager: ShutdownHookManager completed shutdown.
… the JVM shutdown hook. Removed unnecessary debug logging.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #1283.