-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make SegmentLoader extensible and customizable #11398
Make SegmentLoader extensible and customizable #11398
Conversation
This pull request fixes 1 alert when merging 0af368e into 8037a54 - view on LGTM.com fixed alerts:
|
This pull request fixes 1 alert when merging 92cbce4 into 497f2a1 - view on LGTM.com fixed alerts:
|
This pull request introduces 2 alerts and fixes 2 when merging 0e8d0bf into 497f2a1 - view on LGTM.com new alerts:
fixed alerts:
|
This pull request introduces 2 alerts and fixes 2 when merging 0408a52 into 17efa6f - view on LGTM.com new alerts:
fixed alerts:
|
This pull request introduces 2 alerts and fixes 2 when merging 2e6fd24 into d5e8d4d - view on LGTM.com new alerts:
fixed alerts:
|
{ | ||
final ReferenceCountingLock lock = createOrGetLock(segment); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the lock acquisition has been removed from here since getSegmentFiles
already does it.
server/src/main/java/org/apache/druid/segment/loading/StorageLocation.java
Outdated
Show resolved
Hide resolved
* @return Segment object wrapped inside {@link ReferenceCountingSegment}. | ||
* @throws SegmentLoadingException | ||
*/ | ||
ReferenceCountingSegment getSegment(DataSegment segment, boolean lazy, SegmentLazyLoadFailCallback loadFailed) throws SegmentLoadingException; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the SegmentLoader guaranteed to return the same ReferenceCountingSegment instance across multiple calls of getSegment? Should it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can do either and the caller is not supposed to depend on that behavior. From the caller's perspective, it is going to get a segment object wrapped inside ReferenceCountingSegment
. Implementations can have optimizations to save on repeated expensive work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be documented in the javadoc.
server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java
Outdated
Show resolved
Hide resolved
* @param segment - Segment to release the location for. | ||
* @return - True if any location was reserved and released, false otherwise. | ||
*/ | ||
boolean release(DataSegment segment); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What calls release()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the same code that calls reserve
can call release
. Idea is that if reserve
is being called explicitly then, same should be done for release
. In case of failures, SegmentLoader
itself should not release the location and leave it to the caller instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @abhishekagarwal87, I left some comments mostly about the interface change.
server/src/main/java/org/apache/druid/segment/loading/SegmentLoader.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/druid/segment/loading/StorageLocation.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/apache/druid/segment/loading/SegmentLoader.java
Outdated
Show resolved
Hide resolved
This PR splits current SegmentLoader into SegmentLoader and SegmentCacheManager. SegmentLoader - this class is responsible for building the segment object but does not expose any methods for downloading, cache space management, etc. Default implementation delegates the download operations to SegmentCacheManager and only contains the logic for building segments once downloaded. . This class will be used in SegmentManager to construct Segment objects. SegmentCacheManager - this class manages the segment cache on the local disk. It fetches the segment files to the local disk, can clean up the cache, and in the future, support reserve and release on cache space. [See https://github.com/Make SegmentLoader extensible and customizable #11398]. This class will be used in ingestion tasks such as compaction, re-indexing where segment files need to be downloaded locally.
This pull request introduces 2 alerts and fixes 2 when merging dd80d1cd3ebd5a81220f5e2ddac722e7c9226223 into 0453e46 - view on LGTM.com new alerts:
fixed alerts:
|
dd80d1c
to
b1c874b
Compare
This pull request introduces 2 alerts and fixes 2 when merging b1c874b into 6ce3b6c - view on LGTM.com new alerts:
fixed alerts:
|
* @param segment - Segment to release the location for. | ||
* @return - True if any location was reserved and released, false otherwise. | ||
*/ | ||
boolean release(DataSegment segment); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you clarify the contract between this method and getSegmentFiles
? For example, what should happen when release
is called if reserve
was not called but getSegmentFiles
was called?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. if reserve
is not called, getSegmentFiles
will reserve
the space. I will document this.
* {@link StorageLocation} since we don't want callers to operate on {@code StorageLocation} directly outside {@code SegmentLoader}. | ||
* {@link SegmentLoader} operates on the {@code StorageLocation} objects in a thread-safe manner. | ||
*/ | ||
boolean reserve(DataSegment segment); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should isSegmentCached
still return false after reserve
is called? Would be worth to document it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. will document it.
* @return Segment object wrapped inside {@link ReferenceCountingSegment}. | ||
* @throws SegmentLoadingException | ||
*/ | ||
ReferenceCountingSegment getSegment(DataSegment segment, boolean lazy, SegmentLazyLoadFailCallback loadFailed) throws SegmentLoadingException; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be documented in the javadoc.
cleanupCacheFiles(loc.getPath(), storageDir); | ||
} | ||
boolean success = loadInLocationWithStartMarkerQuietly(loc, segment, storageDir, true); | ||
if (success) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems that loc.release
should be called when success
is false? Please add some test to verify this behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
loadInLocationWithStartMarkerQuietly(loc, segment, storageDir, true);
this method will release the location since true
is passed as the value of releaseLocation
flag.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah got it. Thanks.
public synchronized boolean isReserved(String segmentDir) | ||
{ | ||
final File segmentFile = new File(path, segmentDir); | ||
return files.contains(segmentFile); | ||
} | ||
|
||
public File segmentDirectoryAsFile(String segmentDir) | ||
{ | ||
return new File(path, segmentDir); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The LGTM error in https://lgtm.com/projects/g/apache/druid/rev/pr-1ff2ba29372f1d2b44941bb55f75b5830f808401 seems like a false alarm. Perhaps we should suppress it for this change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @abhishekagarwal87.
This pull request introduces 2 alerts and fixes 2 when merging a549256 into 167c452 - view on LGTM.com new alerts:
fixed alerts:
|
This pull request introduces 1 alert and fixes 2 when merging 89cd7e5 into 167c452 - view on LGTM.com new alerts:
fixed alerts:
|
LGTM error has been suppressed but it will take effect after the PR gets merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Description
This PR refactors the code related to segment loading specifically SegmentLoader and SegmentLoaderLocalCacheManager. SegmentLoader is marked
UnstableAPI
which means, it can be extended outside core druid in custom extensions. Here is a summary of changesSegmentLoader
returns an instance ofReferenceCountingSegment
instead ofSegment
. Earlier,SegmentManager
was wrappingSegment
objects insideReferenceCountingSegment
. That is now moved toSegmentLoader
. With this, a custom implementation can track the references of segments. It also allows them to create customReferenceCountingSegment
implementations. For this reason, the constructor visibility inReferenceCountingSegment
is changed fromprivate
toprotected
.SegmentCacheManager
has two additional methods called -reserve(DataSegment)
andrelease(DataSegment)
. These methods let the caller reserve or release space without callingSegmentLoader#getSegment
. We already had similar methods inStorageLocation
and now they are available inSegmentCacheManager
too which wraps multiple locations.SegmentCacheManager
wherever possible. There is no change in the functionality.Key changed/added classes in this PR
SegmentLoader
SegmentLoaderLocalCacheManager
StorageLocation
This PR has: