Make SegmentLoader extensible and customizable #11398

abhishekagarwal87 · 2021-06-30T09:04:58Z

Description

This PR refactors the code related to segment loading specifically SegmentLoader and SegmentLoaderLocalCacheManager. SegmentLoader is marked UnstableAPI which means, it can be extended outside core druid in custom extensions. Here is a summary of changes

SegmentLoader returns an instance of ReferenceCountingSegment instead of Segment. Earlier, SegmentManager was wrapping Segment objects inside ReferenceCountingSegment. That is now moved to SegmentLoader. With this, a custom implementation can track the references of segments. It also allows them to create custom ReferenceCountingSegment implementations. For this reason, the constructor visibility in ReferenceCountingSegment is changed from private to protected.
SegmentCacheManager has two additional methods called - reserve(DataSegment) and release(DataSegment). These methods let the caller reserve or release space without calling SegmentLoader#getSegment. We already had similar methods in StorageLocation and now they are available in SegmentCacheManager too which wraps multiple locations.
Refactoring to simplify the code in SegmentCacheManager wherever possible. There is no change in the functionality.

Key changed/added classes in this PR

SegmentLoader
SegmentLoaderLocalCacheManager
StorageLocation

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

lgtm-com · 2021-06-30T10:14:19Z

This pull request fixes 1 alert when merging 0af368e into 8037a54 - view on LGTM.com

fixed alerts:

1 for Uncontrolled data used in path expression

lgtm-com · 2021-07-06T11:09:48Z

This pull request fixes 1 alert when merging 92cbce4 into 497f2a1 - view on LGTM.com

fixed alerts:

1 for Uncontrolled data used in path expression

lgtm-com · 2021-07-06T13:54:03Z

This pull request introduces 2 alerts and fixes 2 when merging 0e8d0bf into 497f2a1 - view on LGTM.com

new alerts:

2 for Uncontrolled data used in path expression

fixed alerts:

2 for Uncontrolled data used in path expression

lgtm-com · 2021-07-06T20:00:21Z

This pull request introduces 2 alerts and fixes 2 when merging 0408a52 into 17efa6f - view on LGTM.com

new alerts:

2 for Uncontrolled data used in path expression

fixed alerts:

2 for Uncontrolled data used in path expression

lgtm-com · 2021-07-07T10:54:52Z

This pull request introduces 2 alerts and fixes 2 when merging 2e6fd24 into d5e8d4d - view on LGTM.com

new alerts:

2 for Uncontrolled data used in path expression

fixed alerts:

2 for Uncontrolled data used in path expression

abhishekagarwal87 · 2021-07-12T09:14:34Z

server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java

  {
-    final ReferenceCountingLock lock = createOrGetLock(segment);


the lock acquisition has been removed from here since getSegmentFiles already does it.

server/src/main/java/org/apache/druid/segment/loading/StorageLocation.java

jon-wei · 2021-07-12T21:22:10Z

server/src/main/java/org/apache/druid/segment/loading/SegmentLoader.java

+   * @return Segment object wrapped inside {@link ReferenceCountingSegment}.
+   * @throws SegmentLoadingException
+   */
+  ReferenceCountingSegment getSegment(DataSegment segment, boolean lazy, SegmentLazyLoadFailCallback loadFailed) throws SegmentLoadingException;


Is the SegmentLoader guaranteed to return the same ReferenceCountingSegment instance across multiple calls of getSegment? Should it?

It can do either and the caller is not supposed to depend on that behavior. From the caller's perspective, it is going to get a segment object wrapped inside ReferenceCountingSegment. Implementations can have optimizations to save on repeated expensive work.

I think this should be documented in the javadoc.

server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java

jon-wei · 2021-07-12T21:33:05Z

server/src/main/java/org/apache/druid/segment/loading/SegmentLoader.java

+   * @param segment - Segment to release the location for.
+   * @return - True if any location was reserved and released, false otherwise.
+   */
+  boolean release(DataSegment segment);


What calls release()?

the same code that calls reserve can call release. Idea is that if reserve is being called explicitly then, same should be done for release. In case of failures, SegmentLoader itself should not release the location and leave it to the caller instead.

jihoonson

Thanks @abhishekagarwal87, I left some comments mostly about the interface change.

server/src/main/java/org/apache/druid/segment/loading/SegmentLoader.java

server/src/main/java/org/apache/druid/segment/loading/StorageLocation.java

server/src/main/java/org/apache/druid/segment/loading/SegmentLoader.java

This PR splits current SegmentLoader into SegmentLoader and SegmentCacheManager. SegmentLoader - this class is responsible for building the segment object but does not expose any methods for downloading, cache space management, etc. Default implementation delegates the download operations to SegmentCacheManager and only contains the logic for building segments once downloaded. . This class will be used in SegmentManager to construct Segment objects. SegmentCacheManager - this class manages the segment cache on the local disk. It fetches the segment files to the local disk, can clean up the cache, and in the future, support reserve and release on cache space. [See https://github.com/Make SegmentLoader extensible and customizable #11398]. This class will be used in ingestion tasks such as compaction, re-indexing where segment files need to be downloaded locally.

lgtm-com · 2021-07-21T06:53:07Z

This pull request introduces 2 alerts and fixes 2 when merging dd80d1cd3ebd5a81220f5e2ddac722e7c9226223 into 0453e46 - view on LGTM.com

new alerts:

2 for Uncontrolled data used in path expression

fixed alerts:

2 for Uncontrolled data used in path expression

lgtm-com · 2021-07-21T09:23:58Z

This pull request introduces 2 alerts and fixes 2 when merging b1c874b into 6ce3b6c - view on LGTM.com

new alerts:

2 for Uncontrolled data used in path expression

fixed alerts:

2 for Uncontrolled data used in path expression

jihoonson · 2021-07-21T21:58:39Z

server/src/main/java/org/apache/druid/segment/loading/SegmentCacheManager.java

+   * @param segment - Segment to release the location for.
+   * @return - True if any location was reserved and released, false otherwise.
+   */
+  boolean release(DataSegment segment);


Can you clarify the contract between this method and getSegmentFiles? For example, what should happen when release is called if reserve was not called but getSegmentFiles was called?

Sure. if reserve is not called, getSegmentFiles will reserve the space. I will document this.

jihoonson · 2021-07-21T21:59:48Z

server/src/main/java/org/apache/druid/segment/loading/SegmentCacheManager.java

+   * {@link StorageLocation} since we don't want callers to operate on {@code StorageLocation} directly outside {@code SegmentLoader}.
+   * {@link SegmentLoader} operates on the {@code StorageLocation} objects in a thread-safe manner.
+   */
+  boolean reserve(DataSegment segment);


Should isSegmentCached still return false after reserve is called? Would be worth to document it.

Yes. will document it.

jihoonson · 2021-07-21T22:09:37Z

server/src/main/java/org/apache/druid/segment/loading/SegmentLoader.java

+   * @return Segment object wrapped inside {@link ReferenceCountingSegment}.
+   * @throws SegmentLoadingException
+   */
+  ReferenceCountingSegment getSegment(DataSegment segment, boolean lazy, SegmentLazyLoadFailCallback loadFailed) throws SegmentLoadingException;


I think this should be documented in the javadoc.

jihoonson · 2021-07-21T22:24:42Z

server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java

-            cleanupCacheFiles(loc.getPath(), storageDir);
-          }
+        boolean success = loadInLocationWithStartMarkerQuietly(loc, segment, storageDir, true);
+        if (success) {


Seems that loc.release should be called when success is false? Please add some test to verify this behavior.

loadInLocationWithStartMarkerQuietly(loc, segment, storageDir, true);
this method will release the location since true is passed as the value of releaseLocation flag.

Ah got it. Thanks.

jihoonson · 2021-07-21T22:28:16Z

server/src/main/java/org/apache/druid/segment/loading/StorageLocation.java

+  public synchronized boolean isReserved(String segmentDir)
+  {
+    final File segmentFile = new File(path, segmentDir);
+    return files.contains(segmentFile);
+  }
+
+  public File segmentDirectoryAsFile(String segmentDir)
+  {
+    return new File(path, segmentDir);
+  }
+


The LGTM error in https://lgtm.com/projects/g/apache/druid/rev/pr-1ff2ba29372f1d2b44941bb55f75b5830f808401 seems like a false alarm. Perhaps we should suppress it for this change.

jihoonson

LGTM. Thanks @abhishekagarwal87.

lgtm-com · 2021-07-22T06:35:37Z

This pull request introduces 2 alerts and fixes 2 when merging a549256 into 167c452 - view on LGTM.com

new alerts:

2 for Uncontrolled data used in path expression

fixed alerts:

2 for Uncontrolled data used in path expression

lgtm-com · 2021-07-22T08:11:42Z

This pull request introduces 1 alert and fixes 2 when merging 89cd7e5 into 167c452 - view on LGTM.com

new alerts:

1 for Uncontrolled data used in path expression

fixed alerts:

2 for Uncontrolled data used in path expression

abhishekagarwal87 · 2021-07-22T08:24:38Z

LGTM error has been suppressed but it will take effect after the PR gets merged.

maytasm

LGTM

Add ability to reserve a location without downloading segment

0af368e

abhishekagarwal87 marked this pull request as draft June 30, 2021 12:12

abhishekagarwal87 marked this pull request as ready for review July 1, 2021 06:00

comments

92cbce4

abhishekagarwal87 changed the title ~~[Draft] Segment loader changes~~ [Test] Segment loader changes Jul 5, 2021

abhishekagarwal87 added 2 commits July 6, 2021 17:03

Add reserve and release methods

1153abb

Mark SegmentLoader unstableAPI

f62af2d

abhishekagarwal87 changed the title ~~[Test] Segment loader changes~~ Make SegmentLoader extensible and customizable Jul 6, 2021

abhishekagarwal87 added Design Review Refactoring Release Notes Area - Segment Balancing/Coordination labels Jul 6, 2021

Fix test compile

0e8d0bf

Add tests

0408a52

Fix test

2e6fd24

abhishekagarwal87 commented Jul 12, 2021

View reviewed changes

jon-wei reviewed Jul 12, 2021

View reviewed changes

jihoonson reviewed Jul 13, 2021

View reviewed changes

abhishekagarwal87 mentioned this pull request Jul 19, 2021

SegmentLoader refactoring #11466

Merged

3 tasks

abhishekagarwal87 added 5 commits July 19, 2021 14:57

SegmentLoader refactoring

215c0f0

Test compilation failures in indexing service

47e5c65

Fix sql tests

1482561

Add docs and typos

bbba704

Merge with segment loader refactoring

c7ce26d

abhishekagarwal87 added 2 commits July 21, 2021 00:16

Merge branch 'master' of github.com:apache/druid into segment-loader-2

cc68ede

Revert some changes in StorageLocation

2873386

Fix compilation

b1c874b

abhishekagarwal87 force-pushed the segment-loader branch from dd80d1c to b1c874b Compare July 21, 2021 08:11

jihoonson reviewed Jul 21, 2021

View reviewed changes

javadoc

a549256

jihoonson approved these changes Jul 22, 2021

View reviewed changes

Suppress LGTM errors

89cd7e5

maytasm approved these changes Jul 22, 2021

View reviewed changes

abhishekagarwal87 merged commit ce1faa5 into apache:master Jul 22, 2021

clintropolis added this to the 0.22.0 milestone Aug 12, 2021

clintropolis mentioned this pull request Sep 3, 2021

[Draft] 0.22.0 Release Notes #11657

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make SegmentLoader extensible and customizable #11398

Make SegmentLoader extensible and customizable #11398

abhishekagarwal87 commented Jun 30, 2021 •

edited

Loading

lgtm-com bot commented Jun 30, 2021

lgtm-com bot commented Jul 6, 2021

lgtm-com bot commented Jul 6, 2021

lgtm-com bot commented Jul 6, 2021

lgtm-com bot commented Jul 7, 2021

abhishekagarwal87 Jul 12, 2021

jon-wei Jul 12, 2021

abhishekagarwal87 Jul 13, 2021

jihoonson Jul 21, 2021

jon-wei Jul 12, 2021

abhishekagarwal87 Jul 13, 2021

jihoonson left a comment

lgtm-com bot commented Jul 21, 2021

lgtm-com bot commented Jul 21, 2021

jihoonson Jul 21, 2021

abhishekagarwal87 Jul 22, 2021

jihoonson Jul 21, 2021

abhishekagarwal87 Jul 22, 2021

jihoonson Jul 21, 2021

jihoonson Jul 21, 2021

abhishekagarwal87 Jul 22, 2021

jihoonson Jul 22, 2021

jihoonson Jul 21, 2021

jihoonson left a comment

lgtm-com bot commented Jul 22, 2021

lgtm-com bot commented Jul 22, 2021

abhishekagarwal87 commented Jul 22, 2021

maytasm left a comment

		{
		final ReferenceCountingLock lock = createOrGetLock(segment);

Make SegmentLoader extensible and customizable #11398

Make SegmentLoader extensible and customizable #11398

Conversation

abhishekagarwal87 commented Jun 30, 2021 • edited Loading

Description

Key changed/added classes in this PR

lgtm-com bot commented Jun 30, 2021

lgtm-com bot commented Jul 6, 2021

lgtm-com bot commented Jul 6, 2021

lgtm-com bot commented Jul 6, 2021

lgtm-com bot commented Jul 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jihoonson left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Jul 21, 2021

lgtm-com bot commented Jul 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jihoonson left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Jul 22, 2021

lgtm-com bot commented Jul 22, 2021

abhishekagarwal87 commented Jul 22, 2021

maytasm left a comment

Choose a reason for hiding this comment

abhishekagarwal87 commented Jun 30, 2021 •

edited

Loading