Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 5460: (SLTS) Add read index virtual block entries to metadata. #5461

Merged

Conversation

sachin-j-joshi
Copy link
Contributor

@sachin-j-joshi sachin-j-joshi commented Jan 5, 2021

Signed-off-by: Sachin Joshi sachin.joshi@emc.com

Change log description
Persistent custom skip list like sparse index entries.

Purpose of the change
Fixes #5460

What the code does
Add custom skip list index metadata using following logic

  • The segment is imagined to divided into fixed size virtual block ( defined in config say 64MB).
  • For each multiple of fixed block size (eg 1B, 2B, 3B etc) add a metadata entry to table store that points to the chunk that contains the first byte in that virtual block.
  • Use virtual block index entry during read.
  • Update write, delete concat , truncate logic to keep these up to date.

How to verify it
All tests must pass

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
… pinned segments as all metadata is in memory already.

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
@sachin-j-joshi sachin-j-joshi marked this pull request as ready for review January 20, 2021 22:42
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
@sachin-j-joshi sachin-j-joshi changed the title Issue 5460: Add read index virtual block entries to metadata. Issue 5460: (SLTS) Add read index virtual block entries to metadata. Feb 3, 2021
…e-5460-virtual-block-index

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>

# Conflicts:
#	segmentstore/storage/src/test/java/io/pravega/segmentstore/storage/chunklayer/ChunkedSegmentStorageTests.java
#	segmentstore/storage/src/test/java/io/pravega/segmentstore/storage/mocks/InMemorySimpleStorageTests.java
@codecov
Copy link

codecov bot commented Feb 4, 2021

Codecov Report

Merging #5461 (1466105) into master (8b88628) will decrease coverage by 0.00%.
The diff coverage is 91.11%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #5461      +/-   ##
============================================
- Coverage     84.78%   84.78%   -0.01%     
- Complexity    13905    13925      +20     
============================================
  Files           917      918       +1     
  Lines         51517    51635     +118     
  Branches       5312     5321       +9     
============================================
+ Hits          43677    43777     +100     
- Misses         4955     4968      +13     
- Partials       2885     2890       +5     
Impacted Files Coverage Δ Complexity Δ
...col/src/main/java/io/pravega/shared/NameUtils.java 89.57% <0.00%> (-0.56%) 65.00 <0.00> (ø)
...segmentstore/storage/chunklayer/ReadOperation.java 86.77% <81.81%> (-3.82%) 44.00 <3.00> (+4.00) ⬇️
...tore/storage/chunklayer/ChunkedSegmentStorage.java 90.02% <89.65%> (-0.12%) 118.00 <7.00> (+6.00) ⬇️
...segmentstore/storage/chunklayer/ChunkIterator.java 95.23% <90.90%> (-4.77%) 7.00 <3.00> (+1.00) ⬇️
...store/storage/metadata/ReadIndexBlockMetadata.java 90.90% <90.90%> (ø) 6.00 <6.00> (?)
...tstore/storage/chunklayer/DefragmentOperation.java 95.94% <95.83%> (-0.12%) 48.00 <3.00> (+3.00) ⬇️
...torage/chunklayer/ChunkedSegmentStorageConfig.java 98.24% <100.00%> (+0.09%) 3.00 <0.00> (ø)
...gmentstore/storage/chunklayer/ConcatOperation.java 92.30% <100.00%> (+0.38%) 28.00 <0.00> (ø)
...egmentstore/storage/chunklayer/ReadIndexCache.java 82.97% <100.00%> (+0.37%) 25.00 <0.00> (+1.00)
...entstore/storage/chunklayer/TruncateOperation.java 93.85% <100.00%> (+0.10%) 33.00 <0.00> (ø)
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b88628...1466105. Read the comment docs.

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
…e-5460-virtual-block-index

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>

# Conflicts:
#	segmentstore/storage/src/main/java/io/pravega/segmentstore/storage/chunklayer/ChunkedSegmentStorage.java
#	segmentstore/storage/src/main/java/io/pravega/segmentstore/storage/chunklayer/ConcatOperation.java
#	segmentstore/storage/src/main/java/io/pravega/segmentstore/storage/chunklayer/DefragmentOperation.java
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
@andreipaduroiu andreipaduroiu merged commit 579db02 into pravega:master Feb 24, 2021
sachin-j-joshi added a commit to sachin-j-joshi/pravega that referenced this pull request Feb 24, 2021
…ravega#5461)

Persistent custom skip list like sparse index entries.

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>

Co-authored-by: Andrei Paduroiu <andreipaduroiu@users.noreply.github.com>
co-jo pushed a commit to co-jo/pravega that referenced this pull request Mar 1, 2021
…ravega#5461)

Persistent custom skip list like sparse index entries.

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>

Co-authored-by: Andrei Paduroiu <andreipaduroiu@users.noreply.github.com>
co-jo pushed a commit to co-jo/pravega that referenced this pull request Mar 9, 2021
…ravega#5461)

Persistent custom skip list like sparse index entries.

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>

Co-authored-by: Andrei Paduroiu <andreipaduroiu@users.noreply.github.com>
sachin-j-joshi added a commit that referenced this pull request Mar 19, 2021
Cherry-picks following changes 0.9

Issue 5518: (SLTS) Fix Integer overflow. (#5520)
Issue 5606: BoundedInputStream::markSupported() should always return false.. (#5615)
Issue 5597 : ExtendedS3ChunkStorage and ExtendedS3Storage should close S3Client. (#5598)
Issue 5535: (SegmentStore) Refactoring ExtendedS3 Test Mocks (#5537)
Issue 5456: (SLTS) - Handle possible failure during deletion of metadata keys can cause version mismatch in next attempt (#5457)
Issue 5458: (SLTS) For read operation, read all chunks in parallel. (#5459)
Issue 4967: (SLTS) Add createWithContent overload that creates object and writes to it in a single call. (#5455)
Issue 5664: SLTS - fix possible thread visibility issues.. (#5665)
Issue 5475: Eliminate unnecessary pessimistic checks. (#5662)
Issue 5570: (SLTS) AsyncBaseChunkStorage latency should not include time spent in waiting to be executed. (#5661)
Issue 4967: (SLTS) No need to check last chunk length for claimOwnership for no append mode. (#5663)
Issue 5673: (SLTS) Fix wrong GC config (#5674)
Issue 5737: (SLTS) Improve metrics (#5746)
Issue 5460: (SLTS) Add read index virtual block entries to metadata. (#5461)
Issue 5772: (SLTS) - BaseMetadataStore does not evict entries from buffer. (#5773)
Issue 5798: (SLTS) Fix close in GarbageCollector, ExtendedS3ChunkStorage and HDFSChunkStorage. (#5800)
Issue 5788: (SLTS) Remove ACL operations from ExtendedS3ChunkStorage (#5790)
Issue 5808: SLTS - Fix Preconditions checks to include enough information in message. (#5809)
Issue 5853: SLTS - BaseMetadataStore.get does not return deep copy when loading from store. #5858
Issue 5866: (SLTS) ReadIndexCache stats not reported. (#5867)


Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
@sachin-j-joshi sachin-j-joshi deleted the issue-5460-virtual-block-index branch August 23, 2021 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(SLTS) Persistent custom skip list like sparse index entries.
2 participants