Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to cache mark and index files for S3 disk #13076

Merged
merged 10 commits into from Aug 7, 2020

Conversation

Jokser
Copy link
Contributor

@Jokser Jokser commented Jul 29, 2020

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added cache layer for DiskS3 (cache to local disk mark and index files).

Detailed description / Documentation draft:
Introduced cache decorator over an IDisk.
Cache decorator tries to cache files by given predicated to a provided local disk.
writeFile() first writes a file to cache and after writing is finished copies this file to underlying disk.
readFile() tries to read from a cache first. If the file doesn't exist it initiates downloading this file from underlying disk to cache and when downloading is finished - returns read buffer from the cache.
Files moving, renaming, deleting are preserved in cache layer as well to prevent cache pollution with stale files.

@robot-clickhouse robot-clickhouse added the pr-improvement Pull request with some product improvements label Jul 29, 2020
@akuzm
Copy link
Contributor

akuzm commented Aug 7, 2020

00933_test_fix_extra_seek_on_compressed_cache is in CI Fest #7
fuzzer failure is #13392

@akuzm akuzm merged commit 6a596d5 into ClickHouse:master Aug 7, 2020
@akuzm akuzm self-assigned this Aug 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-improvement Pull request with some product improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants