Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*wip* Query cache persistence #63091

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rschu1ze
Copy link
Member

@rschu1ze rschu1ze commented Apr 28, 2024

Fixes: #58228
Fixes: #52141

30% done, not review-able yet. Uploading it as PR only in case my laptop catches fire.

The server now stores/loads the content of the query cache (i.e. cached query results) to/from disk during server shutdown/startup.

The data is stored in subdirectory query_cache/ relative to the base directory (<path>...</path>) in the server configuration).

To support future changes of the persistence format, a file query_cache/format_version.txt stores the version of the data, e.g. 1. Backward compatibility will likely not be implemented, i.e. future ClickHouse versions which support query cache persistence format versions > 1 will not be able to read earlier formats (existing data will simply be deleted).

In format v1, query cache entries are serialized like this:

  • For each query result, a sub-folder is created, with the AST tree hash as folder name, e.g. query_cache/496d3a8d63e7/.
  • [...]

Query cache serialization/deserialization can be enabled using a server setting (default: true):

<query_cache>
    <persist_cache>true</persist_cache>
</query_cache>

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

The server now stores/loads the content of the query cache (i.e.
cached query results) to/from disk during server shutdown/startup.

The data is stored in subdirectory "query_cache/" relative to the base
directory (<path>...</path> in the server configuration).

To support future changes of the persistence format, a file
"query_cache/format_version.txt" stores the version of the data, e.g.
"1". Backward compatibility will likely not be implemented, i.e. future
ClickHouse versions which support query cache persistence format
versions > 1 will not be able to read earlier formats (existing data
will simply be deleted).

In format v1, query cache entries are serialized like this:
- For each query result, a sub-folder is created, with the AST tree hash
  as folder name, e.g. "query_cache/496d3a8d63e7/".
- [...]

Query cache serialization/deserialization can be enabled using a server
setting (default: true):

<query_cache>
    <persist_cache>true</persist_cache>
</query_cache>
@rschu1ze rschu1ze added the do not test disable testing on pull request label Apr 28, 2024
@robot-ch-test-poll2 robot-ch-test-poll2 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Apr 28, 2024
@robot-ch-test-poll2
Copy link
Contributor

robot-ch-test-poll2 commented Apr 28, 2024

This is an automated comment for commit 993cbbd with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks
Check nameDescriptionStatus
CI runningA meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR✅ success
Docs checkBuilds and tests the documentation✅ success
Mergeable CheckChecks if all other necessary checks are successful✅ success
PR CheckThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success

@kssenii kssenii self-assigned this Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do not test disable testing on pull request pr-not-for-changelog This PR should not be mentioned in the changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Query cache on disk (RFC) Allow using cache disk for query result cache.
3 participants