Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve query when no enough memory cache #847

Merged
merged 9 commits into from
May 30, 2023

Conversation

hengfeiyang
Copy link
Contributor

@hengfeiyang hengfeiyang commented May 30, 2023

We have some environments to control the memory cache:

option default value description
ZO_MEMORY_CACHE_ENABLED true you can disable memory cache, default is enable
ZO_MEMORY_CACHE_CACHE_LATEST_FILES false you can auto cache the latest files for accelerate query, default is false, this need more memory
ZO_MEMORY_CACHE_MAX_SIZE 30% default use 30% of total memory, you can configure it by a value (MB)

And if a query needs to load parquet files more than 30% of total memory cache, the query will skip memory cache.

For example:

I have 500MB memory cache.

  • a query needs to load 100MB parquet files, ok, let's use the memory cache.
  • a query needs to load >250MB parquet files, ok, too big, directly load files from storage. disk / s3.

@hengfeiyang hengfeiyang merged commit 23f4dd4 into main May 30, 2023
5 checks passed
@hengfeiyang hengfeiyang deleted the fix/query-object-store branch May 30, 2023 08:11
@hengfeiyang
Copy link
Contributor Author

fix #312

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants