Add option for keeping local cache of recent data. #585

trueleo · 2023-12-22T16:09:12Z

Description

This PR adds local cache / hot tier for Parseable.

This option can be enabled by setting following env vars
P_CACHE_DIR - Local Path for file cache
P_CACHE_SIZE - Size for cache in human readable size ( mb/mib/gib/gb )

When these flags are set, sync flow will move the parquet from staging into the specified cache directory instead of deleting it. Any LRU cached entry is deleted to satisfy the cache constraint upon insertion. LocalCacheManager is responsible for updating and persisting this data structure.

This PR has:

been tested to ensure log ingestion and log query works.
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added documentation for new or modified features or behaviors.

server/src/option.rs

nitisht · 2023-12-23T12:14:48Z

server/src/query/table_provider.rs

@@ -1,140 +0,0 @@
-/*


Why deleted?

This file is unused now

Let's create a separate cleanup PR with details on why files are not used. this change is clearly unrelated to caching.

This change is clearly unrelated to caching.

This file was being used to combine two execution plans ( remote and memory ), but now because of conditional addition of caching there are now three plans to combine. I had to refactor related code in the schema provider file.

Please fix commit message with details

nitisht

Fix the commit message with more details on why we added this and how it works.

nitisht · 2023-12-24T05:03:34Z

Missing items

Prometheus stats on cache as we discussed.
Cache info printed on the startup banner under storage.

nitisht · 2023-12-24T05:07:14Z

What does this mapping in the .cache.json file mean?

[ 
"home/nitish/Development/parseablehq/parseable/data/teststream/date=2023-12-24/hour=04/minute=57/nitish-x1.data.parquet",
"/home/nitish/Development/parseablehq/parseable/cache/teststream/date=2023-12-24.hour=04.minute=57.nitish-x1.data.parquet"
]

nitisht · 2023-12-24T05:20:45Z

If I restart the server and change the cache size via P_CACHE_SIZE, the .cache.json still has old value. We need to update the .cache.json so current specified size is honoured.

theteachr · 2023-12-24T05:41:49Z

What does this mapping in the .cache.json file mean?

@nitisht It's a mapping from store path to where the file is stored in the local cache.

nitisht · 2023-12-24T05:53:24Z

What does this mapping in the .cache.json file mean?

[ 
"home/nitish/Development/parseablehq/parseable/data/teststream/date=2023-12-24/hour=04/minute=57/nitish-x1.data.parquet",
"/home/nitish/Development/parseablehq/parseable/cache/teststream/date=2023-12-24.hour=04.minute=57.nitish-x1.data.parquet"
]

What does this mapping in the .cache.json file mean?

@nitisht It's a mapping from store path to where the file is stored in the local cache.

That is clear, but

Both files are local, still there is difference in path format - leading / vs no /
Where do we use this mapping - that info should be added as comments and commit message

trueleo · 2023-12-25T06:09:37Z

Missing items

* Prometheus stats on cache as we discussed.

* Cache info printed on the startup banner under storage.

I will do this in a separate PR.

nitisht · 2023-12-25T06:10:16Z

Missing items

* Prometheus stats on cache as we discussed.

* Cache info printed on the startup banner under storage.

I will do this in a separate PR.

Let's finish in this PR

This PR adds local cache / hot tier for Parseable. This option can be enabled by setting following env vars P_CACHE_DIR - Local Path for file cache P_CACHE_SIZE - Size for cache in human readable size ( mb/mib/gib/gb ) When these flags are set, sync flow will move the parquet from staging into the specified cache directory instead of deleting it. Any LRU cached entry is deleted to satisfy the cache constraint upon insertion. LocalCacheManager is responsible for updating and persisting this data structure.

server/src/option.rs

server/src/storage/s3.rs

nitisht · 2023-12-25T07:33:32Z

server/src/localcache.rs

+            version: "v1".to_string(),
+            current_size: 0,
+            capacity,
+            files: Cache::new(100),


Why 100 only here?

backing hashlru is capacity based. The capacity is arbitrary and is increased when needed.

who increases? user or the backing cache?

nitisht · 2023-12-25T07:36:56Z

Prometheus stats on cache as we discussed.
Cache info printed on the startup banner under storage.
Both files are local, still there is difference in path format - leading / vs no /
Where do we use this mapping - that info should be added as comments and commit message
If I restart the server and change the cache size via P_CACHE_SIZE, the .cache.json still has old value. We need to update the .cache.json so current specified size is honoured.
What is the minimum allowed value for P_CACHE_SIZE? We need to fail if smaller values are set. IMO 1GiB is a good number.

trueleo changed the title ~~Localcache~~ Add option for keeping local cache of recent data. Dec 22, 2023

nitisht requested review from theteachr and nitisht December 22, 2023 16:20

nitisht reviewed Dec 23, 2023

View reviewed changes

nitisht requested changes Dec 23, 2023

View reviewed changes

nitisht requested a review from nikhilsinhaparseable December 24, 2023 05:35

trueleo force-pushed the localcache branch from 56b0ce8 to 6f9a362 Compare December 25, 2023 06:23

trueleo force-pushed the localcache branch from 6f9a362 to e01ceac Compare December 25, 2023 06:31

nitisht reviewed Dec 25, 2023

View reviewed changes

server/src/option.rs Outdated Show resolved Hide resolved

nitisht reviewed Dec 25, 2023

View reviewed changes

server/src/storage/s3.rs Outdated Show resolved Hide resolved

trueleo added 3 commits December 25, 2023 12:42

Add banner

1b722da

Remove unused

5ff79be

Add more parse option

e7fec8e

nitisht reviewed Dec 25, 2023

View reviewed changes

Fix banner

cbdee0b

theteachr mentioned this pull request Dec 25, 2023

Fix help messages for environment variables / flags #588

Merged

3 tasks

trueleo added 5 commits December 25, 2023 18:15

Add Counter for cache hit

b659668

Add comment

8c04230

Add size requirement

4c8037e

Use capacity from cache manager

334fc2b

Rename

9d99043

trueleo added 5 commits December 26, 2023 12:08

Disable cache for local-store

8ca5f8a

refactor

bd41dce

Dynamically set local cache for stream

fba9260

Add perms

7975ac3

Change action

bebdbf2

nitisht approved these changes Dec 27, 2023

View reviewed changes

nitisht merged commit 631c8f4 into parseablehq:main Dec 27, 2023
6 checks passed

github-actions bot locked and limited conversation to collaborators Dec 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option for keeping local cache of recent data. #585

Add option for keeping local cache of recent data. #585

trueleo commented Dec 22, 2023 •

edited

nitisht Dec 23, 2023

trueleo Dec 25, 2023

nitisht Dec 25, 2023

trueleo Dec 25, 2023 •

edited

nitisht Dec 25, 2023

nitisht left a comment

nitisht commented Dec 24, 2023

nitisht commented Dec 24, 2023

nitisht commented Dec 24, 2023

theteachr commented Dec 24, 2023 •

edited

nitisht commented Dec 24, 2023 •

edited

trueleo commented Dec 25, 2023

nitisht commented Dec 25, 2023

nitisht Dec 25, 2023

trueleo Dec 25, 2023

nitisht Dec 25, 2023

nitisht commented Dec 25, 2023

Add option for keeping local cache of recent data. #585

Add option for keeping local cache of recent data. #585

Conversation

trueleo commented Dec 22, 2023 • edited

Description

nitisht Dec 23, 2023

Choose a reason for hiding this comment

trueleo Dec 25, 2023

Choose a reason for hiding this comment

nitisht Dec 25, 2023

Choose a reason for hiding this comment

trueleo Dec 25, 2023 • edited

Choose a reason for hiding this comment

nitisht Dec 25, 2023

Choose a reason for hiding this comment

nitisht left a comment

Choose a reason for hiding this comment

nitisht commented Dec 24, 2023

nitisht commented Dec 24, 2023

nitisht commented Dec 24, 2023

theteachr commented Dec 24, 2023 • edited

nitisht commented Dec 24, 2023 • edited

trueleo commented Dec 25, 2023

nitisht commented Dec 25, 2023

nitisht Dec 25, 2023

Choose a reason for hiding this comment

trueleo Dec 25, 2023

Choose a reason for hiding this comment

nitisht Dec 25, 2023

Choose a reason for hiding this comment

nitisht commented Dec 25, 2023

trueleo commented Dec 22, 2023 •

edited

trueleo Dec 25, 2023 •

edited

theteachr commented Dec 24, 2023 •

edited

nitisht commented Dec 24, 2023 •

edited