Fix: caching layer persistent

[onlyjackfrost]

Description

• Make caching layer datasource can be persistent
  • It's will reduce the memory usage when loading parquet file to cache
• Skip exporting data if parquet file exists when loading cache
  • avoid exporting/downloading data if the parquet files in path is still fresh

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
vulcan-sql-document	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 18, 2023 9:54am

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.05% 🎉

Comparison is base (de4fb81) 90.09% compared to head (453d8a5) 90.14%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #283      +/-   ##
===========================================
+ Coverage    90.09%   90.14%   +0.05%     
===========================================
  Files          353      353              
  Lines         6077     6090      +13     
  Branches       824      829       +5     
===========================================
+ Hits          5475     5490      +15     
+ Misses         440      438       -2     
  Partials       162      162              

Flag	Coverage Δ
build	89.57% <ø> (ø)
core	93.96% <100.00%> (+0.03%)	⬆️
extension-api-caller	90.00% <ø> (ø)
extension-authenticator-canner	78.37% <ø> (ø)
extension-dbt	97.43% <ø> (ø)
extension-debug-tools	98.11% <ø> (ø)
extension-driver-bq	84.93% <ø> (ø)
extension-driver-canner	84.65% <ø> (ø)
extension-driver-clickhouse	88.09% <ø> (ø)
extension-driver-duckdb	91.80% <ø> (ø)
extension-driver-ksqldb	90.33% <ø> (+0.84%)	⬆️
extension-driver-pg	96.11% <ø> (ø)
extension-driver-snowflake	96.26% <ø> (ø)
extension-huggingface	86.25% <ø> (ø)
extension-store-canner	97.54% <ø> (ø)
integration-testing	90.27% <ø> (ø)
serve	87.17% <ø> (ø)
test-utility	∅ <ø> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
packages/core/src/models/artifact.ts	91.66% <ø> (ø)
packages/core/src/containers/modules/dataSource.ts	95.65% <100.00%> (ø)
...kages/core/src/lib/cache-layer/cacheLayerLoader.ts	100.00% <100.00%> (ø)
packages/core/src/models/cacheLayerOptions.ts	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andreashimin]

LGTM

[andreashimin]

(NIT) Prefer functional programming for filtering

const parquetFiles = files.filter(file => /\.parquet$/.test(file))