Cache size propagete to libdeeplake. #2607

khustup · 2023-09-20T22:41:09Z

🚀 🚀 Pull Request

Impact

Bug fix (non-breaking change which fixes expected existing functionality)
Enhancement/New feature (adds functionality without impacting existing logic)
Breaking change (fix or feature that would cause existing functionality to change)

Description

Pass cache size to the indra.

Things to be aware of

Things to worry about

Additional Context

codecov · 2023-09-20T23:51:09Z

Codecov Report

Patch coverage: 66.66% and project coverage change: -0.60% ⚠️

Comparison is base (799f547) 83.99% compared to head (14dedc0) 83.40%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2607      +/-   ##
==========================================
- Coverage   83.99%   83.40%   -0.60%     
==========================================
  Files         224      224              
  Lines       24674    24677       +3     
==========================================
- Hits        20726    20582     -144     
- Misses       3948     4095     +147

Flag	Coverage Δ
unittests	`83.40% <66.66%> (-0.60%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
deeplake/enterprise/convert_to_libdeeplake.py	`61.81% <66.66%> (-0.80%)`	⬇️

... and 11 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nvoxland · 2023-09-21T01:02:43Z

The formatter needs to be ran on this

FayazRahman · 2023-09-21T11:37:00Z

deeplake/enterprise/convert_to_libdeeplake.py

@@ -212,6 +213,9 @@ def dataset_to_libdeeplake(hub2_dataset: Dataset):
        libdeeplake_dataset = hub2_dataset.libdeeplake_dataset

    assert libdeeplake_dataset is not None
+    libdeeplake_dataset._max_cache_size = max(
+        hub2_dataset.storage.cache_size * MB, libdeeplake_dataset._max_cache_size
