Enhancement/disk utilization #642
Conversation
Add temporary directory and files
Multiple files can have same content hash within an allocation. Remove iterProgress use Remove . import
Add mock for file manager
db.commit to unblock postgres transaction
Makefile
Outdated
| @@ -56,7 +56,7 @@ local-run: | |||
| --hostname 127.0.0.1 \ | |||
| --deployment_mode 0 \ | |||
| --keys_file ../docker.local/keys_config/b0bnode1_keys.txt \ | |||
| --files_dir ./data/blobber/files \ | |||
| --mount_point ./data/blobber/files \ | |||
There was a problem hiding this comment.
please keep it with --files_dir, if not. it will break all helm charts.
There was a problem hiding this comment.
or please update it on dockerfile , cli and helm charts too
There was a problem hiding this comment.
I think its better to update in cli and helm charts. For files storage, It must be mountpoint. So the flag name should be appropriate as well.
Codecov Report
@@ Coverage Diff @@
## staging #642 +/- ##
===========================================
+ Coverage 19.70% 21.99% +2.28%
===========================================
Files 66 69 +3
Lines 7530 7844 +314
===========================================
+ Hits 1484 1725 +241
- Misses 5819 5857 +38
- Partials 227 262 +35
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
Put common function in new file
Add nil checking for integration testing
…ain/blobber into enhancement/disk-utilization
Changes
This PR manages files in blobber so that millions files won't get crowded in same directory.
Using fixed number of files to put in a directory is not feasible because allocation can update its size.
Like previous method, I have used random property of 256 hash of allocation and content hash. Allocation hash(id), to create parent directories for all files belonging to that allocation, and content hash to create directories and file name for some file.
Previous method had overuse of directories and number of directories that an allocation could have was hard coded.
It used content_hash's first 9 characters to create directory for a file.
So there could be 16^9 = 68719476736(68 billion directories).
Now blobber can change this number as per its capacity.
Also in previous method, it would put all allocation in same directory i.e. /data/blobber/files. This would allow a directory to put millions allocation in a single directory thus making lookup inefficient. This also has been fixed.
Fixes
on_cloudfield. If other refs too had been referring to same content hash then there is no way to get data for those refs.