Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

count space and inodes usage for each directory #3257

Merged
merged 48 commits into from
Mar 1, 2023
Merged

count space and inodes usage for each directory #3257

merged 48 commits into from
Mar 1, 2023

Conversation

Hexilee
Copy link
Contributor

@Hexilee Hexilee commented Feb 20, 2023

No description provided.

@Hexilee
count space and inodes usage for each directory
0b1f6df 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
adjust idents
6a0755e 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
adjust an ident
f19c5c8 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
revise
c06eab3 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
merge dirUsedSpaceKey and dirUsedInodesKey in tkv
79b8182 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
add storage structures
793c303 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
implement doIncreDirUsage and doGetDirUsage for all meta engines
618be58 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
implement dirUsageBatch for baseMeta
348e32d 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
use increDirUsage in some IO call
52ac44c 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
use increParentUsage in all meta engines
236e9a9 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
merge from master
b7cc826 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
fix nil attr bugs
b2ba498 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
fix sql conflict
5827c1e 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
remove jfs.db
bce8a5b 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
remove jfs.db
c739730 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
fix doIncreDirUsage for dbMeta
05c81c9 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
dirUsageBatch: commit blocking
cd65027 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
increDirUsage directly if blocked
f6e02a0 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
remove jfs.db
d03d302 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
try to sync dirUsage
d378b1a 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
sync dir usage when necessary
524d9b9 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
fix doGetDirUsage in sql
8f0f9ac 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
implement doSyncDirUsage for tkv
06e6fad 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee Hexilee marked this pull request as ready for review February 22, 2023 15:52
@Hexilee
add logs
9d8e137 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
use Lookup instead doLookup
eaf5e1a 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
try to sync dirUsage in tkv
7fa6256 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
return attr from doUnlink
a283b59 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
sync dirUsage table when necessary
d70b109 
Signed-off-by: xixi <hexilee@juicedata.io>
@github-actions
Copy link

Mutate Test Report

Usage: https://github.com/juicedata/juicefs/blob/main/.github/scripts/mutate/how_to_use_mutate_test.md

@Hexilee
fix tkv
4f7e07d 
Signed-off-by: xixi <hexilee@juicedata.io>
SandyXSD
SandyXSD reviewed Feb 27, 2023
View reviewed changes
pkg/meta/base.go Outdated Show resolved Hide resolved
pkg/meta/redis.go Outdated Show resolved Hide resolved
pkg/meta/base.go Outdated Show resolved Hide resolved
pkg/meta/base.go Outdated Show resolved Hide resolved
@SandyXSD
Copy link
Contributor

Also remove the jfs.db from the changed files.

@Hexilee
remote jfs.db
8598eae 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
add tryIncreDirUsage and mustIncreDirUsage
8f41325 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
Copy link
Contributor Author

Hexilee commented Feb 27, 2023

Also remove the jfs.db from the changed files.

removed

@Hexilee
fix increParentUsage
2ff72f9 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
resolve comments
81e64d2 
Signed-off-by: xixi <hexilee@juicedata.io>
SandyXSD
SandyXSD reviewed Feb 28, 2023
View reviewed changes
pkg/meta/tkv.go Outdated Show resolved Hide resolved
pkg/meta/tkv.go Outdated Show resolved Hide resolved
SandyXSD
SandyXSD reviewed Feb 28, 2023
View reviewed changes
pkg/meta/sql.go Outdated Show resolved Hide resolved
@Hexilee
sql: count usage outside of txn
9b173b0 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
deal with err
3b691c2 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
replace channel with locked map in dir usage batch
45239fe 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
remove dirUsageBatchSize
2fa5f09 
Signed-off-by: xixi <hexilee@juicedata.io>
davies
davies reviewed Feb 28, 2023
View reviewed changes
pkg/meta/base.go Outdated Show resolved Hide resolved
davies
davies reviewed Feb 28, 2023
View reviewed changes
pkg/meta/base.go Outdated Show resolved Hide resolved
@Hexilee
remove tryIncreDirUsage
24f08e4 
Signed-off-by: xixi <hexilee@juicedata.io>
@SandyXSD
Copy link
Contributor

@Hexilee We should delete the usage key when removing a directory.

@Hexilee
remove dir usage when rmdir
f026466 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
Copy link
Contributor Author

Hexilee commented Feb 28, 2023

@Hexilee We should delete the usage key when removing a directory.

Done

@Hexilee
fix redis txn panic
e2c9942 
Signed-off-by: xixi <hexilee@juicedata.io>
@Hexilee
fix sql txn panic
022bfe1 
Signed-off-by: xixi <hexilee@juicedata.io>
@davies davies merged commit ebeada9 into main Mar 1, 2023
@davies davies deleted the dir-used-stat branch March 1, 2023 06:49
@Hexilee Hexilee mentioned this pull request Mar 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davies davies davies approved these changes

@SandyXSD SandyXSD Awaiting requested review from SandyXSD

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Hexilee @codecov-commenter @SandyXSD @davies