curvefs: support space deallocate for curvebs volume as backend #2313

ilixiaocui · 2023-03-15T03:36:05Z

What problem does this PR solve?

Issue Number: #2160

Problem Summary: #2294

What is changed and how it works?

What's Changed:

How it Works:

Side effects(Breaking backward compatibility? Performance regression?):

Check List

Relevant documentation/comments is changed or added
I acknowledge that all my contributions will be made under the project's license

Wine93 · 2023-06-30T08:15:45Z

curvefs/src/mds/fs_info_wrapper.cpp

@@ -50,8 +49,13 @@ FsInfoWrapper::FsInfoWrapper(const ::curvefs::mds::CreateFsRequest* request,
    fsInfo.set_enablesumindir(request->enablesumindir());
    fsInfo.set_txsequence(0);
    fsInfo.set_txowner("");
+    // TODO(@lixiaocui1): Currently, curveadm does not support setting recycling


It will be supported.
BTW: is there a default value for recyleTimeHour if user not specified?

recycleTimeHour is optional in message FsInfo. it doesn't have a default value.

recycleTimeHour is optional in message FsInfo. it doesn't have a default value.

Okay~ i will give a default value (1 hours) in curveadm.

Wine93 · 2023-06-30T09:05:34Z

curvefs/src/mds/fs_manager.cpp

+    if (wrapper.GetFsType() == FSType::TYPE_VOLUME) {
+        auto volumeSpace = spaceManager_->GetVolumeSpace(wrapper.GetFsId());
+        if (volumeSpace == nullptr) {
+            LOG(ERROR) << "handle fs mount point timeout fail, get volume "


Maybe it can be more readable:

LOG(ERROR) << "handle fs mount point timeout fail, get volume " << "space fail, fsName = " << fsName << ", mountpoint = " << mountpoint.ShortDebugString();

this is the default fomat result.

Wine93 · 2023-06-30T09:07:38Z

curvefs/src/mds/fs_manager.cpp

+        if (ret != space::SpaceOk) {
+            LOG(ERROR)
+                << "handle fs mount point timeout fail,release block groups "
+                   "fail, fsName = "


Wine93 · 2023-06-30T09:11:18Z

curvefs/src/mds/space/volume_space.cpp

@@ -156,8 +172,10 @@ SpaceErrCode VolumeSpace::AllocateBlockGroups(
        return err;
    }

-    for (auto& group : *blockGroups) {
+    for (auto &group : *blockGroups) {


Its better to use const auto&.

Wine93 · 2023-06-30T09:16:55Z

curvefs/src/mds/space/volume_space.cpp

@@ -317,6 +352,8 @@ SpaceErrCode VolumeSpace::ReleaseBlockGroups(
    LockGuard lk(mtx_);

    for (auto& group : blockGroups) {
+        VLOG(3) << "VolumeSpace need release block group:"
+                  << group.DebugString();


indentation.

outdated. fix

Wine93 · 2023-06-30T09:19:24Z

curvefs/src/mds/space/volume_space.cpp

@@ -267,6 +286,22 @@ SpaceErrCode VolumeSpace::AcquireBlockGroup(uint64_t blockGroupOffset,
 SpaceErrCode VolumeSpace::AcquireBlockGroupInternal(uint64_t blockGroupOffset,
                                                    const std::string& owner,
                                                    BlockGroup* group) {
+    if (owner.empty()) {


Maybe it can be more readable:

if (owner.empty()) { auto it = deallocatingGroups_.find(blockGroupOffset); if (it == deallocatingGroups_.end()) { return SpaceErrNotFound; } *group = it->second; VLOG(6) << "VolumeSpace recieve acquire blockgroup=" << group->DebugString() << " rquest from metaserver, current block group is under " "deallocating"; return SpaceOk; }

good idea. fix.

Wine93 · 2023-06-30T09:27:30Z

curvefs/src/metaserver/inode_storage.cpp

@@ -571,5 +620,207 @@ MetaStatusCode InodeStorage::GetVolumeExtentByOffset(uint32_t fsId,
    return MetaStatusCode::STORAGE_INTERNAL_ERROR;
 }

+MetaStatusCode InodeStorage::GeAllBlockGroup(


typo: GetAllBlockGroup

Wine93 · 2023-06-30T09:35:16Z

curvefs/src/metaserver/partition.cpp

    }
 }

+#define PRECHECK(fsId, inodeId)                                                \
+    do {                                                                       \
+        if (!IsInodeBelongs((fsId), (inodeId))) {            \


indentation for \

Wine93 · 2023-06-30T09:35:45Z

curvefs/src/metaserver/partition.cpp

+        }                                                                      \
+    } while (0)
+
+#define PRECHECKFSID(fsId)                                                     \


Maybe PRECHECK_FSID is more readable.

Wine93 · 2023-06-30T09:38:41Z

curvefs/src/metaserver/space/inode_volume_space_deallocate.cpp

+
+    // key is volume offset
+    DeallocatableBlockGroupMap increase;
+    while (iter->Valid()) {


for (iter->SeekToFirst(); iter->Valid(); iter->Next()) { // do something }

Wine93 · 2023-06-30T09:42:10Z

curvefs/src/metaserver/space/volume_space_manager.cpp

+    if (it == spaceManagers_.end()) {
+        if (!GenerateSpaceManager(fsId)) {
+            return nullptr;
+        }


Is it return spaceManagers_[fsId] if GenerateSpaceManager return true even if the item not exist in the spaceManagers_?

yes. GenerateSpaceManager will insert new generated manager to spaceManagers_.

Wine93 · 2023-06-30T09:45:02Z

curvefs/src/metaserver/storage/converter.cpp

+std::string Key4DeallocatableBlockGroup::SerializeToString() const {
+    return absl::StrCat(keyType_, kDelimiter, fsId, kDelimiter, volumeOffset);
+}
+


redundant newline.

Wine93 · 2023-07-02T12:52:59Z

If willing, please requested review from me again after all problem solved and all suggestion has been handled :)

ilixiaocui · 2023-07-13T09:26:53Z

If willing, please requested review from me again after all problem solved and all suggestion has been handled :)

okok

ilixiaocui · 2023-07-13T11:18:44Z

cicheck

Wine93

LGTM!

ilixiaocui · 2023-07-19T06:33:26Z

cicheck

wu-hanqing · 2023-07-19T11:49:25Z

curvefs/src/client/async_request_closure.cpp

-                   << MetaStatusCode_Name(st)
-                   << ", inodeid: " << inode_->GetInodeId();
-        inode_->MarkInodeError();
+        if (inode_ != nullptr) {


The inode may be evicted by mistake, here is a defense.

evict an inode won't cause inode_ to become a nullptr, the reason might be UpdateVolumeExtentClosure was created from a null inode.

wu-hanqing · 2023-07-19T11:57:33Z

curvefs/src/metaserver/inode_storage.cpp

+        }
+
+        if (item.has_increase()) {
+            if (!out.IsInitialized()) {


!s.ok() && !s.IsNotFound() means status wrongand inode is found

At this point, we either found a DeallocatableBlockGroup or not found and without any error.

And if we found one, then out must be initialized, otherwise out is not initialized if it isn't found.

If this is true, using s.IsNotFound() will be more explicit, because people may not know why we test out is initialized here.

wu-hanqing · 2023-07-19T12:09:31Z

curvefs/src/metaserver/inode_storage.h

+    MetaStatusCode Increase(Transaction txn, uint32_t fsId,
+                            const IncreaseDeallocatableBlockGroup &increase,
+                            DeallocatableBlockGroup *out);
+
+    MetaStatusCode Decrease(const DecreaseDeallocatableBlockGroup &decrease,
+                            DeallocatableBlockGroup *out);
+
+    MetaStatusCode Mark(const MarkDeallocatableBlockGroup &mark,
+                        DeallocatableBlockGroup *out);


these functions are all private, which are used for processing corresponding to different fields in message DeallocatableBlockGroup.

Although they're private, member functions should access or modify data members not just
function parameters. And another good reason to put them outside of this class is it'll be more test friendly.

wu-hanqing · 2023-07-19T12:12:13Z

curvefs/src/metaserver/space/inode_volume_space_deallocate.cpp

+                       << ", blockGroupOffset:" << blockGroupOffset
+                       << ", to sliceList:" << sliceList.DebugString()
+                       << " failed";
+            return false;


it's still leaked in this case, BTW, you don't have to allocate closure on heap

wu-hanqing · 2023-07-19T12:13:57Z

curvefs/src/metaserver/space/inode_volume_space_deallocate.cpp

+                extents->begin(), extents->end(),
+                [&](const VolumeExtent &extent) {
+                    bool res = (extent.volumeoffset() >= blockGroupOffset) &&
+                               (extent.volumeoffset() < boundary);


because of the existence of 4k headers, the length of the extent will not cross the block group, so there is no need to add length to judge.

consider add a comment to explain this implicit condition

wu-hanqing · 2023-07-19T12:17:05Z

curvefs/proto/common.proto

@@ -31,6 +31,13 @@ enum BitmapLocation {
    AtEnd = 2;
 }

+message EmptyMsg {}


just include google/protobuf/empty.pb.h header where used and protobuf dependency should have been satisfied

Signed-off-by: ilixiaocui <ilixiaocui@163.com>

ilixiaocui · 2023-07-20T06:58:12Z

cicheck

ilixiaocui force-pushed the volumespacerecycle branch 17 times, most recently from be3376e to 60cbecd Compare March 22, 2023 03:33

ilixiaocui force-pushed the volumespacerecycle branch 13 times, most recently from d9c6864 to 8a06440 Compare March 24, 2023 10:05

Wine93 reviewed Jun 30, 2023

View reviewed changes

ilixiaocui force-pushed the volumespacerecycle branch 2 times, most recently from a2bfc88 to b6742eb Compare July 13, 2023 09:26

ilixiaocui force-pushed the volumespacerecycle branch 3 times, most recently from 666ff36 to 40bfa91 Compare July 13, 2023 11:18

Wine93 approved these changes Jul 16, 2023

View reviewed changes

ilixiaocui force-pushed the volumespacerecycle branch 2 times, most recently from 8ae2b6a to 64daf02 Compare July 19, 2023 06:21

wu-hanqing reviewed Jul 19, 2023

View reviewed changes

curvefs: support space deallocate for curvebs volume as backend

fcdb00e

Signed-off-by: ilixiaocui <ilixiaocui@163.com>

ilixiaocui force-pushed the volumespacerecycle branch from 64daf02 to fcdb00e Compare July 20, 2023 06:45

wu-hanqing approved these changes Jul 20, 2023

View reviewed changes

ilixiaocui merged commit f011606 into opencurve:master Jul 20, 2023
3 checks passed

curvefs: support space deallocate for curvebs volume as backend #2313

curvefs: support space deallocate for curvebs volume as backend #2313

Conversation

ilixiaocui commented Mar 15, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Wine93 Jun 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wine93 Jun 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wine93 Jun 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wine93 Jun 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Wine93 commented Jul 2, 2023

ilixiaocui commented Jul 13, 2023

ilixiaocui commented Jul 13, 2023

Wine93 left a comment

Choose a reason for hiding this comment

ilixiaocui commented Jul 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilixiaocui commented Jul 20, 2023

ilixiaocui commented Mar 15, 2023 •

edited

Loading

Wine93 Jun 30, 2023 •

edited

Loading

Wine93 Jun 30, 2023 •

edited

Loading

Wine93 Jun 30, 2023 •

edited

Loading

Wine93 Jun 30, 2023 •

edited

Loading