curvebs: support poolset #1988

jolly-sy · 2022-10-19T07:23:25Z

What problem does this PR solve?

Issue Number: #xxx

Problem Summary:

What is changed and how it works?

What's Changed:

How it Works:

Side effects(Breaking backward compatibility? Performance regression?):

Check List

Relevant documentation/comments is changed or added
I acknowledge that all my contributions will be made under the project's license

jolly-sy · 2022-10-19T08:03:07Z

recheck

jolly-sy · 2022-10-19T09:13:38Z

recheck

jolly-sy · 2022-10-19T09:14:38Z

recheck

jolly-sy · 2022-10-19T11:11:37Z

recheck

jolly-sy · 2022-10-19T12:05:45Z

recheck

jolly-sy · 2022-10-19T12:12:12Z

recheck

jolly-sy · 2022-10-19T12:13:44Z

recheck

jolly-sy · 2022-10-20T02:46:48Z

recheck

jolly-sy · 2022-10-20T05:52:48Z

recheck

wu-hanqing

BTW, delete src/mds/nameserver2/curveadm.

And, write a document for this pr, explain how to upgrade old cluster

wu-hanqing · 2022-11-19T12:17:42Z

tools/curvefsTool.cpp

+        LOG(ERROR) << "No poolsets in cluster map";
+        return -1;
+    }
+    for (const auto poolset : clusterMap_[kPoolsets]) {


using const reference to avoid unnecessary copying.

using const reference to avoid unnecessary copying.

fix

tools/curvefsTool.cpp

wu-hanqing · 2022-11-19T12:27:32Z

tools/curvefsTool.cpp

+        } else if (s == "nvme") {
+            poolsetData.type = PoolsetType::NVME;
+        } else {
+            poolsetData.type = PoolsetType::SSD;


it's better to report an error in this case instead of giving it a default type.

it's better to report an error in this case instead of giving it a default type.

fix

wu-hanqing · 2022-11-19T12:30:38Z

tools/curvefsTool.cpp

+        if (!server[kPhysicalPool].isString()) {
+            LOG(ERROR) << "server physicalpool must be string";
+            return -1;
+        }
+        serverData.physicalPoolName = server[kPhysicalPool].asString();


it seems these code are identical with L402-L406.

it seems these code are identical with L402-L406.

fix

wu-hanqing · 2022-11-19T12:31:37Z

tools/curvefsTool.cpp

+            poolsetData.type = PoolsetType::SSD;
+        }
+
+        poolsetDatas.emplace_back(poolsetData);


poolsetDatas.emplace_back(std::move(poolsetData))

poolsetDatas.emplace_back(std::move(poolsetData))

fix

wu-hanqing · 2022-11-20T03:06:23Z

src/mds/topology/topology_item.cpp

@@ -213,6 +233,7 @@ bool PhysicalPool::ParseFromString(const std::string &value) {
    id_ = data.physicalpoolid();
    name_ = data.physicalpoolname();
    desc_ = data.desc();
+    poolsetId_ = data.poolsetid();


ditto, you should check whether data has poolset id or not, if it don't have it, poolsetId_ should set to UNINTIALIZE_ID.

fix
if(data.has_poolsetid()){
poolsetId_ = data.poolsetid();
}

wu-hanqing · 2022-11-20T03:08:31Z

src/mds/topology/topology_id_generator.h

@@ -97,6 +98,7 @@ class DefaultIdGenerator : public TopologyIdGenerator {
        std::atomic<T> idMax_;
    };

+    IdGenerator<PoolsetIdType> poolsetIdGentor_;


it seems poolsetIdGentor_ doesn't inited

it seems poolsetIdGentor_ doesn't inited

fix, poolsetIdGentor_ is inited

poolsetIdGentor_ init in class TopologyIdGenerator

where does poolsetIdGentor_ inited?
in class TopologyIdGenerator

where does poolsetIdGentor_ inited?

in class TopologyIdGenerator

wu-hanqing · 2022-11-20T03:43:53Z

src/mds/topology/topology.cpp

+    }
+}
+
+// int TopologyImpl::RemovePhysicalPoolNotInPoolset(PoolIdType id) {


remove unused code

remove unused code

fix, the unused code were removed

wu-hanqing · 2022-11-20T04:08:09Z

src/mds/nameserver2/curvefs.cpp

            auto ifok = chunkSegAllocator_->AllocateChunkSegment(
                            fileInfo.filetype(), fileInfo.segmentsize(),
-                            fileInfo.chunksize(), offset, segment);
+                            fileInfo.chunksize(), poolset.GetType(),


noop, segment should be allocated by poolset name instead of type, because we can have more than one poolsets with same type.

noop, segment should be allocated by poolset name instead of type, because we can have more than one poolsets with same type.

fix, parameter has been changed to poolsetName

wu-hanqing · 2022-11-20T04:11:56Z

src/mds/nameserver2/curvefs.cpp

@@ -1251,9 +1258,15 @@ StatusCode CurveFS::GetOrAllocateSegment(const std::string & filename,
            return  StatusCode::kSegmentNotAllocated;
        } else {
            // TODO(hzsunjianliang): check the user and define the logical pool
+            if (!fileInfo.has_poolsetname()) {
+                fileInfo.set_poolsetname("ssdPoolset1");


it's no a good idea, a proper way is adding a config item in mds configuration to store the default poolsetname for allocating segment for those files that don't have a poolsetname in fileinfo.

it's no a good idea, a proper way is adding a config item in mds configuration to store the default poolsetname for allocating segment for those files that don't have a poolsetname in fileinfo.

fix, defalut poolsetName has been set in mds.conf

jolly-sy · 2022-12-13T08:42:58Z

cicheck

wu-hanqing · 2023-05-10T09:12:12Z

cicheck

wu-hanqing · 2023-05-10T11:30:07Z

cicheck

wu-hanqing · 2023-05-12T02:21:41Z

cicheck

xu-chaojie · 2023-06-02T02:07:56Z

include/client/libcurve.h

- *
- * @return: success return 0, fail return less than 0
- */
-int Create2(const char* filename,


why remove Create2?

xu-chaojie · 2023-06-02T03:13:34Z

proto/nameserver2.proto

@@ -93,6 +93,7 @@ message FileInfo {

    optional    FileThrottleParams throttleParams = 17;
    optional    uint64      epoch = 18;
+    optional    string      poolset = 19;


Wouldn't it be better to use poolsetid instead?

Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

Co-authored-by: jolly-sy <757050468@qq.com> Co-authored-by: Hanqing Wu <wuhanqing@corp.netease.com> Signed-off-by: jolly-sy <757050468@qq.com> Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

wu-hanqing · 2023-06-03T11:26:57Z

cicheck

wu-hanqing · 2023-06-07T12:13:17Z

cicheck

wu-hanqing · 2023-06-07T12:20:20Z

cicheck

…directories Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

wu-hanqing · 2023-06-07T12:26:47Z

cicheck

ilixiaocui · 2023-06-08T02:49:52Z

Please add a poolset-related ReadMe in the appropriate section， for exmaple: how to use

wu-hanqing · 2023-06-08T11:17:38Z

Please add a poolset-related ReadMe in the appropriate section， for exmaple: how to use

I will update curveadm wiki later.

jolly-sy force-pushed the fixPoolsetConflic branch from b2aeffd to 793262a Compare October 19, 2022 07:32

jolly-sy force-pushed the fixPoolsetConflic branch 2 times, most recently from a46f82b to 3c50c00 Compare October 19, 2022 09:42

jolly-sy force-pushed the fixPoolsetConflic branch 2 times, most recently from 789c384 to 5d0e610 Compare October 19, 2022 12:01

jolly-sy force-pushed the fixPoolsetConflic branch 2 times, most recently from 48932f1 to 670ac54 Compare October 20, 2022 02:44

jolly-sy force-pushed the fixPoolsetConflic branch from 670ac54 to d0883cb Compare November 2, 2022 09:27

wu-hanqing requested changes Nov 20, 2022

View reviewed changes

jolly-sy force-pushed the fixPoolsetConflic branch 2 times, most recently from 345b4cf to 44cdbc2 Compare November 21, 2022 12:36

jolly-sy force-pushed the fixPoolsetConflic branch from 44cdbc2 to 63c8756 Compare November 28, 2022 23:34

jolly-sy force-pushed the fixPoolsetConflic branch 7 times, most recently from 7ff7e02 to b339c17 Compare December 13, 2022 08:31

jolly-sy force-pushed the fixPoolsetConflic branch from b339c17 to 78498af Compare December 13, 2022 08:57

wu-hanqing force-pushed the fixPoolsetConflic branch from a7e269d to 3309c84 Compare May 10, 2023 11:29

wu-hanqing force-pushed the fixPoolsetConflic branch from 3309c84 to 02a60bd Compare May 12, 2023 02:13

xu-chaojie reviewed Jun 2, 2023

View reviewed changes

xu-chaojie approved these changes Jun 2, 2023

View reviewed changes

wu-hanqing and others added 3 commits June 3, 2023 19:11

curvebs: fix segfault when enable O_DSYNC

3a03126

Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

curvebs/client: mds address should not be empty

5105878

Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

curvebs: support poolset

6f2f0e5

Co-authored-by: jolly-sy <757050468@qq.com> Co-authored-by: Hanqing Wu <wuhanqing@corp.netease.com> Signed-off-by: jolly-sy <757050468@qq.com> Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

wu-hanqing force-pushed the fixPoolsetConflic branch from 02a60bd to 6f2f0e5 Compare June 3, 2023 11:25

wu-hanqing force-pushed the fixPoolsetConflic branch from f7028c0 to 73a8a8f Compare June 7, 2023 12:19

curvebs/mds: add poolset rules to support assigning poolset based on …

fe08f0c

…directories Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

wu-hanqing force-pushed the fixPoolsetConflic branch from 73a8a8f to fe08f0c Compare June 7, 2023 12:21

xu-chaojie approved these changes Jun 7, 2023

View reviewed changes

ilixiaocui approved these changes Jun 8, 2023

View reviewed changes

wu-hanqing merged commit 191a001 into opencurve:master Jun 8, 2023
3 checks passed

curvebs: support poolset #1988

curvebs: support poolset #1988

Conversation

jolly-sy commented Oct 19, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

jolly-sy commented Oct 19, 2022

jolly-sy commented Oct 19, 2022

jolly-sy commented Oct 19, 2022

jolly-sy commented Oct 19, 2022

jolly-sy commented Oct 19, 2022

jolly-sy commented Oct 19, 2022

jolly-sy commented Oct 19, 2022

jolly-sy commented Oct 20, 2022

jolly-sy commented Oct 20, 2022

wu-hanqing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilixiaocui Dec 22, 2022 • edited by jolly-sy Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jolly-sy commented Dec 13, 2022

wu-hanqing commented May 10, 2023

wu-hanqing commented May 10, 2023

wu-hanqing commented May 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wu-hanqing commented Jun 3, 2023

wu-hanqing commented Jun 7, 2023

wu-hanqing commented Jun 7, 2023

wu-hanqing commented Jun 7, 2023

ilixiaocui commented Jun 8, 2023

wu-hanqing commented Jun 8, 2023

ilixiaocui Dec 22, 2022 •

edited by jolly-sy

Loading