rgw: add DAOS SAL implementation #45888

zalsader · 2022-04-13T03:37:22Z

Add DAOS backend RGW SAL (Storage Abstraction Layer) implementation.

To build with DAOS backend, use -DWITH_RADOSGW_DAOS=YES cmake option. daos-devel rpm should be installed beforehand.

To connect to DAOS pool, add the following configuration parameters to ceph.conf:

    [client]
            ...
            rgw backend store = daos
            daos pool = tank

A pool could be created using the following command:

    dmg pool create --size=<size> <pool_name>

To install daos-devel do:

    sudo wget -O /etc/yum.repos.d/daos-packages.repo https://packages.daos.io/v2.0/CentOS8/packages/x86_64/daos_packages.repo
    sudo rpm --import https://packages.daos.io/RPM-GPG-KEY
    sudo yum install -y epel-release daos-server daos-client daos-devel

Note: this is an initial stage of the implementation, so some things still may not work perfectly.

Design and additional documentation could be found here.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

zalsader · 2022-04-13T03:49:09Z

src/rgw/rgw_admin.cc

+    std::string rgw_store = "rados";
+    const auto& config_store = g_conf().get_val<std::string>("rgw_backend_store");
+    #ifdef WITH_RADOSGW_DBSTORE
+    if (config_store == "dbstore") {
+      rgw_store = "dbstore";
+    }
+    #endif
+
+    #ifdef WITH_RADOSGW_MOTR
+    if (config_store == "motr") {
+      rgw_store = "motr";
+    }
+    #endif
+
+    #ifdef WITH_RADOSGW_DAOS
+    if (config_store == "daos") {
+      rgw_store = "daos";
+    }
+    #endif
+
+    lsubdout(cct, rgw, 1) << "RGW CONF BACKEND STORE = " << config_store << dendl;
+    lsubdout(cct, rgw, 1) << "RGW BACKEND STORE = " << rgw_store << dendl;
+
    if (raw_storage_op) {
-      store = StoreManager::get_raw_storage(dpp(), g_ceph_context, "rados");
+      store = StoreManager::get_raw_storage(dpp(), g_ceph_context, rgw_store);
    } else {
-      store = StoreManager::get_storage(dpp(), g_ceph_context, "rados", false, false, false,
+      store = StoreManager::get_storage(dpp(), g_ceph_context, rgw_store, false, false, false,


This allows radosgw-admin to create, update and delete users. Another PR (#45247 ) includes a similar change. We are working together to make sure that radosgw-admin works well with daos and motr.

This bit is going to be part of #45987 so it can be dropped from this PR.

cbodley · 2022-04-20T15:35:04Z

jenkins test make check

dang · 2022-04-20T15:35:21Z

jenkins test api

src/rgw/store/daos/README.md

src/rgw/rgw_sal_daos.cc

soumyakoduri · 2022-04-25T07:18:21Z

src/rgw/rgw_sal_daos.cc

+
+  // XXX: Combine multiple streams into one as motr does
+  uint64_t data_size = data.length();
+  ret = obj.write(dpp, std::move(data), offset);


How are the concurrent writes handled? In case of Rados (& also dbstore), a unique object-ID is created for each object data write to prevent any overwrite & in AtomicWriter::complete() that object-ID is stored in the object metadata to refer to in object::read()

We did not handle this before, I will sketch something up to handle this. Adding a TODO for now

dang

#45623 merged, so this now needs changes for the zonegroup API

To build with DAOS backend, use -DWITH_RADOSGW_DAOS=YES cmake option. `daos-devel` rpm should be installed beforehand. To connect to DAOS pool, add the following configuration parameters to ceph.conf: ``` [client] ... rgw backend store = daos daos pool = tank ``` A pool could be created using the following command: ``` dmg pool create --size=<size> <pool_name> ``` To install `daos-devel` do: ``` sudo wget -O /etc/yum.repos.d/daos-packages.repo https://packages.daos.io/v2.0/CentOS8/packages/x86_64/daos_packages.repo sudo rpm --import https://packages.daos.io/RPM-GPG-KEY sudo yum install -y epel-release daos-server daos-client daos-devel ``` Co-authored-by: Walter Warniaha <walter.warniaha@seagate.com> Signed-off-by: Zuhair AlSader <zuhair.alsader@seagate.com>

zalsader · 2022-04-28T02:10:02Z

@dang @soumyakoduri The PR has been rebased on master and the required changes were made, please have a look, thank you!

zalsader · 2022-04-28T02:35:22Z

make check passed before because the option to build DAOS (WITH_RADOSGW_DAOS=ON) was not passed to the make check script. It is failing now because it cannot find DAOS on ubuntu, I am working with the DAOS team to get an ubuntu release for this.

dang

Looks good otherwise.

dang · 2022-04-29T15:22:08Z

src/rgw/rgw_admin.cc

+    std::string rgw_store = "rados";
+    const auto& config_store = g_conf().get_val<std::string>("rgw_backend_store");
+    #ifdef WITH_RADOSGW_DBSTORE
+    if (config_store == "dbstore") {
+      rgw_store = "dbstore";
+    }
+    #endif
+
+    #ifdef WITH_RADOSGW_MOTR
+    if (config_store == "motr") {
+      rgw_store = "motr";
+    }
+    #endif
+
+    #ifdef WITH_RADOSGW_DAOS
+    if (config_store == "daos") {
+      rgw_store = "daos";
+    }
+    #endif
+
+    lsubdout(cct, rgw, 1) << "RGW CONF BACKEND STORE = " << config_store << dendl;
+    lsubdout(cct, rgw, 1) << "RGW BACKEND STORE = " << rgw_store << dendl;
+
    if (raw_storage_op) {
-      store = StoreManager::get_raw_storage(dpp(), g_ceph_context, "rados");
+      store = StoreManager::get_raw_storage(dpp(), g_ceph_context, rgw_store);
    } else {
-      store = StoreManager::get_storage(dpp(), g_ceph_context, "rados", false, false, false,
+      store = StoreManager::get_storage(dpp(), g_ceph_context, rgw_store, false, false, false,


This bit is going to be part of #45987 so it can be dropped from this PR.

github-actions · 2022-04-29T17:00:42Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

zalsader · 2022-08-17T18:18:58Z

Closing in favor of a new PR that incorporates libds3

github-actions bot added build/ops common documentation rgw labels Apr 13, 2022

zalsader marked this pull request as ready for review April 13, 2022 03:38

zalsader requested a review from a team as a code owner April 13, 2022 03:38

zalsader commented Apr 13, 2022

View reviewed changes

soumyakoduri requested review from cbodley, mattbenjamin, dang and soumyakoduri April 13, 2022 16:35

markhpc added the performance label Apr 14, 2022

dang approved these changes Apr 22, 2022

View reviewed changes

soumyakoduri reviewed Apr 25, 2022

View reviewed changes

src/rgw/store/daos/README.md Outdated Show resolved Hide resolved

soumyakoduri reviewed Apr 25, 2022

View reviewed changes

src/rgw/rgw_sal_daos.cc Outdated Show resolved Hide resolved

soumyakoduri reviewed Apr 25, 2022

View reviewed changes

src/rgw/rgw_sal_daos.cc Outdated Show resolved Hide resolved

soumyakoduri reviewed Apr 25, 2022

View reviewed changes

dang requested changes Apr 26, 2022

View reviewed changes

zalsader force-pushed the add-daos branch from ecdb521 to f3f38bb Compare April 28, 2022 02:05

zalsader requested review from a team as code owners April 28, 2022 02:05

zalsader requested review from avanthakkar and nizamial09 and removed request for a team April 28, 2022 02:05

github-actions bot added bluestore ceph-volume cephadm labels Apr 28, 2022

github-actions bot added cephfs Ceph File System CI Continuous Integration core crimson dashboard mon monitoring orchestrator pybind rbd tests labels Apr 28, 2022

zalsader force-pushed the add-daos branch from f3f38bb to 207ba33 Compare April 28, 2022 02:08

github-actions bot added this to In progress in Dashboard Apr 28, 2022

zalsader requested review from dang and soumyakoduri April 28, 2022 02:11

dang reviewed Apr 29, 2022

View reviewed changes

github-actions bot added the needs-rebase label Apr 29, 2022

nizamial09 removed this from In progress in Dashboard May 9, 2022

djgalloway changed the base branch from master to main May 25, 2022 20:00

idryomov removed the rbd label May 26, 2022

zalsader closed this Aug 17, 2022

zalsader deleted the add-daos branch August 19, 2022 22:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rgw: add DAOS SAL implementation #45888

rgw: add DAOS SAL implementation #45888

zalsader commented Apr 13, 2022 •

edited

zalsader Apr 13, 2022

dang Apr 29, 2022

cbodley commented Apr 20, 2022

dang commented Apr 20, 2022

soumyakoduri Apr 25, 2022

zalsader Apr 27, 2022

dang left a comment

zalsader commented Apr 28, 2022

zalsader commented Apr 28, 2022

dang left a comment

dang Apr 29, 2022

github-actions bot commented Apr 29, 2022

zalsader commented Aug 17, 2022

rgw: add DAOS SAL implementation #45888

rgw: add DAOS SAL implementation #45888

Conversation

zalsader commented Apr 13, 2022 • edited

Checklist

zalsader Apr 13, 2022

Choose a reason for hiding this comment

dang Apr 29, 2022

Choose a reason for hiding this comment

cbodley commented Apr 20, 2022

dang commented Apr 20, 2022

soumyakoduri Apr 25, 2022

Choose a reason for hiding this comment

zalsader Apr 27, 2022

Choose a reason for hiding this comment

dang left a comment

Choose a reason for hiding this comment

zalsader commented Apr 28, 2022

zalsader commented Apr 28, 2022

dang left a comment

Choose a reason for hiding this comment

dang Apr 29, 2022

Choose a reason for hiding this comment

github-actions bot commented Apr 29, 2022

zalsader commented Aug 17, 2022

zalsader commented Apr 13, 2022 •

edited