cephfs: support WORM(Write Once Read Many) feature #26691

weiqiaomiao · 2019-02-28T11:09:18Z

Write Once Read Many (WORM) technology is used in many applications for data integrity reasons and because of the accepted legal admissibility of stored files that use the technology. And we can see some distribute filesystem such as glusterfs support WORM feature. Driven by the needs of our customer , we are working on Cephfs support WORM(write once read only) feature.
In our implementation, the worm feature is disabled default, and can enabled on a exist empty dir . A dir which enable worm feature called a worm root dir. When a subdir or a file create under worm root dir, it auto inherit the worm attibute from worm root dir. A file which enable worm feature called a worm file. The worm file has three state：
(1) normal state： the initial state of a file. in this state, the worm file is mutable and deletable。
(2) retained state: in this state, the worm file is immutable or undeletable.
(3) expired state: in this state, the worm like as the normal file, it is mutable and deletable (the worm file is immutable but deletable in some other storage).

A file makes transition between these 3 states. Methods of making WORM/Retention transition:
(1) manual commit: Using chattr +i command to make the worm file through into retained state immediately.
(2) automatic commit: Automatic transition between the different states of a file. Dormant Files (files which are untouched for a long time) will be converted into WORM files based on auto-commit period .

Retention Profiles/Policies: Configurable Policies that guide the WORM/Retention behavior. Contains,
(1) Default Retention Period: default time period till which a WORM file should be retained(undeletable)
(2) Auto-commit Period: a worm file get into retained state if it is untouched in auto commit period.
(3) minimum retention period: minimum time period for users config on a worm file
(4) max retention period: max time period for user config on a worm file

Cephfs Modification:
(1) add worm state and retention policies into inode per file
(2) worm feature can enable on any empty dir, but once it enabled, it can't disabled. we can enable worm feature by setfattr like:

setfattr -n ceph.worm -v "enable=1 auto_commit_period=120 min_retention_period=240 max_retention_period=7200 retention_period=360" dir1

or

setfattr -n ceph.worm.enable -v 1 dir1
setfattr -n ceph.worm.enable -v 120 dir1
setfattr -n ceph.worm.min_retention_period -v 240 dir1
setfattr -n ceph.worm.max_retention_period -v 7200 dir1
setfattr -n ceph.worm.retention_period -v 360 dir1

and get worm state and policies by getfattr like:

 getfattr -n ceph.worm dir1

(3) Lazy Auto-commit: IO Triggered Using timeouts for untouched files. The modify IO（write、truncate、unlink.etc) will cause the transition.

(4) Using the file atime save WORM file protection expiration time.
manual commit : atime = mtime + retention_period
atomatic commit: atime = mtime + auto_commit_period + retention_period
user can change the worm file retention period ( must between minimum retention period ~ max retention period) through change file Atime.

(5) hard link don't support if worm feature enable.

Signed-off-by: Wei Qiaomiao wei.qiaomiao@zte.com.cn

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

xiexingguo · 2019-02-28T11:24:40Z

src/common/options.cc

+     .set_default(120)
+     .set_description("default auto commit period for worm feature"),
+
+    Option("mds_worm_retention_period", Option::TYPE_UINT, Option::LEVEL_ADVANCED)


Seems these three options are all sharing the same default value. Did you do that deliberately?

Also I've observed you silently changed the perm of all files you touched from 100644->100755, which I guess should definitely need to be addressed first..

These options had been redesigned， I think mds_worm_retention_period may be equal to mds_worm_min_retention_period and set them to "1h", mds_worm_max_retention_period may be larger than 'mds_worm_min_retention_period` and set it to "30 years", What do you think?

the modified file mode already restore to 100644, thanks!

You might switch to the inline .set_min & .set_max helpers to perform the boundary checking, then

fixed it，thanks

xiexingguo · 2019-03-01T09:35:04Z

src/common/options.cc

+    .set_description("rate of decay in seconds for calculating request load average"),
+
+    Option("mds_worm_commit_period", Option::TYPE_UINT, Option::LEVEL_ADVANCED)
+     .set_default(7200)


s/7200/2_hr/

fixed it，thanks

xiexingguo · 2019-03-01T09:35:20Z

src/common/options.cc

+     .set_description("default auto commit period for worm feature"),
+
+    Option("mds_worm_retention_period", Option::TYPE_UINT, Option::LEVEL_ADVANCED)
+     .set_default(3600)


s/3600/1_hr/

fixed it，thanks

ukernel · 2019-03-01T13:17:02Z

src/common/legacy_config_opts.h

+OPTION(mds_worm_commit_period, OPT_U64)
+OPTION(mds_worm_retention_period, OPT_U64)
+OPTION(mds_worm_min_retention_period, OPT_U64)
+OPTION(mds_worm_max_retention_period, OPT_U64)


no need to add options to legacy_config_opts.h

ok，fixed it，thanks

ukernel · 2019-03-01T13:40:31Z

src/mds/mdstypes.h

+    ENCODE_FINISH(bl);
+  }
+  void decode(bufferlist::iterator& p) {
+    DECODE_START_LEGACY_COMPAT_LEN(1, 1, 1, p);


DECODE_START()

fixed it，thanks

ukernel · 2019-03-01T13:49:23Z

src/mds/CInode.cc

@@ -3687,6 +3687,9 @@ int CInode::encode_inodestat(bufferlist& bl, Session *session,
    }
  }

+  mempool_inode *policy_i = ppolicy ? pi : oi;
+  ::encode(policy_i->worm, bl);
+


needs to bump encoding version

fixed it，thanks

ukernel · 2019-03-01T13:51:33Z

src/messages/MClientReply.h

@@ -254,6 +256,8 @@ struct InodeStat {
        change_attr = 0;
      }
    }
+
+    ::decode(worm, p);


needs to check encoding version

fixed it，thanks

weiqiaomiao · 2019-03-04T10:25:35Z

@ukernel @batrick @jtlayton I reorganized all the commit, and makes the code more readable now, can you take a look it？

jtlayton · 2019-03-04T13:11:49Z

src/client/Client.cc

+    req->head.args.setattr.mode = stx->stx_mode;
+    req->inode_drop |= CEPH_CAP_AUTH_SHARED;
+
+    if (in->worm.is_enable() && !in->worm.is_retain() && !(stx->stx_mode & 00222)) {


I'm not sure I really like treating a mode with no write bits as special.

What about POSIX ACLs here? Suppose there are no write bits in the mode but there is a POSIX ACL set that explicitly allows write access for another user? Should that trigger this transition?

A better mechanism might be to wire up the appropriate ioctl so that you could use "chattr -i" here. That would make it a privileged operation, but that seems reasonable for this.

Thanks for your suggestion, jtlaytion!

My original intention was to find a simple way to trigger the conversion of the state and removing the Write property in mode was just one of the methods, and I didn't take into account the POSIX ACL scene.

I think your suggestion is more reasonable, but I'm not familiar with chattr, so I think I need a little time to learn how to do.

Basically, you'd just need to wire up FS_IOC_GETFLAGS and FS_IOC_SETFLAGS and fix them up to handle the IMMUTABLE flag. FUSE has an ioctl handler, so it should be possible to do this for both the kernel and fuse.

I'm very sorry for delay in reply. I add ll_set_flags() function to handle IMMUTABLE flag (make
the worm state transition in it) and ll_get_flags() function to get the flags . Can you take a look at it?

renhwztetecs · 2019-03-05T11:43:53Z

As a document backup application scenario, WORM is an important function. It can protect files from being deleted and modified easily.

renhwztetecs · 2019-03-05T12:08:17Z

src/client/Client.cc

-      if (mask & CEPH_SETATTR_ATIME)
-        in->atime = utime_t(stx->stx_atime);
+      if (mask & CEPH_SETATTR_ATIME) {
+        uint32_t min_atime_sec = in->mtime.sec()+ in->worm.min_retention_period;


Since the acquisition time comes from the local, there is a modification of the local time to cause the folder to enter the protection period in advance. It may be provided by providing an external standard clock source interface or in the document: "Synchronize the time before setting the WORM function"

What do you think? @ukernel @batrick @jtlayton

I think it'd be better to not overload the atime field. While it's not terribly common, some applications do use it. Why not store this in a separate field in the inode and present it via xattr?

OK. Already drop to use atime and add a new field in worm info struct insteed.

batrick · 2019-03-19T23:36:19Z

This looks interesting but before I dig into it I'd like you to cleanup the commit history. For example, 58fd92a needs rewritten to remove the addition to src/common/legacy_config_opts.h. Use git rebase -i and edit the commits as necessary. Please also make sure you don't add unnecessary whitespace changes. I also noticed a lot of trailing whitespace in your patches.

weiqiaomiao · 2019-05-09T00:05:13Z

jenkins，retest this please

jtlayton

I think you mean "auto-inherit" in the title.

jtlayton · 2019-05-28T16:40:12Z

src/client/Client.cc

@@ -8736,14 +8744,21 @@ int Client::_open(Inode *in, int flags, mode_t mode, Fh **fhp,
    return -EROFS;
  }

+  int result = 0;
+  if (in->is_file() && (flags & (O_WRONLY | O_RDWR  | O_TRUNC | O_APPEND))&& in->worm.is_enable()) {


nit: please fix the whitespace above

fixed it，thanks

jtlayton · 2019-05-28T16:42:00Z

src/mds/Server.h

@@ -347,6 +347,8 @@ class Server {
  void handle_conf_change(const ConfigProxy& conf,
                          const std::set <std::string> &changed);

+ // int worm_state_transition(Inode *in, int op);
+


This doesn't look related to the rest of the patch.

already deleted

jtlayton · 2019-05-28T16:46:11Z

src/client/Client.cc

@@ -11690,6 +11705,62 @@ int Client::ll_removexattr(Inode *in, const char *name, const UserPerm& perms)
  return _removexattr(in, name, perms);
 }

+int Client::worm_state_transition(Inode *in, const UserPerm& perms, int op)


A comment header would be nice here. What, specifically, is this function intended to do?

jtlayton · 2019-05-28T16:49:37Z

src/client/fuse_ll.cc

+      if (*flags & (~FS_IMMUTABLE_FL)) {
+        fuse_reply_err(req, EOPNOTSUPP);
+        return;
+      }


I'm a little concerned with this. Suppose I accidentally set this on the wrong file. How do I fix it?

I think may be only wait for file protection time expires 。any suggestion？

I don't think waiting will be acceptable -- we will need to be able to fix this in some fashion, as we can't just throw away the filesystem and start over (as you would with a CD-R or something).

Maybe this could only be allowed from certain clients using a new cephfs client capability flag, and those clients would be able to undo this as well?

http://docs.ceph.com/docs/master/cephfs/client-auth/

OK ! Already add a new cap flag "i" and "chattr -i"" to expired a file which is in retained state.

jtlayton · 2019-05-28T16:52:33Z

src/client/Client.cc

+  if (dir->worm.is_enable() || in->worm.is_enable()) {
+    ldout(cct, 8) << "_link" << "worm feature doesn't support link " << dendl;
+    return -EPERM;
+  }


We expect all files in the worm directory to be controlled，but it hardlink is supported，if a non-worm file linked to a file under worm directory, it can't controlled by worm.

jtlayton · 2019-05-29T13:55:25Z

src/mds/Server.cc

+         parse_worm_vxattr(mdr, cur, "worm.enable", "1", worm);
+      }
+
+      if (q == 0xffffffff) {                    // retetion period ulimit


Use UINT_MAX instead of 0xffffffff (here and below)

jtlayton · 2019-05-29T13:56:08Z

src/mds/mdstypes.h

+#define WORM_EXPIRE  8     // the worm file is in expired state
+#define WORM_RETEN_PERIOD_UNLIMIT 16  //the worm file retention period is unlimit
+#define MAX_MIN_RETEN_PERIOD 946080000 // 30 years
+#define MIN_MAX_RETEN_PERIOD 2207520000 // 70 years


Why these max/min values?

Our customers require a minimum retention period that can be set from 0 to 30 years, and a maximum retention period from 0 to 70 years. And to avoid other people being confused like you, I will remove them and put the restrictions on the configuration of our own management system.

jtlayton · 2019-06-07T11:08:05Z

src/client/fuse_ll.cc

+      if (*flags & (~FS_IMMUTABLE_FL)) {
+        fuse_reply_err(req, EOPNOTSUPP);
+        return;
+      }


I don't think waiting will be acceptable -- we will need to be able to fix this in some fashion, as we can't just throw away the filesystem and start over (as you would with a CD-R or something).

Maybe this could only be allowed from certain clients using a new cephfs client capability flag, and those clients would be able to undo this as well?

http://docs.ceph.com/docs/master/cephfs/client-auth/

stale · 2019-08-07T01:40:35Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

weiqiaomiao · 2019-08-24T02:11:48Z

jenkins retest this please

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

we can set multiple worm attribute at once: `setfattr -n ceph.worm -v "enable=1 auto_commit_period=60 retention_period=120 min_retention_period=120 max_retention_period=240" dir1` or set one worm attribute at a time like: ` setfattr -n ceph.worm.enable -v 1 dir1 setfattr -n ceph.worm.auto_commit_period -v 60 dir1 setfattr -n ceph.worm.min_retention_period -v 120 dir1 setfattr -n ceph.worm.max_retention_period -v 240 dir1 setfattr -n ceph.worm.retention_period -v 120 dir1 ` we can get all file worm attibute: `getfattr -n ceph.worm dir1` or get one file attribute at a time. `getfattr -n ceph.state file1` Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

…nd worm state transition Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

stale · 2019-11-01T11:25:07Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

stale · 2020-01-30T12:22:24Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

xiexingguo added feature cephfs Ceph File System labels Feb 28, 2019

xiexingguo requested review from batrick and ukernel February 28, 2019 11:16

xiexingguo reviewed Feb 28, 2019

View reviewed changes

weiqiaomiao force-pushed the wqm-wip-worm branch from 53c92a4 to 7e2fae9 Compare March 1, 2019 08:53

xiexingguo reviewed Mar 1, 2019

View reviewed changes

ukernel reviewed Mar 1, 2019

View reviewed changes

weiqiaomiao force-pushed the wqm-wip-worm branch 5 times, most recently from 51ba90b to bcfb8f0 Compare March 2, 2019 08:32

xiexingguo requested a review from jtlayton March 4, 2019 06:52

jtlayton reviewed Mar 4, 2019

View reviewed changes

renhwztetecs reviewed Mar 5, 2019

View reviewed changes

weiqiaomiao force-pushed the wqm-wip-worm branch 2 times, most recently from 9d111d9 to 607134b Compare March 16, 2019 06:36

weiqiaomiao force-pushed the wqm-wip-worm branch 5 times, most recently from fecc6a8 to 32e2560 Compare April 2, 2019 12:30

weiqiaomiao force-pushed the wqm-wip-worm branch from 92f9ff6 to ad46612 Compare May 8, 2019 00:44

weiqiaomiao force-pushed the wqm-wip-worm branch from ad46612 to de8f1de Compare May 9, 2019 11:18

jtlayton reviewed May 28, 2019

View reviewed changes

weiqiaomiao force-pushed the wqm-wip-worm branch from de8f1de to f82d62f Compare June 4, 2019 08:25

jtlayton reviewed Jun 7, 2019

View reviewed changes

stale bot added the stale label Aug 7, 2019

batrick removed their assignment Aug 7, 2019

stale bot removed the stale label Aug 7, 2019

weiqiaomiao force-pushed the wqm-wip-worm branch 5 times, most recently from cacf095 to 830fea1 Compare August 23, 2019 02:50

weiqiaomiao added 8 commits September 2, 2019 18:46

cephfs: define and initial worm file attribute info

ecbd280

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

cephfs: new file and subdir auto-inherit parent dir worm attribute, a…

24b3ca7

…nd worm state transition Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

cephfs: modify expire time

f89eed2

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

cephfs: chattr +i to trigger worm state transition

549c961

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

cephfs: broadcast to client when worm attribute modify

88ba110

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

cephfs: worm feature doesn't support hard link

fee71d0

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

cephfs: add chattr -i to expired a worm file which is in retained state

f2a5842

Signed-off-by: Wei Qiaomiao <wei.qiaomiao@zte.com.cn>

weiqiaomiao force-pushed the wqm-wip-worm branch from 830fea1 to f2a5842 Compare September 2, 2019 10:57

stale bot added the stale label Nov 1, 2019

stale bot closed this Jan 30, 2020

cephfs: support WORM(Write Once Read Many) feature #26691

cephfs: support WORM(Write Once Read Many) feature #26691

Conversation

weiqiaomiao commented Feb 28, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weiqiaomiao commented Mar 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renhwztetecs commented Mar 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

batrick commented Mar 19, 2019

weiqiaomiao commented May 9, 2019

jtlayton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stale bot commented Aug 7, 2019

weiqiaomiao commented Aug 24, 2019

stale bot commented Nov 1, 2019

stale bot commented Jan 30, 2020

weiqiaomiao commented Feb 28, 2019 •

edited