Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mds: add protection from clients without fscrypt support #45073

Closed

Conversation

luis-henrix
Copy link
Contributor

This is just an RFC, as I don't think this is ready for getting merged. I'm trying to prevent clients that don't have fscrypt support from breaking encrypted data. There are several ways a client can currently do that, and this patch tries to fix some of them by preventing a client from executing a certain number of operations.

This patch doesn't try to fix every way a file can be damaged. For example, a client can still append data to an encrypted file if it has the right permission for doing so. This could probably be solved by preventing some caps to be given to such clients. But I'm not sure that's something easily done and I'm not sure if that's really a problem -- clients can corrupt files as long as they can write to them, encrypted or not.

Anyway, this is just my initial attempt to fix the problem. I've several questions:

  • Is this the best approach?
  • Are these the Ops to be blocked, or am I missing some? (or maybe some of them should be allowed...?)
  • I've no idea if the locking is correct (or needed).

(I've no idea how to explicitly request reviews from some people, so I'll just mention a few github users here: @jtlayton @lxbsz @vshankar )

Signed-off-by: Luís Henriques lhenriques@suse.de

@github-actions github-actions bot added the cephfs Ceph File System label Feb 17, 2022
@jtlayton jtlayton requested review from lxbsz, vshankar and a team February 17, 2022 14:12
src/mds/Server.cc Outdated Show resolved Hide resolved
@luis-henrix luis-henrix force-pushed the wip-mds-fscrypt-protection branch 2 times, most recently from 59c750e to 93fffb5 Compare February 18, 2022 15:13
@luis-henrix
Copy link
Contributor Author

luis-henrix commented Feb 18, 2022

I pushed a new iteration. While running a few more tests I was seeing the MDS crashing, and the reason was that I was getting a NULL for the request session. TBH, I'm not sure if this may open a hole and allow clients to run these operations. I expect that, if a client is trying to execute one of these operations, the session will be there.

Also, I still have no idea if the locking here is correct.

Edit: the only operation where this NULL session is happening seems to be CEPH_MDS_OP_RENAME. Does that make sense? Any idea why?

@lxbsz
Copy link
Member

lxbsz commented Feb 20, 2022

I pushed a new iteration. While running a few more tests I was seeing the MDS crashing, and the reason was that I was getting a NULL for the request session. TBH, I'm not sure if this may open a hole and allow clients to run these operations. I expect that, if a client is trying to execute one of these operations, the session will be there.

Also, I still have no idea if the locking here is correct.

Edit: the only operation where this NULL session is happening seems to be CEPH_MDS_OP_RENAME. Does that make sense? Any idea why?

When the MDS is doing the 'reintegrate_stray' it will also send a CEPH_MDS_OP_RENAME request to other MDSes, I think this is the root cause why the there has no client session for it. In this case you can use mdr->reqid.name.is_client() to identify it.

src/mds/Server.cc Outdated Show resolved Hide resolved
src/mds/Server.cc Outdated Show resolved Hide resolved
@luis-henrix
Copy link
Contributor Author

I pushed a new iteration. While running a few more tests I was seeing the MDS crashing, and the reason was that I was getting a NULL for the request session. TBH, I'm not sure if this may open a hole and allow clients to run these operations. I expect that, if a client is trying to execute one of these operations, the session will be there.
Also, I still have no idea if the locking here is correct.
Edit: the only operation where this NULL session is happening seems to be CEPH_MDS_OP_RENAME. Does that make sense? Any idea why?

When the MDS is doing the 'reintegrate_stray' it will also send a CEPH_MDS_OP_RENAME request to other MDSes, I think this is the root cause why the there has no client session for it. In this case you can use mdr->reqid.name.is_client() to identify it.

Ah! That makes sense, I guess. I'll use mdr->reqid.name.is_client() and try to confirm this is indeed the case. Thank you for the clarification.

src/mds/Server.cc Outdated Show resolved Hide resolved
@luis-henrix
Copy link
Contributor Author

I've pushed another revision of the patch. The only thing that is still missing (if I didn't forget anything) is VXATTR, which hasn't been merged into master yet.

@lxbsz
Copy link
Member

lxbsz commented Feb 22, 2022

I've pushed another revision of the patch. The only thing that is still missing (if I didn't forget anything) is VXATTR, which hasn't been merged into master yet.

This version LGTM :-)

@luis-henrix luis-henrix changed the title RFC: mds: add protection from clients without fscrypt support mds: add protection from clients without fscrypt support Mar 2, 2022
@luis-henrix
Copy link
Contributor Author

Thanks, @lxbsz I've just removed the 'RFC' from the PR subject.

Copy link
Contributor

@vshankar vshankar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lxbsz
Copy link
Member

lxbsz commented Mar 28, 2022

https://pulpito.ceph.com/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757483/

This is introducing the inifinit looping for one create request:

2022-03-26T16:56:59.800+0000 7f2bfe917700 10 mds.0.cache fw to auth for [dir 0x10000000354 /client.0/tmp/ffsb/autom4te.cache/ [2,head] rep@1.1 dir_auth=1 state=0 f(v0 m2022-03-26T16:51:44.581046+0000 3=3+0) n(v1 rc2022-03-26T16:52:45.849739+0000 b229240 3=3+0)/n(v1 rc2022-03-26T16:51:44.581046+0000 b225928 3=3+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 dnwaiter=0 child=0 frozen=0 subtree=1 replicated=0 dirty=0 waiter=0 authpin=0 tempexporting=0 0x556207209200]
2022-03-26T16:56:59.800+0000 7f2bfe917700  7 mds.0.cache request_forward request(client.4682:1278 nref=3 cr=0x55620794cb00) to mds.1 req client_request(client.4682:1278 create #0x10000000354/requests 2022-03-26T16:56:59.199339+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) v4
2022-03-26T16:56:59.800+0000 7f2bfe917700 20 mds.0.4 get_session have 0x556206ebc800 client.4682 v1:172.21.15.136:0/907805804 state open
2022-03-26T16:56:59.800+0000 7f2bfe917700 10 mds.0.4 send_message_client client.4682 v1:172.21.15.136:0/907805804 client_request_forward(1278 to mds.1 num_fwd=256 client_must_resend) v1

After num_fwd=256 the kclient will abort the request.

@lxbsz
Copy link
Member

lxbsz commented Mar 28, 2022

https://pulpito.ceph.com/vshankar-2022-03-24_09:22:49-fs-wip-vshankar-fscrypt-20220324-121321-testing-default-smithi/6757483/

This is introducing the inifinit looping for one create request:

2022-03-26T16:56:59.800+0000 7f2bfe917700 10 mds.0.cache fw to auth for [dir 0x10000000354 /client.0/tmp/ffsb/autom4te.cache/ [2,head] rep@1.1 dir_auth=1 state=0 f(v0 m2022-03-26T16:51:44.581046+0000 3=3+0) n(v1 rc2022-03-26T16:52:45.849739+0000 b229240 3=3+0)/n(v1 rc2022-03-26T16:51:44.581046+0000 b225928 3=3+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 dnwaiter=0 child=0 frozen=0 subtree=1 replicated=0 dirty=0 waiter=0 authpin=0 tempexporting=0 0x556207209200]
2022-03-26T16:56:59.800+0000 7f2bfe917700  7 mds.0.cache request_forward request(client.4682:1278 nref=3 cr=0x55620794cb00) to mds.1 req client_request(client.4682:1278 create #0x10000000354/requests 2022-03-26T16:56:59.199339+0000 caller_uid=1000, caller_gid=1252{6,36,1000,1252,}) v4
2022-03-26T16:56:59.800+0000 7f2bfe917700 20 mds.0.4 get_session have 0x556206ebc800 client.4682 v1:172.21.15.136:0/907805804 state open
2022-03-26T16:56:59.800+0000 7f2bfe917700 10 mds.0.4 send_message_client client.4682 v1:172.21.15.136:0/907805804 client_request_forward(1278 to mds.1 num_fwd=256 client_must_resend) v1

After num_fwd=256 the kclient will abort the request.

Actually it is not inifinite loop dues to the bug in mds, that is the num_fwd is __u8, which is buggy, and I have created one PR to fix this #45669. We need to handle this in libcephfs and kclient too.

src/mds/Server.cc Outdated Show resolved Hide resolved
@luis-henrix
Copy link
Contributor Author

Actually it is not inifinite loop dues to the bug in mds, that is the num_fwd is __u8, which is buggy, and I have created one PR to fix this #45669. We need to handle this in libcephfs and kclient too.

Does this mean that this isn't an issue with this PR, but a real MDS bug triggered by it?

@lxbsz
Copy link
Member

lxbsz commented Mar 29, 2022

Actually it is not inifinite loop dues to the bug in mds, that is the num_fwd is __u8, which is buggy, and I have created one PR to fix this #45669. We need to handle this in libcephfs and kclient too.

Does this mean that this isn't an issue with this PR, but a real MDS bug triggered by it?

Yeah, just your PR makes that bug to be visible more easily. If the metadata pool is full we will also hit the same issue.

Why the create request is bouncing between two MDSes is that there has one dir, which the CInode's auth the in mds.0 while the CDir's auth is in mds.1. So when creating a file under this dir the client will send the request to CDir's auth mds.1, but your code will try to get the auth CInode, which is in mds.0, so it just forwarded to mds.0.

And when creating a file it must in the CDir's auth mds, so then the mds.0 just forwarded the request back.

@lxbsz
Copy link
Member

lxbsz commented Aug 3, 2023

To fix this in the MDCache::path_traverse() should be proper and then we can hold all the locks needed and then do the fscrypt_auth check, if it's old client and the file enables fscrypt then just fail the request and then drop all the locks.

@luis-henrix
Copy link
Contributor Author

To fix this in the MDCache::path_traverse() should be proper and then we can hold all the locks needed and then do the fscrypt_auth check, if it's old client and the file enables fscrypt then just fail the request and then drop all the locks.

OK, makes sense but it would take me quite some time to figure that out. So, feel free to give it a try -- I'm available to have a look too, and test.

Also, the patch I was testing is this one:

diff --git a/src/include/fs_types.h b/src/include/fs_types.h
index c1932bfcc30e..c14d7c3cc17d 100644
--- a/src/include/fs_types.h
+++ b/src/include/fs_types.h
@@ -48,6 +48,7 @@ class JSONObj;
 #define CEPHFS_EFAULT          14
 #define CEPHFS_EISCONN         106
 #define CEPHFS_EMULTIHOP       72
+#define CEPHFS_ENOKEY          126
 
 // taken from linux kernel: include/uapi/linux/fcntl.h
 #define CEPHFS_AT_FDCWD        -100    /* Special value used to indicate
diff --git a/src/mds/Server.cc b/src/mds/Server.cc
index 6d91d5fcda22..9de341cb2225 100644
--- a/src/mds/Server.cc
+++ b/src/mds/Server.cc
@@ -2650,7 +2650,27 @@ void Server::dispatch_client_request(MDRequestRef& mdr)
     respond_to_request(mdr, mdr->more()->peer_error);
     return;
   }
-  
+
+  Session *session = mds->get_session(req);
+  if (session && !session->info.has_feature(CEPHFS_FEATURE_ALTERNATE_NAME)) {
+    CInode *cur = mdcache->get_inode(req->get_filepath().get_ino());
+    if (!cur)
+      return;
+
+    MutationImpl::LockOpVec lov;
+    /* We need 'As' caps for the fscrypt context */
+    lov.add_rdlock(&cur->authlock);
+    if (!mds->locker->acquire_locks(mdr, lov))
+      return;
+
+    if (!cur->get_inode()->fscrypt_auth.empty()) {
+      dout(10) << "blocking '" << ceph_mds_op_name(req->get_op())
+              << "' operation in encrypted node" << dendl;
+      respond_to_request(mdr, -CEPHFS_ENOKEY);
+      return;
+    }
+  }
+
   if (is_full) {
     CInode *cur = try_get_auth_inode(mdr, req->get_filepath().get_ino());
     if (!cur) {

But, as you said above, it may introduce a deadlock.

@luis-henrix
Copy link
Contributor Author

luis-henrix commented Aug 7, 2023 via email

@lxbsz
Copy link
Member

lxbsz commented Aug 8, 2023

Xiubo Li @.***> writes:
To fix this in the MDCache::path_traverse() should be proper and then we can hold all the locks needed and then do the fscrypt_auth check, if it's old client and the file enables fscrypt then just fail the request and then drop all the locks.
Ok, here's another iteration: I've tried the patch below and it seems to fix block operations from old clients without the locking complexity. However, it still allows an old client to corrupt files (for example, by appending data to a file). diff diff --git a/src/mds/MDCache.cc b/src/mds/MDCache.cc index 014e9bb09c2b..7d916daffca1 100644 --- a/src/mds/MDCache.cc +++ b/src/mds/MDCache.cc @@ -8315,9 +8315,30 @@ int MDCache::path_traverse(MDRequestRef& mdr, MDSContextFactory& cf, if (flags & MDS_TRAVERSE_CHECK_LOCKCACHE) mds->locker->find_and_attach_lock_cache(mdr, cur); - if (mdr && mdr->lock_cache) { - if (flags & MDS_TRAVERSE_WANT_DIRLAYOUT) + if (mdr) { + if (mdr->lock_cache && (flags & MDS_TRAVERSE_WANT_DIRLAYOUT)) mdr->dir_layout = mdr->lock_cache->get_dir_layout(); + const cref_t<MClientRequest> &req = mdr->client_request; + Session *session = mds->get_session(req); + if (session && !session->info.has_feature(CEPHFS_FEATURE_ALTERNATE_NAME) && + req->get_op() != CEPH_MDS_OP_LOOKUP && + req->get_op() != CEPH_MDS_OP_LOOKUPHASH && + req->get_op() != CEPH_MDS_OP_LOOKUPPARENT && + req->get_op() != CEPH_MDS_OP_LOOKUPINO && + req->get_op() != CEPH_MDS_OP_LOOKUPNAME && + req->get_op() != CEPH_MDS_OP_LOOKUPSNAP && + req->get_op() != CEPH_MDS_OP_RMSNAP && + req->get_op() != CEPH_MDS_OP_LSSNAP && + req->get_op() != CEPH_MDS_OP_GETATTR && + req->get_op() != CEPH_MDS_OP_READDIR && + req->get_op() != CEPH_MDS_OP_UNLINK && + req->get_op() != CEPH_MDS_OP_RMDIR) { + if (!cur->get_inode()->fscrypt_auth.empty()) { + dout(10) << "blocking '" << ceph_mds_op_name(req->get_op()) + << "' operation in encrypted node" << dendl; + return -CEPHFS_EROFS; + } + } } else if (rdlock_snap) { int n = (flags & MDS_TRAVERSE_RDLOCK_SNAP2) ? 1 : 0; if ((n == 0 && !(mdr->locking_state & MutationImpl::SNAP_LOCKED)) ||

BTW, where did you get the authlock before checking the cur->get_inode()->fscrypt_auth ?

@luis-henrix
Copy link
Contributor Author

BTW, where did you get the authlock before checking the cur->get_inode()->fscrypt_auth ?

Hmm... good point, I guess I'll need to add that too.

But right now what I'm still struggling to figure out is: how to prevent old clients from getting caps that allow them write to a file. I've been trying to hack something into Locker::handle_client_caps() but no luck so far.

I really hope to sort this out before the 6.6 kernel merge window opens. (I'm obviously assuming 6.6 is still the current target for the kernel client fscrypt code to be merged.)

@lxbsz
Copy link
Member

lxbsz commented Aug 8, 2023

BTW, where did you get the authlock before checking the cur->get_inode()->fscrypt_auth ?

Hmm... good point, I guess I'll need to add that too.

But right now what I'm still struggling to figure out is: how to prevent old clients from getting caps that allow them write to a file. I've been trying to hack something into Locker::handle_client_caps() but no luck so far.

I really hope to sort this out before the 6.6 kernel merge window opens. (I'm obviously assuming 6.6 is still the current target for the kernel client fscrypt code to be merged.)

The kernel patches will be merged in 6.6 window most possibly.

Clients that do not support fscrypt can execute operations that may cause
unrecoverable data loss.  Add protection on the MDS so that it prevents
these clients from executing some operations.

Note, however, that clients will still be able corrupt encrypted files by
appending data to them.  And they will still be able to read encrypted
data from those files.

Signed-off-by: Luís Henriques <lhenriques@suse.de>
@luis-henrix
Copy link
Contributor Author

OK, I've updated my branch with the latest patch. As I mentioned above, I was trying to prevent old clients from reading encrypted files and from corrupting data in encrypted files by writing to them. And I failed, the code is way too complex and I simply can't get anywhere trying to have the MDS from handing caps.

@lxbsz
Copy link
Member

lxbsz commented Aug 9, 2023

OK, I've updated my branch with the latest patch. As I mentioned above, I was trying to prevent old clients from reading encrypted files and from corrupting data in encrypted files by writing to them. And I failed, the code is way too complex and I simply can't get anywhere trying to have the MDS from handing caps.

It's hard to change the code the prevent the caps IMO, because you may need to change the Locker code and Locker state in MDS, which is not acceptable mostly.

From my side I think just try to acquire the authlock in path_traverse() is the best approach. But please make sure the lock order.

And then check the fscrypt in each request handler instead of in path_traverse().

@luis-henrix
Copy link
Contributor Author

luis-henrix commented Aug 9, 2023 via email

@lxbsz
Copy link
Member

lxbsz commented Aug 9, 2023

From my side I think just try to acquire the authlock in path_traverse() is the best approach. But please make sure the lock order. And then check the fscrypt in each request handler instead of in path_traverse().
OK, I'm confused. You're saying that, for each CEPH_MDS_OP_* that is not allowed in old clients, I'll need to repeat this check? Can't we identify a single point where this can be done for all of them instead? Why can't path_traverse() be used for that? Also, regarding the locking and the locking order, I really don't know what else to do :-(

That's just my investigation and maybe yours is better. Please go on and push your change and let's whether could we fix it.

@luis-henrix
Copy link
Contributor Author

luis-henrix commented Aug 9, 2023 via email

@lxbsz
Copy link
Member

lxbsz commented Aug 22, 2023

jenkins retest this please

req->get_op() != CEPH_MDS_OP_RMDIR) {
MutationImpl::LockOpVec lov;
/* We need 'As' caps for the fscrypt context */
lov.add_rdlock(&cur->authlock);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for late, I was on PTO last week.

This will work IMO, but as I mentioned this will break the Locker order.

Could we do this in the following code ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luis-henrix ping?

@luis-henrix
Copy link
Contributor Author

luis-henrix commented Sep 4, 2023 via email

@lxbsz
Copy link
Member

lxbsz commented Sep 7, 2023

Venky Shankar @.***> writes:
@luis-henrix ping?
Oops, sorry I was on vacations and missed initial comment from Xiubo.
This will work IMO, but as I mentioned this will break the Locker order. Could we do this in the following code ?
Unfortunately, as I said above, I still don't understand what's broken and how to fix it :-( Can someone with a better understand of the MDS/Locker code help fixing this?

@luis-henrix No worry, let me work on it. Thanks.

@lxbsz
Copy link
Member

lxbsz commented Sep 11, 2023

@luis-henrix Could you append and try 7fa942f ?

@luis-henrix
Copy link
Contributor Author

luis-henrix commented Sep 11, 2023 via email

@lxbsz
Copy link
Member

lxbsz commented Sep 12, 2023

Xiubo Li @.***> writes:
@luis-henrix Could you append and try 7fa942f ?
Thanks a lot for your help, Xiubo! I'll probably be able to give it a try tomorrow. Just to clarify, this commit has to be applied on top this PR, right?

Yeah, right. Finally we should fold them.

@luis-henrix
Copy link
Contributor Author

luis-henrix commented Sep 12, 2023 via email

@lxbsz
Copy link
Member

lxbsz commented Sep 12, 2023

prevents clients from writing (or creating files, etc) into encrypted directories,

Sorry, please try lxbsz@54bd96e. If this still doesn't work I will try more test. Thanks.

@luis-henrix
Copy link
Contributor Author

luis-henrix commented Sep 13, 2023 via email

Copy link

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@github-actions github-actions bot added the stale label Nov 12, 2023
@lxbsz
Copy link
Member

lxbsz commented Nov 30, 2023

prevents clients from writing (or creating files, etc) into encrypted directories, Sorry, please try lxbsz@54bd96e. If this still doesn't work I will try more test. Thanks.
Thank you Xiubo. I've done a quick test and it seems to be working (but I haven't tested it thoroughly, of course). Since the bulk of the code is in your commit, I'd like to suggest that you pick the code in my commit, merge into yours and simply create a new PR (and this one should be closed, of course).

@luis-henrix I will take over this PR. Thanks for your nice work :-)

@lxbsz
Copy link
Member

lxbsz commented Nov 30, 2023

Raised a new PR to fix this #54725. Just close this one.

@lxbsz lxbsz closed this Nov 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cephfs Ceph File System wip-vshankar-fscrypt CephFS fscrypt changes
Projects
None yet
4 participants