Wip hostpapa 18.2.7 skip error low level apple - #7
Open
sajibreadd wants to merge 55 commits into
Open
Conversation
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit a27ed27) Conflicts: src/test/libcephfs/CMakeLists.txt libcephfs/client: pin inode/dentry for an opened directory Fixes: https://tracker.ceph.com/issues/69092 Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 7b8a86f) test/libcephfs: remove warning in Windows build Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit c9d7345) common/ceph_fs: Enable O_DIRECTORY|O_NOFOLLOWUP flags translation under Win in ceph_flags_sys2wire() Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 4badc83) cephfs-mirror-hostpapa-reef (cherry picked from commit e904f62436db6a2eb295b7a128a5b5d42f7df68b) cephfs/client: dir_reset_t::reset() - add missing fd reset. Likely ceph#55619 missed that. Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 6f1d21c)
(cherry picked from commit ec8274db230a86c89dce0f29d9e157c568c6362b)
To address a bug and future ones where batching lookup/getattr does not help "kick" the MDS in switching state more quickly (e.g. flushing the MDS journal). Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit 0201f86)
Directories and dentries are initialized with value 0 which makes detecting a null (or placeholder) dentry created via Client::get_or_create difficult. We already do checks to see if a dentry is invalid when the directory's shared_gen changes so use an invalid value for these synthetic dentries until they can be appropriately updated from Client::update_dentry_lease. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit 4f8d1d3)
When passing an empty string to filepath, it would test if the first character is '/' which is an invalid access through std::string_view. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit e62fce4)
This gets converted in messenger douts. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit da50c6e)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit d6a5656)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit 8e88fb0)
For debug prints. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit aeb07bb)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit a84c396)
So logs show when they are modified. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit be954e0)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit d27eb1a)
To be used in subsequent commits. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit 5f53a23)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit 3c3d56f)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit ff42d86)
There is no functional change. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit c2e5512)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit 5c0fa11)
It doesn't need to be public; the API does not expose alternate_name. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit 45ff499)
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit f346e9a)
Client::update_dentry_lease also moves alternate_name from the lease which causes it to zero out alternate_name on the second move. Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit 9c276fd)
The dentry alternate_name was exposed primarily to facilitate testing. Instead, put these methods in the TestClient scaffolding to allow reading/manipulating alternate_name. Because we will be using alternate_name to handle encrypted names and the normalized / case folded name, we do not want the application to change the meaning of the metadata out-of-band. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> Fixes: https://tracker.ceph.com/issues/66373 (cherry picked from commit 1e34963)
This condition:
(!dn->inode || dn->inode->caps_issued_mask(mask, true))
is already checked in the surrounding `if`.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit 41a3f09)
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit bac86c2)
That we do not get unexpected ENOENT following readdirs (particularly thinking of cap_shared_gen differences). Test-case-for: https://tracker.ceph.com/issues/70100 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit 0f2abc9)
That the cap_shared_gen value does not result in unexpected ENOENT. Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit aaaa042)
This is groundwork to ensure that all code paths properly apply dentry name
transformations during traversal, specifically for casefolding.
Note some ancillary changes:
* readlink now correctly does lookup permission checks during path walk (previously a TODO).
* There has been some tweaking of the method signatures to accept an `const
InodeRef&`. This was convenient when passing walk_dentry_result::diri around
but the cascade in changes prompted me to leave it partially done to avoid
blowing up this changeset.
It's worth noting that this change is somewhat "half done". I had made an
effort to have all operations (whether fuse or libcephfs) call path_walk once.
This was done for Client::_mkdir and a few others. See for example:
- do_mkdirat -> _mkdir
- mksnap -> _mkdir
- ll_mkdir -> _mkdir
- mkdirs -> _mkdir (this had other significant simplification/cleanup);
however: note that we still have two calls to path_walk for any given
directory to be created. This is not a problem because:
Client::path_walk is now more tolerant of being called in a openat-style way.
The method now accepts an anchor directory and relative path (which may be a
single dentry name for ll_* calls). It is also tolerant of the directory inode
in fact referencing the target inode with relative path == "". That is a useful
property for many openat-style APIs (although largely unofficially). It also
means that if we resolve a path then we can pass the resolved Inode (directory)
/ string (Dentry name) pair to another method which may also call path_walk
with minimal replication of path walk work. This is done a few times in this
changeset.
Fixes: https://tracker.ceph.com/issues/66373
Fixes: https://tracker.ceph.com/issues/70100
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit b8662fd)
Conflicts:
src/client/Client.cc
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit 92df0fc)
ll_walk was wrongly coded to make the input file path relative to ino=0. That inode does not exist and isn't considered the root inode. Fixes: https://tracker.ceph.com/issues/70573 Fixes: e08210d Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit 25eb99d)
…ded. This is required when more entries with the same name don't fit into the fragment. With the existing means for fragment offset specification such a splitting to be prohibited. Fixes: https://tracker.ceph.com/issues/72518 Signed-off-by: Igor Fedotov <igor.fedotov@croit.io> (cherry picked from commit 24955e6)
Fixes: https://tracker.ceph.com/issues/75885 Signed-off-by: Md Mahamudur Rahaman Sajib <mahamudur.sajib@croit.io> (cherry picked from commit a886d63) Conflicts: src/include/cephfs/libcephfs.h
Fixes: http://tracker.ceph.com/issues/70287 Signed-off-by: Venky Shankar <vshankar@redhat.com> (cherry picked from commit 33c6f23)
The lock cache is buggy and we need to disable it as a workaround. Fixes: https://tracker.ceph.com/issues/65607 Signed-off-by: Xiubo Li <xiubli@redhat.com> (cherry picked from commit a86ee49) Conflicts: src/mds/MDSRank.cc
In case the current request has lock cache attached and then the lock cache must have already acquired the wrlock of filelock. So currently the path_traverse() will acquire the wrlock twice and possibly caused deadlock by itself. Fixes: https://tracker.ceph.com/issues/65607 Signed-off-by: Xiubo Li <xiubli@redhat.com> Signed-off-by: Sunnatillo <sunnat.samadov@est.tech> (cherry picked from commit 276b81a) Conflicts: src/mds/MDSRank.cc
If there are trailing newlines in a string (like long_desc), then the generated C++ code is invalid (because the newline will not be escaped). Signed-off-by: Patrick Donnelly <pdonnell@ibm.com> (cherry picked from commit 73792b7)
A getattr/lookup can cause a wrlock or xlock to become unstable after a request
(like rename) acquires it but before early reply. The MDS will not nudge the
log in this situation and the getattr/lookup will need to wait for the eventual
journal flush before the lock is released.
Now looks like:
2025-06-27T19:41:32.043+0000 7f11d21a9640 5 mds.0.log _submit_thread 25185408~2845 : EUpdate rename [metablob 0x1, 3 dirs]
2025-06-27T19:41:32.043+0000 7f11d99b8640 1 -- [v2:172.21.10.4:6868/56297870,v1:172.21.10.4:6869/56297870] --> [v2:172.21.10.4:6818/999439823,v1:172.21.10.4:6819/999439823] -- osd_op(unknown.0.23:135 2.e 2:7bf7e7b5:::200.00000006:head [write 19584~2865 [fadvise_dontneed] in=2865b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e72) -- 0x563e9b730000 con 0x563e9a674800
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (dn xlock x=1 by request(client.4393:20702 nref=6 cr=0x563e9b83ae00)) on [dentry #0x1/a/file [2,head] auth (dn xlock x=1 by request(client.4393:20702 nref=6 cr=0x563e9b83ae00)) (dversion lock w=1 last_client=4393) pv=14558 v=14556 ap=2 ino=0x10000000002 remote_ino=0x0 referent_inode_ptr=(nil) referent_ino=0x0 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x563e95629900]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (dn xlock x=1 by request(client.4393:20702 nref=6 cr=0x563e9b83ae00)) on [dentry #0x1/b/file [2,head] auth NULL (dn xlock x=1 by request(client.4393:20702 nref=6 cr=0x563e9b83ae00)) (dversion lock w=1 last_client=4393) pv=14557 v=14555 ap=2 ino=(nil) remote_ino=0x0 referent_inode_ptr=(nil) referent_ino=0x0 state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 waiter=0 authpin=1 0x563e95629b80]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (dversion lock w=1 last_client=4393) on [dentry #0x1/a/file [2,head] auth (dn xlock x=1 by request(client.4393:20702 nref=6 cr=0x563e9b83ae00)) (dversion lock w=1 last_client=4393) pv=14558 v=14556 ap=2 ino=0x10000000002 remote_ino=0x0 referent_inode_ptr=(nil) referent_ino=0x0 state=1610612736 | request=1 lock=2 inodepin=1 dirty=1 waiter=0 authpin=1 0x563e95629900]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (dversion lock w=1 last_client=4393) on [dentry #0x1/b/file [2,head] auth NULL (dn xlock x=1 by request(client.4393:20702 nref=6 cr=0x563e9b83ae00)) (dversion lock w=1 last_client=4393) pv=14557 v=14555 ap=2 ino=(nil) remote_ino=0x0 referent_inode_ptr=(nil) referent_ino=0x0 state=1610612736 | request=1 lock=2 inodepin=0 dirty=1 waiter=0 authpin=1 0x563e95629b80]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (ifile excl w=1) on [inode 0x10000000001 [...2,head] /b/ auth v29159 pv29165 ap=1 DIRTYPARENT f(v0 m2025-06-27T19:41:32.017971+0000) n(v0 rc2025-06-27T19:41:32.017971+0000 1=0+1) (isnap sync r=1) (inest lock w=1) (ifile excl w=1) (iversion lock w=1 last_client=4393) caps={4393=pAsLsXsFsx/-@955},l=4393 | request=0 lock=4 dirfrag=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x563e9a6f5080]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (inest lock w=1 dirty) on [inode 0x1 [...2,head] / auth v179 snaprealm=0x563e9a6ce6c0 f(v0 m2025-06-27T19:09:29.187695+0000 2=0+2) n(v45 rc2025-06-27T19:41:31.249722+0000 3=0+3)/n(v0 rc2025-06-27T19:08:43.024940+0000 1=0+1) (isnap sync r=2) (inest lock w=1 dirty) caps={4359=pAsLsXsFs/-@83,4393=pAsLsXsFs/-@1245} | dirtyscattered=1 request=0 lock=2 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x563e9a6f4b00]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (inest lock w=1) on [inode 0x10000000001 [...2,head] /b/ auth v29159 pv29165 ap=1 DIRTYPARENT f(v0 m2025-06-27T19:41:32.017971+0000) n(v0 rc2025-06-27T19:41:32.017971+0000 1=0+1) (isnap sync r=1) (inest lock w=1) (ifile excl w=1) (iversion lock w=1 last_client=4393) caps={4393=pAsLsXsFsx/-@955},l=4393 | request=0 lock=4 dirfrag=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x563e9a6f5080]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (iquiesce lock w=1 last_client=4393) on [inode 0x10000000000 [...2,head] /a/ auth v29161 pv29163 ap=3 DIRTYPARENT f(v0 m2025-06-27T19:41:32.017971+0000 1=1+0) n(v0 rc2025-06-27T19:41:32.017971+0000 2=1+1) (isnap sync r=2) (inest lock w=1) (ifile lock->sync w=1) (iversion lock w=1 last_client=4393) (iquiesce lock w=1 last_client=4393) caps={4359=pAsLsXs/-@109,4393=pAsLsXs/-@955} | request=1 lock=5 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=1 authpin=1 0x563e9a6f5600]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (isnap sync r=1) on [inode 0x10000000001 [...2,head] /b/ auth v29159 pv29165 ap=1 DIRTYPARENT f(v0 m2025-06-27T19:41:32.017971+0000) n(v0 rc2025-06-27T19:41:32.017971+0000 1=0+1) (isnap sync r=1) (inest lock w=1) (ifile excl w=1) (iversion lock w=1 last_client=4393) caps={4393=pAsLsXsFsx/-@955},l=4393 | request=0 lock=4 dirfrag=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x563e9a6f5080]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (isnap sync r=2) on [inode 0x1 [...2,head] / auth v179 snaprealm=0x563e9a6ce6c0 f(v0 m2025-06-27T19:09:29.187695+0000 2=0+2) n(v45 rc2025-06-27T19:41:31.249722+0000 3=0+3)/n(v0 rc2025-06-27T19:08:43.024940+0000 1=0+1) (isnap sync r=2) (inest lock w=1 dirty) caps={4359=pAsLsXsFs/-@83,4393=pAsLsXsFs/-@1245} | dirtyscattered=1 request=0 lock=2 dirfrag=1 caps=1 openingsnapparents=0 dirty=1 authpin=0 0x563e9a6f4b00]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (iversion lock w=1 last_client=4393) on [inode 0x10000000000 [...2,head] /a/ auth v29161 pv29163 ap=3 DIRTYPARENT f(v0 m2025-06-27T19:41:32.017971+0000 1=1+0) n(v0 rc2025-06-27T19:41:32.017971+0000 2=1+1) (isnap sync r=2) (inest lock w=1) (ifile lock->sync w=1) (iversion lock w=1 last_client=4393) (iquiesce lock w=1 last_client=4393) caps={4359=pAsLsXs/-@109,4393=pAsLsXs/-@955} | request=1 lock=5 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=1 authpin=1 0x563e9a6f5600]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log NO (iversion lock w=1 last_client=4393) on [inode 0x10000000001 [...2,head] /b/ auth v29159 pv29165 ap=1 DIRTYPARENT f(v0 m2025-06-27T19:41:32.017971+0000) n(v0 rc2025-06-27T19:41:32.017971+0000 1=0+1) (isnap sync r=1) (inest lock w=1) (ifile excl w=1) (iversion lock w=1 last_client=4393) caps={4393=pAsLsXsFsx/-@955},l=4393 | request=0 lock=4 dirfrag=1 caps=1 dirtyparent=1 dirty=1 authpin=1 0x563e9a6f5080]
2025-06-27T19:41:32.043+0000 7f11d99b8640 10 mds.0.locker nudge_log YES (ifile lock->sync w=1) on [inode 0x10000000000 [...2,head] /a/ auth v29161 pv29163 ap=3 DIRTYPARENT f(v0 m2025-06-27T19:41:32.017971+0000 1=1+0) n(v0 rc2025-06-27T19:41:32.017971+0000 2=1+1) (isnap sync r=2) (inest lock w=1) (ifile lock->sync w=1) (iversion lock w=1 last_client=4393) (iquiesce lock w=1 last_client=4393) caps={4359=pAsLsXs/-@109,4393=pAsLsXs/-@955} | request=1 lock=5 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=1 authpin=1 0x563e9a6f5600]
2025-06-27T19:41:32.043+0000 7f11d99b8640 20 mds.0.locker : request(client.4393:20702 nref=5 cr=0x563e9b83ae00)
2025-06-27T19:41:32.043+0000 7f11d99b8640 20 mds.0.log _submit_entry EUpdate rename [metablob 0x1, 3 dirs]
Easily reproducible with two ceph-fuse clients. One session doing:
$ mkdir a b
$ touch a/file
$ while true; do mv -v a/file b/file; mv -v b/file a/file; done
and the other
$ while true; do stat a/file; done
You can observe the rename/stat stalls (up to 5 seconds; MDS tick interval) without this patch.
Fixes: https://tracker.ceph.com/issues/71876
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
(cherry picked from commit db5c9dc)
Conflicts:
src/mds/Locker.h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an
xbetween the brackets:[x]. Spaces and capitalization matter when checking off items this way.Checklist
Show available Jenkins commands
jenkins test classic perfJenkins Job | Jenkins Job Definitionjenkins test crimson perfJenkins Job | Jenkins Job Definitionjenkins test signedJenkins Job | Jenkins Job Definitionjenkins test make checkJenkins Job | Jenkins Job Definitionjenkins test make check arm64Jenkins Job | Jenkins Job Definitionjenkins test submodulesJenkins Job | Jenkins Job Definitionjenkins test dashboardJenkins Job | Jenkins Job Definitionjenkins test dashboard cephadmJenkins Job | Jenkins Job Definitionjenkins test apiJenkins Job | Jenkins Job Definitionjenkins test docsReadTheDocs | Github Workflow Definitionjenkins test ceph-volume allJenkins Jobs | Jenkins Jobs Definitionjenkins test windowsJenkins Job | Jenkins Job Definitionjenkins test rook e2eJenkins Job | Jenkins Job DefinitionYou must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.