New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoid getting meta sync lock if no need #17172
Avoid getting meta sync lock if no need #17172
Conversation
cc @elega |
Thanks for the contribution! |
Did you see this issue in the master branch? If not, why is this only triggered in your branch? |
@tcrain any thoughts? |
No. We didn't use the master branch in our production clusters. |
I have not tested that if the master branch also has the following problem:
If yes, then the PR is still needed. If no, then it may just be an issue in our internal branch. I think this point can be confirmed from code. |
Could you please help double check to see if the master branch has this problem? If the master branch has no such problem, there must be some problem someplace else triggered this issue in your branch. |
Since yimin confirmed, I am fine with this pr. Thanks for fixing it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
alluxio-bot, merge this please |
### What changes are proposed in this pull request? Avoid getting meta sync lock if no need. ### Why are the changes needed? After backporting Alluxio#16241, "ls /" encountered the following exception ``` 2023-03-30 14:00:51,657 ERROR FileSystemMasterClientServiceHandler - Exit (Error): ListStatus: request=path: "/" options { loadMetadataType: ONCE commonOptions { syncIntervalMs: 86400000 ttl: -1 ttlAction: DELETE } recursive: false loadMetadataOnly: false } java.lang.RuntimeException: Call cancelled by trackers: GRPC_CLIENT_TRACKER at alluxio.master.file.RpcContext.throwIfCancelled(RpcContext.java:107) at alluxio.master.file.InodeSyncStream.sync(InodeSyncStream.java:322) at alluxio.master.file.DefaultFileSystemMaster.syncMetadata(DefaultFileSystemMaster.java:3816) at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:1042) ... ``` We found that "ls /" was waiting for the WRITE lock of "/" in MetadataSyncLockManager even it did not need to sync metadata. As other clients were "ls" other paths and they hold READ lock on "/", "ls /" got stuck. ### Does this PR introduce any user facing changes? NO pr-link: Alluxio#17172 change-id: cid-405084e218b150b4ffcce08294b30b3dd867a309
### What changes are proposed in this pull request? Avoid getting meta sync lock if no need. ### Why are the changes needed? After backporting Alluxio#16241, "ls /" encountered the following exception ``` 2023-03-30 14:00:51,657 ERROR FileSystemMasterClientServiceHandler - Exit (Error): ListStatus: request=path: "/" options { loadMetadataType: ONCE commonOptions { syncIntervalMs: 86400000 ttl: -1 ttlAction: DELETE } recursive: false loadMetadataOnly: false } java.lang.RuntimeException: Call cancelled by trackers: GRPC_CLIENT_TRACKER at alluxio.master.file.RpcContext.throwIfCancelled(RpcContext.java:107) at alluxio.master.file.InodeSyncStream.sync(InodeSyncStream.java:322) at alluxio.master.file.DefaultFileSystemMaster.syncMetadata(DefaultFileSystemMaster.java:3816) at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:1042) ... ``` We found that "ls /" was waiting for the WRITE lock of "/" in MetadataSyncLockManager even it did not need to sync metadata. As other clients were "ls" other paths and they hold READ lock on "/", "ls /" got stuck. ### Does this PR introduce any user facing changes? NO pr-link: Alluxio#17172 change-id: cid-405084e218b150b4ffcce08294b30b3dd867a309
### What changes are proposed in this pull request? Avoid getting meta sync lock if no need. ### Why are the changes needed? After backporting Alluxio#16241, "ls /" encountered the following exception ``` 2023-03-30 14:00:51,657 ERROR FileSystemMasterClientServiceHandler - Exit (Error): ListStatus: request=path: "/" options { loadMetadataType: ONCE commonOptions { syncIntervalMs: 86400000 ttl: -1 ttlAction: DELETE } recursive: false loadMetadataOnly: false } java.lang.RuntimeException: Call cancelled by trackers: GRPC_CLIENT_TRACKER at alluxio.master.file.RpcContext.throwIfCancelled(RpcContext.java:107) at alluxio.master.file.InodeSyncStream.sync(InodeSyncStream.java:322) at alluxio.master.file.DefaultFileSystemMaster.syncMetadata(DefaultFileSystemMaster.java:3816) at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:1042) ... ``` We found that "ls /" was waiting for the WRITE lock of "/" in MetadataSyncLockManager even it did not need to sync metadata. As other clients were "ls" other paths and they hold READ lock on "/", "ls /" got stuck. ### Does this PR introduce any user facing changes? NO pr-link: Alluxio#17172 change-id: cid-405084e218b150b4ffcce08294b30b3dd867a309
What changes are proposed in this pull request?
Avoid getting meta sync lock if no need.
Why are the changes needed?
After backporting #16241, "ls /" encountered the following exception
We found that "ls /" was waiting for the WRITE lock of "/" in MetadataSyncLockManager even it did not need to sync metadata. As other clients were "ls" other paths and they hold READ lock on "/", "ls /" got stuck.
Does this PR introduce any user facing changes?
NO