Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NFSv4 Permission denied even with correct permissions #2575

Open
tailtwo opened this Issue Apr 11, 2019 · 3 comments

Comments

Projects
None yet
2 participants
@tailtwo
Copy link

tailtwo commented Apr 11, 2019

Issue Report

Bug

Some files hosted on a NFSv4 export with correct permissions randomly cannot be read with any user (Permission denied). A ls -l (emphasis on th -l) solve the problem. Maybe a getattr command is "fixing" it?

Container Linux Version

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2023.5.0
VERSION_ID=2023.5.0
BUILD_ID=2019-03-09-0138
PRETTY_NAME="Container Linux by CoreOS 2023.5.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Container Linux on bare metal and ignition config.

Expected Behavior

Reading a file with correct permissions on a NFS mount returns the file content without problems.

Actual Behavior

Reading a file returns Permission denied with any account (even root) even when the correct permissions.

Reproduction Steps

  1. Mount a NFS export on the host with the following options (Kubernetes default) : rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.20.6,local_lock=none,addr=192.168.20.4
  2. Write a file in the mounted export
  3. Issue a command that read the file, sometimes and randomly the file maybe unreadable because of a permission problem.

Other Information

This behaviour appeared on 2023.5.0, the previous version (1967.6.0) worked for months without this issue. I suspect a kernel issue. I attached a tcpdump capture during a cat on a faulty file (sorry for the amount of packets but I couldn't narrow my scope more).

capture.zip

EDIT: Here is a strace of a cat command on a faulty file :

execve("/usr/bin/cat", ["cat", "vol-01/chap-06/d0eac605-08c0-43f"...], [/* 26 vars */]) = 0
brk(0)                                  = 0x563ba621f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=31149, ...}) = 0
mmap(NULL, 31149, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1888404000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\20\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1869216, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1888402000
mmap(NULL, 3976832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1887e1d000
mprotect(0x7f1887fde000, 2097152, PROT_NONE) = 0
mmap(0x7f18881de000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c1000) = 0x7f18881de000
mmap(0x7f18881e4000, 16000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f18881e4000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f1888403500) = 0
mprotect(0x7f18881de000, 16384, PROT_READ) = 0
mprotect(0x563ba4a3e000, 4096, PROT_READ) = 0
mprotect(0x7f188840c000, 4096, PROT_READ) = 0
munmap(0x7f1888404000, 31149)           = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
openat(AT_FDCWD, "vol-01/chap-06/d0eac605-08c0-43fd-9f3e-0e3c90e8de24.properties", O_RDONLY) = -1 EACCES (Permission denied)
write(2, "cat: ", 5cat: )                    = 5
write(2, "vol-01/chap-06/d0eac605-08c0-43f"..., 62vol-01/chap-06/d0eac605-08c0-43fd-9f3e-0e3c90e8de24.properties) = 62
write(2, ": Permission denied", 19: Permission denied)     = 19
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++
@ajeddeloh

This comment has been minimized.

Copy link

ajeddeloh commented Apr 11, 2019

I agree this sounds like a kernel issue. Can you reproduce on the latest alpha or beta? They're running slightly newer kernels which will tell us if the fix is already upstream.

@tailtwo

This comment has been minimized.

Copy link
Author

tailtwo commented Apr 12, 2019

Mounted the NFS export on manjaro with kernel 5.0.5-1-MANJARO and the problematic files are perfectly readable without any workaround. Will try CoreOS beta branch next week.

@ajeddeloh

This comment has been minimized.

Copy link

ajeddeloh commented Apr 12, 2019

It's worth noting the CL will remain on the 4.19.x stable kernel branch through it's end of life. It's useful to know that it works with a 5.0.x kernel, but we won't be shipping a 5.0.x kernel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.