Stale file handle during reading rewritten object with cached metadata

[mhnap]

Mountpoint for Amazon S3 version

mount-s3 1.4.0

AWS Region

us-east-2

Describe the running environment

Running in local Ubuntu 22.04.4 LTS.

Mountpoint options

mount-s3 mhnap-bucket /media/mount-s3 --read-only --log-directory=/tmp/mount-s3/logs --debug --debug-crt --cache /tmp/mount-s3 --metadata-ttl 1000000

What happened?

This issue is similar to #782 which I previously created, but here I rewrite the object.

I am not sure whether this is expected behavior, so I would be grateful for clarifying.

The test I ran:

$ dd if=/dev/urandom of=file bs=1 count=10
10+0 records in
10+0 records out
10 bytes copied, 0.000198077 s, 50.5 kB/s
$ aws s3 cp file s3://mhnap-bucket/dir/file
upload: ./file to s3://mhnap-bucket/dir/file
$ mount-s3 mhnap-bucket /media/mount-s3 --read-only --log-directory=/tmp/mount-s3/logs --debug --debug-crt --cache /tmp/mount-s3 --metadata-ttl 1000000
bucket mhnap-bucket is mounted at /media/mount-s3
$ ls -lh /media/mount-s3/dir/file
-rw-r--r-- 1 mhnap mhnap 10 Feb 26 11:31 /media/mount-s3/dir/file
$ dd if=/dev/urandom of=file bs=1 count=100
100+0 records in
100+0 records out
100 bytes copied, 0.000530112 s, 189 kB/s
$ aws s3 cp file s3://mhnap-bucket/dir/file
upload: ./file to s3://mhnap-bucket/dir/file
$ ./read /media/mount-s3/dir/file
Err(Os { code: 116, kind: StaleNetworkFileHandle, message: "Stale file handle" })
$ ./read /media/mount-s3/dir/file
Ok([200, 84, 37, 49, 213, 169, 102, 191, 249, 109, 23, 138, 141, 212, 234, 50, 58, 244, 106, 160, 44, 217, 177, 118, 211, 2, 57, 181, 93, 134, 29, 84, 89, 75, 49, 37, 65, 232, 62, 250, 247, 50, 209, 144, 151, 55, 133, 102, 221, 216, 26, 133, 12, 191, 11, 201, 77, 137, 136, 120, 253, 28, 0, 143, 205, 0, 120, 14, 137, 251, 40, 167, 17, 28, 184, 127, 226, 24, 102, 194, 127, 39, 180, 231, 221, 236, 79, 63, 252, 46, 236, 0, 66, 97, 35, 224, 152, 65, 49, 45])
$ umount /media/mount-s3/

This case is different cause I can read successfully on the second attempt.

Also, if I skip ls -lh /media/mount-s3/dir/file then I would see a successful read on the first attempt.

Cannot paste logs, so uploaded them here: mountpoint-s3-2024-02-26T09-31-13Z.log

Relevant log output

No response

[passaro]

This is a similar scenario as in #782, except when trying to get the content, Mountpoint reports that the version that is in the (metadata) cache is stale compared to the one in S3.
I cannot reproduce the successful second attempt, though. When using --cache and --metadata-ttl, Mountpoint will not try to refresh its metadata cache until the TTL expires, except on specific operations, like ls on the containing directory, or when opening the file with the O_DIRECT flag - see details here.

The logs you included do in fact show lines like this:

2024-02-26T09:31:40.493833Z DEBUG readdirplus{req=366 ino=2 fh=28 offset=0}: mountpoint_s3::inode: inode needs to be recreated parent=2 name="file" ino=3

which are likely the result of a ls finding a stale metadata entry and replacing it with a fresh one.

[passaro]

Closing as expected behavior. Feel free to reopen or raise a new issue if you have more questions.