Input/output error during reading deleted object with cached metadata

[mhnap]

Mountpoint for Amazon S3 version

mount-s3 1.4.0

AWS Region

us-east-2

Describe the running environment

Running in local Ubuntu 22.04.4 LTS.

Mountpoint options

mount-s3 mhnap-bucket /media/mount-s3 --read-only --log-directory=/tmp/mount-s3/logs --debug --debug-crt --cache /tmp/mount-s3 --metadata-ttl 1000000

What happened?

I was trying to use mount-s3 with a really long --metadata-ttl and observe how caching would behave. I'm mounting with --read-only flag and expect stale content or metadata cause of caching. However, I thought that metadata caching would not interfere with content caching.

The test I ran:

$ dd if=/dev/urandom of=file bs=1 count=10
10+0 records in
10+0 records out
10 bytes copied, 0.00019309 s, 51.8 kB/s
$ aws s3 cp file s3://mhnap-bucket/dir/file
upload: ./file to s3://mhnap-bucket/dir/file
$ mount-s3 mhnap-bucket /media/mount-s3 --read-only --log-directory=/tmp/mount-s3/logs --debug --debug-crt --cache /tmp/mount-s3 --metadata-ttl 1000000
bucket mhnap-bucket is mounted at /media/mount-s3
$ ls -lh /media/mount-s3/dir/file
-rw-r--r-- 1 mhnap mhnap 10 Feb 24 01:17 /media/mount-s3/dir/file
$ aws s3 rm s3://mhnap-bucket/dir/file
delete: s3://mhnap-bucket/dir/file
$ ./read /media/mount-s3/dir/file
Err(Os { code: 5, kind: Uncategorized, message: "Input/output error" })
$ ./read /media/mount-s3/dir/file
Err(Os { code: 5, kind: Uncategorized, message: "Input/output error" })
$ umount /media/mount-s3/

So, in this test, I cache file metadata, delete it in S3, and then try to read it. (./read is basically std::fs::read)

There was no cached content, but there was cached metadata, and while getting the content for the first time I got an error during reading. The behavior I was expecting is to get object content if it was not cached before, despite metadata caching.

Worth noting that if I skip ls -lh /media/mount-s3/dir/file then I would see the correct read:

$ ./read /media/mount-s3/dir/file
Err(Os { code: 2, kind: NotFound, message: "No such file or directory" })

Cannot paste logs, so uploaded them here: mountpoint-s3-2024-02-23T23-17-27Z.log

Relevant log output

No response

[passaro]

What you describe is the expected behavior when using --cache and --metadata-ttl:

• ls discovers file and caches its metadata (no content though)
• file is removed from a different client
• reading from file then includes multiple system calls:
  • open, which succeeds (within the specified TTL)
  • read, which will fail with "Input/output error" since the content of file has never been cached before and is not on S3 anymore

[passaro]

Closing as expected behavior. Feel free to reopen or raise a new issue if you have more questions.