Clarify the expected behavior of the --metadata-ttl option set to zero

[mhnap]

In an attempt to disable the metadata cache, I set the --metadata-ttl=0, referencing #616 as a potential guide. However, this resulted in a significant increase in List and Head requests, exceeding even the number observed with the cache completely disabled.

While I'm uncertain about the intended behavior of --metadata-ttl=0, I assume it should either disable the metadata cache (if feasible) or be prevented from being accepted as a CLI option.

[jamesbornholt]

Yes, --metadata-ttl=0 disables metadata caching entirely, which is almost certainly not a good thing, as filesystem performance with zero metadata caching is very bad. For Mountpoint, that means sending many more LIST/HEAD requests than necessary, as you're seeing.

When caching is disabled (i.e. no --cache or --metadata-ttl is set), the actual setting is roughly the same as --metadata-ttl=0.1 (noting that you can't actually set that as a float), as this comment explains:

mountpoint-s3/mountpoint-s3/src/fs.rs

Lines 303 to 313 in 022f915

	// We want to do as little caching as possible by default,
	// but Linux filesystems behave badly when the TTL is exactly zero.
	// For example, results from `readdir` will expire immediately, and so
	// the kernel will immediately re-lookup every entry returned from `readdir`. So we apply
	// small non-zero TTLs. The goal is to be small enough that the impact on consistency is
	// minimal, but large enough that a single cache miss doesn't cause a cascading effect where
	// every other cache entry expires by the time that cache miss is serviced. We also apply a
	// longer TTL for directories, which are both less likely to change on the S3 side and
	// checked more often (for directory permissions checks).
	let file_ttl = Duration::from_millis(100);
	let dir_ttl = Duration::from_millis(1000);

This 0.1 setting is also the reason our documentation has a caveat around consistency:

However, Mountpoint may return stale metadata for an existing object within 1 second of the object being modified or deleted in your S3 bucket by another client. This occurs only if the object was accessed through Mountpoint immediately before being modified or deleted in your S3 bucket. The stale metadata will only be visible through metadata operations such as stat on individual files. Directory listings will never be stale and always reflect the current metadata. These cases do not apply to newly created objects, which are always immediately visible through Mountpoint. Stale metadata can be refreshed by either opening the file or listing its parent directory.

We probably should just forbid --metadata-ttl=0, or convert it to mean the same thing as the default value, as I can't imagine a setting where it would actually work well. Until then, I think --metadata-ttl=1 is likely the effect you're looking for.

[mhnap]

@jamesbornholt Thank you for such a detailed explanation. Now it's clear to me why I saw such behavior for the --metadata-ttl=0 case. I just didn't realize that --metadata-ttl=0 would result in different settings than disabled caching.

Now, as I understand that --metadata-ttl=0 is technically valid and produces expected behavior, the only proposition I have is to add more details about the current default TTLs settings to the documentation. Not sure whether any changes are needed on the CLI side.

[passaro]

#855 introduces a warning when using --metadata-ttl=0:

Warning

The --metadata-ttl 0 setting is no longer supported and will be removed in a future release. The new --metadata-ttl minimal has a similar effect, but behaves better when latency for S3 requests is high.

We have also added more details in the documentation, see the Metadata Cache section in CONFIGURATION.md.