Make fewer lookup requests when inode type might be known

[jamesbornholt]

Right now our lookup implementation does two concurrent ListObjectsV2 requests. List requests are expensive, both cost-wise and performance-wise. We should think about a better strategy here, perhaps starting with a HeadObject request. If there's an existing inode, we could also use that as a hint as to which requests are most likely to succeed (only as a hint, because the bucket might have changed).

[jamesbornholt]

#69 improved this by replacing one ListObjects with a HeadObject. What's left to do is to use cached state as a hint to optimize the requests here—if we already suspect something is a file (or directory), we can start with the HeadObject (or ListObjects) and only do the other request if it fails.

[plundra]

Not sure if applicable, I'm not familiar with internals of FUSE, however:

In mount.fuse(8) I see two options:

entry_timeout=T
       The timeout in seconds for which name lookups will be cached. The default is 1.0 second. For all the timeout options, it is possible to give fractions of a second as well (e.g. entry_timeout=2.8)

attr_timeout=T
       The timeout in seconds for which file/directory attributes are cached.  The default is 1.0 second.

Which both sounds excellent, if mount-s3 could set/use these?

We have a very static object structure, so caching would be great and work very well.
Did some measurements and for one test it's 30-50% list-type requests, which are costly both in round trip time, but also money :-)

(Thanks for a very exciting and promising project btw!)

[jamesbornholt]

Yeah, we actually do set those timeouts, here:

mountpoint-s3/mountpoint-s3/src/fs.rs

Lines 201 to 216 in b632bbe

	impl Default for CacheConfig {
	fn default() -> Self {
	// We want to do as little caching as possible, but Linux filesystems behave badly when the
	// TTL is exactly zero. For example, results from `readdir` will expire immediately, and so
	// the kernel will immediately re-lookup every entry returned from `readdir`. So we apply
	// small non-zero TTLs. The goal is to be small enough that the impact on consistency is
	// minimal, but large enough that a single cache miss doesn't cause a cascading effect where
	// every other cache entry expires by the time that cache miss is serviced. We also apply a
	// longer TTL for directories, which are both less likely to change on the S3 side and
	// checked more often (for directory permissions checks).
	let file_ttl = Duration::from_millis(100);
	let dir_ttl = Duration::from_millis(1000);
	Self { file_ttl, dir_ttl }
	}
	}

We set them very low because we want to preserve S3's strong consistency model by default. But we know for some workloads, the bucket doesn't change very much/at all, and so we could cache that metadata much longer. We're tracking that as a roadmap item in #255.

This issue is tracking a smaller improvement we could make: if we've listed a file/directory previously, then when the cache expires we could speculate that it's still a file/directory when we try to look it up from S3 again. I think with the way that lookup works right now, that would allow us to skip some HeadObject requests, but probably not any ListObjects requests.