S3Path.exists() returns True on partial matches

[maciejb]

S3Path.exists() is returning True for anything that begins with the same as a real file. For instance, if we have a file named foobar, we get a false positive when querying for foo.

>>> S3Path('s3://my-bucket/foobar').exists()  # This file is really there
True
>>> S3Path('s3://my-bucket/foo').exists()  # This file does NOT exist
True

[jayqi]

Looks like this is a bug in S3Client._s3_file_query.

cloudpathlib/cloudpathlib/s3/s3client.py

Lines 153 to 165 in 2084e20

	# else, confirm it is a dir by filtering to the first item under the prefix
	except ClientError:
	return next(
	(
	obj
	for obj in (
	self.s3.Bucket(cloud_path.bucket)
	.objects.filter(Prefix=cloud_path.key)
	.limit(1)
	)
	),
	None,
	)

This branch of logic is too permissive and needs to also check if there's a /.

Also a blind spot in our tests that this isn't caught.

[pjbull]

This branch of logic is too permissive and needs to also check if there's a /.

I think append a / to the key if there isn't one in this fallback path—that said, we'll need to double check if that handles the fake directory scenario as well.

[jayqi]

Whoops, closed this by accident.

[frndmg]

Hi. Looking for some eyes on #244 if anybody has the time. We stumbled upon the same problem and fixed it by patching it but wanted to submit a fix.

I think append a / to the key if there isn't one in this fallback path—that said, we'll need to double check if that handles the fake directory scenario as well.

It is also checking that condition as well.