S3 identifier with @ in key raises RuntimeError

[jayantj]

Trying to open a file on S3 with a @ in the filename or prefix raises a RuntimeError

uri = 's3://bucketname/docs/x@t.ai/test.pdf'
f = smart_open(uri, 'rb')

RuntimeError                              Traceback (most recent call last)
<ipython-input-60-6156dc49e12c> in <module>()
----> 1 f = smart_open(uri, 'rb')

/data/environs/nlp/local/lib/python2.7/site-packages/smart_open/smart_open_lib.pyc in smart_open(uri, mode, **kw)
    120         # this method just routes the request to classes handling the specific storage
    121         # schemes, depending on the URI protocol in `uri`
--> 122         parsed_uri = ParseUri(uri)
    123 
    124         if parsed_uri.scheme in ("file", ):

/data/environs/nlp/local/lib/python2.7/site-packages/smart_open/smart_open_lib.pyc in __init__(self, uri, default_scheme)
    251                 # Bucket names can contain lowercase letters, numbers, and hyphens.
    252                 # Each label must start and end with a lowercase letter or a number.
--> 253                 raise RuntimeError("invalid S3 URI: %s" % uri)
    254         elif self.scheme == 'file':
    255             self.uri_path = parsed_uri.netloc + parsed_uri.path

RuntimeError: invalid S3 URI: s3://bucketname/docs/x@t.ai/test.pdf

[tmylk]

The name doesn't follow AWS best practices. What is the reason for using it?

The "@" symbol is not considered a safe character in AWS guide. It is referenced as "character that requires special handling".

[tmylk]

Need to add logic to distinguish s3://key:secret@bucket/object vs s3://bucket/obj@ect and s3://bucket/ob:j@ect

[piskvorky]

Is it possible to do so cleanly & unambiguously?

Is the S3 URI syntax described somewhere formally, so we know for sure?

[piskvorky]

Maybe if slashes / are not allowed in credentials (I'm not sure -- maybe they are), a simple check would be "credentials before @ contain a / => they're not really credentials, treat @ as part of filename".

[jayantj]

/ is allowed in the credentials - secret keys usually have a slash in them

I think these are the two additional tests we need -

        # correct uri, key contains @
        parsed_uri = smart_open.ParseUri("s3://mybucket/mykey@mydir")
        self.assertEqual(parsed_uri.scheme, "s3")
        self.assertEqual(parsed_uri.bucket_id, "mybucket")
        self.assertEqual(parsed_uri.key_id, "mykey@mydir")
        self.assertEqual(parsed_uri.access_id, None)
        self.assertEqual(parsed_uri.access_secret, None)

        # correct uri with credentials, key contains @
        parsed_uri = smart_open.ParseUri("s3://ACCESSID456:acces/sse_cr-et@mybucket/mykey@mydir")
        self.assertEqual(parsed_uri.scheme, "s3")
        self.assertEqual(parsed_uri.bucket_id, "mybucket")
        self.assertEqual(parsed_uri.key_id, "mykey@mydir")
        self.assertEqual(parsed_uri.access_id, "ACCESSID456")
        self.assertEqual(parsed_uri.access_secret, "acces/sse_cr-et")

Seems slightly difficult to do cleanly, lots of edge cases.

[tmylk]

The Access Key ID is 20 alpha-numeric characters, so a regex can locate the s3://key:secret@bucket/object. Will send a fix later today.

[menshikh-iv]

The problem still exists