dsdt remote does not attach a remote context as expected

[post2web]

I have a local context:

$ dsdt --version
Running Disdat version 0.9.13
$ dsdt context
*	test	[None@None]

I am trying to attach a remote context to an existing bucket with a non-existing key but I get a RuntimeError:

$ dsdt remote new s3://my-bucket/non-existing-key
Error code 404
Unable to bind context new because URL s3://my-bucket/non-existing-key does not exist.
Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/F7DBigBang/bin/dsdt", line 11, in <module>
    load_entry_point('disdat==0.9.13', 'console_scripts', 'dsdt')()
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/dsdt.py", line 91, in main
    args.func(args)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1579, in <lambda>
    remote_p.set_defaults(func=lambda args: _remote(fs, args))
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1363, in _remote
    fs.remote_add(args.context, args.s3_url)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1348, in remote_add
    ctxt_obj.bind_remote_ctxt(remote_context, s3_url)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/data_context.py", line 151, in bind_remote_ctxt
    raise RuntimeError
RuntimeError

If fails in the same way if the key exists:

$ touch test && aws s3 cp test s3://my-bucket/non-existing-key/test
upload: ./test to s3://my-bucket/non-existing-key/test
 $ dsdt remote new s3://my-bucket/non-existing-key
Error code 404
Unable to bind context new because URL s3://my-bucket/non-existing-key does not exist.
Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/F7DBigBang/bin/dsdt", line 11, in <module>
    load_entry_point('disdat==0.9.13', 'console_scripts', 'dsdt')()
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/dsdt.py", line 91, in main
    args.func(args)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1579, in <lambda>
    remote_p.set_defaults(func=lambda args: _remote(fs, args))
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1363, in _remote
    fs.remote_add(args.context, args.s3_url)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1348, in remote_add
    ctxt_obj.bind_remote_ctxt(remote_context, s3_url)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/data_context.py", line 151, in bind_remote_ctxt
    raise RuntimeError
RuntimeError

It succeeds to attach a remote context if I point to the file that I uploaded for the previous test:

 $ dsdt remote new s3://my-bucket/non-existing-key/test
 $ dsdt context
*	test	[new@s3://my-bucket/non-existing-key/test]

Am I using it wrong? It doesn't seem to be the expected behavior.

[kyocum]

Looking into this issue, we wanted to only attach to remotes on existing "paths" in S3.

The current code allows attaching to just buckets, which is fine.

But for / paths we required the last key to refer to an existing object, where the object is a "folder" in s3, i.e., it ends with '/'. However the current code also allows one to attach to a remote without that requirement.

[post2web]

It throws the same error even if the path ends with '/'.

$ dsdt remote new s3://my-bucket/non-existing-key/
Error code 404
Unable to bind context new because URL s3://my-bucket/non-existing-key/ does not exist.
Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/F7DBigBang/bin/dsdt", line 11, in <module>
    load_entry_point('disdat==0.9.13', 'console_scripts', 'dsdt')()
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/dsdt.py", line 91, in main
    args.func(args)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1579, in <lambda>
    remote_p.set_defaults(func=lambda args: _remote(fs, args))
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1363, in _remote
    fs.remote_add(args.context, args.s3_url)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/fs.py", line 1348, in remote_add
    ctxt_obj.bind_remote_ctxt(remote_context, s3_url)
  File "/mnt/data/.virtualenvs/F7DBigBang/lib/python3.6/site-packages/disdat/data_context.py", line 151, in bind_remote_ctxt
    raise RuntimeError
RuntimeError

The only way dsdt remote works for me if I point it to a file on S3.

[kyocum]

Current behavior
There is only one "folder" at s3://disdat-prod, which is testbind
Both dsdt remote somecontext s3://disdat-prod/something
and
dsdt remote somecontext s3://disdat-prod/something/
succeed, and following bundles pushes also succeed.
And dsdt remote somecontext s3://disdat-prod/testbind also works.
Closing issue.