Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 API is not compatible with libcloud #863

Closed
douardda opened this Issue Feb 26, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@douardda
Copy link

douardda commented Feb 26, 2019

When using the S3 endpoint from libcloud in Python, there are several issues:

  • list_containers() returns an empty list
  • container.list_objects() crash with the a traceback

Note that adding a container/bucket or an object using libcloud does work.

System Setup
On a debian stretch system, from a fresh python3.5 virtualenv in which libtool is installed using pip:

$ mkvirtualenv -p /usr/bin/python3 seaweedfs
(seaweedfs) $ pip list
Package         Version   
--------------- ----------
apache-libcloud 2.4.0     
certifi         2018.11.29
chardet         3.0.4     
idna            2.8       
pip             19.0.3    
pkg-resources   0.0.0     
requests        2.21.0    
setuptools      40.8.0    
urllib3         1.24.1    
wheel           0.33.1    

with a seaweedfs system running locally in a docker:

$ s3cmd ls s3://
2019-02-22 10:44  s3://newbucket
2019-02-22 10:37  s3://newbucket3
2019-02-25 18:08  s3://toto

Executing:

from libcloud.storage import providers
from libcloud.storage.types import Provider

drv = providers.get_driver(Provider.S3)
cnx = drv(key='any', secret='any', host='localhost', port=8333, secure=False)
print('containers', cnx.list_containers())
ct = cnx.get_container('toto')
print(ct.list_objects())

produces:

(seaweedfs) $ python test-libcloud.py 
containers []
Traceback (most recent call last):
  File "test-libcloud.py", line 8, in <module>
    print(ct.list_objects())
  File "/home/ddouard/.virtualenvs/seaweedfs/lib/python3.5/site-packages/libcloud/storage/base.py", line 137, in list_objects
    return self.driver.list_container_objects(container=self)
  File "/home/ddouard/.virtualenvs/seaweedfs/lib/python3.5/site-packages/libcloud/storage/drivers/s3.py", line 271, in list_container_objects
    ex_prefix=ex_prefix))
  File "/home/ddouard/.virtualenvs/seaweedfs/lib/python3.5/site-packages/libcloud/storage/drivers/s3.py", line 308, in iterate_container_objects
    xpath='IsTruncated', namespace=self.namespace)).lower()
AttributeError: 'NoneType' object has no attribute 'lower'

For this error, it looks like the S3 API does not generates a complete XML (or libcloud does make erroneous assumptions on the xml resturned by an s3 enpoint), missing the IsTruncated tag.

@douardda

This comment has been minimized.

Copy link
Author

douardda commented Feb 26, 2019

A quick follow up on this. If I set the namespace attribute of the libcloud driver, it works as expected. So I'm not sure if this is expected, or is a configuration caveat of the seaweedfs storage.
By default libcloud uses 'http://s3.amazonaws.com/doc/2006-03-01/' as namespace for the s3 driver.

If this behavior is expected, then we might think about adding a entry to the wiki about using the Amazon S3 API.

@chrislusf

This comment has been minimized.

Copy link
Owner

chrislusf commented Feb 26, 2019

@chrislusf

This comment has been minimized.

Copy link
Owner

chrislusf commented Feb 27, 2019

fixed with
70ac2f6
cb07d15

@chrislusf chrislusf closed this Feb 27, 2019

@douardda

This comment has been minimized.

Copy link
Author

douardda commented Mar 4, 2019

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.