How do I configure s3 to use with depot and flask application factory way?

[rlam3]

How do I configure s3 settings to use with depot? I'm looking for an example.
What should the Depot manager syntax look like?

So far, I've been trying to look at your code and documentation to figure this out but nothing comes up.

from depot.manager import DepotManager
from depot.io.awss3 import S3Storage

S3_LOCATION = 'test.s3-website-us-west-2.amazonaws.com'
S3_KEY = 'XXXX'
S3_SECRET = 'XXXX'
S3_UPLOAD_DIRECTORY = 'media'
S3_BUCKET = 'test'
#
s3 = S3Storage(S3_KEY,S3_SECRET,bucket=S3_BUCKET)

DepotManager.configure('media',{
    'depot.backend': depot.io.awss3.S3Storage????
})

# DepotManager.configure('media', {'depot.storage_path': 'media/'})

app.wsgi_app = DepotManager.make_middleware(app.wsgi_app)

Thanks!

[rlam3]

I found out that I could configure depot in the following manner for S3

S3_BUCKET = 'test'

DepotManager.configure('test',{
    'depot.backend':'depot.io.awss3.S3Storage',
    'depot.access_key_id':S3_KEY,
    'depot.secret_access_key':S3_SECRET,
    'depot.bucket':S3_BUCKET,
})

because 'test' is the bucket... is there a way to upload to certain folders within the bucket? Or would it be better to have multiple buckets instead of one overall encompassing?

If it is recommended that I use multiple buckets. How do I configure multiple buckets using one DepotManager?

ex:

/app_bucket
    /profile-pic-folder
    /user-images-folder

OR

profile-pic-bucket
user-images-bucket

Thanks!

[rlam3]

I'm getting an error when I upload certain files above 1MB

Have any of you ran into thej following problem before?

Traceback (most recent call last):
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/depot/middleware.py", line 187, in __call__
    return self.app(environ, start_response)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/johnsmith/Projects/flaskPlayground/uploadPlayground/testDepotS3/flaskdepot.py", line 222, in upload_pdfs
    doc = PDFs(name=filefromlist.filename,pdf=filefromlist)
  File "<string>", line 4, in __init__

  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 260, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line 526, in _declarative_constructor
    setattr(self, k, kwargs[k])
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 226, in __set__
    instance_dict(instance), value, None)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 693, in set
    value, old, initiator)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 712, in fire_replace_event
    state, value, previous, initiator or self._replace_token)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/sqlalchemy/orm/events.py", line 1669, in wrap
    return fn(target, value, *arg)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/depot/fields/sqlalchemy.py", line 74, in _field_set
    value = upload_type(value, column_type._upload_storage)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/depot/fields/interfaces.py", line 52, in __init__
    self.process_content(content)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/depot/fields/upload.py", line 37, in process_content
    file_path, file_id = self.store_content(content, filename, content_type)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/depot/fields/upload.py", line 48, in store_content
    file_id = self.depot.create(content, filename, content_type)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/depot/io/awss3.py", line 129, in create
    self.__save_file(key, content, filename, content_type)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/depot/io/awss3.py", line 115, in __save_file
    encrypt_key=self._encrypt_key)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
    retry_handler=retry_handler
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/Users/johnsmith/.virtualenvs/test/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
    raise ex
error: [Errno 32] Broken pipe

[rlam3]

Looks like I'm not the only one having the same problem.

boto/boto#621

And it seems like one of the many ways to solve this is wait for the dns?

http://www.jacobtomlinson.co.uk/2014/07/31/amazon-s3-s3cmd-put-errno-32-broken-pipe/

[amol-]

Noticed now, have you tried setting explicitly the host in your DepotManager.configure call?

[rlam3]

@amol- Adding depot.host doesn't work either... I think it has to do with the dns on amazon.

[amol-]

@rlam3 if it's related to dns it should have solved now after a few hours. Do you still face the issue?

[rlam3]

@amol- . yes it is. I was wondering if it was possible to be able to find out how the transfer rate from depot or if you are familiar of a feature like this.

From what I understand so far, please correct me if I'm wrong... One depot configuration = one bucket.

I'd like to know how I'm able to specify the path for which my objects will be created in because if I were to use s3 with cloudfront as cdn with depot, it would require me to make several cdn cloudfront as opposed to one cloudfront with sub folders like (avatar,user_images). I tried doing something along the lines of renaming the file before upload ex:

ImageUpload(db.Model)
    ...
    user_image = db.Column(UploadedFileField(upload_type=UploadedImageWithMaxSize,upload_storage='s3_depot'))
    ...

...
if request.POST is "POST"
    user1_image_file = request.files.get('document')
    user1_image_file.filename = "/".join(['userImages',user1_image_file.filename])
    image_upload = ImageUpload(user_image=user1_image_filename)
    db.session.add(image_upload)
    db.session.commit()
...

I'm trying to aim for:

app_depot
    /user_images
    /static_assets

If you don't have this implemented. How do you suggest I do this? Thanks for the help!

[rlam3]

@amol- I'm not entirely sure if this is the proper way to do it with your depot code. But I managed to add a new attribute to the UploadedFileField so it can take in the folder name/ sub directory name for S3. Its more of a quick hack.
So far, it seems to be working for me on s3 for uploads.

Would love to know what your thoughts on this are:

rlam3@81d0660

Now it looks like this:

... some model with SQLAlchemy

class User(db.Model)
    image = db.Column(UploadedFileField(upload_storage='app_name/bucket_name',upload_directory='image_subdir'))

...

if request.POST is "POST"
    user1_image_file = request.files.get('document')
    user = User(image=user1_image_file)
    db.session.add(user)
    db.session.commit()

Thanks!

[amol-]

@rlam3 UploadedFileField should only contain options that apply to all storage backends, upload_directory might not apply to some backends (like GridFS) so it's better suited to be configured in S3Storage.
That way you can configure multiple S3Storage storages (even with the same bucket) but pointing to different paths and then upload the files to the right storage to make them appear at the required path.

DEPOT philosophy is that the only thing that identifies a file is a storagename/fileid pair, if the upload_directory is configured outside of the storage that constraint is not true anymore.

[rlam3]

@amol- Considering my changes made within rlam3/depot@81d0660. which lines are the ones that are configured outside of the storage?

I accidentally closed this case. Could you please reopen for me? Thanks

[amol-]

@rlam3 I meant that you are currently passing upload_directory to create and UploadedFileField, while it should probably go into the constructor of the S3Storage like the access key and so on. So that you can still point to where the file is just by storagename/fileid

It should definitely not go into DepotFileInfo as there are storages that don't support the concept of directories.

[rlam3]

@amol- Okay thanks I think I understand you now. And now I have a quick question about multiple files. Should I create another model for individual files or is it possible to save multiple files through 1 UploadFileField?

[amol-]

This was solved by #17