Feature Request: Upload/Download data from Cloud Storage directly

[icoxfog417]

Recently, many user will have their data on cloud storage like S3/GCS etc.

For that reason, it'll be good to upload data from cloud storage directly, and also save annotated data to it. If this feature implemented, there are no need to have the data on local.

[c-w]

Great idea. I would recommend to use the Apache libcloud library which offers a unified interface to many storage providers like AWS, Azure, GCP, etc.

[bahiamartins]

Use django storages and you are good to go. https://django-storages.readthedocs.io/en/latest/

[c-w]

@Hironsan @icoxfog417 This feature will be very useful for one of my upcoming projects, so I'd like to start working on implementing it.

Here are my initial thoughts, please let me know what you think so that we're aligned on how to best build this feature.

I was thinking of adding a new button next to the file upload button to trigger the cloud upload feature:

This button can open a modal that lets the user input all the data required to connect to the cloud storage via Apache Libcloud:

When the user clicks the upload button, the form is sent to the TextUploadAPI. In this request, there will be no file present so the backend API checks for that. If the file is not present, the API opens the cloud file as a stream and passes it to the parser. The rest of the code doesn't have to change. Something along these lines:

class TextUploadAPI(APIView):
  # ... the code before this point doesn't have to change
  def post(self, request, *args, **kwargs):
        project = get_object_or_404(Project, pk=self.kwargs['project_id'])

        if 'file' in request.data:
            data = request.data['file']
        else:
            libcloud_client = ... # code to unpack libcloud fields here and create client
            data = ... # code to open file from cloud storage as stream here

        parser = self.select_parser(request.data['format'])
        data = parser.parse(data)
        # ... the rest of the code below this point doesn't have to change

# instead of the if-statement, we could also make a new API that handles only cloud upload and
# move the shared code from the file upload API into utility functions so they can be re-used

One benefit of the approach of using Apache Libcloud is that the cloud upload piece is completely decoupled from the rest of the Django configuration so for each upload the user could select a different cloud storage, e.g. import one set of data from AWS S3 and another from Google Cloud Storage, without having to change anything about the server configuration: all the cloud connection information is passed from the client.

In the future, the modal could also be extended to include a full browser of the cloud storage to provide a nice file explorer like functionality (e.g. using django-cloud-browser) but this will require a lot more effort (e.g. that project doesn't support Azure Storage currently which is a requirement for me). I believe the approach with the modal proposed here may provide a nice first-step/MVP to enable the cloud-based file import workflow and we can do improvements on top of this base later.

Let me know what you think and looking forward to hearing your thoughts soon!

[c-w]

See #213 for an implementation of cloud data import via django-cloud-browser.

[echan00]

Just tried out this feature, seems like transfer from AWS will still timeout on Heroku 😢

[c-w]

@echan00 I suspect the timeout will be solved by implementing an approach like #241 (comment) which moves the generation of the export file out of the HTTP loop and into a background worker.