Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trial uppy as library for upload SDK (including adjustment for Azure) #7

Closed
5 of 6 tasks
rufuspollock opened this issue Aug 5, 2020 · 3 comments
Closed
5 of 6 tasks
Assignees

Comments

@rufuspollock
Copy link
Member

rufuspollock commented Aug 5, 2020

We want to research if we can use uppy for our file upload. https://uppy.io/ is a fully featured client library for managing uploads with many adapters. Features include:

  • support for many cloud storage sources out of the box
  • import direct from google drive, dropbox etc
  • resumable file uploads
  • i18n
  • recovery from crashes
  • etc

Acceptance

Answers to questions like:

  • Is uppy good?
  • How easy is it to use the features?
  • ❗ can we integrate uppy with what we are doing? In particular, can we integrate a bespoke function for signing files for cloud storage upload?

Tasks

@kmanaseryan
Copy link
Contributor

Progress

  • Read the blob upload flow
  • Understood that this is also related to Git LFS, this is where the upload signing and upload verification happen.
  • So in brief the flow is [get token from ckan] -> [get signed upload url] -> [upload blob] -> [verify upload]
  • Uppy consists of plugins. There are plugins for destinations (where blobs are being uploaded) and plugins for sources (from where the blobs should be fetch for the uploads).
    • The only available plugin as a destination which matches with our current implementation is AWS S3.
    • So it seems that we can use it only for S3? No, the trick here is that this AWS plugin can be used for other Storage providers. There are actually S3-alternatives, so basically any storage provider which has identical implementation then can be used as a "aws-s3" plugin in Uppy.
    • So it seems if we override the signing related functions in aws-s3 plugin using the auth related functions in ckan-client-js, we can use the Uppy, this seems promising, but has to be tested.

Answers to the questions

  • Is uppy good? - Yes, very well documented, good community and the number of stars is +20k
  • How easy is it to use the features? - So the features are very well documented and very intuitive. I would ask how easy we can integrate in our system? see the propoasl
  • ❗ can we integrate uppy with what we are doing? In particular, can we integrate a bespoke function for signing files for cloud storage upload? - The current aws-s3 plugin in Uppy seems very promising as you can see it has been used for different storage providers. At first glance checking the signing related functions, seems that we have all the params getting from CKAN Client JS SDK, so we just to reuse this methods for to override some functions of the plugin, but this is not 100% guaranteed, needs to be tested. Please also note that if we want to upload from remote sources then we have to have a Campanion based proxy server

Proposal

So there are @uppy/aws-s3-multipart and @uppy/aws-s3 packages. We can create our own custom plugin inherited from one of this packages and use also ckan-client-js for the auth and signing urls. Then this can be used in Uppy as a custom plugin created by us.

@rufuspollock
Copy link
Member Author

rufuspollock commented Sep 22, 2020 via email

@rufuspollock
Copy link
Member Author

FIXED.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants