@pameyer and I talked about this a bit at http://irclog.iq.harvard.edu/dataverse/2016-01-08#i_29469 and it's definitely something that's of interest for @sbgrid . Their in-house system uses rsync for this. I'm not sure the best way to implement this in Dataverse. This feature also seems strongly related to the Data IO Component BRD @raprasad is working on so I left a comment on it to check if it's in scope or not.

@bmckinney I'm giving you this issue to think about and discuss with @pameyer with regard to the needs of @sbgrid. Definitely open to suggestions. 😄

Thanks, I definitely have some thoughts on this. I'll plan to discuss with @pameyer and share a google doc.

See: https://docs.google.com/document/d/1JDCKh8UzB2fQoRkmlCU8BrY0iwH0dEGnOAgJhRZVepM/edit?usp=sharing

@bmckinney nice! Please see also the diagram by @michbarsinai at #2728 (comment) and the BRD: Data IO Component doc @raprasad has been working on.

@bmckinney I just mentioned your desktop app idea to @landreev and a standup and he agrees that you need something on the client side to coordinate resumption of uploads and downloads. Also, I'm reminded of a demo @astrofrog gave to me, @mcrosas (and others?) in May 2015. I just looked at http://irclog.iq.harvard.edu/search.pl?channel=dataverse&q=trobitaille but I can't find a reference to any code anywhere but it was written in Python. I do see a mention of Dataverse at https://github.com/glue-viz/glue/blob/16bc539736cb580ac4d42ccf5f6741ada553fde1/glue/main.py#L257 and there was a little discussion at dvn/swordpoc#3 . Anyway, if we do decide to pursue some sort of desktop client for Dataverse, let's create a new GitHub issue to track it.

before i forget, this project/protocol is somewhat relevant: http://tus.io/ caveats: only deals with uploads, currently no java server implementation. it could be a nice option for "somewhat large" file uploads

"Resumable upload supports uploading videos that are up to 1.75GB and 45 minutes long." https://developers.facebook.com/docs/graph-api/video-uploads#resumable via http://socialwg.indiewebcamp.com/irc/social/2016-06-10/line/1465619059043

Too small for our requirements but potentially interesting.

This issue is highly related since rsync supports the ability to resume uploads and downloads: Data Capture Module (rsync support) #3145

Truth be told, this issue is a little more about the download side of things and the DCM is about uploads.

DCM is about uploads - but integration of components for rsync downloads are in other issues.

wget-able links will help here as well (aka - users will be able to do resumable downloads without other changes to dataverse, within the restrictions of files/datasets that will display a download link.

@pameyer good point about wget. I chronicled my experience with wget --continue at #5092 and it sounds like we're going to promote the use of non-browser tools like wget in #4949.

Stumbled across this download issue, and saw the wget conversation. Reminded me that we added a bit of help text to the Download Package File from S3 #4949 issue.

We currently display a Download URL on the file pg, which provides download access via the API.

I suggest that we could resolve this issue by displaying similar help text above this Download URL field that explains the wget and download manager methods, as well as a link to the User Guide.

@mheppler fine with me.