Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SWORD upload limit, change from int to long #2169

Closed
2 tasks
raprasad opened this issue May 14, 2015 · 7 comments
Closed
2 tasks

SWORD upload limit, change from int to long #2169

raprasad opened this issue May 14, 2015 · 7 comments

Comments

@raprasad
Copy link
Contributor

raprasad commented May 14, 2015

Currently the SWORD library returns a Java int for getMaxUploadSize

see: https://github.com/IQSS/dataverse/blob/master/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordConfigurationImpl.java#L126

Change from an int to a long``to accommodate 10gb files*. Or anything beyond Java's``Integer.MAX_SIZE of 2,147,483,647

  • contact sword team
  • potentially change upstream code, etc.

Related: #2149

@pdurbin
Copy link
Member

pdurbin commented May 14, 2015

From looking at the DVN 3.x code at https://github.com/IQSS/dvn/blob/97fc08b8d9364ddcacd24045483086d4ed7610d6/DVN-root/DVN-web/src/main/java/edu/harvard/iq/dvn/api/datadeposit/SwordConfigurationImpl.java#L96 it looks we put in equivalent behavior in DVN 3.x.

We would try Integer.parseInt(maxUploadInBytes) and catch NumberFormatException and return unlimited and log the following:

logger.fine("Could not convert " + maxUploadInBytes + " from JVM option " + jvmOption + " to int. Setting Data Deposit APU max upload size limit to unlimited.");

(Whoops. "API" that should have been!) 👎

The point is that DVN 3.6 and Dataverse 4.0 are going to work the same way in this regard. Good, I guess but I agree that this should be an long rather than an int. An int is too small. Big data, here we come. 👍

Let me reach out to http://www.mail-archive.com/sword-app-tech@lists.sourceforge.net/ and get a pulse on this. My last pull request at swordapp/JavaServer2.0#6 still hasn't been commented on and there was no reply to http://www.mail-archive.com/sword-app-tech@lists.sourceforge.net/msg00421.html but I'd like to try.

The maintainer (@richard-jones) is looking for help in general: http://www.mail-archive.com/sword-app-tech@lists.sourceforge.net/msg00400.html

@scolapasta scolapasta added this to the Candidates for 4.0.3 milestone Jun 1, 2015
@scolapasta scolapasta modified the milestones: Candidates for 4.3, Candidates for 4.2 Sep 17, 2015
@scolapasta scolapasta assigned pdurbin and unassigned scolapasta Sep 17, 2015
@mercecrosas mercecrosas modified the milestones: Candidates for 4.3, In Review Nov 30, 2015
@pdurbin pdurbin removed their assignment Jan 21, 2016
@scolapasta scolapasta removed this from the Not Assigned to a Release milestone Jan 28, 2016
@kcondon
Copy link
Contributor

kcondon commented Apr 4, 2016

The intent of our platform is to not have many built in limits but in practice we are not allowing 10GB uploads currently. So, since there is a dependency on SWORD we will mark this change as a feature.

@pdurbin
Copy link
Member

pdurbin commented Apr 4, 2016

For the record, I have not yet reached out the to the SWORD mailing list yet as I hinted I might (almost a year ago): #2169 (comment)

@tomck
Copy link
Contributor

tomck commented Jun 23, 2016

Could I get clarification on this? The docs, referencing this issue, say "If the MaxFileUploadSizeInBytes is NOT set, uploads, including SWORD may be of unlimited size." I removed the limit by using curl -x DELETE, but it seems the text above the file upload box isn't variable and still displays "File upload limit is 2GB per file."

Attempting to upload via Dropbox and my temp uploads seem to hit 7.4GB and halt. An upload of a tar.xz file got stuck after uploading (I can't find it in the actual data storage folder on the system and it's not showing up in the file listing, is the system trying to extract it?).

edit: issue was due to multiple temp files filling disk, switching to using a larger storage disk and relocating files and using symlinks for temp files seems to have resolved it mostly.

@pdurbin
Copy link
Member

pdurbin commented Jun 24, 2016

@tomck it sounds like you've hit bug #3071 (File Upload: Make upload limit text dynamic based on setting) which is fixed in the "develop" branch and will be part of Dataverse 4.4.

If the docs on MaxFileUploadSizeInBytes aren't clear, please let us know.

Since you seem to be operating on somwhat large files, you may be interested in the work going on at #3145 (still early days).

@pdurbin
Copy link
Member

pdurbin commented Jul 12, 2018

They're working on SWORD v3: https://groups.google.com/d/msg/dataverse-community/D7lRTA3f8Hc/nX0v50MKAgAJ

Hopefully this issue will be addressed there. This issue doesn't seem to affect very many people. Closing.

@pdurbin pdurbin closed this as completed Jul 12, 2018
@pdurbin
Copy link
Member

pdurbin commented Mar 17, 2022

These days we're running @poikilotherm's version of the SWORD library (🎉 see PR #8000) so I just opened gdcc/sword2-server#43 about the upload limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants