feat(upload): make upload file storage path configurable #6656 #8983
Conversation
As outlined in IQSS#6656, files will be stored in `domaindir/generated/jsp/dataverse` during upload before being moved to our temporary ingest file space at `$dataverse.files.directory/temp`. With this commit, we enable to configure a different place for these kind of generated temporary files by using MPCONFIG variable substitution inside of glassfish-web.xml. Also sorts the content of glassfish-web.xml into order as specified by the XSD. Documentation of the setting is provided.
- Instead of a /docroot, add a more generic /dv which is owned by payara:payara and can be used to either store data in a single volume using subfolders or use subfolders with different backing volumes. Anyway, data is not written to overlay FS this way. (As long as an app image points to this location) - Also define /secrets and /dumps as volumes, so data flowing into these locations is again not added to the overlay FS (which might cause severe damage in case of heap dumps!) - Document the different locations in the base image guide. - Remove the /docroot workaround for uploaded files. This will be solved at application level (either by moving the workaround there) or IQSS#8983
|
I tested this today in UI Upload - 1 GiB random data :: Dataset
|
|
This looks great, we should be able to move it along and merge it in the next sprint. |
pdurbin
added
the
Size: 30
|
Small code change. Good docs. Gave it a size of 33. |
There was a problem hiding this comment.
Looks great! Thanks, @poikilotherm!! Approved (I made small doc changes). @landreev also made a positive comment: #8983 (comment)
The only testing I did was to confirm that the new unconfigured JSF upload default directory is /usr/local/payara5/glassfish/domains/domain1/uploads. I assuming configuration works fine.
Moving to QA.
What this PR does / why we need it:
As outlined in #6656, files will be stored in
domaindir/generated/jsp/dataverseduring upload before being moved to our temporary ingest file space at$dataverse.files.directory/temp.With this commit, we enable to configure a different place for these kind of generated temporary files by using MPCONFIG variable substitution inside of glassfish-web.xml. This is kinda hacky, but at least it allows to store the data in distinguished places.
Also sorts the content of glassfish-web.xml into order as specified by the XSD. Documentation of the setting is provided.
Which issue(s) this PR closes:
Closes #6656
Special notes for your reviewer:
Please see screenshots below. You can also try with
docker-aio.I added a few notes about temporary upload storage to the configuration guide with the gist of knowledge from #6656 so admins can learn about potential problems with large file uploads.
The container base image in #8933 certainly will include something using this if this PR would be merged before. Otherwise I'd extend this PR to include something for the container scenario.
Suggestions on how to test this:
Configure your deployment with a system property pointing
dataverse.files.uploadsto a different place and watch data being stored there.Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Nope.
Is there a release notes update needed for this change?:
Do we need one? This might be a scenario for admins as outlined by a bunch of folks. Please leave a comment.
Additional documentation:
🔋 included.