apache2 proxy timeouts and large file uploads

[BrandonKowalski]

This is a followup based on this discussion in the Google Group and may be a solution to #3634.

We first saw this issue at Cornell when our data archivist assistant attempted to upload a ~5 GB file. This failed due to MaxFileUploadSizeInBytes & TabularIngestSizeLimit being set too low.

Once changed, the UI properly reflected the fact that we now accept files up to 10 GB in size.

This unfortunately proved to be half of the problem.

The other half I debugged today via the network monitor in FF showed that once the upload bar in the UI was completely full the server killed the request and sent back a 500 error. The body of this error was the default apache2 error page.

Inspecting the apache error log I saw that the ajp proxy timed out. I added timeout=600 (units in seconds) and restarted the apache service and the large file was then successfully uploaded.

The change I made was located in /etc/httpd/conf.d/ssl.conf and looks like the following:

# pass everything else to Glassfish
ProxyPass / ajp://localhost:8009/ timeout=600

Perhaps the documentation should mention that this timeout should be changed for larger file uploads.

Hopefully this helps others out.

[pdurbin]

@BrandonKowalski thanks for opening this issue and doing all the research for the fix. It looks like #3645 may have been related as well.

I spoke with @kcondon this afternoon and it sounds like he found a similar fix that involved putting the timeout in a more global location that the ProxyPass line, a similar fix but also undocumented. This issue can be about deciding what we document in the Dataverse guides.

In the guides, we should probably be documenting the configuration of Apache in a more global way (as discussed in #4392) but for historical reasons it's currently documented in the "Shibboleth" section: https://github.com/IQSS/dataverse/blob/v4.8.5/doc/sphinx-guides/source/installation/shibboleth.rst#configure-apache . For now, perhaps we could simply tweak the ProxyPass line that's shown there and write a little about the timeout fix. I love one line fixes. They're well worth documenting. 😄

@BrandonKowalski or others, are you interested in making a pull request? I can help walk you through the process.

[BrandonKowalski]

I can help out with the documentation fix. I've never used sphinx but this would be a decent opportunity to get my feet wet.

[pdurbin]

@BrandonKowalski nice! Maybe http://guides.dataverse.org/en/4.8.5/developers/documentation.html will help you get set up. It boils down to installing Sphinx and executing make html after you make some edits. The format is reStructuredText rather than Markdown but you can probably figure it out without much study. I'm much more comfortable in Markdown myself. 😄

You might also want to take a look at http://guides.dataverse.org/en/4.8.5/developers/version-control.html#how-to-make-a-pull-request but we're not picky. It's just a pull request. Any way you can make it is fine. 😄

[BrandonKowalski]

Finally got around to it. Sorry for the delay!

[pdurbin]

@BrandonKowalski pull request #4459 looks good and I approved it but I just moved this issue to code review at https://waffle.io/IQSS/dataverse in case anyone else wants to take a look. We'll talk about it at standup this morning. Thanks!

[kcondon]

See guides: http://guides.dataverse.org/en/4.8.6/installation/shibboleth.html#edit-apache-ssl-conf-file
1. Backup ssl.conf
2. edit ssl.conf:
3. ProxyPass / ajp://localhost:8009/ timeout=600 defines a timeout of 600 seconds.
4. restart apache