Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large video file can't be moved to Fedora (test case #23) #1436

Closed
dflitner opened this issue Feb 10, 2020 · 18 comments · Fixed by #1468
Closed

Large video file can't be moved to Fedora (test case #23) #1436

dflitner opened this issue Feb 10, 2020 · 18 comments · Fixed by #1468
Milestone
1.1.0

Comments

@dflitner
Copy link

dflitner commented Feb 10, 2020 •
edited
Loading

I found an old closed issue which I'm not sure is related.

I tried uploading a 2.5GB mp4 file during the test sprint on both my local machine and one that Eli had set up on his computer. We of course had to adjust the Drupal settings to allow large files to be uploaded in the first place.

Multiple attempts gave the same results.

It sat spinning for quite a while and then halted with an error message:
File upload error. Could not move uploaded file.

Watchdog has two entries:

Upload error. Could not move uploaded file 1984-clio-awards-19117.mp4 to destination fedora://2020-02/1984-clio-awards-19117.mp4.

Warning: move_uploaded_file(): Unable to move '/tmp/phpNGE6s6' to 'fedora://2020-02/1984-clio-awards-19117.mp4' in Drupal\Core\File\FileSystem->moveUploadedFile() (line 85 of /var/www/html/drupal/web/core/lib/Drupal/Core/File/FileSystem.php)

#0 /var/www/html/drupal/web/core/includes/bootstrap.inc(600): _drupal_error_handler_real(2, 'move_uploaded_f...', '/var/www/html/d...', 85, Array)
#1 [internal function]: _drupal_error_handler(2, 'move_uploaded_f...', '/var/www/html/d...', 85, Array)
#2 /var/www/html/drupal/web/core/lib/Drupal/Core/File/FileSystem.php(85): move_uploaded_file('/tmp/phpNGE6s6', 'fedora://2020-0...')
#3 /var/www/html/drupal/web/core/modules/file/file.module(1106): Drupal\Core\File\FileSystem->moveUploadedFile('/tmp/phpNGE6s6', 'fedora://2020-0...')
#4 /var/www/html/drupal/web/core/modules/file/file.module(926): _file_save_upload_single(Object(Symfony\Component\HttpFoundation\File\UploadedFile), 'field_media_vid...', Array, 'fedora://2020-0...', 0)
#5 /var/www/html/drupal/web/core/modules/file/file.module(813): file_save_upload('field_media_vid...', Array, 'fedora://2020-0...', NULL, 0)
#6 /var/www/html/drupal/web/core/modules/file/file.module(1444): _file_save_upload_from_form(Array, Object(Drupal\Core\Form\FormState))
#7 /var/www/html/drupal/web/core/modules/file/src/Element/ManagedFile.php(74): file_managed_file_save_upload(Array, Object(Drupal\Core\Form\FormState))
#8 /var/www/html/drupal/web/core/modules/file/src/Plugin/Field/FieldWidget/FileWidget.php(324): Drupal\file\Element\ManagedFile::valueCallback(Array, Array, Object(Drupal\Core\Form\FormState))
#9 [internal function]: Drupal\file\Plugin\Field\FieldWidget\FileWidget::value(Array, Array, Object(Drupal\Core\Form\FormState))
#10 /var/www/html/drupal/web/core/lib/Drupal/Core/Form/FormBuilder.php(1256): call_user_func_array(Array, Array)
#11 /var/www/html/drupal/web/core/lib/Drupal/Core/Form/FormBuilder.php(992): Drupal\Core\Form\FormBuilder->handleInputElement('media_video_add...', Array, Object(Drupal\Core\Form\FormState))
#12 /var/www/html/drupal/web/core/lib/Drupal/Core/Form/FormBuilder.php(1062): Drupal\Core\Form\FormBuilder->doBuildForm('media_video_add...', Array, Object(Drupal\Core\Form\FormState))
#13 /var/www/html/drupal/web/core/lib/Drupal/Core/Form/FormBuilder.php(1062): Drupal\Core\Form\FormBuilder->doBuildForm('media_video_add...', Array, Object(Drupal\Core\Form\FormState))
#14 /var/www/html/drupal/web/core/lib/Drupal/Core/Form/FormBuilder.php(1062): Drupal\Core\Form\FormBuilder->doBuildForm('media_video_add...', Array, Object(Drupal\Core\Form\FormState))
#15 /var/www/html/drupal/web/core/lib/Drupal/Core/Form/FormBuilder.php(563): Drupal\Core\Form\FormBuilder->doBuildForm('media_video_add...', Array, Object(Drupal\Core\Form\FormState))
#16 /var/www/html/drupal/web/core/lib/Drupal/Core/Form/FormBuilder.php(320): Drupal\Core\Form\FormBuilder->processForm('media_video_add...', Array, Object(Drupal\Core\Form\FormState))
#17 /var/www/html/drupal/web/core/lib/Drupal/Core/Controller/FormController.php(91): Drupal\Core\Form\FormBuilder->buildForm(Object(Drupal\media\MediaForm), Object(Drupal\Core\Form\FormState))
#18 [internal function]: Drupal\Core\Controller\FormController->getContentResult(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\RouteMatch))
#19 /var/www/html/drupal/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(123): call_user_func_array(Array, Array)
#20 /var/www/html/drupal/web/core/lib/Drupal/Core/Render/Renderer.php(573): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber{closure}()
#21 /var/www/html/drupal/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(124): Drupal\Core\Render\Renderer->executeInRenderContext(Object(Drupal\Core\Render\RenderContext), Object(Closure))
#22 /var/www/html/drupal/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(97): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array)
#23 /var/www/html/drupal/vendor/symfony/http-kernel/HttpKernel.php(151): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber{closure}()
#24 /var/www/html/drupal/vendor/symfony/http-kernel/HttpKernel.php(68): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#25 /var/www/html/drupal/web/core/lib/Drupal/Core/StackMiddleware/Session.php(57): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#26 /var/www/html/drupal/web/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php(47): Drupal\Core\StackMiddleware\Session->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#27 /var/www/html/drupal/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(106): Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#28 /var/www/html/drupal/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(85): Drupal\page_cache\StackMiddleware\PageCache->pass(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#29 /var/www/html/drupal/web/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php(47): Drupal\page_cache\StackMiddleware\PageCache->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#30 /var/www/html/drupal/web/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php(52): Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#31 /var/www/html/drupal/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#32 /var/www/html/drupal/web/core/lib/Drupal/Core/DrupalKernel.php(694): Stack\StackedHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#33 /var/www/html/drupal/web/index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#34 {main}
@dannylamb
Copy link
Contributor

I'm surprised you're even allowed to upload it. Isn't 2GB the PHP upload limit?

@elizoller
Copy link
Member

Locally, I upped the php_memory_limit to 8GB. In your test VM it looked it was still set to 32MB I believe (which is why we turned to my local install).

@dannylamb
Copy link
Contributor

@elizoller I was thinking of upload_max_filesize. I did some quick googling to confirm my previous statements, and maybe I'm showing my age. The sneaky 2GB max size may have been just a 32-bit thing 👴

Regardless, I'm adding this to the milestone. I'll play around with some large files on my end and report back.

@dannylamb dannylamb added this to the 1.1.0 milestone Feb 11, 2020
@elizoller
Copy link
Member

locally i have upload_max_filesize set to 8GB as well

@dannylamb
Copy link
Contributor

After bumping my upload and post max sizes and php memory, I run into this as well with a 2.4 GB binary.

@dannylamb
Copy link
Contributor

@ajstanley You did some work on uploading large binaries for UPEI, do you recall what the limits were before needing to use something like tus or plupload?

@mjordan
Copy link
Contributor

mjordan commented Feb 12, 2020

Could PHP's max execution time setting have anything to do with this? In other words, the file is so large, PHP is timing out.

@dflitner
Copy link
Author

dflitner commented Feb 12, 2020 via email

@elizoller
Copy link
Member

I'll try it.

@elizoller
Copy link
Member

I upped the max execution time to 12000 and then tried a 3.88GB file and got "The file could not be saved. An unknown error has occurred." with no further message in the recent logs.

@dflitner
Copy link
Author

Could it be a problem with the available memory for tmp files? I'm not sure if Islandora has tweaked the process but, generally, a file will upload completely to the local Drupal tmp folder and then transfer to the endpoint. Is it just that our VBs don't have space for big files?

I fixed the problem on another site by making sure the Drupal directory/partition had a lot of available space and using File Resumable Upload to help.

@elizoller
Copy link
Member

^ That idea came up in the tech call yesterday. I think @dannylamb was going to try one on a hosted VM instead of the Vagrant VM to hopefully surpass issues like that

@dannylamb
Copy link
Contributor

I've tried three times to upload a big file to http://138.197.139.139 and it just results in an indefinitely spinning wheel :(

@dannylamb
Copy link
Contributor

Yeah, still doing this. Tried again today and ended with another endlessly spinning wheel. Maybe someone with something better than my country internets can try.

@dflitner
Copy link
Author

dflitner commented Feb 19, 2020 via email

@elizoller
Copy link
Member

Do you mean another tester that is pointing your VM or a different VM?

@dannylamb
Copy link
Contributor

I think @dflitner just recreated what I've been experiencing. I was hoping for better behaviour from a remote environment, but it's worse 😭

At this point in the very least we should confirm the max size before it starts to crap out so we can state it as a known issue. E.g. "If you're uploading a file bigger than 2GB, consider using plupload or migrate_islandora_csv" in the release notes/documentation/UI. FWIW in Islandora 7 you've got to do pretty much the same. Either it's plupload or a custom batch for files bigger than 2GB.

I'm thinking @ajstanley and @alxp may have some insight on large files from their work at UPEI.

@dannylamb
Copy link
Contributor

After a recent conversation, it seems most likely that this is due to running small vagrant evnironments on our laptops. @dflitner's done it on D8 sites and @alxp has done it on an I8 site, but those are both properly configured production environments. Chances are the vagrant environment just doesn't have the resources to do something in the GB range of file size. So maybe the best we can do is just document what you need to keep in mind when configuring a live site. Is that an acceptable outcome to this issue?

In general, though, large files over HTTP are generally problematic, and we should investigate using @ajstanley's TUS module for the UI. Also, @wgillingham has successfully done large files over FTP in I7, so we should probably check out the FTP plugins for the migrate API too.

@dannylamb dannylamb mentioned this issue Mar 17, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.1.0
Development

Successfully merging a pull request may close this issue.

Small page on uploading large files dannylamb/CLAW
4 participants
@mjordan @elizoller @dflitner @dannylamb