File upload triggers exceptions

[gszathmari]

I just upgraded to 1.1.0 and media uploads seems to trigger a few exceptions. This brings the whole site down (my visitors get HTTP 500) until I remove the cache/compiled/files file manually.

The weird thing is that files is an actual file when I get the 500 error, but in other cases it is a directory. I think something creates files as a file, thus mkdir files raises and exception somewhere.

Logs from my error.log

2016/07/15 20:02:39 [warn] 15056#15056: *23 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000018, client: 115.187.195.164, server: privacyforjournalists.org.au, request: "POST /admin/media/guides/encrypt-your-hard-drive-with-bitlocker-windows.json/task:addmedia HTTP/1.1", host: "privacyforjournalists.org.au", referrer: "https://privacyforjournalists.org.au/admin/pages/guides/encrypt-your-hard-drive-with-bitlocker-windows"
2016/07/15 20:02:39 [error] 15056#15056: *23 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught exception 'RuntimeException' with message 'Creating directory '/var/www/privacyforjournalists.org.au/html/cache//compiled/files' failed on error mkdir(): File exists' in /var/www/privacyforjournalists.org.au/html/vendor/rockettheme/toolbox/File/src/File.php:420

#0 /var/www/privacyforjournalists.org.au/html/vendor/rockettheme/toolbox/File/src/File.php(195): RocketTheme\Toolbox\File\File->mkdir('/var/www/privac...')
#1 /var/www/privacyforjournalists.org.au/html/vendor/rockettheme/toolbox/File/src/File.php(316): RocketTheme\Toolbox\File\File->lock()
#2 /var/www/privacyforjournalists.org.au/html/vendor/rockettheme/toolbox/File/src/PhpFile.php(31): RocketTheme\Toolbox\File\File->save(Array)
#3 /var/www/privacyforjournalists.org.au/html/system/src/Grav/Common/File/CompiledFile.php(66): RocketTheme\Toolbox\File\PhpFile->save(Array)
#4 /var/www/privacyforjournalists.org.au/html/system/src/Grav/Common/Config/Setup.php(165): Grav\Common\File\CompiledYamlFile->con" while reading response header from upstream, client: 115.187.195.164, server: privacyforjournalists.org.au, request: "POST /admin/media/guides/encrypt-your-hard-drive-with-bitlocker-windows.json/task:addmedia HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm-web1.sock:", host: "privacyforjournalists.org.au", referrer: "https://privacyforjournalists.org.au/admin/pages/guides/encrypt-your-hard-drive-with-bitlocker-windows"```

[rhukster]

That's a weird one. This happens during the actual upload process? Nginx it appears? What file are you uploading? Can you provide me a sample file I can test with?

[w00fz]

Can you check the images folder and see if you get any folder locked up?

[rhukster]

In the meantime I suggest uploading files manually via ftp or something.

[gszathmari]

I am uploading compressed .png files to a new page using the 'Default' template. I have nginx indeed with php-fpm.

I have attached one of the offending files. However, this happens randomly.

[gszathmari]

@w00fz not sure how can I do that, could you please advise?

root@ubuntu:/var/www/website1/html/images# ls -l
total 56
drwxr-xr-x 4 web1 web1 4096 Jul 15 20:04 0
drwxr-xr-x 4 web1 web1 4096 Jul 15 20:04 1
drwxr-xr-x 6 web1 web1 4096 Jul 15 20:04 2
drwxr-xr-x 4 web1 web1 4096 Jul 15 20:04 3
drwxr-xr-x 3 web1 web1 4096 Jul 15 20:04 4
drwxr-xr-x 3 web1 web1 4096 Jul 15 20:04 5
drwxr-xr-x 4 web1 web1 4096 Jul 15 20:06 6
drwxr-xr-x 6 web1 web1 4096 Jul 15 20:06 7
drwxr-xr-x 3 web1 web1 4096 Jul 15 20:04 8
drwxr-xr-x 4 web1 web1 4096 Jul 15 20:04 9
drwxr-xr-x 4 web1 web1 4096 Jul 15 20:04 a
drwxr-xr-x 5 web1 web1 4096 Jul 15 20:06 b
drwxr-xr-x 4 web1 web1 4096 Jul 15 20:04 c
drwxr-xr-x 5 web1 web1 4096 Jul 15 20:06 e

[w00fz]

Yah I can reproduce with my own PNG file. Looks like it's a PNG issue.
The upload never truly finishes for me and I am forced to restart PHP-FPM. After that, everything is kinda broken.
What I noticed is that the images folder are locked up, they have no access property

[gszathmari]

@w00fz What helped me is removing the files file from cache/compiled manually

[w00fz]

How big in size is the image you are trying to upload and fails?
Also you should zip the image because I think it gets compressed if uploaded directly in here.

[gszathmari]

Average size is 50-60 KB, I was uploading 31 images

[rhukster]

31 images at the same time? does it happen when you do them individually? BTW, your error above led me to this:

http://serverfault.com/questions/511789/nginx-client-request-body-is-buffered-to-a-temporary-file

can you increase your client_body_buffer_size? http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size ?

[gszathmari]

I raised it to 1M and did not help

[gszathmari]

This fixes the problem and restores the website:

root@ubuntu:/var/www/website1/html/cache/compiled# ls -l
total 0
-rw-r--r-- 1 web1 web1 0 Jul 15 21:13 files
root@ubuntu:/var/www/website1/html/cache/compiled# rm files

[rhukster]

yes, i was able to reproduce the exact error on my raspberry pi with PHP7 and nginx+php-fpm. It was not while uploading, it was while deleting a bunch of files via the admin. it happened randomly, like part of the filename was lost, and it renamed the folder files/ to a file called file and that really broke things. I did get an invalid ajax method..

The only problem is that I only got it twice and can't get it to happen again for some reason. I'll keep trying this weekend though and digging in a bit further.

[rhukster]

Fixed via admin.

[gszathmari]

awesome, thanks a lot

[johncarter-oz]

@gszathmari indeed your solution works! A BIG thank you!
May I just explain how I got into the need for this solution? Because I cannot find it anywhere.

In our Linux configuration (BlueOnyx) the site storage allocation and the "user" allocation is entirely different and as there is no database in grav, my user was clocking up data for backups etc.
Eventually, I was installing a theme and I got an error that Grav could not perform MKDIR.
It half created the template (it just had a letter "v" in the top left corner) and I had to remove it via CLI and clear cache as described above. Increased my storage allocation for user and everything working great again.
I don't know if Grav could perform a simple MD5 checksum when it installs a theme or plugin and to drop the package if it does not match? Just a thought...