New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[appserver-io/server] Problems with file uploads > 1MB #941

Closed
wick-ed opened this Issue Dec 14, 2015 · 12 comments

Comments

Projects
None yet
3 participants
@wick-ed
Member

wick-ed commented Dec 14, 2015

There seems to be a problem with file uploads for files bigger than 1MegaByte.
Posted example code to reproduce the error looks as following:

<!DOCTYPE html>
<html>

<head>
<title></title>
</head>

<body>

<form enctype="multipart/form-data" method="post" name="fileinfo" action="mailer/m.php">

<input type="file" name="field-file" required />
<br>
<input type="submit" value="submit" />

</form>
<div></div>

</body>

</html>
<?php
    $uploads_dir = "./sample_upload";
    $upload_error = 0;
    $fileName = $_FILES['field-file']['name'];
    $fileType = $_FILES['field-file']['type'];
    $tmp_name = $_FILES['field-file']["tmp_name"];
    $err_ok = $_FILES['field-file']['error'];
    if ( $err_ok != UPLOAD_ERR_OK ) 
        $upload_error = 1;
    if ( !$upload_error ) {
        $codedFilename = time()."_". basename( $fileName );
        $stored_file = $uploads_dir ."/".$codedFilename;
        if ( !move_uploaded_file( $tmp_name , $stored_file ) ) 
            $upload_error = 2;
    }
    if($upload_error) {
        if ($upload_error === 1) 
            $err_msg = "(1 - ".codeToMessage($err_ok).")"; else $err_msg = "(2)";
        echo "Error upload screenshot :: err $err_msg";
        exit;
    }
    echo "File uploaded";

    function codeToMessage($code)
    {
        switch ($code) {
            case UPLOAD_ERR_INI_SIZE:
               $message = "The uploaded file exceeds the upload_max_filesize directive in php.ini";
            break;
            case UPLOAD_ERR_FORM_SIZE:
                $message = "The uploaded file exceeds the MAX_FILE_SIZE directive that was specified in the HTML form";
                break;
            case UPLOAD_ERR_PARTIAL:
                $message = "The uploaded file was only partially uploaded";
            break;
            case UPLOAD_ERR_NO_FILE:
                $message = "No file was uploaded";
            break;
            case UPLOAD_ERR_NO_TMP_DIR:
                $message = "Missing a temporary folder";
            break;
            case UPLOAD_ERR_CANT_WRITE:
                $message = "Failed to write file to disk";
            break;
            case UPLOAD_ERR_EXTENSION:
                $message = "File upload stopped by extension";
            break;
            default: 
                $message = "Unknown upload error"; 
            break; 
        } 
        return $message; 
    }

@wick-ed wick-ed added the bug label Dec 14, 2015

@wick-ed

This comment has been minimized.

Show comment
Hide comment
@wick-ed

wick-ed Dec 14, 2015

Member

/opt/appserver/var/log/php_errors.log shows the following message:

 PHP Notice: fwrite(): send of 8192 bytes failed with errno=11 Resource temporarily unavailable in /opt/appserver/vendor/appserver-io/fastcgi/src/Connection.php on line 157

Test script will report The uploaded file was only partially uploaded.

Issues does not seem to be related to a fixed file size as several tries of an upload bigger than 1MB brought different results.
Even a 2.5MB test file could be uploaded successfully on the third try.

Member

wick-ed commented Dec 14, 2015

/opt/appserver/var/log/php_errors.log shows the following message:

 PHP Notice: fwrite(): send of 8192 bytes failed with errno=11 Resource temporarily unavailable in /opt/appserver/vendor/appserver-io/fastcgi/src/Connection.php on line 157

Test script will report The uploaded file was only partially uploaded.

Issues does not seem to be related to a fixed file size as several tries of an upload bigger than 1MB brought different results.
Even a 2.5MB test file could be uploaded successfully on the third try.

@al-Obaidi

This comment has been minimized.

Show comment
Hide comment
@al-Obaidi

al-Obaidi Dec 14, 2015

When the error message appeared, if you Refresh the page, and click "Continue" to Confirm Form Resubmission, the file will be uploaded.

al-Obaidi commented Dec 14, 2015

When the error message appeared, if you Refresh the page, and click "Continue" to Confirm Form Resubmission, the file will be uploaded.

@wick-ed

This comment has been minimized.

Show comment
Hide comment
@wick-ed

wick-ed Dec 21, 2015

Member

Hi @Ferfoor ,

sorry for the long wait.

The problem is linked to a non-blocking socket connection which is not been tested for readiness on write.
Therefor parts of the message get dropped when the resource (PHP-FPM) has too much load.

One thing you can do for sure is give FPM more power by spawning more server. You can adjust that in /opt/appserver/etc/php-fpm.conf.

Secondly, You can test this patch I wrote:
https://github.com/wick-ed/FastCGI/tree/941
It is no definite solution (and therefor not merged) as I see problems with performance but it should solve your specific problem.
I will expand on this patch and merge it as soon as I got some more time.

Regards,
Bernhard

Member

wick-ed commented Dec 21, 2015

Hi @Ferfoor ,

sorry for the long wait.

The problem is linked to a non-blocking socket connection which is not been tested for readiness on write.
Therefor parts of the message get dropped when the resource (PHP-FPM) has too much load.

One thing you can do for sure is give FPM more power by spawning more server. You can adjust that in /opt/appserver/etc/php-fpm.conf.

Secondly, You can test this patch I wrote:
https://github.com/wick-ed/FastCGI/tree/941
It is no definite solution (and therefor not merged) as I see problems with performance but it should solve your specific problem.
I will expand on this patch and merge it as soon as I got some more time.

Regards,
Bernhard

@al-Obaidi

This comment has been minimized.

Show comment
Hide comment
@al-Obaidi

al-Obaidi Dec 21, 2015

Hi Berhard,

Thank you very much for your feedback.
I will test your suggestion at the end of the day.

Best Regards.

al-Obaidi commented Dec 21, 2015

Hi Berhard,

Thank you very much for your feedback.
I will test your suggestion at the end of the day.

Best Regards.

@al-Obaidi

This comment has been minimized.

Show comment
Hide comment
@al-Obaidi

al-Obaidi Dec 27, 2015

Hi @wick-ed,

I've installed appserver to make a new round of testing. It is version appserver/1.1.0-3.
To my surprise that I didn't get an error message when I've made the first upload. I've tried another upload with increased file size, also it succeed. I've ended up uploading 1mb, 13mb, 36mb, and 73mb files (after setting the upload size limit in php.ini), all succeed :).

The only thing came to my mind that appserver developer team have updated the source code and when I've installed it again (as I removed it after the last bug and installed apache) the new code work as expected without problem.

I'm appreciating your effort to fix this bug. I like appserver and I will continue testing it. Last time I'v tried to install magento 2, but unfortunately it is stuck at 90% of installation. I will give it another try with current version. My eye to use magento 2 on appserver with php 7.

Thanks again.

al-Obaidi commented Dec 27, 2015

Hi @wick-ed,

I've installed appserver to make a new round of testing. It is version appserver/1.1.0-3.
To my surprise that I didn't get an error message when I've made the first upload. I've tried another upload with increased file size, also it succeed. I've ended up uploading 1mb, 13mb, 36mb, and 73mb files (after setting the upload size limit in php.ini), all succeed :).

The only thing came to my mind that appserver developer team have updated the source code and when I've installed it again (as I removed it after the last bug and installed apache) the new code work as expected without problem.

I'm appreciating your effort to fix this bug. I like appserver and I will continue testing it. Last time I'v tried to install magento 2, but unfortunately it is stuck at 90% of installation. I will give it another try with current version. My eye to use magento 2 on appserver with php 7.

Thanks again.

@wagnert

This comment has been minimized.

Show comment
Hide comment
@wagnert

wagnert Dec 28, 2015

Member

@Ferfoor I can reproduce that bug, so i think it's still there and we've to find a solution. But it's fine for sure, that you didn't have that problem any more :)
Regarding Magento 2 i also had the problem, that the setup stopped at 90 %, but i was able to install it from the commandline!

Member

wagnert commented Dec 28, 2015

@Ferfoor I can reproduce that bug, so i think it's still there and we've to find a solution. But it's fine for sure, that you didn't have that problem any more :)
Regarding Magento 2 i also had the problem, that the setup stopped at 90 %, but i was able to install it from the commandline!

@al-Obaidi

This comment has been minimized.

Show comment
Hide comment
@al-Obaidi

al-Obaidi Dec 28, 2015

Hi wagnert

After reading your comment , I've tried again to reproduce the error. Unfortunately I could reproduced it again as you mentioned. So it seems it happens randomly. We still need to find out what the problem is.

About magento 2, I succeed installing it by increasing maximum execution time, I've set to max_execution_time = 3000 in /opt/appserver/etc/php-fpm-fcgi.ini. I've installed it with data sample. look nice.

al-Obaidi commented Dec 28, 2015

Hi wagnert

After reading your comment , I've tried again to reproduce the error. Unfortunately I could reproduced it again as you mentioned. So it seems it happens randomly. We still need to find out what the problem is.

About magento 2, I succeed installing it by increasing maximum execution time, I've set to max_execution_time = 3000 in /opt/appserver/etc/php-fpm-fcgi.ini. I've installed it with data sample. look nice.

@wagnert

This comment has been minimized.

Show comment
Hide comment
@wagnert

wagnert Dec 28, 2015

Member

@Ferfoor It's an issue for sure and @wick-ed will have an eye on it. Thanks for the tip with Magento 2. We'll raise the max_execution_time by default to an apropriate value :)

Member

wagnert commented Dec 28, 2015

@Ferfoor It's an issue for sure and @wick-ed will have an eye on it. Thanks for the tip with Magento 2. We'll raise the max_execution_time by default to an apropriate value :)

@wick-ed

This comment has been minimized.

Show comment
Hide comment
@wick-ed

wick-ed Dec 29, 2015

Member

Hi @Ferfoor , @wagnert ,

the issue is related to high load on php-fpm and the socket connection to the appserver therefor not being able to write data (e.g. a file currently being uploaded).
So it makes perfect sense for this issue to occur randomly.

Advised solution is to check and resend (that is what my patch is doing for you) but this decreases performance for certain use cases.

Will merge the patch when I get a handle on the performance problems. :)

Member

wick-ed commented Dec 29, 2015

Hi @Ferfoor , @wagnert ,

the issue is related to high load on php-fpm and the socket connection to the appserver therefor not being able to write data (e.g. a file currently being uploaded).
So it makes perfect sense for this issue to occur randomly.

Advised solution is to check and resend (that is what my patch is doing for you) but this decreases performance for certain use cases.

Will merge the patch when I get a handle on the performance problems. :)

@al-Obaidi

This comment has been minimized.

Show comment
Hide comment
@al-Obaidi

al-Obaidi Dec 29, 2015

Hi @wick-ed
What the parameter/value I have to change in "/opt/appserver/etc/php-fpm.conf" ? regarding your suggestion:-

"One thing you can do for sure is give FPM more power by spawning more server. You can adjust that in /opt/appserver/etc/php-fpm.conf."

al-Obaidi commented Dec 29, 2015

Hi @wick-ed
What the parameter/value I have to change in "/opt/appserver/etc/php-fpm.conf" ? regarding your suggestion:-

"One thing you can do for sure is give FPM more power by spawning more server. You can adjust that in /opt/appserver/etc/php-fpm.conf."

@wick-ed

This comment has been minimized.

Show comment
Hide comment
@wick-ed

wick-ed Jan 5, 2016

Member

Hi @Ferfoor ,

this would most notebly be:

  • pm.max_children
  • pm.start_servers
  • pm.min_spare_servers
  • pm.max_spare_servers
  • pm.max_requests

See also http://php.net/manual/en/install.fpm.configuration.php

With these values you can manage how much child processes php-fpm uses and keeps open for requests. This can (but without guaranty) improve throughput of your FastCGI connection.

But as Tim already said: this is most definitely a bug on our side.

Member

wick-ed commented Jan 5, 2016

Hi @Ferfoor ,

this would most notebly be:

  • pm.max_children
  • pm.start_servers
  • pm.min_spare_servers
  • pm.max_spare_servers
  • pm.max_requests

See also http://php.net/manual/en/install.fpm.configuration.php

With these values you can manage how much child processes php-fpm uses and keeps open for requests. This can (but without guaranty) improve throughput of your FastCGI connection.

But as Tim already said: this is most definitely a bug on our side.

@al-Obaidi

This comment has been minimized.

Show comment
Hide comment
@al-Obaidi

al-Obaidi Jan 14, 2016

Thank you @wick-ed for response.

al-Obaidi commented Jan 14, 2016

Thank you @wick-ed for response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment