Uploaded file to PHP FPM server is still base64 encoded

[ZizzyZizzy]

Expected Behavior

When a file is uploaded, it should be decoded automatically.

Actual Behavior

File is still base64 encoded

Steps to Reproduce the Problem

Use fast-cgi-client to upload a file to a native PHP 7.2 FPM server. (I'm using Ubuntu)
Save the uploaded file and notice it's still base64 encoded

Specifications

fast-cgi-client Build 9b392b0

`> # uname -a

Linux mc40 4.15.0-115-generic #116-Ubuntu SMP Wed Aug 26 14:04:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/rel

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

php -v

PHP 7.2.24-0ubuntu0.18.04.6 (cli) (built: May 26 2020 13:09:11) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.24-0ubuntu0.18.04.6, Copyright (c) 1999-2018, by Zend Technologies

php -m

[PHP Modules]
bcmath
bz2
calendar
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
gettext
hash
iconv
imagick
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
yaml
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache`

Further comments

I honestly can't yet tell if this is an issue with the way fast-cgi-client is sending the file, or if it's something strange with PHP FPM on the server side.

Here's the file I'm sending:

ls -al bow_arrow_2d.png
-rwxrw-r-- 1 www-data www-data 3352 Aug  2  2019 bow_arrow_2d.png

# file bow_arrow_2d.png
bow_arrow_2d.png: PNG image data, 64 x 64, 8-bit/color RGBA, non-interlaced

My debugging shows the inflated size of the file and the wrong mime type on the FPM server side:

> 2020-09-22 20:54: ERROR - wrongmimetype
> 2020-09-22 20:54: Bad mime type 'text/plain'. Expecting one of:
> Array
> (
>     [0] => image/png
> )
>
> Array
> (
>     [file] => Array
>         (
>             [name] => bow_arrow_2d.png
>             [type] => application/octet-stream
>             [tmp_name] => /tmp/phpuTUfwo
>             [error] => 0
>             [size] => 4588
>         )
>
> )

When I use move_uploaded_file to /var/tmp/, it's clear that the image file is base-64 encoded. Decoded with "base64 -di", it is exactly the same file and size that I uploaded.

I've spent the better part of an hour checking php.ini, www.conf, and everywhere else I can think of to try and figure out why FPM is not decoding the image.

The code I'm using to send the image file is nearly identical to the one under the examples that comes with the source code. I'm just passing the path to the image file. Here's the debug output right before it sends $postdata and $files:

> Sending Post Request:
> Array
> (
>     [action] => seticon
>     [id] => 2806
> )
>
> Array
> (
>     [file] => /var/www/html/img/bow_arrow_2d.png
> )

Here's a snippet of the code I'm using to send.

>                 $multipartFormData = new MultipartFormData($postdata, $files);
>                 $postRequest = new PostRequest($fpmurl, $multipartFormData->getContent());
>             } catch (\Exception $e) {
>                 $errorMessage="Error getting form data! Did you forget to change integers to strings?\n" . print_r($e->getMessage(),TRUE);
>                 self::DebugLog($errorMessage);
>                 return json_encode(['status'=>'error','message'=>'Unexpected error. Please call support.','admin'=>$errorMessage]);
>             }
>             $postRequest->setContentType($multipartFormData->getContentType());
>             $postRequest->setRemoteAddress($CallingIP);
>             $response = $client->sendRequest($connection, $postRequest);

[ZizzyZizzy]

I used tcpflow to debug the connection to FPM. It's not doing anything special. The POSTed file is base64 encoded properly.

--__X_FASTCGI_CLIENT_BOUNDARY__
Content-Disposition: form-data; name="file"; filename="mimetest.php"
Content-Type: application/octet-stream
Content-Transfer-Encoding: base64

PD9waHAKZnVuY3Rpb24gZ2V0VmFycyhhcnJheSAkb3B0aW9ucyA9IFtdKSB7CgkvLyBTZXQgdGhl
IGRlZmF1bHQgdmFsdWVzLgoJJGRlZmF1bHRzID0gWwoJCSdmaWxlJyA9PiAnJwoJXTsKCSRvcHRp
b25zICs9ICRkZWZhdWx0czsKCgkvLyBTdWZmaWNpZW50IGVub3VnaCBjaGVjayBmb3IgQ0xJLgoJ
I2lmICgnY2xpJyA9PT0gUEhQX1NBUEkpIHsKCWlmIChwaHBfc2FwaV9uYW1lKCkgPT0gImNsaSIp
IHsKCQlyZXR1cm4gZ2V0b3B0KCcnLCBbJ2ZpbGU6J10pICsgJG9wdGlvbnM7Cgl9Cn0KJHZhcnM9
Z2V0VmFycygpOwokZmlsZT0kdmFyc1snZmlsZSddOwokbWltZVR5cGU9IG1pbWVfY29udGVudF90
eXBlKCRmaWxlKTsKcHJpbnQgIiRtaW1lVHlwZVxuIjsKaWYocHJlZ19tYXRjaCgnL15pbWFnZS8n
LCRtaW1lVHlwZSkpIHsKCXByaW50ICJcbkltYWdlIGRhdGE6XG4iIC4gcHJpbnRfcihnZXRpbWFn
ZXNpemUoJGZpbGUpLCBUUlVFKSAuICJcbiI7Cn0K
--__X_FASTCGI_CLIENT_BOUNDARY__--

[hollodotme]

@ZizzyZizzy Thank you for your patience! I had a look into this and sadly have to confirm that this is an implementation issue in the library.

The "problem" is that PHP-FPM doesn't handle the given Content-Transfer-Encoding in any way. It just reads the data in the block and puts it into a file as is. That's why you only see base64 encoded files after sending a multipart request.

After some research and testing I figured that the whole encoding stuff is not necessary at all and the contents of files can be sent as is, no matter what encoding or format.

The solutions was to remove the Content-Transfer-Encoding: base64 header and the base64 encoding of the file's content.

I added tests with an image to verify that the server stores the same file as uploaded.

The fixed version is already pushed to development branch. I'll release it as v3.1.5 as soon as I can.

[ZizzyZizzy]

@hollodotme This is great news! I will test it out in the next day or so, but it sounds like this will fix the issue!

It will actually be nice not to have to deal with the 30%+ size inflation of uploads, or attempt to Base64 decode with PHP and watch it crash out of memory for very large uploads!

[ZizzyZizzy]

@hollodotme It works perfectly, thank you!