Weird Size Issue - 600KB video file results in 18MB upload traffic. 8 MB file results in 228MB outgoing traffic

[acpadhi]

I am trying to upload a video to you tube using the latest code samples.
The file size is around 600 KB (KiloBytes).

While the sample uploads the video fine using the chunked upload method, I found something really strange.

A file size of 600KB results in an upload traffic of around 17 to 18 MB.
Assuming, a base64_encode of the same file results in a physical file size of : 884KB.

Can someone explain, if the large upload traffic is natural or there is some issue ?

If it is normal, then this is an extremely inefficient way of uploading data. Youtube should develop better methods.

I also tested with a physical file of size 8MB and it resulted in an upload traffic of whooping 228 MB.

Can any one throw some light on this ? if someone wants, I can paste the entire code of the script I am using,

I can see why people are complaining about slow uploads too : #50

Edit : Tested with the previous version of the code. google-api-php-client-1.0.0-alpha - This works fine, and upload traffic is equal to file size.

[acpadhi]

google-api-php-client-1.0.0-alpha does not have this problem. Upload Traffic Size is almost equal to file size.

So some issue cropped up after the gzip stuff was added

[ianbarber]

Very odd, I can see the same thing when uploading to drive. Disabling the gzip seems to bring it back to normal, which is rather confusing. I'll poke around a little more, but we may just need to automatically disable gzip for file upload, which is straightforward.

Workaround is to disable it manually using: $client->setClassConfig("Google_Http_Request", "disable_gzip", true);

[ianbarber]

So what I can see is compress.zlib doing an absolute tonne of writes on the upload, which is not very helpful. Unfortunately, I can't find a wonderful work around that lets us deal with the stream filter in a decent way, so I think disabling on uploads will be the best bet.

[ianbarber]

Patched in #65

[AdamWill]

I'm seeing something very like this with ownCloud's Google Drive support (which I'm still working to get ported to 1.x). It's using extremely simple upload code, ripped straight from the examples:

public function writeBack($tmpFile) {
    if (isset(self::$tempFiles[$tmpFile])) {
        $path = self::$tempFiles[$tmpFile];
        $parentFolder = $this->getDriveFile(dirname($path));
        if ($parentFolder) {
            // TODO Research resumable upload
            $mimetype = \OC_Helper::getMimeType($tmpFile);
            $data = file_get_contents($tmpFile);
            $params = array(
                'data' => $data,
                'mimeType' => $mimetype,
                'uploadType' => 'media'
            );
            $result = false;
            if ($this->file_exists($path)) {
                $file = $this->getDriveFile($path);
                $result = $this->service->files->update($file->getId(), $file, $params);
            } else {
                $file = new \Google_Service_Drive_DriveFile();
                $file->setTitle(basename($path));
                $file->setMimeType($mimetype);
                $parent = new \Google_Service_Drive_ParentReference();
                $parent->setId($parentFolder->getId());
                $file->setParents(array($parent));
                $result = $this->service->files->insert($file, $params);
            }
            if ($result) {
                $this->setDriveFile($path, $result);
            }
        }
        unlink($tmpFile);
    }
}

and I'm currently testing with version 1.0.6 of the library. But I seem to be seeing just the same issue as the OP here - uploading a simple ~4MB file results in over 100MB of traffic. Any ideas?

[AdamWill]

Ah. I think the fix for this may have only fixed it for use of the Google_Http_MediaFileUpload class, not the simple files->insert method. Let me see if my theory holds up.

[AdamWill]

Hum. Actually, I think I was using a packaged copy of 1.0.3 I had lying around and somehow with 1.0.6 the problem doesn't happen any more, though I can't clearly see why it would affect 1.0.3 but not 1.0.6. But hey, so long as it works...hey ho.

Aha! So I think I'm not hitting it in 1.0.6 because it's using Curl instead of Stream and bypassing the whole thing. If I force use of Stream instead of Curl, I can cause it to happen again.

In MediaFileUpload.php , the bit where it conditionally disables gzip for uploads is only in nextChunk(), which is only used for resumable uploads. I've tried copying it into process() (with the obvious change to $this->request instead of $httpRequest), and the code gets hit (I put a debug log in to check), but it doesn't seem to do the job - the amount of data transferred is still huge. But if I disable gzip with the big hammer - $client->setClassConfig("Google_Http_Request", "disable_gzip", true); - that does solve the problem. Not quite sure why just trying to disable it in process() doesn't work.