Error uploading big video file to Kaltura with chunked upload

[stesteau]

Hello,

We use Kaltura SAAS Hercule hosted on rackforcecloudvideo.com

I posted on the KalturaGeneratedAPIClientsJava, but I guess this is the right place.

I am trying to upload a big file using chunked upload. I litterally copied and pasted the code from the android project

https://raw.githubusercontent.com/kaltura/AndroidReferenceApp/master/DemoApplication/src/com/kaltura/services/UploadToken.java

I just removed unused code such as

setChanged();
notifyObservers(getUploadedFileSize());

The upload and converting works well with small/medium size files, anything under 2G actually.

But with a 3.1G file, the upload is still working, I can see the file in the admin «converting». But after a while It falls in error and the contibution file (the uploaded file) hasn't been rebuilt into an mp4; it's a .dat

Thank you

[kaltura-hooks]

Hi @moskiteau,

Thank for you reporting an issue and helping improve Kaltura!

To get the fastest response time, and help the maintainers review and test your reported issues or suggestions, please ensure that your issue includes the following (please comment with more info if you have not included all this info in your original issue):

• Is the issue you're experiencing consistent and across platforms? or does it only happens on certain conditions?
  please provide as much details as possible.
• Which Kaltura deployment you're using: Kaltura SaaS, or self-hosted?
  If self hosted, are you using the RPM or deb install?
• Packages installed.
  When using RPM, paste the output for:

    # rpm -qa "kaltura*"

For deb based systems:

    # dpkg -l "kaltura-*"

• If running a self hosted ENV - provide the MySQL server version used
• If running a self hosted ENV - is this a single all in 1 server or a cluster?
• If running a self hosted ENV, while making the problematic request, run:

    # tail -f /opt/kaltura/log/*.log /opt/kaltura/log/batch/*.log | grep -A 1 -B 1 --color "ERR:\|PHP\|trace\|CRIT\|\[error\]"

and paste the output.

• When relevant, provide any screenshots or screen recordings showing the issue you're experiencing.

For general troubleshooting see:
https://github.com/kaltura/platform-install-packages/blob/Jupiter-10.13.0/doc/kaltura-packages-faq.md#troubleshooting-help

If you only have a general question rather than a bug report, please close this issue and post at:
http://forum.kaltura.org

Thank you in advance,

[jessp01]

Hello,

Can you please attach the full code so we can try to reproduce? Also, what ver of the clientlibs are you using? You should generate the clientlibs on the server you are working against.
You can do so by running

php /opt/kaltura/app/generator/generate.php java

In addition, Kaltura SaaS refers to the version on www.kaltura.com, the version you work against is either an OnPrem or CE install of Hercules, I gather, correct?

[KalturaCommunity-zz]

It's also possible that chunk upload was not yet introduced in Hercules as it's a rather old version. Please check the clientlibs generated on the server you work against to make sure.

Sent from Outlook Mobilehttps://aka.ms/blhgte

On Wed, Nov 25, 2015 at 11:45 AM -0800, "Jess Portnoy" <notifications@github.commailto:notifications@github.com> wrote:

Hello,

Can you please attach the full code so we can try to reproduce? Also, what ver of the clientlibs are you using? You should generate the clientlibs on the server you are working against.
You can do so by running

php /opt/kaltura/app/generator/generate.php java

In addition, Kaltura SaaS refers to the version on www.kaltura.comhttp://www.kaltura.com, the version you work against is either an OnPrem or CE install of Hercules, I gather, correct?

Reply to this email directly or view it on GitHubhttps://github.com//issues/3549#issuecomment-159712045.

[stesteau]

@jessp01
You are correct. We are using a kaltura reseller solution http://rackforcecloudvideo.com/products/kaltura-management-console
Unfortunately, I can't access the server by ssh since we are not managing the server.

@KalturaCommunity
That is not the case since I uploaded successfully and converted multiple video files. Only big files seems to fail.

I will attach the code in a bit.

[etameran]

If you are using one upload request (instead of uploading chunk by chunk) you will be limited to 2GB by the php code on the server.

Eran

---

From: Sébastien Testeau notifications@github.com
Sent: Wednesday, November 25, 2015 10:01 PM
To: kaltura/server
Subject: Re: [server] Error uploading big video file to Kaltura with chunked upload (#3549)

@jessp01https://github.com/jessp01
You are correct. We are using a kaltura reseller solution http://rackforcecloudvideo.com/products/kaltura-management-console
Unfortunately, I can't access the server by ssh since we are not managing the server.

@KalturaCommunityhttps://github.com/KalturaCommunity
That is not the case since I uploaded successfully and converted multiple video files. Only big files seems to fail.

I will attach the code in a bit.

Reply to this email directly or view it on GitHubhttps://github.com//issues/3549#issuecomment-159716912.

[stesteau]

@etameran

I know that, it's the reason why I am using the chunk by chunk upload

[KalturaCommunity-zz]

I did not mean the upload API is not present is Hercules, it has been present forever. I meant that the chunk option may not yet been introduced.

Sent from Outlook Mobilehttps://aka.ms/blhgte

On Wed, Nov 25, 2015 at 12:02 PM -0800, "Sébastien Testeau" <notifications@github.commailto:notifications@github.com> wrote:

@jessp01https://github.com/jessp01
You are correct. We are using a kaltura reseller solution http://rackforcecloudvideo.com/products/kaltura-management-console
Unfortunately, I can't access the server by ssh since we are not managing the server.

@KalturaCommunityhttps://github.com/KalturaCommunity
That is not the case since I uploaded successfully and converted multiple video files. Only big files seems to fail.

I will attach the code in a bit.

Reply to this email directly or view it on GitHubhttps://github.com//issues/3549#issuecomment-159716912.

[stesteau]

@KalturaCommunity
Yes, that is what I said, the chunk option. I uploaded with chunks multiples files successfully already on hercule

[stesteau]

ok, here is the code
https://github.com/moskiteau/KalturaGeneratedAPIClientsJava

You just need to change the video file to a bigger one

[tan-tan-kanarek]

Could you also share the logs, or the exact exception stack and message?

[stesteau]

@tan-tan-kanarek
I don't have access to the logs since it is an hosted solution (argghh!)
Also, there is no upload error, none, zero, nada. The error is after upload when status is «ok», while trying to encode the contribution file. (see screenshot)

But, when I try to download the contribution file I get this error:

An error occurred while processing your request.
Reference #172.134ea7ce.1448549119.e88a0

[jessp01]

@moskiteau, can you please open a free trial account on our SaaS and see if this happens to you when uploading there? It will be easier for us to debug. If it does happen, can you provide the entry ID?

[stesteau]

@jessp01
Ok, it is uploaded and it is currently converting.

entryId: 1_n5a1qqvy

[jessp01]

Thanks, @moskiteau.
I see the result is much the same.
We will investigate.

[tan-tan-kanarek]

I see nothing wrong with this entry: http://www.kaltura.com/tiny/lvz6a

[stesteau]

It is indeed working on the SaaS

[jessp01]

Is the server you are working against OnPrem or CE? if it is OnPrem, please have the customer open a ticket for support. If it is CE, then we will need the logs to try and help further.

[stesteau]

It is onPrem, thank you for you help guys

[jessp01]

Most welcome. Please open a support ticket and they can further assist you.

[jessp01]

@moskiteau - I was able to reproduce the issue on SOME machines/JRE versions.
What I saw is simply that, when readSum which is defined as int and passed as resumeAt to addChunk(), is larger than 2140160000, doing this:
readSum += numRead;
causes it to become a negative value, which naturally, will not lead to overly positive results:)

Changing:
private int readSum = 0;
to:
private long readSum = 0;

and the addChunk() method signature to:
private boolean addChunk(KalturaClient client, String uploadTokenId, File outFile, boolean resume,boolean finalChunk, long resumeAt)

Solved the issue and I was able to upload a 3G file.

In order to test it from CLI, I created this little class here:
import com.kaltura.client.enums.;
import com.kaltura.client.types.;
import com.kaltura.client.services.*;
import com.kaltura.client.KalturaApiException;
import com.kaltura.client.KalturaClient;
import com.kaltura.client.KalturaConfiguration;
import myupload.MyUploadToken;
class UploadTest
{
public static void main(String[] argv){
try{
if (argv.length < 4){
System.out.println("Usage: </path/to/file>\n");
System.exit (1);
}
KalturaConfiguration config = new KalturaConfiguration();
config.setEndpoint(argv[0]);
KalturaClient client = new KalturaClient(config);

                    String secret = argv[2];
                    String userId = null;
                    int partnerId = Integer.parseInt(argv[1]);
                    String privileges = null;
                    //String ks = client.generateSessionV2(secret, null, KalturaSessionType.ADMIN, partnerId, 86400, "");
                    KalturaSessionService sessionService = client.getSessionService();
                    String ks = sessionService.start(secret, "admin",KalturaSessionType.ADMIN,partnerId, 86400, "");
                    client.setSessionId(ks);
                    System.out.println(ks);
                    KalturaMediaEntry entry = new KalturaMediaEntry();
                    entry.name = "Linux012";
                    entry.type = KalturaEntryType.MEDIA_CLIP;
                    entry.mediaType = KalturaMediaType.VIDEO;
                    KalturaMediaEntry newEntry = client.getMediaService().add(entry);
                    MyUploadToken ut = new MyUploadToken("TAG", 100000, client);
                    ut.setStartUpload(true);
                    ut.uploadMediaFileAndAttachToEmptyEntry("TAG",newEntry,argv[3]);
            } catch (Exception exc) {
                    // TODO: handle exception
            }
    }

}

Can you change the code in stesteau/KalturaGeneratedAPIClientsJava@22f4545 accordingly and run a simple test from CLI with:
$ java -cp testAPI/:. UploadTest service_url partner_id admin_secret /path/to/large/file

And let me know as to the results?

of course, change -cp to where ever you've got the Kaltura classes and required deps..

[stesteau]

Indeed, that will work better.

Ok, so It does work now.

But... That brings another problem which is why it worked on the SAAS and the 2.4G file was uploaded successfully!?

My guess is the overflow doesn't break the upload and although it is an error in the code it is still luckily working.

But that's in you hand now :)

Cheers

[jessp01]

The reason it works with the latest code base stems from changes made to: alpha/apps/kaltura/lib/kUploadTokenMgr.php
commit e4e29c8
Date: Thu Jul 17 12:52:28 2014 +0300

@@ -249,14 +238,21 @@
         */
        protected function resumeFile($sourceFilePath, $targetFilePath, $resumeAt = -1)
        {
-               $sourceFileResource = fopen($sourceFilePath, 'rb');
                $targetFileResource = fopen($targetFilePath, 'r+b');

-        if ($resumeAt != -1)
+               fseek($targetFileResource, 0, SEEK_END);
+               if ($resumeAt != -1)
+               {
+                       if ($resumeAt > ftell($targetFileResource))
+                       {
+                               fclose($targetFileResource);
+                               throw new kUploadTokenException("Temp file [$targetFilePath] attempted to resume at invalid position $resumeAt", kUploadTokenException::UPLOAD_TOKEN_RESUMING_INVALID_POSITION);
+                       }
                        fseek($targetFileResource, $resumeAt, SEEK_SET);
-        else
-                       fseek($targetFileResource, 0, SEEK_END);
-                                               
+               }
+
+               $sourceFileResource = fopen($sourceFilePath, 'rb');
+
                while(!feof($sourceFileResource))
                {
                        $data = fread($sourceFileResource, 1024*100);

If you take the latest file from https://github.com/kaltura/server/blob/Kajam-11.4.0/alpha/apps/kaltura/lib/kUploadTokenMgr.php and override the original, it will work even if readSum is declared as an int.

However, since:
Integer.MAX_VALUE = 2147483647
Integer.MIN_VALUE = -2147483648

Long.MAX_VALUE = 9223372036854775807
Long.MIN_VALUE = -9223372036854775808

Declaring readSum as long is better anyhow. No good reason to overflow, even if it does work in newer Kaltura server versions.