Video File is not downloaded Properly

[kingbode]

I tried to download video files using Tumblethree, files are being downloaded, but when I run them, it seems that they are not downloaded properly as the file play stops in the middle of the video, so is this common issue , or not ??

thanks in advance

[johanneszab]

It seems?

I need an exact video url/post where I can check this. Otherwise I cannot help you. I've just tested 3 videos, which were downloaded exactly the same as they are on the website. And note, one of those videos is cut off right in the middle. I think there was a size limit for uploading videos some time ago?

It's not that I don't believe you. I've asked the same here (#12), but since I don't have much time for the development at all right now, I've briefly tested it and it seems to work. Since there was no response I've not further investigated this.

[kingbode]

Thanks Zab for your prompt response, I checked video downloading using your app, an d also try it on my trial dev app using my own version ( inherited from helena tumblOne code ), I'm using asyn download,,, and both got me at the end videos , seems to be not downloaded completely, if you have time to test, check it on below blog,

honestly I still don't understand the limit issue of Tumblr, if you would direct me to a topic that explain this I would appreciate it, what i understood that Tumblr will use Api ver 2 and this Api has limit of transaction per day.. so what about api ver 1?? are they doing to stop it ?

[johanneszab]

If you lower the Parallel Connections it works for me.

There are some Exceptions with: Received an unexpected EOF or 0 bytes from the transport stream and I'm not sure right if thats from my throttle code or a server response.

[johanneszab]

honestly I still don't understand the limit issue of Tumblr, if you would direct me to a topic that explain this I would appreciate it, what i understood that Tumblr will use Api ver 2 and this Api has limit of transaction per day.. so what about api ver 1?? are they doing to stop it ?

Here is the official documentation of the Tumblr api. Usually, you create APIs to have an interface for other programmers that guarantees that this interface always behaves the same way. So if you use it as a programmer you shouldn't have to worry that your application someday stops working because the company decided to change something, like the website or its layout. If we parse the website that could happen.
So the best solution is probably read the website and just use regular expression for specific terms, like http://media*.tumblr.com/*{gif|jpg|png}. That should most likely always work.

Most companies have some regulation for their apis. You can check that for yourself. If you open TumblThree in visual studio and untick the Limit Tumblr Api checkbox or just request something from the old api (version 1) several times per second, you'll see that at some point the server does not respond anymore, but sends the 429 exception.
That wasn't the case for the old api before mid-February.

[kingbode]

I agree with you that parsing the website is better than depending on website structure that may change at any point of time, I was about to do this with tumblr pages in the beginning, but I stuck on how to get the the whole content of the page as it seems having JS function that fetches the next page and so on, I need to check your code and how you achieved this in Tumblr with parser not with API, does latest version depend on parsing or only API ? , for me I tried to load the code of latest version and it supposed to work normally in VS but it gives me these errors, could you help , why I'm getting these errors ??

[johanneszab]

I just replaced my download&throttle code with the
WebClient.DownloadFileTaskAsync(url,filelocation)
and started 20 downloads on my slow home network. You can't have easier code than this and it should work, but the downloads are also incomplete:

Exception thrown: 'System.IO.IOException' in System.dll ("Unable to read data from the transport connection: The connection was closed.")
Exception thrown: 'System.Net.WebException' in mscorlib.dll ("An exception occurred during a WebClient request.")

So it's probably a server response if the download takes to long or something.

[johanneszab]

hmm, those $ signs are interpolated strings. Actually, I've never looked into the RateLimiter code since it's from google, I thought its good/better than the other rate limiter.

Thats a c# 6.0 feature, so relative new. According to stackoverflow here, and here, you need at least a Visual Studio 2015 or a NuGet package which updates your compiler.

I didn't know this, sorry. We can replace the strings with string.format() methods so people don't need the newest Visual Studio.

Edit: I'll just do this and upload the changes. Shouldn't take too long.

[kingbode]

Thanks Zab for your efforts, one more request but not necessary to be fulfilled ( as you wish I mean ), I'm zero in WPF structure , so is there any intention to have form application version ?? or what is the advantage to be WPF version over form application.

[johanneszab]

Could you try again with the newest source and see if it compiles for you?

I'm interested in that it works for other people. I've had some emails/comments where the source didn't compile and I couldn't help because there were zero issues here on 3 different machines.

[kingbode]

this snap shot with source of today 04-04-2017
TumblThree-1.0.4.36

[johanneszab]

Ah, you've probably downloaded the release source code again, although I've meant the current master source. I've re-uploaded the release thus it contains the current state of the source now.

Please try again, both sources (master,release) should work now.

[kingbode]

below is what I got right after loading the solution

and after rebuild I got below errors

[johanneszab]

Same problem.

I probably won't fix that today anymore, I'm really tired now. Thanks for testing! I guess that's then all errors. The 10-16 are just follow up errors because you couldn't compile the source with VS 2013.

[kingbode]

yes may be you are right, I will try to update my VS and check again

[johanneszab]

I will try to update my VS and check again

Nah, you don't have to. It's a pain to update/install VS and it's good to see if it works with VS 2013. It's a easy fix too and nothing important there. Just ouput/excepption messages.

Right now we don't need anything from C# 6.0, but I don't code when I'm tired. It's most of the time now worth the hassle. I'll fix that tomorrow.

[kingbode]

Ok Zab , I appreciate your help, I'm waiting for the fix

[johanneszab]

Okay, try again please and let me know what happens!

[johanneszab]

We probably can resume the broken video downloads.

The could catch the specific EOF/null byte/unable to read from transport stream exceptions and try to resume the file until it's successfully downloaded. Might be worth a try.

[kingbode]

is there a way to get file size from the server before download is complete to double check if it's downloaded successfully or not ? I think IDM is doing the same, and in case of any reason download fail, then exception can display the reason.
also the same for image resolutions\image file size , to limit from the beginning , I think your setting it's applied , I don;t remember

[johanneszab]

Here is a C# implementation with nice code: https://github.com/Avira/.NetFileDownloader
And here is the bare stuff: https://www.codeproject.com/Tips/307548/Resume-Suppoert-Downloading

Use FileInfo to get how many bytes have been downloaded to the file already, use httpwebrequest.Range() to start at that byte and use HEAD to get the total file size.

Seems to be common practice at larger files. It's actually my first "net" application ever. I mean, quite obviously that the internet isn't the most reliable connection.

[johanneszab]

I've implemented a quick hack and I think it must be on purpose.

There is a regular pattern, like all 120 seconds were the EOF/0 byte gets send to every connection, thus killing the connection and renders in incomplete files. If I however download just one video with a heavily throttled connection there is not a single EOF/0 byte, even if the download takes more than 10 minutes. And all files are complete ..

[johanneszab]

Ok, I've got some working code now.

Maybe it's wise now to implement a resume method if there is already a incomplete file on the disk from a previous download session. Right now we just overwrite them and start from scratch. That never was a issue with pictures since they aren't that big and it only happens for pictures which weren't complete downloaded if the download was stopped.