Fix timeouts when sending large files

[tsnoam]

This issue is in order to track the various tasks of a bug in master branch.
The problem is that urllib3 doesn't honour the read_timeout when using a newly created connection. However, with an existing connection it works just fine.
Regardless of working with upstream urllib3 to fix the issue, we need to fix the issue for v6.0 milestone.

• Create a unitest to recreate the problem. See in travis that the problem is recreated.
• "Vendor" a fork of urllib3 with a fix with python-telegram-bot package.
• Check that the unitest is now fixed.
• Modify defaults of timeouts to sane values:
  • connect_timeout = 5 seconds
  • read_timeout = 5 seconds
  • sendDocument, sendAudio & similar methods = 20 seconds.

[Eldinnie]

Is it possible this has to do with this? https://core.telegram.org/api/optimisation#downloading-files-and-uploading-data-to-the-server It mentions:

It makes sense to download files over several connections (optimally to have a pool). When uploading data to a server one connection is enough to achieve the best results.

I'm not really into this connectionstuff myself, but as far as I read the code, sending of files goes via the same connection_pool as all other tasks. It seems TG is advising against this.

[tsnoam]

Unitesting is tricky as this one is hard to recreate.
We're going to release v6 with our vendored urllib3 version and see how to go forward from there.
Keeping this issue open for reference,

[Eldinnie]

@tsnoam do we still need this issue open? Or can we safely say this issue has been resolved?

[tsnoam]

@Eldinnie i am not yet convinced that the problem is fully fixed and/or that we don't left overs. I do know that the situation is much better than the past (or people stopped complaining...).
There are a few reports of timeouts issue which require more thorough investigation.
please keep this open.

[Eldinnie]

@tsnoam 👍

[EnricoVerzegnassi]

I have this problem too on my raspberry with ArchLinux Arm.
When I send files bigger than ~5 Mb, python stops uploading, therefore I do not receive the file in the telegram chat.

[tsnoam]

@ilverz It doesn't necessarily be the same issue. Have you tried increasing the timeout?

[EnricoVerzegnassi]

@tsnoam Actually no, I'm sorry, but how can I set the timeout?
I just use the "bot" object from which I call the method "send_document".

[Eldinnie]

See the timeout parameter:
http://python-telegram-bot.readthedocs.io/en/stable/telegram.bot.html#telegram.Bot.send_document

[EnricoVerzegnassi]

@Eldinnie Solved! Thank you guys, and sorry for the stupid error.

[5p4k]

Hello, I am seeing a massive amount of timeout when sending photos or videos (fails in almost all cases). I see data being transferred, but if it doesn't get transferred within the amount of time specified in the timeout, it will fail nonetheless with a timeout error. To send some 500KB of photo, I had to set manually the timeout parameter to 20 (I know it was supposed to be default, but unless I do that, it fails).

Shouldn't the timeout trigger when the specified amount of time elapses without any data being transferred? It seems that timeout refers, in behaviour, to the maximum amount of time by which the transfer has to be completed. This doesn't make sense to me, as it would imply that the timeout scales up with the size of the media being transferred.

I could not understand exactly the gist of this issue from the description, but it seems very closely related – if that's not the case, I'd be happy to open a new issue.

[5p4k]

Hello, I am seeing massive timeouts when sending photos and videos above 1MB. I did not understand the gist of this issue from the description, but it seems related (if not, I'd be happy to open a new one). I have noticed two things

1. Although the default timeout is said to be 20 seconds, I actually had to specify it explicitly in send_video and send_photo for it to be taken in effect (otherwise it would time out in ~5 secs)
2. The timeout parameter, seem to refer to the overall time the transfer takes. Shouldn't it be the maximum time that is allowed to elapse without transferring any data?

More on 2.: I see data being transferred and maybe a few seconds more would have allowed the file to go through. Also, with the current behaviour, the timeout should be scaled up with the file size, and this does not make sense to me.

[tsnoam]

We've done what we can to fix the issue and it seems like it was (or at least significantly improved to the state that the community is happy).
Unfortunately we don't see how to implement unitests to recreate the timeout issue - we just fail to recreate this in "lab".

I'm closing this issue to remove "noise" from the issue lists.

[tsnoam]

@LizardM4
If you're not there already, you're welcome to join our community ask usage questions there. If eventually you find a bug with the library please open a new issue in github.