Failed to download files from a CDN

[mohammad2611373]

I had problem downloading big files. I opened an issue but you said there is no problem. After trying I faced same error. I copy and paste InteractiveTelegramClient into my code and just created an instance by my api_id and ... .
But when downloading big files it only downloads 1MB.

[Lonami]

Is there any code that always triggers the issue that I can use? Paste a log, maybe?

[mohammad2611373]

The code is exactly InteractiveTelegramClient That you wrote it yourself.

[Lonami]

Okay, any file I can try that always triggers this? I can send a big file to myself and try downloading it and it will fail?

[mohammad2611373]

for example I cannot download media of this channel : https://t.me/akhbarefori/110387

[mohammad2611373]

just call run() method of InteractiveTelegramClient instance and insert this channel and just type "!dm 110387".
It only downloads 1MB

[Lonami]

Enter a message: !dm 2864124
Downloading media to usermedia/...
Downloaded 128.00KB out of 3.10MB (4.03%)
Downloaded 256.00KB out of 3.10MB (8.05%)
-snip-
Downloaded 3.00MB out of 3.10MB (96.62%)
Downloaded 3.10MB out of 3.10MB (100.00%)
Media downloaded to usermedia/wm_5a7ed9bf31d38_330305.mp4!

It correctly downloaded the file for me (I forwarded it to my chat, and then downloaded it). Can you please import logging and then logging.basicConfig(level=logging.DEBUG)?

[mohammad2611373]

Downloading media to usermedia/...
INFO:telethon.telegram_client:Downloading file in chunks of 131072 bytes
DEBUG:telethon.telegram_bare_client:Invoking GetFileRequest
DEBUG:telethon.network.mtproto_sender:Processing MsgsAck result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Updates result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
INFO:telethon.telegram_client:File lives in a CDN
DEBUG:telethon.telegram_bare_client:Invoking GetConfigRequest
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.telegram_bare_client:Invoking GetCdnConfigRequest
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
INFO:telethon.telegram_bare_client:Creating new CDN client
INFO:telethon.telegram_bare_client:Connecting to 91.108.21.1:443...
INFO:telethon.telegram_bare_client:Connection success!
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
INFO:telethon.telegram_bare_client:Initializing a new connection while invoking
ERROR:telethon.telegram_bare_client:Authorization key seems broken and was invalid!
WARNING:telethon.telegram_bare_client:Invoking ['GetCdnFileRequest(file_token=b\'%\\x9a\\xe9\\x104 _\\xa8\\x90e\\xde\\xcay\\xd2\\x11\\x9e(\\x8a_\\xfd\\x86\\x831e\\xbc\\xd5\\t\\x83H}\\xe6:v<\\xeb\\x95\\xa6\\xd8\\xa3\\xfd)(\\x13x\\xfd#\\x85\\x88K\\xbf\\x81\\xac\\xd3"\\x9a:C)\\x875\\x0b\\xeb\\xb6H\\xe4_\\xe1\\r\\xf5\\x12\\xe14\\x12)\\xbbU\\x9e1\\xe44+^\\x18M\\x89\\xce\\x9aJ\\xc9\\xf4\\xe7\\xb3m\\xb4\\x04@\\xa3e\\x83\\x86\\x8ar\\xe4[\\xb4\\xf8\\xc6\\xf0\\xbf\\xf8\\x13U]\\x9f#\\xeaB\\xad;*DG\\xa7:\\x84\\x8a!\\x95\\x85\\x94Bc\\x9a\\xadJU\\x86\\xaaZ\\x1b\\x04\\x1a`}\\x95\\xf3\\xb3G\\x11\\xcc\\xd7\\xd0\\xe0\\xbc4BP$|-\\xb0\\xaag\\x0e\\x84\\xae\\xbe\\xa2\\xc9\\x0cQ\\xd6:Q(\\x8a\\x00\\xf7]\\x83\\xc7\\x0e\\x07\\xa7\\x08I\\x9e1\\xff\\x87\\x92O>}\\xdc\\x82k\\xbd\\xea\\xd2}\\xe6\\x03"h\\x8c\\xadK4\\xbb4\\x8f\\x85\\x0f\\xdb\\xe9,\\xe2W\\xad\\xa3\\x83s,\\x8fK\\xa3\\xee\\xbf\\x86{\\xf8\\xc7W\\x01NI\\x10\\xe9\\x9f\\x1cM\\x8a\\x81\\xcb\\xec\\x0e\\x16\\xbb\\xea4_\\x02\\xae\\xe2/\', limit=131072, offset=0)'] failed 1 times, reconnecting and retrying
INFO:telethon.telegram_bare_client:Reconnection aborted: already connected
INFO:telethon.telegram_bare_client:Need to generate new auth key before invoking
INFO:telethon.telegram_bare_client:Initializing a new connection while invoking
DEBUG:telethon.network.mtproto_sender:Processing BadServerSalt result
DEBUG:telethon.network.mtproto_sender:Processing container result
DEBUG:telethon.network.mtproto_sender:Processing NewSessionCreated result
DEBUG:telethon.network.mtproto_sender:Processing MsgsAck result
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 128.00KB out of 5.68MB (2.20%)
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 256.00KB out of 5.68MB (4.40%)
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 384.00KB out of 5.68MB (6.61%)
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 512.00KB out of 5.68MB (8.81%)
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 640.00KB out of 5.68MB (11.01%)
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 768.00KB out of 5.68MB (13.21%)
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
DEBUG:telethon.telegram_bare_client:Receiving items from the network timed out
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Pong result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Updates result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Updates result
DEBUG:telethon.telegram_bare_client:Receiving items from the network...
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 896.00KB out of 5.68MB (15.41%)
DEBUG:telethon.telegram_bare_client:Invoking GetCdnFileRequest
DEBUG:telethon.network.mtproto_sender:Processing Remote Procedure Call result
DEBUG:telethon.telegram_client:Saved 131072 more bytes
Downloaded 1.00MB out of 5.68MB (17.61%)
INFO:telethon.telegram_bare_client:Disconnecting...
DEBUG:telethon.telegram_bare_client:Stopping all workers...
DEBUG:telethon.telegram_bare_client:Disconnecting the socket...
Media downloaded to usermedia/ونگر.mp4!

[mohammad2611373]

As you see after downloading only 1MB, downloading finishes

[Lonami]

Right, you live close to some CDNs and Telegram decided to use those to download the file. So what fails is downloading files from a CDN.

[mohammad2611373]

Is my country a concern? I'm from Iran

[Lonami]

Yeah using Telegram from places like Iran cause some big channels to have the media stored in Content Distributed Networks (CDNs) which Telegram uses for increased speed and to avoid censorship I guess. So the library is failing to fetch the file from there.

[mohammad2611373]

So what about using a proxy?

[Lonami]

Or you could wait until I fix this. But sure, a proxy may work.

[mohammad2611373]

Thanks a lot

[iranpak]

I have same issue and i know that using a proxy works. but when i use proxy the rest of my script (that is uploading the downloaded media on my server) will be very slow so i don't want to use a proxy. so my question is that if my country (IRAN) is concern, why and how the official telegram clients can download these medias without using any proxy by me?

[Lonami]

Because the library (and not the official clients) seems to have a bug regarding CDN files. I don't live anywhere close to a CDN so sadly I can't seem to find a way to test those, I'd need a proxy.

[iranpak]

So don't you have any idea except using proxy?
another question is that does telegram give you (your library) a direct link for downloading files?
if yes can we access that direct link? and if no can we generate that somehow?

[Lonami]

No it's not a link (unless a link to the message of a public channel shows the media in your browser, which I'm not sure but might). They give you an ID/hash like combination to download the file.

[Lonami]

Does anyone know of a public proxy I can use that triggers CDN downloads?

[Lonami]

Until I can get a proxy and a link a message of a public channel with some media that triggers the issue, I can't do anything.

[zackpollard]

I'm having the same issue right now, I can help you debug if needed, feel free to message me here or on telegram on https://telegram.me/zackpollard. I've been triggering this issue by uploading files to a supergroup and then trying to download them immediately via the telethon library. It gets to byte 1032192 (just below 1MB) and then fails to download that chunk with the following error
telethon.errors.rpcerrorlist.LimitInvalidError: An invalid limit was provided. See https://core.telegram.org/api/files#downloading-files
I'm not using a proxy and I live in the UK. Also as I said i'm uploading to a supergroup but it's only got me and one other person in it, and the files were uploaded within minutes of me testing the download.
Any help would be appreciated, cheers!

[Lonami]

@zackpollard LimitInvalidError is definitely not the error you should be getting if you're triggering a CDN download with the latest version since they are literally "unimplemented" (it should raise an error like such).

[zackpollard]

@Lonami ah I see, hmmm, is there a way I can contact you privately to discuss this separate issue then as the way we're using the library is a bit different which may be causing the issues (we've modified downloads.py slightly) as we need to download only certain chunks of a file at a time.

[Lonami]

You can contact me on Telegram as @Lonami but make sure to tell me your GitHub username so I know who you are.

[Lonami]

In order to fix this one has to find a popular-enough channel with media from certain countries (or using a proxy). If the download fails, most likely it's because it's a CDN download. In that case, one will need to implement the code to download files from CDNs with raw API. If anyone is interested on working in this and has a file it can be tested on, leave a comment and I'll provide further instructions.

[Lonami]

Layer 109 added cdn_supported to getFile. This flag defaults to False, which is what has been since that layer was added which means the download will be performed via the main servers. Although slower, it works, so there's no longer an issue to fix here, just an enhancement I'd rather not track unless there's pressing interest.