Download fails: Migrate error loop #198
Comments
|
Hi! Thank you for your report, we are investigating this problem. The The The Can you please answer following questions:
|
|
I've tried (roughtly) following code for couple of minutes: func run(ctx context.Context) error {
client := telegram.NewClient(telegram.TestAppID, telegram.TestAppHash, telegram.Options{})
d := downloader.NewDownloader()
raw := tg.NewClient(client)
sender := message.NewSender(raw)
return client.Run(ctx, func(ctx context.Context) error {
peer, err := sender.Resolve("https://t.me/ahangify").AsInputPeer(ctx)
if err != nil {
return xerrors.Errorf("resolve: %w", err)
}
return messages.NewQueryBuilder(raw).Search(peer).Video().
ForEach(ctx, func(ctx context.Context, elem messages.Elem) error {
msg, ok := elem.Msg.(*tg.Message)
if !ok {
return nil
}
media, ok := msg.Media.(*tg.MessageMediaDocument)
if !ok {
return xerrors.Errorf("unexpected type %T", msg.Media)
}
doc, ok := media.Document.AsNotEmpty()
if !ok {
return xerrors.Errorf("unexpected type %T", media.Document)
}
_, err := d.Download(raw, doc.AsInputDocumentFileLocation()).Stream(ctx, io.Discard)
return err
})
})
}This downloads all videos from @ahangify channel (just selected most popular Iranian channel for test). I've encountered no errors so far. My primary DC was 4, and channel DC was 2, so I've got lots of My primary hypothesis is that you are downloading files by references that are already expired. |
|
Thank for looking into it! I'm not so sure it's about expired hashes. I've run a test now for a few hours on some active channels to try to answer your questions. Also made some new discoveries. Generally see little difference between time till download from receiving and time to finished download between successful and unsuccessful attempts. Worth noting is that I'm not scraping specific channels but saving attachments to message that come as update streams. Otherwise what I'm doing looks similar to yours. Does retries help? E.g. if you are trying to download failed file again, does it eventually succeed? How long it takes between start of download to failure? How long it takes between getting file access hash to failure? What is your session main DC? It usually appear in log. Having run my test now for a few hours it appears to be getting worse, higher failure rate with |
|
I'll try doing same on active channels to reproduce this issue. My second hypothesis is that telegram doing some weird throttling, so you can try rate limiting your downloads. |
|
Can you please try |
|
Sorry for late reply, I've been trying to figure this out but not gotten really far. 27.1 helped in debugging however. Problem seems to be Here's a log that might help, I get a few messages indicating a reconnect / re-auth, and right after I get the first Download start / download failed are log messages i add to keep track of my downloads. �[36mtelegram_1 |�[0m 2021-03-14 18:44:11.899816 I | Successful document download on attempt: 1 |
|
Hi @patrikhermansson! Thank you for your report! I found some bugs related to session management, fixed and released v0.28.1. Can you try it please? |
|
Thanks! v0.28.1 solves the problem! I've been running it for a few hours now without a single |
|
Great! Closing this for now. |
Hi, sorry if this is not a clearly defined issue. I'm getting a very high number of errors on file downloads using v0.27.0 but this happened with previous versions as well.
Somewhere between 20-60% percent of file downloads fail (depending on time I try) which seems too high but it's notably not all of them, many succeeds as well and I can't differentiate between the ones who succeeds and fails. They usually fail with error:
get file: get next chunk: rpcDoRequest: rpc error code 400: FILE_ID_INVALIDwhich could be my fault but info logs looks strange. Once in a while they fail withget file: get next chunk: invoke pool: rpcDoRequest: rpc error code 401: AUTH_KEY_UNREGISTERED.Before the download errors with either of these messages the INFO log prints, the line below:
Got migrate error: Creating sub-connectionseveral hundred times before I usually get400: FILE_ID_INVALID, occaccionally I getFILE_ID_INVALIDright away. However, in some cases it also succeeds but the amount of migrate error seems strange?This is the line that gets printed hundreds of times, sometimes for up to to several minutes, before the download fails:
INFO telegram/invoke_raw.go:23 Got migrate error: Creating sub-connection {"v": "v0.27.0", "error": "FILE_MIGRATE", "dc": 5}INFO telegram/invoke_raw.go:23 Got migrate error: Creating sub-connection {"v": "v0.27.0", "error": "FILE_MIGRATE", "dc": 5}INFO telegram/invoke_raw.go:23 Got migrate error: Creating sub-connection {"v": "v0.27.0", "error": "FILE_MIGRATE", "dc": 5}I'm calling both
downloader.NewDownloader().DownloadDirect(c, loc).Stream(ctx, f)as well asdownloader.NewDownloader().Download(c, loc).Stream(ctx, f)so there doesn't appear to be a difference when I use CDN.Happy to try to provide more info if you tell me what could be useful.
The text was updated successfully, but these errors were encountered: