Connection problems

[micheleselea]

I'm experimenting a lot of connection issue since today 13:30. Basically tdlib client cannot connext to telegram server and the state change between connecting and connected and disconnected continuoilusly
Are there some telegram server issue? Or it's my connection problem?

[levlam]

There was no any issues in the last few hours..

[micheleselea]

So it's very strange because I'm receiving thousand of mwssages like

updateConnectionState {
state = connectionStateUpdating {
}
}

updateConnectionState {
state = connectionStateReady {
}
}

updateConnectionState {
state = connectionStateConnecting {
}
}

In last 2 hours, sometimes it's ok, and sometimes give me
UpdateMessageSendFail
With retry_after
I don't know what is going on but was all warking until 13:30

[micheleselea]

@levlam about retry_after it's a time you should wait before send any new message or a message to that id?
the strange thing is that I see even retry_after 2400 seconds....probably somthing is somehow heppening with telegeam because never seen that error

[micheleselea]

3 hours....same problems

[levlam]

@micheleselea This particular message. A failed to send message can be retried to send with resendMessages.

[levlam]

@micheleselea Send the full log with verbosity level 4 to @tdlib_bot, so I can tell you the cause.

[micheleselea]

I cannot now, I have to wayt monday I just hope the problem will be fixed before monday...

[micheleselea]

The problem is still there... I think that is the worst service issue since I use telegram for bot so since 2015. I think I have to find another sistem to make my notification more reliable...I'm realy sorry about because I like telegram and I like bot and tdlib but I cannot have all that problems...

[micheleselea]

@levlam what I can'rlt figure out is that: does tdlib can be use to develop a BOT that has thousand of users? Because ij 2/3 months of use I hit a lot of constraint and limitation that make me think the lib are only for "user like" use....it's a pity if so, because BOT are great but they do not suport secret chat, tdlib was to be the perfect solution but I thino cannot do what I have to do..I'm I wrong?

[levlam]

@micheleselea TDLib can handle millions of chats simultaneously, so thousands of users shouldn't be a problem for it. But usage by ordinary users is still the primary use case for TDLib.

[micheleselea]

Ok so, suppose your tdlib client have to send 1000messages every minutes to your thousand client, I mean 1000 secretchat messages...I think today I reach the limit because the server could bot satisfy request and drop my messages or disconnect the client. So do you think this is not a environment I can use? There is a way to tell telegram server thet the tdlib client is not a human but a bot? Or Have I to change notification system?
As I told you with bot api there is not such limitation, the problem is only the secret chat missing. @levlam what do you suggest in that situation?

[levlam]

I would suggest not to invent theories about limits reaching before TDLib logs show the real reason of reconnections.

[micheleselea]

I'm not "invent" theory I was just reporting you the facts. Yestetday I disabled using of tdlib for sending notification falling back on the standard BOT and all is working wonderfull, even the tdlib receiving messages are very fast. I think the problem is that the server consider my tdlib client hostile because I sent too much messages. That is the more logic conclusion. So I'll give another chance to tdlib reproducing yesterday behavior and give you the log. If, I hope not, the problem is that, I can't waste other time I have to move on some other system...and I'll be very sad because I like Telegram platform.
Anyway in the mean while che question I want to ask you as in my prev message: do you think you can build using tdlib a BOT clinet that can send thousands messages per minute to your thousands client? Is there the possibility that server consider you as spammer and block you as yesterday? Keep in mind we are always speeking about secretchat

[levlam]

"I think the problem is" is exactly inventing a theory without facts. You have zero evidence for now that server did any kind of blocking for you. And the behavior you described is definitely not the way in which spammers are blocked. The real reason of reconnections can be seen only in TDLib logs.

[micheleselea]

@levlam ok

[micheleselea]

@levlam can you answer my question please?

[levlam]

Do you think you can build using tdlib a BOT clinet that can send thousands messages per minute to your thousands client?

Telegram limits for ordinary users are much lower. Only bots are allowed to send so much messages.

Is there the possibility that server consider you as spammer and block you as yesterday?

The server fails particular requests with 429th error if flood limits are exceeded, but not interfere with user connections.

[micheleselea]

Ok I'll try to reproduce the problem and let you view logs. I hope we find the reason, but from your preview message I understand that probably I took the wrong way anyway

[micheleselea]

this morning still no problem with the same version of Saturday and quite the same Traffic, maybe more traffic due to the work activity

[micheleselea]

@levlam about the retry_after fields of updateMessageSendFailed.
do you think that could happen if my tdlib BOT send too much messages? the retry after is for any kind of new message to be sent or for a specific type of message or receipient?

[micheleselea]

in log I found a lot of lines like:

error_code = 429
error_message = "Too Many Requests: retry after 150"

or

error_code = 429
error_message = "Too Many Requests: retry after 3019"

429 error code you told me is the code of "flood limits"

[levlam]

@micheleselea 429 is returned exactly when flood limit is exceeded. The number in the error is a number of seconds to wait, before the message could be retried to be sent. The same number is returned in updateMessageSendFailed.retry_after. There are different kinds of flood limits, so it isn't possible to tell which one is exceeded.

[micheleselea]

my message was about updateMessageSendFailed.retry_after I usually do not resend message that fail to send because but I was asking if that retry_after is only about that message or is a time to wait before send next message in general

[levlam]

And I answered that there are different kinds of flood limits. The value of retry_after is related to a particular message. Usually other messages will also fail to send with the same retry_after time, but sometimes they will be succesfully sent.

[micheleselea]

ok understand, anyway the problems I had saturday generates a lot of that error logs. Today I did not yet see any of that logs. I'll keep you informed

[micheleselea]

12 hours without problems. I think saturday were something wrong with connection or telegram servere..

[micheleselea]

Ok after 2 days the system was working this morning I'm in the same saturday situation.
I have messages like:

updateMessageSendFailed Too Many Requests: retry after 599[code:429]
updateMessageSendFailed Too Many Requests: retry after 599[code:429]

updateConnectionState {
  state = connectionStateReady {
  }
}

updateConnectionState {
  state = connectionStateConnecting {
  }
}

updateHavePendingNotifications {
  have_delayed_notifications = false
  have_unreceived_notifications = true
}
updateConnectionState {
  state = connectionStateReady {
  }
}
updateConnectionState {
  state = connectionStateConnecting {
  }
}

updateHavePendingNotifications {
  have_delayed_notifications = false
  have_unreceived_notifications = true
}
updateConnectionState {
  state = connectionStateReady {
  }
}
updateConnectionState {
  state = connectionStateConnecting {
  }
}
updateHavePendingNotifications {
  have_delayed_notifications = false
  have_unreceived_notifications = true
}
updateConnectionState {
  state = connectionStateReady {
  }
}

updateConnectionState about 2000 messgages received in 30 minutes.
I'm trying collecting logs

[micheleselea]

@levlam Ok I have the log collected, tell me where you want me to send. I temporary have to disable using the client for sending because it does not work, and even new message notification are very slow I think it takes about 1 minute from the moment I send message and the momento tdlib receive it

[micheleselea]

@levlam I think it's all related to Error : 429 : Too Many Requests: retry after 139
you will see the log.
Do you know exactly what is the limit after I can start receiving 429 error? so I can do a check on my application to verify if I'm near to that limit?

[levlam]

@micheleselea Unfortunately, the load is so high during problems that 756MB of the log contains only last 184 seconds. So it can't be seen why the issue appeared, but at least it is clear how to make it self-repair after the issue happened.

[micheleselea]

@levlam but is it depend from 429 flood limit?

[micheleselea]

Anyway take in mind the log switch happened at 13:14 and the problem start at about 13:30 so the 774Mb of log file should be start before the event

[levlam]

The log autorotates every 1GB (as set by the app), so it starts only at 13:41:41 and ends at 13:44:45.

[micheleselea]

@levlam I see a log.txt.old when it rotate, can be usefull? I don't think I have that now, but for the next time

[levlam]

In this case log.txt.old contained 1GB of data, which also shouldn't be enough to see the beginning of the issue.

[micheleselea]

So what you think it's the max ammount of time I can wait before stop application and grab log?

[levlam]

You can extrapolate 756MB in 3 minutes.

[micheleselea]

Ok so next time it happens, if there's no fix yet, I'll try to stop it as soon as I can

[micheleselea]

@micheleselea Unfortunately, the load is so high during problems that 756MB of the log contains only last 184 seconds. So it can't be seen why the issue appeared, but at least it is clear how to make it self-repair after the issue happened.

@levlam do you think the "self-repair" will be a tdlib patch or a server side change?

[micheleselea]

@levlam I sent another 2 logs captured this morning with the problem.
Tell me if you find it useful or if I can sto capturing log

[micheleselea]

This morning I had problem since 11:08 till now....a lot of 429 errors and some connection/reconnection problem

[micheleselea]

@levlam I really need to have an idea, even an approximate idea, of how can I reach that 429 limit. Because today happens dozen times, and the 3 days before never happened.
I need to understand if I completely miss the target so I'm using something I can never figure out to run well or if I just to make it "better".
For example latest 24 hour I sent about 6270 to 440 SecretChat. Do you think that is too much for a client developed with TDLib or it's still in the "good range"? If not, consider my tdlib application has only one instance from one server running (so just one client with that api id and that phone number logged in) there's a way to instruct Telegram Server that I'm not a human? but I have only one tdlib instance running?

[levlam]

This is a secret chats, so nothing should depend on the server. The fix, if possible, should be in TDLib.

The last log catched the moment when the problem started, so there is no need in more logs.

[levlam]

I've checked the log and found that the issue was triggered by 136 messages sent in batches from 1591174611 to 1591174622. To avoid the issue you can avoid sending more than 100 messages in any 30 seconds.

[micheleselea]

I've checked the log and found that the issue was triggered by 136 messages sent in batches from 1591174611 to 1591174622. To avoid the issue you can avoid sending more than 100 messages in any 30 seconds.

Ok I'll try, at least now I have a way to make it work better, thanks

[micheleselea]

I put a 100 msg max in 30seconds, and I immediatly hit the condition :)
Let see what happen next few days, but I think probably I choose the wrong way,
Probably the best thing should be if BOTs support SecretChat, @levlam do you think one day there could be the possibility for a BOT to handle SecretChat? that probably should be the solution to all those problems

[micheleselea]

I put the 100 messages max in 30 seconds but the 429 error still happen sometime with a retry_after time very high for example 128 or 400 seconds...still no solution for SecretChat I think. should I abandon my tdlib BOT SecretChat project? probably....

[levlam]

@micheleselea You also need to consider, that messages with files aren't sent immediately. If the file is uploaded for 10 seconds, it is sent only after 10 seconds. I would also suggest to leave some gap for internal service messages and reduce the limit to 80 messages per 30 seconds.

[micheleselea]

ok I'll try

[micheleselea]

nope, after restarting with 80 messages in 30seconds limit, without hitting that limit yet I start receiving
updateMessageSendFailed Too Many Requests: retry after 561[code:429]
and

updateConnectionState {
  state = connectionStateConnecting {
  }
}
updateConnectionState {
  state = connectionStateReady {
  }
}

I don't understand but or I'm making a bad code, and I hope so, or something else is wrong

[levlam]

Have you reached the limit previously? You need to wait the specified time first. Also if there are pending unsent messages, they should be also counted.

[micheleselea]

consider everytime I restart the application che queue is deleted, so I start over, so should not be any message queued on startup

[levlam]

@micheleselea This is not true. TDLib continues to send messages after restart, nothing is lost. Moreover, secret chat can't work without that, otherwise the server would be able to tamper with messages and remove arbitrary one from the middle.

[micheleselea]

Understand. I'll try again next week with secretchat and 80messages limit because for now was not succesfull....I give one other shot next week than Inhavr to give up

[micheleselea]

All my try to limit message even 80 on 30seconds make me receive 429 and than connecting disconnectig. Ok fine, I have to try another way because this one is not the right one, it's a shame but I cannot make it work as I need. Thank you anyway, bye

[micheleselea]

I tried not to send more than 4 messages every 2 seconds....and I got 429 anyway....that way I can't use it, I need to move away from SecretChat structures. Bye @levlam

[levlam]

In secret chats message deletions are also counted for message sending and in fact they was the primary cause of the issue. The behavior was improved in the latest version from master, so it should work better now. The server-side flood control still remains the same and you are limited to ~90 message actions (including deletions) in any 30 seconds.

[micheleselea]

Understand