[BUG] Exception while shutdown #3893

et-ness · 2023-09-16T16:16:51Z

Steps to Reproduce

run an example
stop with CTRL+C
randomly the shutdown got an exception

Expected behaviour

Close the bot without exceptions

Actual behaviour

These exceptions is very similar to issue #1563 if I run the same code on python3.9 I got the exception described in #1563 while if I run on python3.11 sometime I got these exceptions:

^CTraceback (most recent call last):
  File "/usr/lib/python3/dist-packages/anyio/streams/tls.py", line 130, in _call_sslobject_method
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 921, in read
    v = self._sslobj.read(len)
        ^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2546)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/httpcore/backends/asyncio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/anyio/streams/tls.py", line 195, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method
    data = await self.transport_stream.receive()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/anyio/_backends/_asyncio.py", line 1265, in receive
    await self._protocol.read_event.wait()
  File "/usr/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/lib/python3/dist-packages/httpcore/backends/asyncio.py", line 32, in read
    with anyio.fail_after(timeout):
  File "/usr/lib/python3/dist-packages/anyio/_core/_tasks.py", line 118, in __exit__
    raise TimeoutError
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/lib/python3/dist-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpcore/_async/http11.py", line 112, in handle_async_request
    raise exc
  File "/usr/lib/python3/dist-packages/httpcore/_async/http11.py", line 91, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpcore/_async/http11.py", line 155, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpcore/_async/http11.py", line 191, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpcore/backends/asyncio.py", line 31, in read
    with map_exceptions(exc_map):
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/lib/python3/dist-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/request/_httpxrequest.py", line 219, in do_request
    res = await self._client.request(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/lib/python3/dist-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/john/test/./bot.py", line 98, in <module>
    main()
  File "/home/john/test/./bot.py", line 95, in main
    app.run_polling()
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_application.py", line 765, in run_polling
    return self.__run(
           ^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_application.py", line 963, in __run
    loop.run_until_complete(self.updater.stop())  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_updater.py", line 752, in stop
    await self._stop_polling()
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_updater.py", line 777, in _stop_polling
    await self.__polling_cleanup_cb()
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_updater.py", line 381, in _get_updates_cleanup
    await self.bot.get_updates(
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_extbot.py", line 553, in get_updates
    updates = await super().get_updates(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/_bot.py", line 394, in decorator
    result = await func(self, *args, **kwargs)  # skipcq: PYL-E1102
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/_bot.py", line 3542, in get_updates
    await self._post(
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/_bot.py", line 482, in _post
    return await self._do_post(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_extbot.py", line 335, in _do_post
    return await super()._do_post(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/_bot.py", line 510, in _do_post
    return await request.post(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/request/_baserequest.py", line 168, in post
    result = await self._request_wrapper(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/request/_baserequest.py", line 288, in _request_wrapper
    raise exc
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/request/_baserequest.py", line 278, in _request_wrapper
    code, payload = await self.do_request(
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/request/_httpxrequest.py", line 236, in do_request
    raise TimedOut from err
telegram.error.TimedOut: Timed out
Exception ignored in: <coroutine object Application._update_fetcher at 0x7f43eb05c400>
Traceback (most recent call last):
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_application.py", line 1101, in _update_fetcher
  File "/usr/lib/python3.11/asyncio/queues.py", line 160, in get
  File "/usr/lib/python3.11/asyncio/base_events.py", line 761, in call_soon
  File "/usr/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

I think depends when I start the shutdown. Something is wrong in shutdown routine.

Operating System

Linux

Version of Python, python-telegram-bot & dependencies

python-telegram-bot 20.5
Bot API 6.8
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]

Relevant log output

File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/request/_httpxrequest.py", line 236, in do_request
    raise TimedOut from err
telegram.error.TimedOut: Timed out
Exception ignored in: <coroutine object Application._update_fetcher at 0x7f43eb05c400>
Traceback (most recent call last):
  File "/home/john/test/.python3/local/lib/python3.11/dist-packages/telegram/ext/_application.py", line 1101, in _update_fetcher

Additional Context

Issue there is only on shutdown, is not a big problem, but is bad to see all those exceptions.

Bibo-Joshi · 2023-09-22T16:45:00Z

Hi. #1563 is not related - that issue predates v20. What I see in your traceback is an ordinary timeout exception in the final call to get_updates which marks the processed updates as read. Please have a look at this wiki page.

et-ness · 2023-09-22T20:26:32Z

Hi. The issue is very random. If I start the bot and after max 10 seconds I stop it sometime I got the exception and sometime no.
It happens very often when I stop it with CTRL+C (SIGINT) and more rarely with SIGTERM.
I ran the bot for 24 hours and no timeout exceptions, I got the exception ONLY when I stop it.
I tried to increase the timeouts but I got always the same exceptions randmonly.
I tried to run the bot on aws server (python 3.9) and local (python 3.11) always random exception on shutdown but only on shutdown never when the bot is working normally.
I will try to create aws server on the same zone of api telegram.

Bibo-Joshi · 2023-09-23T17:20:26Z

Updater.stop calls get_updates one more time so that the offset parameter is passed and TG knows which updates where already processed. The timeout exception above happens during this call. The timeout values used for this call are the same value as for the get_updates calls during runtime. Why the timeout happens more frequently during shutdown than during runtime is not clear to me.
Please note that there is both ApplicationBuilder.read_timeout and ApplicationBuilder.get_updates_read_timeout and similar for the other timeouts.

et-ness · 2023-09-24T09:01:15Z

This is the test code:

#!/usr/bin/env python3

from telegram import Update, ForceReply
from telegram.ext import ApplicationBuilder, CommandHandler, ContextTypes, filters, MessageHandler


async def hello(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    await update.message.reply_text(f'Hello {update.effective_user.first_name}')

async def start(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    """Send a message when the command /start is issued."""
    user = update.effective_user
    await update.message.reply_html(
        rf"Hi {user.mention_html()}!",
        reply_markup=ForceReply(selective=True),
    )

async def help_command(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    """Send a message when the command /help is issued."""
    await update.message.reply_text("Help!")

async def echo(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    """Echo the user message."""
    await update.message.reply_text(update.message.text)

app = ApplicationBuilder() \
         .token("XXXXXXXXXXXXXXXXXXX") \
         .read_timeout(60) \
         .get_updates_read_timeout(60) \
         .build()

app.add_handler(CommandHandler("start", start))
app.add_handler(CommandHandler("hello", hello))
app.add_handler(CommandHandler("help", help_command))

app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, echo))

app.run_polling()

60 seconds timeout, some tests with time, just to understand that the timeout is never reached.
I think the timeout exception is the consequence of stopping the coroutine.
I'm not expert of asyncio and coroutine, but I see asyncio.exceptions.CancelledError, so I think if the coroutine is stopped while is reading we have an I/O exception which is masked with a timeout exception.

In my tests you can see also the difference between SIGTERM and SIGINT. Nothing should change, because they are simple signals all managed in the same way but you can see some differences

ec2-user@server 08:36:14 ~/test$
ec2-user@server 08:36:14 ~/test$
ec2-user@server 08:36:14 ~/test$
ec2-user@server 08:36:14 ~/test$
ec2-user@server 08:36:14 ~/test$
ec2-user@server 08:36:14 ~/test$./bot2.py 
^CTraceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 133, in _call_sslobject_method
    result = func(*args)
  File "/usr/lib64/python3.9/ssl.py", line 889, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 115, in fail_after
    yield cancel_scope
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 198, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1103, in receive
    await self._protocol.read_event.wait()
  File "/usr/lib64/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f26f0563370

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in fail_after
    raise TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 219, in do_request
    res = await self._client.request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/./bot2.py", line 38, in <module>
    app.run_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 765, in run_polling
    return self.__run(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 963, in __run
    loop.run_until_complete(self.updater.stop())  # type: ignore[union-attr]
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 752, in stop
    await self._stop_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 777, in _stop_polling
    await self.__polling_cleanup_cb()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 381, in _get_updates_cleanup
    await self.bot.get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 553, in get_updates
    updates = await super().get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 394, in decorator
    result = await func(self, *args, **kwargs)  # skipcq: PYL-E1102
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 3542, in get_updates
    await self._post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 482, in _post
    return await self._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 335, in _do_post
    return await super()._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 510, in _do_post
    return await request.post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 168, in post
    result = await self._request_wrapper(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 288, in _request_wrapper
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 278, in _request_wrapper
    code, payload = await self.do_request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 236, in do_request
    raise TimedOut from err
telegram.error.TimedOut: Timed out
Task was destroyed but it is pending!
task: <Task pending name='Application:6638977760:update_fetcher' coro=<Application._update_fetcher() running at /home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py:1101> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f26f0552280>()]>>
Exception ignored in: <coroutine object Application._update_fetcher at 0x7f26f0f8c240>
Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 1101, in _update_fetcher
  File "/usr/lib64/python3.9/asyncio/queues.py", line 168, in get
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 751, in call_soon
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
ec2-user@server 08:36:33 ~/test$./bot2.py 
^CTraceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 133, in _call_sslobject_method
    result = func(*args)
  File "/usr/lib64/python3.9/ssl.py", line 889, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 115, in fail_after
    yield cancel_scope
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 198, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1103, in receive
    await self._protocol.read_event.wait()
  File "/usr/lib64/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f898eb840d0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in fail_after
    raise TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 219, in do_request
    res = await self._client.request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/./bot2.py", line 38, in <module>
    app.run_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 765, in run_polling
    return self.__run(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 963, in __run
    loop.run_until_complete(self.updater.stop())  # type: ignore[union-attr]
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 752, in stop
    await self._stop_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 777, in _stop_polling
    await self.__polling_cleanup_cb()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 381, in _get_updates_cleanup
    await self.bot.get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 553, in get_updates
    updates = await super().get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 394, in decorator
    result = await func(self, *args, **kwargs)  # skipcq: PYL-E1102
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 3542, in get_updates
    await self._post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 482, in _post
    return await self._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 335, in _do_post
    return await super()._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 510, in _do_post
    return await request.post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 168, in post
    result = await self._request_wrapper(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 288, in _request_wrapper
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 278, in _request_wrapper
    code, payload = await self.do_request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 236, in do_request
    raise TimedOut from err
telegram.error.TimedOut: Timed out
Task was destroyed but it is pending!
task: <Task pending name='Application:6638977760:update_fetcher' coro=<Application._update_fetcher() running at /home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py:1101> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f898ebf0280>()]>>
Exception ignored in: <coroutine object Application._update_fetcher at 0x7f898f62a240>
Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 1101, in _update_fetcher
  File "/usr/lib64/python3.9/asyncio/queues.py", line 168, in get
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 751, in call_soon
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
ec2-user@server 08:37:02 ~/test$

---------------------------------------------------------------------------------------------------------

ec2-user@server 08:38:18 ~/test$
ec2-user@server 08:38:18 ~/test$
ec2-user@server 08:38:19 ~/test$
ec2-user@server 08:38:19 ~/test$
ec2-user@server 08:38:19 ~/test$./bot2.py &
[1] 80034
ec2-user@server 08:38:23 ~/test$kill 80034
ec2-user@server 08:38:31 ~/test$
[1]+  Done                    ./bot2.py
ec2-user@server 08:38:32 ~/test$
ec2-user@server 08:38:33 ~/test$./bot2.py &
[1] 80036
ec2-user@server 08:38:35 ~/test$
ec2-user@server 08:38:37 ~/test$
ec2-user@server 08:38:37 ~/test$
ec2-user@server 08:38:38 ~/test$
ec2-user@server 08:38:38 ~/test$kill 80036
ec2-user@server 08:38:44 ~/test$
[1]+  Done                    ./bot2.py
ec2-user@server 08:38:44 ~/test$
ec2-user@server 08:38:45 ~/test$
ec2-user@server 08:38:45 ~/test$./bot2.py &
[1] 80093
ec2-user@server 08:38:48 ~/test$
ec2-user@server 08:38:48 ~/test$
ec2-user@server 08:38:49 ~/test$kill 80093
ec2-user@server 08:39:00 ~/test$
ec2-user@server 08:39:01 ~/test$
ec2-user@server 08:39:01 ~/test$
ec2-user@server 08:39:02 ~/test$Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 133, in _call_sslobject_method
    result = func(*args)
  File "/usr/lib64/python3.9/ssl.py", line 889, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 115, in fail_after
    yield cancel_scope
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 198, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1103, in receive
    await self._protocol.read_event.wait()
  File "/usr/lib64/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f959fc23370

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in fail_after
    raise TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 219, in do_request
    res = await self._client.request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/./bot2.py", line 38, in <module>
    app.run_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 765, in run_polling
    return self.__run(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 963, in __run
    loop.run_until_complete(self.updater.stop())  # type: ignore[union-attr]
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 752, in stop
    await self._stop_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 777, in _stop_polling
    await self.__polling_cleanup_cb()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 381, in _get_updates_cleanup
    await self.bot.get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 553, in get_updates
    updates = await super().get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 394, in decorator
    result = await func(self, *args, **kwargs)  # skipcq: PYL-E1102
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 3542, in get_updates
    await self._post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 482, in _post
    return await self._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 335, in _do_post
    return await super()._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 510, in _do_post
    return await request.post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 168, in post
    result = await self._request_wrapper(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 288, in _request_wrapper
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 278, in _request_wrapper
    code, payload = await self.do_request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 236, in do_request
    raise TimedOut from err
telegram.error.TimedOut: Timed out
Task was destroyed but it is pending!
task: <Task pending name='Application:6638977760:update_fetcher' coro=<Application._update_fetcher() running at /home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py:1101> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f959fc12520>()]>>
Exception ignored in: <coroutine object Application._update_fetcher at 0x7f95a064c240>
Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 1101, in _update_fetcher
  File "/usr/lib64/python3.9/asyncio/queues.py", line 168, in get
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 751, in call_soon
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed

[1]+  Exit 1                  ./bot2.py
ec2-user@server 08:39:03 ~/test$
ec2-user@server 08:39:06 ~/test$

----------------------------------------------------------------------------------------------------------------------------------

ec2-user@server 08:40:00 ~/test$
ec2-user@server 08:40:00 ~/test$
ec2-user@server 08:40:00 ~/test$
ec2-user@server 08:40:01 ~/test$./bot2.py &
[1] 80098
ec2-user@server 08:40:02 ~/test$
ec2-user@server 08:40:03 ~/test$
ec2-user@server 08:40:03 ~/test$
ec2-user@server 08:40:03 ~/test$
ec2-user@server 08:40:03 ~/test$fg
./bot2.py
^Cec2-user@server 08:40:06 ~/test$
ec2-user@server 08:40:07 ~/test$
ec2-user@server 08:40:07 ~/test$./bot2.py &
[1] 80100
ec2-user@server 08:40:09 ~/test$fg
./bot2.py
^CTraceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 133, in _call_sslobject_method
    result = func(*args)
  File "/usr/lib64/python3.9/ssl.py", line 889, in read
    v = self._sslobj.read(len)
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2633)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 115, in fail_after
    yield cancel_scope
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 198, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
    data = await self.transport_stream.receive()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1103, in receive
    await self._protocol.read_event.wait()
  File "/usr/lib64/python3.9/asyncio/locks.py", line 226, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f6253d94340

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in fail_after
    raise TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 219, in do_request
    res = await self._client.request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/test/./bot2.py", line 38, in <module>
    app.run_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 765, in run_polling
    return self.__run(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 963, in __run
    loop.run_until_complete(self.updater.stop())  # type: ignore[union-attr]
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 752, in stop
    await self._stop_polling()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 777, in _stop_polling
    await self.__polling_cleanup_cb()
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_updater.py", line 381, in _get_updates_cleanup
    await self.bot.get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 553, in get_updates
    updates = await super().get_updates(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 394, in decorator
    result = await func(self, *args, **kwargs)  # skipcq: PYL-E1102
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 3542, in get_updates
    await self._post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 482, in _post
    return await self._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_extbot.py", line 335, in _do_post
    return await super()._do_post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/_bot.py", line 510, in _do_post
    return await request.post(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 168, in post
    result = await self._request_wrapper(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 288, in _request_wrapper
    raise exc
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_baserequest.py", line 278, in _request_wrapper
    code, payload = await self.do_request(
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/request/_httpxrequest.py", line 236, in do_request
    raise TimedOut from err
telegram.error.TimedOut: Timed out
Task was destroyed but it is pending!
task: <Task pending name='Application:6638977760:update_fetcher' coro=<Application._update_fetcher() running at /home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py:1101> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f6253d8b520>()]>>
Exception ignored in: <coroutine object Application._update_fetcher at 0x7f62547c5240>
Traceback (most recent call last):
  File "/home/ec2-user/test/.python3/lib/python3.9/site-packages/telegram/ext/_application.py", line 1101, in _update_fetcher
  File "/usr/lib64/python3.9/asyncio/queues.py", line 168, in get
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 751, in call_soon
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed
ec2-user@server 08:40:14 ~/test$
ec2-user@server 08:40:16 ~/test$
ec2-user@server 08:40:16 ~/test$

----------------------------------------------------------------------------

ec2-user@server 08:42:39 ~/test$ 
ec2-user@server 08:42:39 ~/test$ 
ec2-user@server 08:42:39 ~/test$ 
ec2-user@server 08:42:39 ~/test$ 
ec2-user@server 08:42:39 ~/test$ kill 80162
ec2-user@server 08:42:44 ~/test$ 
[1]+  Done                    ./bot2.py
ec2-user@server 08:42:44 ~/test$ 
ec2-user@server 08:42:44 ~/test$ 
ec2-user@server 08:42:45 ~/test$ 
ec2-user@server 08:42:45 ~/test$ ./bot2.py &
[1] 80219
ec2-user@server 08:42:47 ~/test$ 
ec2-user@server 08:42:48 ~/test$ 
ec2-user@server 08:42:48 ~/test$ 
ec2-user@server 08:42:48 ~/test$ kill 80219
ec2-user@server 08:42:52 ~/test$ 
[1]+  Done                    ./bot2.py
ec2-user@server 08:42:52 ~/test$ 
ec2-user@server 08:42:53 ~/test$ 
ec2-user@server 08:42:53 ~/test$ 
ec2-user@server 08:42:53 ~/test$ 
ec2-user@server 08:42:54 ~/test$ ./bot2.py &
[1] 80221
ec2-user@server 08:42:55 ~/test$ 
ec2-user@server 08:42:56 ~/test$ 
ec2-user@server 08:42:56 ~/test$ 
ec2-user@server 08:42:56 ~/test$ kill 80221
ec2-user@server 08:43:01 ~/test$ 
[1]+  Done                    ./bot2.py
ec2-user@server 08:43:01 ~/test$ 
ec2-user@server 08:43:01 ~/test$ 

----------------------------------------------------------------------------

Why the timeout happens more frequently during shutdown than during runtime is not clear to me.

I never saw a timeout while the bot ran for 24 hours.
While the shutodown too many.

zoedberg · 2023-09-25T14:56:11Z

I've also encountered this issue. It doesn't seem related to user-configurable timeouts because I've set all possible timeouts like this

        .get_updates_connect_timeout(30)
        .get_updates_pool_timeout(30)
        .get_updates_read_timeout(30)
        .get_updates_write_timeout(30)
        .connect_timeout(30)
        .pool_timeout(30)
        .read_timeout(30)
        .write_timeout(30)

and I'm still receiving the error around 3 seconds after pressing ctrl+c.

This error seems new, I'm not encountering it with version 20.4

Bibo-Joshi · 2023-09-25T16:08:29Z

I dug around in the code a bit and I found the issue:

Application.run_polling and Updater.start_polling both accept keyword arguments for the different timeouts, which override the values set via the builder pattern
In both methods mentioned above and an (ExtBot).get_updates, the default value for read_timeout is currently 2, overriding the value set via the builder pattern. The default 2 is in fact documented, but the overriding behavior is still not inuitive. Note that for the other timeout parameters, everything should work as expected, i.e. the values set by the builder pattern are respected.

This usually doesn't show at runtime, because due to long-polling the actual read-timeout is way longer.
A quick workaround is to pass the higher read-timeout value to Application.run_polling (or Updater.start_polling).

However, I do see a need to change interfaces here.

Since we set values for get_updates timeouts via the builder pattern (or if builder pattern is not used, by passing a custom request object to parameter get_updates_request of Bot). So there is probably no actual added value in having the parameters available again in run/start_polling. I vote to deprecate them.
The default behavior of the parameter read_timeout of get_updates should be to respect the specified default value of the get_updates_request object of the Bot. For now, only a deprecation warning should be issued as this would change documented behavior.

@Poolitzer @harshil21 would you agree?

et-ness · 2023-09-26T11:28:07Z

A quick workaround is to pass the higher read-timeout value to Application.run_polling (or Updater.start_polling).

Your workaround

app.run_polling(read_timeout=30)

resolve my issue.

Poolitzer · 2023-09-26T11:38:06Z

Not sure I would like to deprecate the Bot arguments. Setting a higher timeout is one of the most changed settings, forcing people to either use application or pass their own request object just to change them isn't end user friendly imo

Bibo-Joshi · 2023-09-26T13:49:53Z

Not sure I would like to deprecate the Bot arguments. Setting a higher timeout is one of the most changed settings, forcing people to either use application or pass their own request object just to change them isn't end user friendly imo

There is a misunderstanding here: I don't want to remove any arguments from Bot.get_updates, but only change the default behavior of the read_timeout argument. A "deprecation" warning would just say "default behavior will change to xy in the future".
Removing arguments is only proposed for Application.run_polling and Updater.start_polling

Actually, I had another thought on making the default behavior match the default read timeout of the get_updates_request: that is likely harder to realize than I first thought, because the timeout parameter defined by TG has to be added to thr value of read_timout. However, we do not have the get_updates_request's default value for read_timeout available within the Bot class. Possible options that come to my mind:

Move the adding of values into (Base/HTTPX)Request. BaseRequest would be better because custom user implementations don't have to adapt. In any cases this would mean a breach of separation of concerns, i.e. the requests backend should not include special handling for different TG endpoints
Add a property/multiple properties to BaseRequest that expose the default value of the different timeout parameters. These would have to be abstract properties for custom implementations to give them proper meaning, i.e. ultimately this would have to be a breaking change. But that way, we could access the value within the Bot class and compute the sum

python-telegram-bot/python-telegram-bot#3893

Poolitzer · 2023-10-20T14:17:37Z

okay yes, I can see the argument removing, makes sense.

I would probably go for decoupling with properties.

et-ness added the bug 🐛 label Sep 16, 2023

Harmon758 added a commit to Harmon758/Harmonbot that referenced this issue Oct 8, 2023

[Telegram] Set read_timeout for Application.run_polling

e34a841

python-telegram-bot/python-telegram-bot#3893

harshil21 mentioned this issue Oct 20, 2023

Review types of optional arguments in methods and classes #3864

Closed

This was referenced Nov 3, 2023

handle _get_updates_cleanup() timeout error #3959

Closed

Adjust read_timeout behavior for get_updates #3963

Merged

Bibo-Joshi mentioned this issue Nov 22, 2023

[BUG] ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2580) #3980

Closed

Bibo-Joshi closed this as completed in #3963 Nov 27, 2023

github-actions bot locked and limited conversation to collaborators Dec 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Exception while shutdown #3893

[BUG] Exception while shutdown #3893

et-ness commented Sep 16, 2023 •

edited by Bibo-Joshi

Bibo-Joshi commented Sep 22, 2023

et-ness commented Sep 22, 2023

Bibo-Joshi commented Sep 23, 2023

et-ness commented Sep 24, 2023 •

edited by Bibo-Joshi

zoedberg commented Sep 25, 2023 •

edited

Bibo-Joshi commented Sep 25, 2023

et-ness commented Sep 26, 2023 •

edited

Poolitzer commented Sep 26, 2023

Bibo-Joshi commented Sep 26, 2023 •

edited

Poolitzer commented Oct 20, 2023

[BUG] Exception while shutdown #3893

[BUG] Exception while shutdown #3893

Comments

et-ness commented Sep 16, 2023 • edited by Bibo-Joshi

Steps to Reproduce

Expected behaviour

Actual behaviour

Operating System

Version of Python, python-telegram-bot & dependencies

Relevant log output

Additional Context

Bibo-Joshi commented Sep 22, 2023

et-ness commented Sep 22, 2023

Bibo-Joshi commented Sep 23, 2023

et-ness commented Sep 24, 2023 • edited by Bibo-Joshi

zoedberg commented Sep 25, 2023 • edited

Bibo-Joshi commented Sep 25, 2023

et-ness commented Sep 26, 2023 • edited

Poolitzer commented Sep 26, 2023

Bibo-Joshi commented Sep 26, 2023 • edited

Poolitzer commented Oct 20, 2023

et-ness commented Sep 16, 2023 •

edited by Bibo-Joshi

et-ness commented Sep 24, 2023 •

edited by Bibo-Joshi

zoedberg commented Sep 25, 2023 •

edited

et-ness commented Sep 26, 2023 •

edited

Bibo-Joshi commented Sep 26, 2023 •

edited