Eventual exception when updating `Log` from from a threader worker #3537

davep · 2023-10-15T09:56:43Z

Coming from a question asked on Discord, it's possible to get an exception when loading up a Log with the content of a large file, when doing so from a threaded worker.

Given this code:

from pathlib import Path

from textual import work
from textual.app import App, ComposeResult
from textual.screen import Screen
from textual.widgets import Header, Log, Footer

class ShowFile(Screen):

    def __init__(self, path: Path, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.path = path

    def on_mount(self) -> None:
        self.sub_title = f"Showing: {self.path.name}"
        self.read_file(self.path)

    def compose(self) -> ComposeResult:
        yield Header(show_clock=True)
        yield Log(classes="status-richlog")
        yield Footer()

    @work(thread=True)
    def read_file(self, path: Path) -> None:
        log = self.query_one(Log)
        with open(path, "r") as f:
            for line in f:
                self.app.call_from_thread(log.write, line)

class LoadUpFileApp(App[None]):

    def on_mount(self) -> None:
        self.push_screen(ShowFile(Path("test.txt")))

if __name__ == "__main__":
    LoadUpFileApp().run()

if I create test.txt as a text file around 2.6M in size, when loading up the file I eventually get this chain of exceptions:

Traceback (most recent call last):
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/message_pump.py", line 536, in _process_messages_loop
    await self._dispatch_message(message)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/message_pump.py", line 597, in _dispatch_message
    await self.on_event(message)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2606, in on_event
    self.screen._forward_event(event)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/screen.py", line 952, in _forward_event
    event.style = self.get_style_at(event.screen_x, event.screen_y)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/screen.py", line 313, in get_style_at
    return self._compositor.get_style_at(x, y)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/_compositor.py", line 822, in get_style_at
    lines = widget.render_lines(Region(0, y, region.width, 1))
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/widget.py", line 2993, in render_lines
    strips = self._styles_cache.render_widget(self, crop)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/_styles_cache.py", line 115, in render_widget
    strips = self.render(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/_styles_cache.py", line 214, in render
    strip = render_line(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/_styles_cache.py", line 409, in render_line
    line = render_content_line(y - gutter.top)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/widgets/_log.py", line 254, in render_line
    strip = self._render_line(scroll_y + y, scroll_x, self.size.width)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/widgets/_log.py", line 272, in _render_line
    line = self._render_line_strip(y, rich_style)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/widgets/_log.py", line 297, in _render_line_strip
    self._render_line_cache[y] = line
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/_cache.py", line 122, in set
    del self._cache[last[2]]  # type: ignore[index]
KeyError: 7652

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2225, in _process_messages
    await run_process_messages()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2192, in run_process_messages
    await self._process_messages_loop()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/message_pump.py", line 541, in _process_messages_loop
    self.app._handle_exception(error)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2064, in _handle_exception
    self._fatal_error()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2069, in _fatal_error
    traceback = Traceback(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 264, in __init__
    trace = self.extract(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 449, in extract
    locals={
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 450, in <dictcomp>
    key: pretty.traverse(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 853, in traverse
    node = _traverse(_object, root=True)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  [Previous line repeated 966 more times]
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 679, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 614, in _traverse
    def iter_rich_args(rich_args: Any) -> Iterable[Union[Any, Tuple[str, Any]]]:
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 309, in inner
    return cached(*args, **kwds)
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 1248, in __hash__
    return hash(frozenset(self.__args__))
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 1038, in __hash__
    return hash((self.__origin__, self.__args__))
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 1351, in run_async
    await app._process_messages(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2230, in _process_messages
    self._handle_exception(error)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2064, in _handle_exception
    self._fatal_error()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2069, in _fatal_error
    traceback = Traceback(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 264, in __init__
    trace = self.extract(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 449, in extract
    locals={
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 450, in <dictcomp>
    key: pretty.traverse(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 853, in traverse
    node = _traverse(_object, root=True)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  [Previous line repeated 968 more times]
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 679, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 614, in _traverse
    def iter_rich_args(rich_args: Any) -> Iterable[Union[Any, Tuple[str, Any]]]:
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 309, in inner
    return cached(*args, **kwds)
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 1248, in __hash__
    return hash(frozenset(self.__args__))
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 1038, in __hash__
    return hash((self.__origin__, self.__args__))
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/davep/develop/python/textual-sandbox/file_reader.py", line 36, in <module>
    LoadUpFileApp().run()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 1400, in run
    asyncio.run(run_app())
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 1389, in run_app
    await self.run_async(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 1361, in run_async
    await app._shutdown()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2424, in _shutdown
    await self._close_all()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2416, in _close_all
    await child._close_messages()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/message_pump.py", line 462, in _close_messages
    await self._task
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/message_pump.py", line 481, in _process_messages
    await self._process_messages_loop()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/message_pump.py", line 541, in _process_messages_loop
    self.app._handle_exception(error)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2064, in _handle_exception
    self._fatal_error()
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/textual/app.py", line 2069, in _fatal_error
    traceback = Traceback(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 264, in __init__
    trace = self.extract(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 449, in extract
    locals={
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/traceback.py", line 450, in <dictcomp>
    key: pretty.traverse(
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 853, in traverse
    node = _traverse(_object, root=True)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  [Previous line repeated 969 more times]
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 838, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 679, in _traverse
    child_node = _traverse(child, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 685, in _traverse
    child_node = _traverse(arg, depth=depth + 1)
  File "/Users/davep/develop/python/textual-sandbox/.venv/lib/python3.10/site-packages/rich/pretty.py", line 614, in _traverse
    def iter_rich_args(rich_args: Any) -> Iterable[Union[Any, Tuple[str, Any]]]:
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 309, in inner
    return cached(*args, **kwds)
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 1248, in __hash__
    return hash(frozenset(self.__args__))
  File "/Users/davep/.pyenv/versions/3.10.10/lib/python3.10/typing.py", line 1038, in __hash__
    return hash((self.__origin__, self.__args__))
RecursionError: maximum recursion depth exceeded while calling a Python object
make: *** [file_reader] Error 1

Often the app will crash with a bell sound and nothing else happens until I do something to cause the app to attempt to refresh (a resize of the terminal being a good example).

If, on the other hand, I run similar code with the same input file, but without updating the Log from within a threaded worker:

from pathlib import Path

from textual.app import App, ComposeResult
from textual.screen import Screen
from textual.widgets import Header, Log, Footer

class ShowFile(Screen):

    def __init__(self, path: Path, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.path = path

    def on_mount(self) -> None:
        self.sub_title = f"Showing: {self.path.name}"
        self.read_file(self.path)

    def compose(self) -> ComposeResult:
        yield Header(show_clock=True)
        yield Log(classes="status-richlog")
        yield Footer()

    def read_file(self, path: Path) -> None:
        log = self.query_one(Log)
        with open(path, "r") as f:
            for line in f:
                log.write(line)

class LoadUpFileApp(App[None]):

    def on_mount(self) -> None:
        self.push_screen(ShowFile(Path("test.txt")))

if __name__ == "__main__":
    LoadUpFileApp().run()

this runs to completion without any issues at all.

The text was updated successfully, but these errors were encountered:

willmcgugan · 2023-10-15T10:14:41Z

What is the contents of "test.txt" ?

davep · 2023-10-15T10:15:41Z

@willmcgugan

if I create test.txt as a text file around 2.6M in size

Literally, just a plain text file; in this case built up with lorem ipsum.

willmcgugan · 2023-10-15T10:19:51Z

Was it multiple lines or one single line?

davep · 2023-10-15T10:22:46Z

Multiple lines. Literally just this sort of thing:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse
sagittis odio at suscipit dignissim. Sed auctor molestie orci iaculis
egestas. Fusce metus augue, rhoncus quis augue a, sodales placerat lacus.
Nam in metus et dui varius convallis convallis sed turpis. Vivamus pharetra
dictum suscipit. Etiam fringilla iaculis magna non commodo. Etiam eros
lorem, laoreet at rhoncus ullamcorper, luctus vestibulum elit. Donec
dignissim vestibulum ante, ac tempus magna elementum eget. Nulla et nibh
lacinia dolor faucibus consectetur. Mauris vel vehicula libero. Vivamus
vitae commodo mi, sed euismod tortor.

Vivamus quis imperdiet tellus, vitae mattis sapien. Nulla facilisi. Sed
pretium suscipit nisl, nec porttitor augue tempor in. Maecenas ullamcorper
blandit nisi, non iaculis enim fermentum laoreet. Aliquam dui leo, auctor
non lorem in, convallis consectetur velit. Integer turpis orci, laoreet sed
nisi vitae, pellentesque interdum purus. Curabitur congue tortor ac velit
consectetur tempor. Quisque rutrum aliquam elit, sit amet venenatis nibh
efficitur nec.

Sed pharetra ullamcorper diam, quis eleifend purus dapibus a. Suspendisse
sit amet pellentesque eros, non elementum sapien. Praesent urna lacus,
pharetra quis molestie id, bibendum vitae dui. Sed ut tincidunt dolor.
Integer sodales neque id libero mattis tristique. Quisque fringilla metus
libero, vel egestas nunc auctor consequat. Proin sit amet euismod libero,
sit amet iaculis libero. Vivamus pulvinar vitae magna sit amet scelerisque.
Donec vulputate arcu et placerat suscipit. Proin in congue tortor, sit amet
dapibus quam. Morbi velit lacus, lacinia quis venenatis eget, lacinia eget
ligula. Aenean varius leo vel quam porttitor, vitae pulvinar nunc commodo.
Cras elementum aliquam ante a congue.

Maecenas non lacinia ante, eu iaculis libero. Aliquam ultricies magna et
diam posuere malesuada. Fusce ut hendrerit turpis. Praesent risus eros,
viverra et ligula at, sodales tristique leo. Quisque malesuada rhoncus
tortor, eu sagittis quam ultricies et. Aliquam eu aliquet ex. Phasellus
tempor ex ut diam hendrerit, quis tristique justo lobortis. Etiam commodo
congue mauris. Pellentesque sed laoreet nisi, semper tincidunt elit. Duis
vel quam porta, posuere neque eget, hendrerit leo. Aliquam leo mauris,
semper quis dolor vulputate, tristique laoreet sem. Vivamus bibendum elit
sed pretium tincidunt. Suspendisse id vulputate lorem. Nulla aliquet elit id
aliquam vehicula.

Proin eget dignissim mi. Sed sit amet purus non augue ornare posuere a ut
nisi. Nam a risus euismod felis porta sagittis. Nunc neque tellus, eleifend
at pretium ac, posuere hendrerit odio. Mauris gravida ante eu risus
porttitor vestibulum nec non justo. Vestibulum sed malesuada purus. Quisque
non urna nibh. Pellentesque id sagittis ex. Mauris metus erat, consequat non
purus eget, venenatis eleifend lectus. Nulla facilisi. Phasellus quis arcu
ac est ornare imperdiet. Praesent interdum volutpat tristique. Duis in odio
accumsan, congue velit a, ultricies sapien.

repeatedly concatenated until the file was large enough to reproduce the failure the dev reported.

willmcgugan · 2023-10-15T10:45:00Z

I can reproduce that.

In the short term, you could recommend reading the entire file writing it in single call to log.write, which works fine.

davep · 2023-10-15T10:45:55Z

I've also noticed that using write_line seems to be fine too and suggested that as a workaround.

willmcgugan · 2023-10-15T11:56:27Z

Interesting bug. Not actually related to workers / threadings, as much as it looked like it!

davep · 2023-10-15T11:58:00Z

Yeah, just looking at the PR now and I'm wondering why the same code only displayed the issue when from a worker.

willmcgugan · 2023-10-15T11:59:57Z

I suspect its a race condition of sorts. The worker just happened to change the order of things enough to trip the issue.

github-actions · 2023-10-17T09:45:42Z

Don't forget to star the repository!

Follow @textualizeio for Textual updates.

davep added bug Something isn't working Task labels Oct 15, 2023

willmcgugan mentioned this issue Oct 15, 2023

fix for cache #3538

Merged

willmcgugan closed this as completed in #3538 Oct 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eventual exception when updating `Log` from from a threader worker #3537

Eventual exception when updating `Log` from from a threader worker #3537

davep commented Oct 15, 2023 •

edited

Loading

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

github-actions bot commented Oct 17, 2023

Eventual exception when updating Log from from a threader worker #3537

Eventual exception when updating Log from from a threader worker #3537

Comments

davep commented Oct 15, 2023 • edited Loading

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

davep commented Oct 15, 2023

willmcgugan commented Oct 15, 2023

github-actions bot commented Oct 17, 2023

Eventual exception when updating `Log` from from a threader worker #3537

Eventual exception when updating `Log` from from a threader worker #3537

davep commented Oct 15, 2023 •

edited

Loading