unhashable type: 'types.SimpleNamespace' when using uninitialized logging or entering alive_bar's context twice #107
Comments
|
You don't need |
|
But I don't understand the error. What are you calling? And what is the complete stacktrace? |
|
Thanks for the reply -- In testing the new version, I saw the incr parameter was no longer defined, so I removed it from my code. After doing that, I'm occasionally getting the error below when calling bar with an integer to increment. The code erroring out now is the "bar(chunkSize)" call below: Full error stack: |
|
Great! Now I can see what did happen. |
|
You use |
|
There isn't logging in this call and no nested progress bars either. It's using a sqlalchemy engine created here via pyodbc for the sourceEngine: Other than that engine being opened by pd.read_sql, the |
|
I have the same problem with │ ------------------------------------------------------------------------------------ log_metrics | │
│ │
│ 168 │ │ if flush: │
│ 169 │ │ │ self._delete_last_lines(len(metrics)) │
│ 170 │ │ else: │
│ ❱ 171 │ │ │ self._console.print("---------------------", style="bold magenta") │
│ 172 │ │ for metric in metrics: │
│ 173 │ │ │ self._console.print(metric) │
│ │
│ ╭──────────────────────────────────── locals ─────────────────────────────────────╮ │
│ │ flush = False │ │
│ │ metrics = [ | |
| | ------------------------------------ │ │
│ │ ] │ │
│ │ self = <------------------.training.utils.ProgressBar object at 0x13edccfd0> │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ -------------------------------------------------------------/.venv/lib/python3.7/site-packages/ │
│ rich/console.py:1615 in print │
│ │
│ 1612 │ │ │ │ ): │
│ 1613 │ │ │ │ │ buffer_extend(line) │
│ 1614 │ │ │ else: │
│ ❱ 1615 │ │ │ │ self._buffer.extend(new_segments) │
│ 1616 │ │
│ 1617 │ def print_json( │
│ 1618 │ │ self, │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ buffer_extend = <built-in method extend of list object at 0x13edd0948> │ │
│ │ crop = True │ │
│ │ emoji = None │ │
│ │ end = '\n' │ │
│ │ extend = <built-in method extend of list object at 0x13edfa408> │ │
│ │ height = None │ │
│ │ highlight = None │ │
│ │ justify = None │ │
│ │ line = [ │ │
│ │ │ Segment( │ │
│ │ │ │ '---------------------', │ │
│ │ │ │ Style( │ │
│ │ │ │ │ color=Color('magenta', ColorType.STANDARD, number=5), │ │
│ │ │ │ │ bold=True │ │
│ │ │ │ ) │ │
│ │ │ ), │ │
│ │ │ Segment('\n',) │ │
│ │ ] │ │
│ │ markup = None │ │
│ │ new_line_start = False │ │
│ │ new_segments = [ │ │
│ │ │ Segment( │ │
│ │ │ │ '---------------------', │ │
│ │ │ │ Style( │ │
│ │ │ │ │ color=Color('magenta', ColorType.STANDARD, number=5), │ │
│ │ │ │ │ bold=True │ │
│ │ │ │ ) │ │
│ │ │ ), │ │
│ │ │ Segment( │ │
│ │ │ │ '\n', │ │
│ │ │ │ Style( │ │
│ │ │ │ │ color=Color('magenta', ColorType.STANDARD, number=5), │ │
│ │ │ │ │ bold=True │ │
│ │ │ │ ) │ │
│ │ │ ) │ │
│ │ ] │ │
│ │ no_wrap = None │ │
│ │ objects = ('---------------------',) │ │
│ │ overflow = None │ │
│ │ render = <bound method Console.render of <console width=176 ColorSystem.TRUECOLOR>> │ │
│ │ render_options = ConsoleOptions(size=ConsoleDimensions(width=176, height=43), │ │
│ │ legacy_windows=False, min_width=1, max_width=176, is_terminal=True, │ │
│ │ encoding='utf-8', max_height=43, justify=None, overflow=None, no_wrap=None, │ │
│ │ highlight=None, markup=None, height=None) │ │
│ │ renderable = <text '---------------------' []> │ │
│ │ renderables = [<text '---------------------' []>] │ │
│ │ self = <console width=176 ColorSystem.TRUECOLOR> │ │
│ │ sep = ' ' │ │
│ │ soft_wrap = False │ │
│ │ style = 'bold magenta' │ │
│ │ width = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ │
│ -------------------------------------------------------------/.venv/lib/python3.7/site-packages/ │
│ rich/console.py:825 in __exit__ │
│ │
│ 822 │ │
│ 823 │ def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: │
│ 824 │ │ """Exit buffer context.""" │
│ ❱ 825 │ │ self._exit_buffer() │
│ 826 │ │
│ 827 │ def begin_capture(self) -> None: │
│ 828 │ │ """Begin capturing console output. Call :meth:`end_capture` to exit capture mod │
│ │
│ ╭─────────────────────── locals ────────────────────────╮ │
│ │ exc_type = None │ │
│ │ exc_value = None │ │
│ │ self = <console width=176 ColorSystem.TRUECOLOR> │ │
│ │ traceback = None │ │
│ ╰───────────────────────────────────────────────────────╯ │
│ │
│ -------------------------------------------------------------/.venv/lib/python3.7/site-packages/ │
│ rich/console.py:784 in _exit_buffer │
│ │
│ 781 │ def _exit_buffer(self) -> None: │
│ 782 │ │ """Leave buffer context, and render content if required.""" │
│ 783 │ │ self._buffer_index -= 1 │
│ ❱ 784 │ │ self._check_buffer() │
│ 785 │ │
│ 786 │ def set_live(self, live: "Live") -> None: │
│ 787 │ │ """Set Live instance. Used by Live context manager. │
│ │
│ ╭───────────────────── locals ─────────────────────╮ │
│ │ self = <console width=176 ColorSystem.TRUECOLOR> │ │
│ ╰──────────────────────────────────────────────────╯ │
│ │
│ -------------------------------------------------------------/.venv/lib/python3.7/site-packages/ │
│ rich/console.py:1866 in _check_buffer │
│ │
│ 1863 │ │ │ │ │ │ │ │ for line in text.splitlines(True): │
│ 1864 │ │ │ │ │ │ │ │ │ write(line) │
│ 1865 │ │ │ │ │ │ │ else: │
│ ❱ 1866 │ │ │ │ │ │ │ │ self.file.write(text) │
│ 1867 │ │ │ │ │ │ │ self.file.flush() │
│ 1868 │ │ │ │ │ │ except UnicodeEncodeError as error: │
│ 1869 │ │ │ │ │ │ │ error.reason = f"{error.reason}\n*** You may need to add PY │
│ │
│ ╭───────────────────── locals ──────────────────────╮ │
│ │ self = <console width=176 ColorSystem.TRUECOLOR> │ │
│ │ text = '\x1b[1;35m---------------------\x1b[0m\n' │ │
│ ╰───────────────────────────────────────────────────╯ │
│ │
│ -------------------------------------------------------------/.venv/lib/python3.7/site-packages/ │
│ alive_progress/core/hook_manager.py:36 in write │
│ │
│ 33 │ │ │ stream.flush() │
│ 34 │ │
│ 35 │ def write(stream, part): │
│ ❱ 36 │ │ buffer = buffers[stream] │
│ 37 │ │ if part != '\n': │
│ 38 │ │ │ # this will generate a sequence of lines interspersed with None, which will │
│ 39 │ │ │ # be rendered as the indent filler to align additional lines under the same │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ base = ( │ │
│ │ │ namespace(flush=functools.partial(<function │ │
│ │ buffered_hook_manager.<locals>.flush at 0x13eb462f0>, <_io.TextIOWrapper │ │
│ │ name='<stdout>' mode='w' encoding='UTF-8'>), isatty=<built-in method isatty │ │
│ │ of _io.TextIOWrapper object at 0x108db2630>, │ │
│ │ write=functools.partial(<function buffered_hook_manager.<locals>.write at │ │
│ │ 0x13eb46378>, <_io.TextIOWrapper name='<stdout>' mode='w' │ │
│ │ encoding='UTF-8'>)), │ │
│ │ │ namespace(flush=functools.partial(<function │ │
│ │ buffered_hook_manager.<locals>.flush at 0x13eb462f0>, <_io.TextIOWrapper │ │
│ │ name='<stderr>' mode='w' encoding='UTF-8'>), isatty=<built-in method isatty │ │
│ │ of _io.TextIOWrapper object at 0x108db2630>, │ │
│ │ write=functools.partial(<function buffered_hook_manager.<locals>.write at │ │
│ │ 0x13eb46378>, <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>)) │ │
│ │ ) │ │
│ │ buffers = defaultdict(<class 'list'>, {}) │ │
│ │ cond_refresh = <Condition(<unlocked _thread.RLock object owner=0 count=0 at 0x13eb613f0>, │ │
│ │ 0)> │ │
│ │ get_header = <function buffered_hook_manager.<locals>.<lambda> at 0x13edf79d8> │ │
│ │ part = '\x1b[1;35m---------------------\x1b[0m\n' │ │
│ │ stream = namespace(flush=functools.partial(<function │ │
│ │ buffered_hook_manager.<locals>.flush at 0x13eb462f0>, <_io.TextIOWrapper │ │
│ │ name='<stdout>' mode='w' encoding='UTF-8'>), isatty=<built-in method isatty │ │
│ │ of _io.TextIOWrapper object at 0x108db2630>, │ │
│ │ write=functools.partial(<function buffered_hook_manager.<locals>.write at │ │
│ │ 0x13eb46378>, <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)) │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: unhashable type: 'types.SimpleNamespace'
During handling of the above exception, another exception occurred: |
|
I can't simulate this at all. |
|
I found that this problem is connected to overlapping context managers. import time
from alive_progress import *
range_1 = alive_it(range(13))
with alive_bar(14) as bar_1:
for item in range(13):
print("hello")
time.sleep(0.1)
bar_1()
with alive_bar(14) as bar_2:
for item in range(10):
print("hello")
time.sleep(0.1)
bar_2()Clearly, it is not a proper way of using alive-progress. However, it occurs with from typing import Iterable, Any
import time
from loguru import logger
from rich.console import Console
from alive_progress import alive_it
from ignite.engine import Engine, Events
class ProgressBar:
def __init__(self, iterable: Iterable, total: int = None, title: str = None, **options) -> None:
self._console = Console()
self._total = total or 0
self._iterable = alive_it(
iterable, total=self._total, title=title, enrich_print=False, receipt_text=True, **options
)
def __len__(self) -> int:
return self._total
def __iter__(self) -> Any:
yield from self._iterable
def text(self, desc: str) -> None:
self._iterable.text(desc)
def desc(self, desc: str) -> None:
self._console.print(desc)
class SimpleIterable:
def __init__(self, length: int) -> None:
self._iterable = range(length)
def __iter__(self) -> int:
for item in self._iterable:
yield item
trainer = Engine(lambda engine, sample: time.sleep(0.2))
evaluator = Engine(lambda engine, sample: time.sleep(0.2))
data_loaders = {"train": SimpleIterable(20), "evaluate": SimpleIterable(13)}
@trainer.on(Events.ITERATION_COMPLETED)
def on_train_batch(engine: Engine) -> None:
engine.state.dataloader.desc("train batch processed")
@evaluator.on(Events.ITERATION_COMPLETED)
def on_eval_batch(engine: Engine) -> None:
engine.state.dataloader.desc("eval batch processed")
@trainer.on(Events.EPOCH_STARTED)
def on_train_epoch_start(engine: Engine) -> None:
engine.set_data(ProgressBar(data_loaders["train"], total=13, title="Train epoch"))
@evaluator.on(Events.EPOCH_STARTED)
def on_eval_epoch_start(engine: Engine) -> None:
engine.set_data(ProgressBar(data_loaders["evaluate"], total=10, title="Evaluate epoch"))
@trainer.on(Events.EPOCH_COMPLETED)
def on_train_epoch_end(engine: Engine) -> None:
logger.info("Train step completed")
evaluator.run(data_loaders["evaluate"])
@evaluator.on(Events.EPOCH_COMPLETED)
def on_eval_epoch_end(engine: Engine) -> None:
logger.info("Evaluate step completed")
if __name__ == "__main__":
trainer.run(data_loaders["train"], max_epochs=3, epoch_length=7) |
So, it really was what I thought... |
|
I definitely don't have nested bars in the case of my code above. I'll try to get some generic reproduceable code together for you. |
|
Got it! Finally figured out the source here. There is a subprocess that gets called in an "unknown length" bar that uses logging. The defined length bar that follows errors out. Check this out: import time
import logging
from alive_progress import alive_bar
def subprocessWithLogging():
logging.info('whoops')
def unknownTest():
with alive_bar(enrich_print=False) as bar:
print('This is an unknown length bar')
time.sleep(1)
bar()
print('This is another step with logging')
subprocessWithLogging()
time.sleep(1)
bar()
def definiteTest():
with alive_bar(100, bar='circles', enrich_print=False) as bar:
for _ in range(100):
time.sleep(.01)
bar()
def main():
"This runs successfully"
definiteTest()
unknownTest()
"This does not"
definiteTest()
main() |
|
I guess you need to instantiate a logger instance with a specified name to fix it. logger = logging.getLogger("alive-progress")
def subprocessWithLogging():
logger.info("whoops")I can not reproduce it with these settings. |
|
Thank you @dbrocha, I see now what's happening... See, at the time you enter But since you are not calling def info(msg, *args, **kwargs):
if len(root.handlers) == 0:
basicConfig() <<<<<<<<<<<<<<<
root.info(msg, *args, **kwargs)That Fortunately the fix is really simple, just call def main():
"This runs successfully"
definiteTest()
unknownTest()
"This does not"
definiteTest()
logging.basicConfig()
main() |
|
Hey @Nemexur, regarding your "totally different problem but with the same outcome", it seems another beast altogether... I don't know anything about pytorch or ignite, but I know that My first approach was to retrieve the generator out of the iterable class, and aggregate them in a tuple: @trainer.on(Events.EPOCH_STARTED)
def on_train_epoch_start(engine: Engine) -> None:
pb = ProgressBar(data_loaders["train"], total=13, title="Train epoch")
engine.set_data((iter(pb), pb))Then both use it: @trainer.on(Events.ITERATION_COMPLETED)
def on_train_batch(engine: Engine) -> None:
engine.state.dataloader[1].desc("train batch processed")and exhaust its generator: @trainer.on(Events.EPOCH_COMPLETED)
def on_train_epoch_end(engine: Engine) -> None:
logger.info("Train step completed")
for _ in engine.state.dataloader[0]:
pass
evaluator.run(data_loaders["evaluate"])OK, IT WORKS! But then I tried to simplify it, and realized that even when I do not exhaust the generator it still works just the same! engine.set_data((iter(pb),))EVEN worse, if I remove the tuple, change to just the generator, it breaks again 😕 engine.set_data(iter(pb))That's it, I'm not sure what's happening... Can you make sense of this? |
rsalmei
changed the title
|
First of all, thanks for the help @rsalmei!! def set_data(self, data_loader):
self.state.dataloader = data_loader
self._dataloader_iter = iter(data_loader)
def run(self, ...):
iter_counter = 0
# ...
while True:
try:
# ...
self.state.batch = next(self._dataloader_iter)
# ...
iter_counter += 1
should_exit = False
except StopIteration:
if should_exit:
# ...
break
self.set_data(self.state.dataloader)
should_exit = True
continue
# Process batch
if self.state.epoch_length is not None and self.state.epoch_length == iter_counter:
break
...Here is the thing, Regarding your solution with Thus, the problem occurs when we try to start a new |
|
You're welcome @Nemexur! I've tried to remove the tuple hack and just sent: engine.set_data(iter(pb))Then, on engine.state.dataloader.close()Simple and clean humm? 👍 |
|
Indeed, very simple and clean. I did not know that about python generators. Thanks!! |
|
Yeah, there's even a |
|
I encountered the same error not when running nested bars directly but when PyTest runs nested bars. The solution is simple: don't use nested bars. But I leave this here for completeness and in case others come across this issue through PyTest. # installed using: pip install alive-progress
# alive-progress version/build/channel: 2.1.0 pypi_0 pypi
# To recreate the error, run this script using pytest command.
from alive_progress import alive_bar
from time import sleep
def bar_1():
items = list('abcdefg')
with alive_bar(len(items), title='Bar-1: ') as bar:
for i, idx in enumerate(items):
sleep(.5)
bar()
def test_nested_bar():
items = list('123456')
with alive_bar(len(items), title='Bar-2: ') as bar:
for i, idx in enumerate(items):
bar_1()
sleep(.5)
bar()Last error message: |
The "incr" param is gone in v2.0 and I believe this arg is occasionally being misassigned.
Error:
Exception has occurred: TypeError
unhashable type: 'types.SimpleNamespace'
The text was updated successfully, but these errors were encountered: