-
-
Notifications
You must be signed in to change notification settings - Fork 311
Open
Labels
bugSomething isn't workingSomething isn't working
Description
🐛 Describe the bug
Summary: When reloading a strategy checkpoint using maybe_load_checkpoint, the loggers for that strategy are unpickled, which calls TextLogger._fobj_deserialize. If the strategy saved text logs to a directory that no longer exists (for example in a temporary directory), then when loading the checkpoint you get FileNotFoundError for any file definition of the form "path:<path to file>".
🐜 To Reproduce
os.makedirs("logs", exist_ok=True)
log_fname = "logs/{}.log".format(datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))
text_logger = TextLogger(file=open(log_fname, 'w'))
eval_plugin = EvaluationPlugin(
loggers=[text_logger]
)
strategy = Naive(
model=model,
optimizer=optimizer,
criterion=criterion,
evaluator=eval_plugin,
)
save_checkpoint(
strategy,
"checkpoint.pt"
)
shutil.rmtree("logs")
strategy, _ = maybe_load_checkpoint(
strategy,
"checkpoint.pt"
)
🐝 Expected behavior
The log directory and file should be created. This is special pickling behavior from avalanche, hence ideally it should either handle all the pickling cases properly or not pickle the text logger.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working