Skip to content

Error on unpickling TextLogger after deleting directory where text log was saved. #1661

@man2machine

Description

@man2machine

🐛 Describe the bug
Summary: When reloading a strategy checkpoint using maybe_load_checkpoint, the loggers for that strategy are unpickled, which calls TextLogger._fobj_deserialize. If the strategy saved text logs to a directory that no longer exists (for example in a temporary directory), then when loading the checkpoint you get FileNotFoundError for any file definition of the form "path:<path to file>".

🐜 To Reproduce

os.makedirs("logs", exist_ok=True)
log_fname = "logs/{}.log".format(datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))
 
text_logger = TextLogger(file=open(log_fname, 'w'))
eval_plugin = EvaluationPlugin(
    loggers=[text_logger]
)

strategy = Naive(
    model=model,
    optimizer=optimizer,
    criterion=criterion,
    evaluator=eval_plugin,
)

save_checkpoint(
    strategy,
    "checkpoint.pt"
)

shutil.rmtree("logs")

strategy, _ = maybe_load_checkpoint(
    strategy,
    "checkpoint.pt"
)

🐝 Expected behavior
The log directory and file should be created. This is special pickling behavior from avalanche, hence ideally it should either handle all the pickling cases properly or not pickle the text logger.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions