Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to log_model="all" with wandb_logger #1641

Closed
robmarkcole opened this issue Oct 10, 2023 · 2 comments · Fixed by #1647
Closed

Unable to log_model="all" with wandb_logger #1641

robmarkcole opened this issue Oct 10, 2023 · 2 comments · Fixed by #1647
Milestone

Comments

@robmarkcole
Copy link
Contributor

Description

Configured to log all checkpoints, but only a single is logged

Steps to reproduce

from lightning.pytorch.loggers import WandbLogger

wandb_logger = WandbLogger(
    project="eurosat-test3", 
    name="imagenet", 
    log_model="all", 
    save_dir = "wandb_logs_tests"
)

trainer = Trainer(
    logger=wandb_logger,
    min_epochs=15,
    max_epochs=25,
)

trainer.fit(model=model, datamodule=datamodule)

glob("wandb_logs_tests/eurosat-test3/*/checkpoints/*.ckpt") # returns only a single checkpoint

Version

0.5.0

@adamjstewart adamjstewart added this to the 0.5.1 milestone Oct 10, 2023
@calebrob6
Copy link
Member

Is this a torchgeo bug or lightning bug?

@robmarkcole
Copy link
Contributor Author

robmarkcole commented Oct 11, 2023

I commented out configure_callbacks(self) and now all logs are created by the callback:

image

However if I comment out the callback, I still only have 1 checkpoint (final) locally
image

However all checkpoints are uploaded
image

It is not clear but perhaps this is intended behaviour and not a bug. Either way I think this rules out this being a torchgeo bug so closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants