Tensorboard Logging

[shukon]

Currently tensorboard_logger is used. I don't think it's actively supported anymore, they point to pytorch's own tensorboard-module on their Github.
Why bother? tensorboard_logger uses a singleton-default-logger and I cannot reset the path to write the tensorboard-eventfiles. That would be helpful though, to distinguish between different configurations when refitting configurations from the incumbent-trajectory :)

[LMZimmer]

I agree and am currently thinking about other ways of logging (also because tb logs are very memory inefficient). Do you have any suggestions? If you want I could also point you to where the logs are performed and you could write a PR

[shukon]

Mh - I'm not sure whether ditching tensorboard in general makes sense 🤔 I think with pytorch and tensorflow using it, that makes it kind of a standard? I found where the logging is done and I will look into it. Do you know why logging would be memory-inefficient? Running the tensorboard-server surely is, but on the logging-side of it, shouldn't the logs just be written to the file regularly? I'll have a look but for now I'll be installing tensorboard_logger from source in the dev-version, they provide an unconfigure()-call :)

[maxmarketit]

In my opinion, it would be better if seperate logging is made for each model.

Something like,

from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter(filename)

Different writer for different model. Each model will have its own logging directory and
we can do tensorboard --log-dir root to see and compare all the models.

I think we can have another writer for monitoring loss for all the models...

But as the number of models gets large, it will be soon very crowded.

But you can filter some of the models.

[shukon]

@maxmarketit Yes! Separate writers for separate models / training processes would be a good idea. I'm just not sure where to create and where to pass the writer in this pipeline...

[maxmarketit]

@shukon I might not understand the situation here but I do not see any problem
because if the each training of NN starts and ends one summaryWriter, is there really any requirement to pass the writer?

I think we can set the root directory and every model trained can saved to subdirectory with appropriate name which convealing main architecture 'resnet-100-20-30' or 'shapedmlpnet-diamond-...-...' and as the optimization goes, it could change to resnet-100-20-30-i01' '-i02' '-i03' etc