Sync tensorboard stops working in upgrade from 0.9.7 to 0.10.2

[ecatkins]

I had been using sync_tensorboard with the Tensorflow Object Detection API (tensorflow==1.15.2), but when upgrading to version 0.10.2 metrics are no longer logged to the wandb dashboard.

I have confirmed when downgrading to 0.9.7 the logging begins working again

[issue-label-bot]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.97. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[raubitsj]

@ecatkins , Thanks for reporting this issue. Can you tell us what version of tensorboard you were using?

In the new version of the library we improved support for tensorflow 2.x and the latest pytorch but this might have inadvertently broken something with tensorflow 1.x support.

Are you manually loggging to a summarywriter or is this something that the tensorflow object detection api is doing?

[ecatkins]

tensorboard==1.15.0

And no, nothing manual. I was just using the events that the tensorflow object detection api logs OOTB

[koulanurag]

Facing same issue. Tensorboard events are no longer being logged in the wandb api.

torch==1.5.1+cu101
tensorboard==2.2.1
tensorboard-plugin-wit==1.7.0

[raubitsj]

@koulanurag, This is fixed now and will be in the next release: 0.10.3. If you want to try it now use:

pip install --upgrade git+git://github.com/wandb/client.git

@ecatkins, we are still looking into your issue as I believe it is different but the fixes we did might address your problem if you were writing your tensorboard log to cloud storage (s3).

[moizsajid]

I was also facing this same problem even in version 0.10.7.dev1. Everything is working fine now after downgrading to version 0.9.7 as suggested by @ecatkins.

[vanpelt]

@moizsajid can you provide an example script and the versions of tensorflow / tensorboard you're using. sync_tensorboard should be fixed for all version of tensorflow >= 1.15 in 0.10.3 so if you're still seeing it we need to address the bug.

[lminer]

I'm having this issue with tensorflow 2.3.1. Works fine if I save my tensorboard logs locally, but breaks when I log to s3.

I can confirm this is still a problem in 0.10.3 and on the latest development branch.

[TusharNimbhorkar]

@vanpelt @ecatkins @raubitsj
Facing the same issue. Is there any update on this?

[vanpelt]

Hey @TusharNimbhorkar what version of tensorboard are you using / how are you instantiating your SummaryWriter?

[TusharNimbhorkar]

@vanpelt
Version:
tensorboard==2.4.0
tensorboardX==2.1

I am using tensorboardX's SummaryWriter. I have installed tensorboard because wandb gives me error without it.

Command to instantiating SummaryWriter:
writer = SummaryWriter(output_folder)

[vanpelt]

@TusharNimbhorkar we just made a branch that addresses support for tboardx, it will go out with in the next release. You can use it now but installing wandb with:

pip install --upgrade git+git://github.com/wandb/client.git@task/tboardx#egg=wandb

[TusharNimbhorkar]

@vanpelt
Thank you this fixed worked. Any idea when the next release is scheduled?

[github-actions]

This issue is stale because it has been open 60 days with no activity.

[sydholl]

In the past year we've majorly reworked the CLI and UI for Weights & Biases. We're closing issues older than 6 months. Please comment to reopen.