You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
model, stat = train_eval(model, train_loader, val_loader, device)
File "bert_unlab_mlm.py", line 175, in train_eval
wandb.log({'train_batch_loss':loss.item()})
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 1422, in log
self._log(data=data, step=step, commit=commit)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 1232, in _log
self._partial_history_callback(data, self.history_step, commit=True)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 1083, in _partial_history_callback
self._backend.interface.publish_partial_history(
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 517, in publish_partial_history
self._publish_partial_history(partial_history)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 61, in _publish_partial_history
self._publish(rec)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/interface/interface_queue.py", line 49, in _publish
raise Exception("The wandb backend process has shutdown")
Exception: The wandb backend process has shutdown
Can you please help with this?
Expected Behavior
No response
Steps To Reproduce
No response
Screenshots
No response
Environment
OS: Linux
Version: wandb==0.12.11; Python 3.8.12
Additional Context
No response
The text was updated successfully, but these errors were encountered:
Hi @kanishk-adapt thank you for reporting this. Can you please provide a bit more information of the training environment and infrastructure where this was running? I've noticed you're using an older wandb SDK version, would it be possible to upgrade to the most recent one (0.13.7)? Also could you provide us with the debug.log and debug-internal.log files of the run that failed? you can find those in /wandb/run-<data-time>-<run-id>/logs relative path from where you executed the script. If you don't want to publicly share them, please send these files to support@wandb.com and mention this GitHub issue, and we can take it from there.
Hi @kanishk-adapt I wanted to follow up on this issue, does it still occur for you and would it be possible to provide us with the information above so that we could investigate further what's the root cause? thanks!
Hi @thanos-wandb , thank you for your response.
I found a similar issue here: #3223
The resolution, to add environment variable: WANDB_START_METHOD="thread" seems to have resolved the issue.
I have now upgraded wandb to 0.13.7 and seems to be working well.
Thanks again,
Current Behavior
My script failed because of the below error,
`
File "bert_unlab_mlm.py", line 300, in
model, stat = train_eval(model, train_loader, val_loader, device)
File "bert_unlab_mlm.py", line 175, in train_eval
wandb.log({'train_batch_loss':loss.item()})
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 1422, in log
self._log(data=data, step=step, commit=commit)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 1232, in _log
self._partial_history_callback(data, self.history_step, commit=True)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/wandb_run.py", line 1083, in _partial_history_callback
self._backend.interface.publish_partial_history(
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/interface/interface.py", line 517, in publish_partial_history
self._publish_partial_history(partial_history)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/interface/interface_shared.py", line 61, in _publish_partial_history
self._publish(rec)
File "/home/kverma/anaconda3/envs/acl/lib/python3.8/site-packages/wandb/sdk/interface/interface_queue.py", line 49, in _publish
raise Exception("The wandb backend process has shutdown")
Exception: The wandb backend process has shutdown
Can you please help with this?
Expected Behavior
No response
Steps To Reproduce
No response
Screenshots
No response
Environment
OS: Linux
Version: wandb==0.12.11; Python 3.8.12
Additional Context
No response
The text was updated successfully, but these errors were encountered: