Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue where sync_tensorboard could die on large histograms #3019

Merged
merged 12 commits into from
Feb 18, 2022

Conversation

KyleGoyette
Copy link
Contributor

@KyleGoyette KyleGoyette commented Dec 10, 2021

Fixes WB-7696

Description

Wraps the call to wandb.Histogram in wandb/integrations/log.py with a try except to keep it from dying when a large histogram is logged.

Testing

On colab using this run: https://wandb.ai/kylegoyette/debug_ray/runs/35d5o7qc?workspace=user-kylegoyette
Locally: https://wandb.ai/kylegoyette/debug_ray/runs/2a988003?workspace=user-kylegoyette

Checklist

  • Name PR "[WB-NNNN][WB-MMMM] Add support for..." similar to entries in CHANGELOG.md
  • Include reference to internal ticket "Fixes WB-NNNN" (and github issue "Fixes #NNNN" if applicable)

@codecov
Copy link

codecov bot commented Dec 10, 2021

Codecov Report

Merging #3019 (00601f5) into master (61ea6a7) will increase coverage by 0.02%.
The diff coverage is 50.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3019      +/-   ##
==========================================
+ Coverage   80.15%   80.17%   +0.02%     
==========================================
  Files         213      213              
  Lines       27872    27875       +3     
==========================================
+ Hits        22340    22348       +8     
+ Misses       5532     5527       -5     
Flag Coverage Δ
functest 57.22% <0.00%> (+<0.01%) ⬆️
unittest 69.57% <50.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
wandb/integration/tensorboard/log.py 86.66% <50.00%> (-1.09%) ⬇️
wandb/sdk/internal/sender.py 91.44% <0.00%> (-0.28%) ⬇️
wandb/sdk/lib/git.py 76.35% <0.00%> (ø)
wandb/sdk/wandb_run.py 88.17% <0.00%> (+0.28%) ⬆️
wandb/sdk/internal/meta.py 90.74% <0.00%> (+3.08%) ⬆️

@KyleGoyette KyleGoyette requested a review from vwrj February 4, 2022 18:58
Copy link
Contributor

@vwrj vwrj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@KyleGoyette KyleGoyette merged commit 412f7eb into master Feb 18, 2022
@KyleGoyette KyleGoyette deleted the bugfix/histogram-crashes-tb-watcher branch February 18, 2022 06:26
@raubitsj raubitsj changed the title [WB-7696] Fix issue where TBWatcher thread could die on large histograms Fix issue where sync_tensorboard could die on large histograms Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants