Metric Logging updates 1/4 #345
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
allenwang28
reviewed
Oct 8, 2025
| # In multiprocessing environments, WandB service tokens can become stale and point | ||
| # to dead service processes. This causes wandb.init() to hang indefinitely trying | ||
| # to connect to non-existent services. Clearing forces fresh service connection. | ||
| from wandb.sdk.lib.service import service_token |
There was a problem hiding this comment.
should this also be imported at the top?
There was a problem hiding this comment.
I dont think so. My opinion is that backends like wandb should be protected, otherwise user is required to have wandb even if they dont use it. Extrapolate that to mlflow, scuba, etc. Let me know if you disagree.
felipemello1
commented
Oct 8, 2025
allenwang28
approved these changes
Oct 8, 2025
14 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
FORGE_DISABLE_METRICSnow disables spawningLocalFetcherActorwhen spawning processesrecord_metrics(...), ifGlobalLoggingActor.init_backendswas not initialized, it would raise errors. Now it just prints a warning. This enables stand alone components to be tested without metric logging.init_backendwas called AFTER initializing the other services. Callingservice_token.clear_service_in_env()in shared mode fixes this issue.