Updates logging to include Async Task Ids. #216
Conversation
This PR adds new logging handler classes for Stream (console) and File output that can add async task ids to the logging message. This makes debugging eaiser and lets us group logging messages based on parameter name without needing to pass the actual parameter around all the time. Additionally, the logging configuration has been unified across all our provided scripts and the env var `GIT_THETA_LOG_LEVEL` can be used to control the level of the logs. Currently, this all still works by configuring the root logger, eventually we should move to a named logger and only configure that one. That will allow us to do things like log to console and file at different levels.
blester125
force-pushed
the
feat/async-logging
branch
from
95bd8f6
to
33bf77d
Compare
| @@ -73,14 +73,15 @@ def __get__(self, obj, objtype=None): | |||
| class EnvVarConstants: | |||
| CHECKPOINT_TYPE = EnvVar(name="GIT_THETA_CHECKPOINT_TYPE", default="pytorch") | |||
| UPDATE_TYPE = EnvVar(name="GIT_THETA_UPDATE_TYPE", default="dense") | |||
| UPDATE_DATA_PATH = EnvVar(name="GIT_THETA_UPDATE_DATA_PATH", default="update.pt") | |||
| UPDATE_DATA_PATH = EnvVar(name="GIT_THETA_UPDATE_DATA_PATH", default="") | |||
There was a problem hiding this comment.
Why doesn't this get a default anymore?
There was a problem hiding this comment.
Basically, when an update is done with side loaded information (like a sparse or lora update), the on-disk/working checkpoint doesn't change, but the checkout in the git history does (it has the side loaded information in it). This causes false positives when you run git status. As such, we include a check for side-loaded information in some of our scripts that will write the updated version to disk so that git doesn't detect a diff.
With the default value, git-theta was always triggering that extra write, even in a dense update where the result is overwriting the model with the same values.
So removing this default value skips that extra smudge in cases where there isn't any side-loaded information.
This PR adds new logging handler classes for Stream (console) and File output that can add async task ids to the logging message. This makes debugging eaiser and lets us group logging messages based on parameter name without needing to pass the actual parameter around all the time.
Additionally, the logging configuration has been unified across all our provided scripts and the env var
GIT_THETA_LOG_LEVELcan be used to control the level of the logs.Currently, this all still works by configuring the root logger, eventually we should move to a named logger and only configure that one. That will allow us to do things like log to console and file at different levels.
I also changed the default value for
GIT_THETA_UPDATE_PATHas the default was triggering an extra smudge, even when using dense updates.