-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Configurable logging #45
Conversation
Hi, @khornlund this is my brief review for this PR. Removing the Moving the model print to the I'm not sure as for above things but using logging levels as verbosity options also seems dangerous. What happens if some imported modules use I don't think I fully understood what the yaml file is doing, but it seems that most of its content is about saving log into Finally, current default logging format seems to have too much information for most of use cases. I noticed that I said only bad things but, I really appreciate you for making this PR. |
Hi @SunQpark , thanks for the speedy response!
Great - it was clunky having two loggers and I don't think the saving/reloading worked anyway.
I think it's really useful to be logging to output of
Could you elaborate on what you don't like about this?
Ok - perhaps there's a use case I'm not understanding where this would be a problem. It's an optional parameter, so it doesn't require being used in a different way. Perhaps you would prefer the
No problem, I'll change it so we're consistent with keras.
The beauty of the logging module is you can control the verbosity for each logger (usually you have a different logger instance per class or module). If you import a module that itself uses You can also filter what is recorded by each
The yaml config allows a clean way of defining logging formats, and definition of
This also works, but you get a lot of functionality for free by using the For example, by using Imagine trying out a new model architecture, and wanting to compare it to the results of a previous version from 2 weeks ago. You can just grep through the saved logs to find the name or datetime of a previous run. The other advantage is users are free to add whatever This might be a lot of information but I hope I'm making the case that by using
I think it's good where possible to separate configuration from code - it makes it easier to track what is a code change and what is a configuration change. Plus, I think yaml is easier to read.
Ok - perhaps we could have a minimal format for printing to the console, but record more information to file so it's easy to search? Let me know what you think and I'll make the appropriate changes :) |
Hi @khornlund, I'm sorry for underestimating what you have suggested. This project is designed for use by people unfamiliar with pytorch (or python). It would be very difficult to such people that reading code written by others and determining which part is important, and which is not. That's why this project tries to hide most of its complex functionality into base classes, and why I think the usage of seemingly difficult modules (like logging) should be kept minimal and subtle. What I don't like for current model printing is not that it requires additional parameter(which was not true), but that it appears to do so. Someone who's not familiar with subclassing Let's discuss concrete things now.
Based on my experiment, defining logger instance in the
I really like this idea. I'd like to have console output having simple format (as before), and Having the configuration as separate yaml file is OK, but I think it could be better hidden somewhere other than project root directory, since it is not important in out project (in that it is not directly related to training model with pytorch). Maybe we can make Tell me if anything seems wrong or unclear. Please proceed as much as you are willing to do and leave comment, then I'll handle the rest (if you don't mind). |
Hi @SunQpark, I've made some new changes - comments below.
This is a fair point - logging gives you lots of cool functionality but it does increase the complexity.
Ok, I've removed the logger from the
This could work - it might simplify things a bit, but you lose some flexibility going this route. Usually multiple loggers are used so you can control the verbosity of each.
I've implemented this - compare the console output to what is logged to file.
I've reworked how this pathing works to address this - see
I think it makes sense to leave the logs outside the timestamp because if you continue training at a different time, it would be better to have a contiguous record of the training across multiple sessions. What do you think?
Ok, I don't have a good suggestion for this - I'll leave that to you to decide. How do you feel about the changes? :) |
I'm working on this now. Is this normal? Then what If not, how can I fix it? |
It will only capture things you log with level >= logging.ERROR eg.
Right now nothing does that, so it won't capture anything. You could wrap calls to train or test, and capture the errors, or just remove the handler to log errors (maybe this is the simplest). |
Configurable logging
Merged PR to the master with the following changes.(Commits are not added here because of my poor usage of git. Sorry.)
Trained models are being saved under
yaml seemed to be a good format, but does not comes with the python installation.
|
Hi guys,
This PR implements configurable logging via YAML.
In particular, it allows for:
I've removed the
train_logger
, as we can use different log levels (info/debug) and the verbosity config to control what is output, and by logging to a file we can avoid having to save the log state out as part of a checkpoint.Let me know what you think.
Cheers,
Karl