-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create a separate *.log file for the worker node *.stdouts. #528
Conversation
4dc03ec
to
276b8e3
Compare
@ntouran Michael has explained his reasoning, but I am curious what you think of this change to our logging. |
This is reasonable in concept for sure. I think it is worth trying to get the same to be synced up with the caseTitle though, because otherwise it will be easy to miss in archiving files, and multiple runs of in the same dir (e.g. param sweeping) will overwrite one another. (though honestly running concurrently in the same dir probably is a bad idea regardless). |
@mgjarrett What do you think of Nick's idea? |
Yeah I think it would be best to have the |
I like this idea. In retrospect, it seems like the However... the Maybe that could be called inside One question I have is WHY isn't this already set to armi/cases/case.py: runLog.LOG.startLog(self.cs.caseTitle)
armi/operators/operator.py: runLog.LOG.startLog(self.cs.caseTitle) |
I think startLog is the log in the execution directory. The logs that Mike is reading are in the fast path. |
Sorry for the late questions on this, but on the discussion level, can we talk a little bit more about the motivation of this? Ideally we'd answer these questions in implementation docstrings in the code itself so it lives on.
|
The goal is not to have ~hundreds of files, but to separate the current The unit tests are passing, but I haven't yet verified that it works with MPI with the latest commit. |
The |
There are times when the workers are useful... so i don't want to get rid of the data entirely. Its just appending all the workers to the main standard does not enable the use of the data though since a lot of time you are just looking at the stdout to see how your run is and then it updates, but the file gets like 50x larger. Setting workers to "error" sounds like an okay solution, but a lot of times you may want the data, but just don't want to view it at the same time you are viewing the normal stdout. There should be an option to store the worker data without having to view it with the main stdout data. |
I say we go ahead with this PR. Since it's just 2 files, that's not a huge QA burden, and even with branchVerbosity options it can be a bit verbose. If stdout gets too big and you're trying to refresh it over a slow VPN there a good case to be made to have them be separated. I would like to see a follow-up ticket, maybe for our internal plugins, that says "get rid of all these totally useless xtra messages in the worker nodes" :) |
Just a heads up, we're still working on getting the |
Hear, hear! As any ecosystem ages, it's important to clean out unused log messages, and push the rarely-used ones to debug-level. |
Ok so is it fully baked now after those last two commits? Ready to be merged? |
Needed one more commit because the |
|
Is it something I can fix? It looks like a bunch of files unrelated to this PR. |
I have already run
If you look at the output from this |
I have fixed the It was unrelated to this PR. |
Description
At the end of a run, all of the logs from the individual MPI processes get concatenated together and appended to the end of the
*.stdout
. If there are a lot of MPI processes, this can make the*.stdout
obnoxiously bloated, with a bunch of information that the user would rarely ever want to look at.Instead, this PR puts all of those worker node logs into a separate file,
.\logs\armi-mpi-workers.log
It would probably be best if the file had the name of the case (i.e., the same as the
*.stdout
and*.stderr
), but I don't know if that name is available to the logger at this level. It's just writing those to the streamssys.stdout
andsys.stderr
. If somebody has a better idea for naming I'm all ears; this was just kind of a temporary name to make sure the code worked.Checklist
If user exposed functionality was added/changed:
doc
folder.setup.py
.