Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Much improved output formatting #45

Merged
merged 51 commits into from Oct 27, 2023
Merged

Much improved output formatting #45

merged 51 commits into from Oct 27, 2023

Conversation

Chroxvi
Copy link
Contributor

@Chroxvi Chroxvi commented Oct 19, 2023

This is a major rework of how we handle outputting information to the console. It consist of:

  • Cotainr build CLI options --verbose/--quiet to adjust the verbosity of the output to the console.
  • Cotainr build CLi option --log-to-file to also save the stdout/stderr messages to files in addition to showing them on the console.
  • Colored console output (colored by severity of message) by default with cotainr build CLI option --no-color to disable colored output.
  • A Python logging based setup for handling all messages being passed within cotainr, handled at the standard Python log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.
  • Passing of verbosity levels to Singularity and Conda.
  • Filtering of messages from Conda to remove partial progress bars and various ANSI escape codes that mess up the console output.
  • Basic Cotainr build CLI progress messages.
  • A console spinner which we prepend to the output message from the Cotainr build CLI while we are waiting for the next output message.
  • A development documentation page detailing how all of this works.
  • A few minor clean-ups.

Once this PR is merged, we should close #32 as this PR supersedes #32.

@Chroxvi Chroxvi requested review from rloewe and eskech October 19, 2023 11:48
@Chroxvi Chroxvi changed the title Much import output formatting Much improved output formatting Oct 20, 2023
assert stdout == stdout_spin_and_final_line
assert stderr == stderr_spin_and_final_line

@pytest.mark.parametrize("spins", [1])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the spins variable slightly confusing when looking at the code. It took me a little bit to figure out that it was an argument to patch_fix_number_of_message_spins. There is a comment about it in the patch file, but I wonder if there should be one where it is used as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added comments for all test cases that make use of this "override a fixture with direct test parametrization"-approach to passing a variable from the test case to the fixture.

See https://stackoverflow.com/questions/18011902/how-to-pass-a-parameter-to-a-fixture-function-in-pytest for a discussion of alternative ways to achieve the same result.

Copy link
Contributor

@eskech eskech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beside the small comment it looks nice.

"increase the verbosity of the output from cotainr. "
"Can be used multiple times: Once for subprocess output, "
"twice for subprocess INFO, three times for DEBUG, "
"and four times for TRACE"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add an example like -v -v e.g. for INFO. is a bit hard to understand. Not a blocker.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I have added it to the backlog.

from . import util

logger = logging.getLogger(__name__)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be a matter of style and different in python but should it not be ok just to have the static logger and alway call on that and not create a copy/ref here? - Not a blocker.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually does provide a static (hierarchy of) logger(s). The logging module keeps track of all loggers added via logging.getLogger and returns a reference to a logger if it has already been created (which is also why we need to reload the logging modules in our tests, to reset this internal state). The pattern logger = logging.getLogger(__name__) is just the recommended way to maintain a hierarchy of loggers in line with the package/module hierarchy (see https://docs.python.org/3/library/logging.html#logger-objects for details).

# When the command to be run in the container provides its own
# log_dispatcher
with custom_log_dispatcher.prefix_stderr_name(
prefix=self.__class__.__name__
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for a future change should we create better names for the outb`put than use the class name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable. I wasn't able to come up with better short names than the class names, though.

Regex from https://stackoverflow.com/a/14693789
"""

ansi_escape_re = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all regex needs a comment on what we are trying to match. Even if we link to stackoverflow. Not a blocker

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a task to the backlog to properly document regex's with examples and a humanly readable template of what we are trying to match.

"""

progress_bar_re = re.compile(
r"^(.+?)\|(.+?)\|[ \#0-9]+?\|[ ]{1,3}[0-9]{1,2}\%"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please comment on what the regex is doing. for a quick read it is hard to understand. - not a blocker.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a task to the backlog to properly document regex's with examples and a humanly readable template of what we are trying to match.

@eskech eskech merged commit aae08d2 into main Oct 27, 2023
11 checks passed
@eskech eskech deleted the output_formatting branch October 27, 2023 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants