Implement quiet mode with reduced console output (fixes #4)

[f4lco]

Hello there 👋

I'm interested in a quiet mode and make a proposal using the stock logging module. The output format stays the same, in addition to the following behaviors:

• No additional argument says "verbose output". This is a good default to newcomers and for debugging.
• -q eliminates all informational output, but keeps displaying all errors / test failures.
• -qq is dedicated to heavily automated environments, which only rely on the exit code of treon.

[amit1rrr]

@f4lco Thank you for your contribution. Couple of suggestions,

• I don't think -qq is required. Even in heavily automated CI system most users would want to have some ability to look at failures etc. We can just keep a single -q/--quiet mode that suppresses all informational output.

• We should print the result summary even under -quiet mode. Result summary is short and it explicitly shows which notebooks ran under treon. It avoids situation where test script succeeded without actually running anything.

[f4lco]

@amit1rrr I understand. I do like the suggested behavior you just laid out, but I do not think that is how a "quiet" mode supposedly works.

I suggest the following: without arguments, show a minimum about of which notebooks are run, and the result table (compare with pytest, for instance). Have a --verbose option, which includes all the details. What do you think?

[amit1rrr]

I suggest the following: without arguments, show a minimum about of which notebooks are run, and the result table (compare with pytest, for instance).

I like this. Those two things you mentioned and stack trace in case of failures is a good default. So default output would look something like,

Testing /workspace/keras/overfit.ipynb
<error tracebcks if any>
Testing /workspace/keras/underfit.ipynb
<error tracebcks if any>

<result table>

Have a --verbose option, which includes all the details. What do you think?

Sure. It would have all the output from the default prepended with some more details like below,

Executing treon version 0.1.0
Recursively scanning /workspace/treon/tmp/docs/ for notebooks...

-----------------------------------------------------------------------
Collected following Notebooks for testing
-----------------------------------------------------------------------
/workspace/treon/tmp/docs/tools/templates/subsite/g3doc/tutorials/notebook.ipynb
/workspace/treon/tmp/docs/site/ru/guide/keras.ipynb
/workspace/treon/tmp/docs/site/ru/guide/eager.ipynb

<followed by default output>

--verbose seems a bit superfluous now but if we setup the logging levels as you have done we can add more useful outputs in future that would help with debugging the failures or even treon issues itself.

What do you think?

@sgugger I remember you requesting this feature at one point. I hope this proposal works for you. Feel free to chime in if you have any comments.

[sgugger]

This proposal seems great and perfectly in line with what I had suggested at one point!

[f4lco]

@amit1rrr finally, seems like we are on the same page now 🙂 I updated the PR accordingly!

[amit1rrr]

Looks good. Couple of minor suggestions.

[amit1rrr]

Thanks for the contribution @f4lco. I have merged in the changes. Will update the package on PyPI in some time.

[f4lco]

Thank you very much for your time and patience, @amit1rrr ! Don't worry, it's not urgent 😉

[amit1rrr]

@f4lco @sgugger FYI, this is release in latest treon version 0.1.2

Thanks!