Configure log path via env var, and catch filesystem errors #975

dsherry · 2020-07-24T14:34:21Z

Fix #956

We'll now support the ability to a) configure and disable the log file and b) ignore file logging if there are filesystem permissions errors during setup.

Behavior

Default: As before, if the env var EVALML_LOG_FILE is not set, the default behavior will be to write logs to evalml_debug.log in the current directory.
Configurable log location: If the env var EVALML_LOG_FILE is set to a valid file path before importing evalml, evalml will write debug-level logs to that location.
- This will occur regardless of whether the file specified already exists, as long as the parent directory exists and is writeable.
Error handling: If at any point the chosen path is a) a directory or b) the program doesn't have write access in the parent directory, a warning and stacktrace will be logged, and logger setup will continue without a debug log file
Disabling: if the env var EVALML_LOG_FILE is set to an empty value, logging setup will not attempt to create a log file.

Examples
Default behavior: log to evalml_debug.log in current repo

python demo.py

Log to a different location (must be filepath, not dir):

EVALML_LOG_PATH="/path/to/logfile.log" python demo.py

To suppress log output entirely:

EVALML_LOG_PATH="" python demo.py

codecov · 2020-07-24T14:40:18Z

Codecov Report

Merging #975 into main will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##             main     #975    +/-   ##
========================================
  Coverage   99.85%   99.86%            
========================================
  Files         179      179            
  Lines        9283     9424   +141     
========================================
+ Hits         9270     9411   +141     
  Misses         13       13

Impacted Files	Coverage Δ
evalml/tests/utils_tests/test_logger.py	`100.00% <100.00%> (ø)`
evalml/utils/logger.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1f1adb...eada442. Read the comment docs.

…silient to filesystem errors.

freddyaboulton

@dsherry I don't think I'm getting the intended behavior when I set EVALML_LOG_FILE to a filepath that does not exist or a directory. Let me know if I am misunderstanding!

evalml/utils/logger.py

freddyaboulton · 2020-07-30T15:10:37Z

evalml/utils/logger.py

        logger.setLevel(logging.DEBUG)

+        evalml_log_path_str = os.environ.get('EVALML_LOG_FILE', 'evalml_debug.log')


I think it would be helpful to print the log file so the user knows where to check.

I agree, that would be helpful. However I don't think we should do that in this PR.

This code gets called once per static invocation of the logger, i.e. once per evalml source file. If we add a print here, it'll occur multiple times. I'd prefer that sort of thing to happen only once. If we want to do that, we'd need a global config check to happen before this code is ever called. We don't have a "config" object in evalml yet. I believe featuretools does.

You are reminding me: we have to add something to the user guide or API docs about logging. I will include that in this PR. That should help a lot.

Yea that's a really good point. For the time being, we display the logfile path whenever we encounter an error during scoring in AutoML search so that should help the user find the logfile when they need it.

evalml/tests/utils_tests/test_logger.py

evalml/utils/logger.py

jeremyliweishih

LGTM other than the fixes that @freddyaboulton proposed!

jeremyliweishih · 2020-07-30T15:28:23Z

@dsherry might need to kick off the build_docs test again. Sometimes it does this.

dsherry · 2020-07-30T16:07:56Z

Yeah thanks @jeremyliweishih , more circleci network hiccups. They seem to have increased a bunch as of a few weeks ago.

angela97lin

Left one comment about potentially using the tmp_dir fixture if applicable but otherwise LGTM :)

angela97lin · 2020-07-30T16:27:51Z

evalml/tests/utils_tests/test_logger.py

+def test_get_logger_path_valid(mock_RotatingFileHandler, monkeypatch, logger_env_cleanup):
+    assert os.environ.get('EVALML_LOG_FILE') is None
+
+    with tempfile.TemporaryDirectory() as temp_dir:


Not 100% sure this is applicable here but there's a tmp_dir fixture you might be able to use: https://docs.pytest.org/en/stable/tmpdir.html :) We've used it in other tests for creating a temporary directory!

dsherry · 2020-07-30T18:37:43Z

evalml/utils/logger.py

+        if len(evalml_log_path_str) == 0:
+            return logger
+        if evalml_log_path.is_dir() or not os.access(evalml_log_path.parent, os.W_OK):
+            print(f'Warning: cannot write debug logs to path "{evalml_log_path}". ' + warning_msg)


This is a bit ugly because it prints out once for every file in our codebase which imports and uses the logger. But its a fine starting place. Better to surface the warning here than to be silent. Thanks @freddyaboulton !

Looks good @dsherry !

freddyaboulton

Looks great!

dsherry · 2020-07-30T19:25:49Z

docs/source/user_guide/utilities.ipynb

+    "\n",
+    "EvalML uses [the standard python logging package](https://docs.python.org/3/library/logging.html). By default, EvalML will log `INFO`-level logs and higher (warnings, errors and critical) to stdout, and will log everything to `evalml_debug.log` in the current working directory.\n",
+    "\n",
+    "If you want to change the location of the logfile, before import, set the `EVALML_LOG_FILE` environment variable to specify a filename within an existing directory in which you have write permission. If you want to disable logging to the logfile, set `EVALML_LOG_FILE` to be empty. If the environment variable is set to an invalid location, EvalML will print a warning message to stdout and will not create a log file."


I added a page to the guide

Might be a little more clear to specify that it needs to be set to an empty string

dsherry force-pushed the ds_956_logger_permissions_fix branch from 4b1121f to 25d37cf Compare July 24, 2020 14:34

dsherry force-pushed the ds_956_logger_permissions_fix branch from 25d37cf to c04c581 Compare July 28, 2020 22:38

dsherry added 4 commits July 30, 2020 09:38

Update logger to have configurable log path via env var, and to be re…

0d07241

…silient to filesystem errors.

Logger tests

2fe0252

Update tests to clear out logger state and clean up env var

010c583

Clean up fixtures

5d0d4a8

dsherry force-pushed the ds_956_logger_permissions_fix branch from c04c581 to 5d0d4a8 Compare July 30, 2020 14:13

dsherry added 4 commits July 30, 2020 10:17

Add an exception handler test

6b1faef

Add a test for empty env var

a15d091

Fix fixture usage

3d88861

Cleanup for last commit

0b212e3

dsherry marked this pull request as ready for review July 30, 2020 14:31

dsherry requested review from angela97lin, freddyaboulton and jeremyliweishih July 30, 2020 14:31

auto-assign bot assigned dsherry Jul 30, 2020

freddyaboulton suggested changes Jul 30, 2020

View reviewed changes

jeremyliweishih approved these changes Jul 30, 2020

View reviewed changes

Changelog

a489b4a

dsherry added 2 commits July 30, 2020 12:13

Remove unneeded line

2186596

Move line up for simplicity

550e66e

angela97lin approved these changes Jul 30, 2020

View reviewed changes

Special-case empty string. Show warning for other cases

ca23ab5

dsherry commented Jul 30, 2020

View reviewed changes

dsherry requested review from freddyaboulton, angela97lin and jeremyliweishih July 30, 2020 18:37

lint

a68b5e2

freddyaboulton approved these changes Jul 30, 2020

View reviewed changes

Add utilities page to user guide

eada442

dsherry commented Jul 30, 2020

View reviewed changes

dsherry merged commit 7b6a469 into main Jul 30, 2020

dsherry deleted the ds_956_logger_permissions_fix branch July 30, 2020 20:02

angela97lin mentioned this pull request Jul 31, 2020

Release v0.12.0 #1007

Merged

		logger.setLevel(logging.DEBUG)

		evalml_log_path_str = os.environ.get('EVALML_LOG_FILE', 'evalml_debug.log')

Configure log path via env var, and catch filesystem errors #975

Configure log path via env var, and catch filesystem errors #975

Uh oh!

Conversation

dsherry commented Jul 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jul 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

freddyaboulton left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

freddyaboulton Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

dsherry Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

freddyaboulton Jul 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jeremyliweishih left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremyliweishih commented Jul 30, 2020

Uh oh!

dsherry commented Jul 30, 2020

Uh oh!

angela97lin left a comment

Choose a reason for hiding this comment

Uh oh!

angela97lin Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

dsherry Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

freddyaboulton Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

freddyaboulton left a comment

Choose a reason for hiding this comment

Uh oh!

dsherry Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

jeremyliweishih Jul 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dsherry commented Jul 24, 2020 •

edited

Loading

codecov bot commented Jul 24, 2020 •

edited

Loading

freddyaboulton left a comment •

edited

Loading

freddyaboulton Jul 30, 2020 •

edited

Loading

jeremyliweishih left a comment •

edited

Loading