Configure logging via TOML config #2483

orsinium · 2018-08-07T16:05:30Z

Description

Now you can fully configure logging from TOML-config.
Option no_configure_logging disable any logging configuring.
Separate logging configuration from other code for better maintainability.
Also I update Apache copyright notice from my previous PR. Cindicator's leader recommend to use "Vote Inc." company name in copyrights.

Motivation and Context

Q. Why do you use same file for Luigi's config and logging config?
A. Because this is all project config. Also in future Pull Requests I plan to integrate in Luigi configs inheritance which allow also use separate file for logging, local settings or something else.

Q. Why do you do this? We already have support for logging.conf.
A. Because logging via ini-file is ugly, since ini format doesn't support inherit dicts. But TOML (and YAML. JSON and other modern formats) can do it.

Q. Why do you change no_configure_logging behavior?
A. Option no_configure_logging useful when you're configure logging in project bypass to Luigi. For example, when you want to use JSON or YAML configs for it. This is intuitive behavior for option with this name.

Q. Why do you change code structure?
A. Old code structure is hard to test, explore and extend. Now it's beautiful :)

Have you tested this? If so, how?

I have included unit tests.
Don't trust me! I update example repository, and you can play with new logging configuring yourself.

And...

Thank you for best pipeline library. And best music streaming service too. All this things is amazing!

Pygments doesn't supports TOML by default https://gitlab.com/gitlab-org/gitlab-pygments-rb/commit/d460ba68716937e59498f95fd34709d98990dea3

Tarrasch · 2018-08-08T06:03:30Z

I'm in favor of this. I'll have to review another time though. :)

orsinium · 2018-08-08T10:21:25Z

Now all tests is fine :)

orsinium · 2018-08-09T10:23:05Z

Thank you for review :) All your advice applied, tests is passed. Do you have any other improvements?

orsinium · 2018-08-20T14:34:23Z

@Tarrasch, how is your review? ^^

Tarrasch

So cool to see that we're getting somewhere with improving the logging experience in luigi!

I just added a lot of comments on the code so that the code for this important core part of luigi keeps a high standard. I hope you can find time to address this stuff. I'm happy to merge (without more approval from me) once the issues are addressed.

Thanks!

Tarrasch · 2018-08-21T17:17:02Z

doc/logging.rst

+5. logging_conf_file option
+6. ``logging`` section
+7. ``--log-level``
+8. log_level option


Thanks for doing the deep-dive and finding these things out.

Tarrasch · 2018-08-21T17:50:30Z

luigi/setup_logging.py

+try:
+    from ConfigParser import NoSectionError
+except ImportError:
+    from configparser import NoSectionError


I recall this was a standard trick, can you link to some stack overflow thread or something?

Alternatively say # codepath for python < 2.7 something

Tarrasch · 2018-08-21T17:51:56Z

luigi/setup_logging.py

+
+
+class DaemonLogging(BaseLogging):
+    configured = False


Is this internal state? Make it _configured maybe?

Tarrasch · 2018-08-21T17:52:20Z

luigi/setup_logging.py

+
+class DaemonLogging(BaseLogging):
+    configured = False
+    log_format = "%(asctime)s %(name)s[%(process)s] %(levelname)s: %(message)s"


Can you document if you can override this? If it's private do as commented above.

Tarrasch · 2018-08-21T17:52:46Z

luigi/setup_logging.py

+        return configured
+
+
+class DaemonLogging(BaseLogging):


Maybe document this is to be used for luigid

Tarrasch · 2018-08-21T17:56:20Z

luigi/setup_logging.py

+        return True
+
+    @classmethod
+    def _default(cls, opts):


add a more descriptive name please

Tarrasch · 2018-08-21T17:57:40Z

test/cmdline_test.py

+        InterfaceLogging.config = get_config()
+
+    def _clean_config(self):
+        DaemonLogging.config = LuigiTomlParser()


Looks weird. Where does toml come in in this generic context?

Should code like this be in a class TomlParserCmdlineTest(unittest.TestCase): context or something??

It is for creating minimal config for tests. It's here for old tests.

Tarrasch · 2018-08-21T18:00:45Z

test/setup_logging_test.py

+        self.assertTrue(result)
+
+
+class PatchedLogging(InterfaceLogging):


Was it possible to mock one layer less?

Tarrasch · 2018-08-21T18:02:54Z

test/cmdline_test.py

-
-    @mock.patch("warnings.warn")
-    @mock.patch("luigi.interface.setup_interface_logging")
-    def test_cmdline_logger(self, setup_mock, warn):


It's so much code that it's hard to review it all. But can you make sure that all tests you delete are added somewhere else?

Yes, all logging well tested in test/setup_logging_test.py

Tarrasch · 2018-08-21T18:06:43Z

test/setup_logging_test.py

+        result = self.cls._cli(opts)
+        self.assertFalse(result)
+
+    def test_section(self):


This test name setup_logging_test.TestDaemonLogging.test_section could have more context if you rename test_section to maybe test_section_returns_true_with_config and another test test_section_returns_false_without_config.

I know we don't follow this style at most places in this code base. But code testing improvements can start somewhere. If you want you can do this is too much you can do this as a follow up.

It's test all what we need test for _section method

The point is that test_section is not saying much compared to test_section_returns_true_with_config.

orsinium · 2018-08-24T16:42:10Z

Added some @Tarrasch's improvements. Other requests commented

orsinium · 2018-09-28T15:37:21Z

CI is OK now. What about this PR? :)

dlstadther · 2018-09-28T16:35:09Z

I'm good here.

Have you been using this feature in production? All is good for your use?

orsinium · 2018-09-28T16:48:50Z

Now I'm in another team in the same company. I'll get actual feedback on this feature by next few days :)

dlstadther · 2018-09-28T17:07:37Z

Congrats on your move @orsinium ! Appreciate the feedback

orsinium · 2018-11-05T08:39:06Z

I got feedback and have fixed example based on this:

disable_existing_loggers should be false for saving already existing loggers.
dot in luigi.server logging means inheritance so we can just configure only luigi logger.

I think, it will be better change luigi-interface to luigi.interface, but I don't do it in this PR because it can broke logging in existing projects.

So. now it is ready, I hope :)

orsinium · 2018-11-22T11:18:40Z

@dlstadther, @Tarrasch, what are we going to do with this Pull Request?

dlstadther · 2018-11-25T02:35:59Z

I just looked back through this PR and I think it LGTM. Given @Tarrasch had some changes requested, I'll wait for him to give his approval that his comments were addressed.

Thanks for continuing to follow up @orsinium !

Tarrasch

I think things can be improved (as always). But anyway you've waited long already (sorry for my slow review), and this PR is actually a big improvement. Thanks!

Tarrasch · 2018-11-25T11:07:29Z

luigi/interface.py

-            'core', 'no_configure_logging', False):
-        setup_interface_logging(logging_conf, env_params.log_level)
+
+    InterfaceLogging.setup(env_params)


Oh nice you even cleaned up this part!

Tarrasch · 2018-11-25T11:09:12Z

luigi/setup_logging.py

+#
+"""
+This module contains helper classes for configuring logging for luigid and
+workers via command line arguments and options from config files.


Do these docs render nicely? Can you be explicit about if other's can use these methods or not?

Tarrasch · 2018-11-25T11:09:54Z

luigi/setup_logging.py

+        return True
+
+    @classmethod
+    def _default(cls, opts):


(just a reminder to feel free to do this)

honnix · 2018-12-18T16:58:28Z

luigi/interface.py

@@ -38,31 +36,7 @@
 from luigi import worker
 from luigi import execution_summary
 from luigi.cmdline_parser import CmdlineParser
-
-
-def setup_interface_logging(conf_file='', level_name='DEBUG'):


A quick question. We call this method directly to set up logging. What is the equivalence now? Or it is not needed anymore?

orsinium · 2018-12-18T18:56:02Z

Yeah, of course, we call InterfaceLogging instead of:

luigi/luigi/interface.py

Line 144 in 235edcc

InterfaceLogging.setup(env_params)

orsinium added 30 commits July 13, 2018 13:45

mv cfg parser

d00f793

split cfg parser and config getter

783ddeb

+TOML config parser

fd4e30f

improve config getter

d712827

+Apache License in new files

bcb59e0

TOML config parser tests

0242248

override configs by keys

debea65

Documentation for TOML config parser

ddfb5be

Merge branch 'config'

d0caa72

FIX lexer error

3261393

Pygments doesn't supports TOML by default https://gitlab.com/gitlab-org/gitlab-pygments-rb/commit/d460ba68716937e59498f95fd34709d98990dea3

FIX lexer error

edf41c8

Pygments doesn't supports TOML by default https://gitlab.com/gitlab-org/gitlab-pygments-rb/commit/d460ba68716937e59498f95fd34709d98990dea3

FIX: no shared instance between different config parsers

1d0d110

rm unused os import

6142834

Merge remote-tracking branch 'origin/config'

d86414a

+getitem for config

b88376a

improve config paths

f4c128a

add config function

ba20f7b

sort imports

22b7368

make additional parsers optional

95fa46b

Merge branch 'improve-config'

beeb92f

improve add_config_path for default parser

913fe19

Merge branch 'improve-config'

03fb9ba

FIX package installation

ae050b0

Merge branch 'improve-config'

a5c85f9

FIX TOML parser raises KeyError, not NoSectionError

284cc26

Merge branch 'improve-config'

4b43134

Merge branch 'improve-config' into config

0d6943a

some improvements after review.

a7047fa

Merge branch 'config'

135d7d1

Merge remote-tracking branch 'spotify/master' into config

1c9129b

FIX pass env_params into logging setup from retcodes

0c8df8f

orsinium changed the title ~~WIP: Configure logging via TOML config~~ Configure logging via TOML config Aug 8, 2018

FIX docstrings style

b7a6c47

Tarrasch suggested changes Aug 21, 2018

View reviewed changes

orsinium added 4 commits August 24, 2018 18:25

+info about config parser import

7bee2aa

make configured option internal

b521db7

make log format private

b42c201

review

d1d2e09

Merge remote-tracking branch 'spotify/master' into logging

9c5f755

orsinium added 2 commits November 5, 2018 11:30

Merge remote-tracking branch 'spotify/master' into logging

00608a3

update config example

9d6139b

Merge remote-tracking branch 'spotify/master' into logging

235edcc

Tarrasch approved these changes Nov 25, 2018

View reviewed changes

Tarrasch merged commit 0a098f6 into spotify:master Nov 25, 2018

orsinium deleted the logging branch November 29, 2018 17:24

honnix mentioned this pull request Dec 12, 2018

AttributeError during logging setup with Python 2.7 #2592

Closed

honnix reviewed Dec 18, 2018

View reviewed changes

Configure logging via TOML config #2483

Configure logging via TOML config #2483

orsinium commented Aug 7, 2018

Tarrasch commented Aug 8, 2018

orsinium commented Aug 8, 2018

orsinium commented Aug 9, 2018

orsinium commented Aug 20, 2018

Tarrasch left a comment

Tarrasch Aug 21, 2018

Tarrasch Aug 21, 2018

Tarrasch Aug 21, 2018

Tarrasch Aug 21, 2018

Tarrasch Aug 21, 2018

Tarrasch Aug 21, 2018

Tarrasch Aug 21, 2018

orsinium Aug 24, 2018

Tarrasch Aug 21, 2018

Tarrasch Aug 21, 2018

orsinium Aug 24, 2018

Tarrasch Aug 21, 2018

orsinium Aug 24, 2018

Tarrasch Aug 25, 2018

orsinium commented Aug 24, 2018

orsinium commented Sep 28, 2018

dlstadther commented Sep 28, 2018

orsinium commented Sep 28, 2018

dlstadther commented Sep 28, 2018

orsinium commented Nov 5, 2018

orsinium commented Nov 22, 2018

dlstadther commented Nov 25, 2018

Tarrasch left a comment

Tarrasch Nov 25, 2018

Tarrasch Nov 25, 2018

Tarrasch Nov 25, 2018

honnix Dec 18, 2018

orsinium commented Dec 18, 2018

		self.assertTrue(result)


		class PatchedLogging(InterfaceLogging):

Configure logging via TOML config #2483

Configure logging via TOML config #2483

Conversation

orsinium commented Aug 7, 2018

Description

Motivation and Context

Have you tested this? If so, how?

And...

Tarrasch commented Aug 8, 2018

orsinium commented Aug 8, 2018

orsinium commented Aug 9, 2018

orsinium commented Aug 20, 2018

Tarrasch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orsinium commented Aug 24, 2018

orsinium commented Sep 28, 2018

dlstadther commented Sep 28, 2018

orsinium commented Sep 28, 2018

dlstadther commented Sep 28, 2018

orsinium commented Nov 5, 2018

orsinium commented Nov 22, 2018

dlstadther commented Nov 25, 2018

Tarrasch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orsinium commented Dec 18, 2018