Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional TOML configs support #2457

Merged
merged 27 commits into from Jul 28, 2018
Merged

Optional TOML configs support #2457

merged 27 commits into from Jul 28, 2018

Conversation

@orsinium
Copy link
Contributor

@orsinium orsinium commented Jul 13, 2018

Description

  • Added parser for TOML configs.
  • TOML config parser can read files from luigi/base.toml and luigi/local.toml. This is useful for effective secrets management via VCS.
  • You can select config parser via LUIGI_CONFIG_PARSER env var.

Motivation and Context

  1. TOML standardized, modern format which supports data types.
  2. It is useful setup common default options in config files, not in tasks.
  3. Now we can write new config parsers for other formats like YAML without pain. Luigi has been more flexible and extendable.

Have you tested this? If so, how?

  1. I have included unit tests.
  2. I ran my jobs with this code and it works for me.
Copy link
Collaborator

@dlstadther dlstadther left a comment

@orsinium , Thanks for your submission!

I like the idea here (enabling other styles of configuration formats), but I'm not yet sold on why we'd want other formats.

Do other configuration formats allow features that ConfigParser doesn't? If so, what are they and how are they applicable to luigi?

(I anticipated your clarification to my concern above and I've gone ahead and reviewed your code. Those comments and questions are below.)

Loading

* ``/etc/luigi/luigi.toml``
* ``luigi.toml``
* ``luigi/base.toml``
* ``luigi/local.toml``
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why base.toml and local.toml are proposed as valid configuration file names?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because implicit better than explicit. Now we can use explicit inheritance via luigi.cfg and LUIGI_CONFIG_PATH=luigi_local.cfg. Motivation of inheritance I describe in Pull Request: this is useful for configuring default settings, that can be overridden inlocal config.

Loading

Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean that it uses local first and then base. Which makes sense so you can have local changes that don't conflict with the base config.

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm removing base and local configs. This need be discussed in future for better realization.

Loading

@@ -23,6 +41,17 @@ The config file is broken into sections, each controlling a different part of th
[core]
scheduler_host=luigi-host.mycompany.foo

Example toml config:

.. code:: python
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is python used here for code syntax highlighting?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because Pygments doesn't have available lexerfor TOML. This is already contributed in some Pygments versions:

But Pygments which used for travis checks doesn't have this lexer yet. I choose python as alternative lexer because toml syntax very similar to hybrid of python and ini :)

Loading

Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And a TODO then please. :)

Loading

streaming-jar = "/usr/lib/hadoop-xyz/hadoop-streaming-xyz-123.jar"
[core]
scheduler_host = "luigi-host.mycompany.foo"
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are single quotes acceptable too?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see TOML spec for more details about syntax

Loading

Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe link to the spec in the docs too?

Loading

@@ -0,0 +1,22 @@
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't these dates include 2018. Probably not that important

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied Apache license notice from original configuration.py. You can improve it, I don't now how it must be look in right way.

Loading

Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I think this is fine ...

Loading

if cls._instance is None:
cls._instance = cls(*args, **kwargs)
loaded = cls._instance.reload()
logging.getLogger('luigi-interface').info('Loaded %r', loaded)
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

side note: am i correct that neither the client nor the server's logging config names are configurable? I've found that interesting - why statically set to luigi-interface. Perhaps we don't want that changed in general.

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't change .instance() logic, just moved it from cfg parser to base.

Loading

Copy link
Collaborator

@dlstadther dlstadther Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was just raising an unrelated question - a topic for future possible PR (configurable logging names).

Nothing for you to do here.

Loading

return cls._instance

@classmethod
def add_config_path(cls, path):
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for changing the order of these classmethods?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What did you mean? I just moved this method from cfg parser to base without any changes.

Loading


class BaseParser:
_instance = None
_config_paths = []
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be an abstract property that is required to be overwritten?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this can be confusable if somebody try to use different parsers in one project. I fix it. Thank you for review :)

Loading

# limitations under the License.
#
import os
import os.path
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why import both os and os.path? import os.path can be removed

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly. import os can be removed, not os.path. Fixed. Thank you.

Loading

}


def get_config(parser=os.environ.get('LUIGI_CONFIG_PARSER', 'cfg')):
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am i correct that if you want to use a config parser other than cfg, you have to define the LUIGI_CONFIG_PARSER environmental variable?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right. I plan to contribute ability to set up config parser as argument for worker's run. But now Luigi supports config parser chainging only via env var. I write it into documentation.

Loading

setup.py Outdated
@@ -39,6 +39,7 @@ def get_static_files(path):
install_requires = [
'tornado>=4.0,<5',
'python-daemon<3.0',
'toml',
Copy link
Collaborator

@dlstadther dlstadther Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No version specification?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I think, we don't need specify library version. This library exposes an API familiar to users of the standard library json, marshal and pickle, and authors will not change it in future.

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 14, 2018

Yes, TOML, opposite to ini, supports data types like integer, float, boolean, list, dict. Ini supports only strings.

In future, I plan to contribute pyproject.toml support, when tools like pip and setuptools will migrate on it. Also you can read in this article other benefits of migration from ini to toml.

Thank you for good question :)

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 14, 2018

By the way, thank you for useful library, guys. Luigi is save many hours for our team :)

Loading

Copy link
Collaborator

@Tarrasch Tarrasch left a comment

Looks like a nice contribution. How long have you used this in prod?

Loading


in increasing order of preference. The order only matters in case of key conflicts (see docs for ConfigParser.read_). These files are meant for both the client and ``luigid``. If you decide to specify your own configuration you should make sure that both the client and ``luigid`` load it properly.
You can choose right parser via ``LUIGI_CONFIG_PARSER`` environment variable. For example, ``LUIGI_CONFIG_PARSER=toml``.
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we instead require that the file must end in .toml? It's very little magic for more convenience.

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think about it, but this is very explicit while we have many possible places for configs. What if we have luigi.cfg and luigi.toml? We can't combine it into one config.

Loading

Copy link
Collaborator

@dlstadther dlstadther Jul 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add an improvement down the road to define a hierarchical level of config extensions. I.e. (.cfg, .toml, .yml, <etc.>). I'm fine with requiring an ENV variable for now

Loading

* ``/etc/luigi/luigi.toml``
* ``luigi.toml``
* ``luigi/base.toml``
* ``luigi/local.toml``
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean that it uses local first and then base. Which makes sense so you can have local changes that don't conflict with the base config.

Loading

@@ -23,6 +41,17 @@ The config file is broken into sections, each controlling a different part of th
[core]
scheduler_host=luigi-host.mycompany.foo

Example toml config:

.. code:: python
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And a TODO then please. :)

Loading

streaming-jar = "/usr/lib/hadoop-xyz/hadoop-streaming-xyz-123.jar"
[core]
scheduler_host = "luigi-host.mycompany.foo"
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe link to the spec in the docs too?

Loading

@@ -0,0 +1,22 @@
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I think this is fine ...

Loading

import logging


class BaseParser:
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inherit from object?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No-no-no, don't do it! In Python2 cfg parser fails if BaseParser inherit from object. This is strange, yes. I will add comment for it

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

Loading

@@ -38,8 +37,10 @@
except ImportError:
from configparser import ConfigParser, NoOptionError, NoSectionError

from .base_parser import BaseParser
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recall relative imports are discouraged, no?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? We already have from .tools import range in project, and it's works. Relative imports little faster and better recognized some tools like isort.

Loading

setup.py Outdated
@@ -39,6 +39,7 @@ def get_static_files(path):
install_requires = [
'tornado>=4.0,<5',
'python-daemon<3.0',
'toml',
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be excluded? Eventually we want to populate this with dependencies needed but it's important that you can run luigi without toml installed.

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Loading

@@ -0,0 +1,34 @@
# -*- coding: utf-8 -*-
Copy link
Collaborator

@Tarrasch Tarrasch Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a better name than getter.py?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 15, 2018

Good question, thank you :) I used it only one day on the dev. I make this feature for our team and send this Pull Request immediately. I can give here feedback after 1-2 weeks of active usage. I think, this is a good idea, thank you.

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 16, 2018

Improved some things:

  • Dropped luigi/base.toml and luigi/local.toml paths. Inheritance must be discussed in future.
  • Added function for adding config in paths list of current active config parser.
  • Now TOML-parser's requirements optional for luigi and can be installed via extras: pip install luigi[toml]
  • For TOML parser we can get sections and options as attributes: config['mongo']['port'].
  • FIX catching config option getting in whole project: cfg parser raises NoSectionError, but TOML parser raises KeyError. Raise from new parsers exceptions from configparser -- bad idea.

Also you can run example project as proof-of-concept. For example, I am getting config['mongo']['port'] and this is returns int, not str. Soooo beautiful :)

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 16, 2018

codecov shows strange things. Don't trust it.

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 23, 2018

I'm fine. Codecov good and fair again :)

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 23, 2018

Today I'll try to run our new pipeline with this improvements. I'll give you feedback soon.

Loading

Copy link
Collaborator

@dlstadther dlstadther left a comment

Let us know the results of your pipeline execution as it relates to this PR :)

Loading

* ``luigi.toml``
* ``LUIGI_CONFIG_PATH`` environment variable

Both config lists reversordered by priotity (from low to high). The order only matters in case of key conflicts (see docs for ConfigParser.read_). These files are meant for both the client and ``luigid``. If you decide to specify your own configuration you should make sure that both the client and ``luigid`` load it properly.
Copy link
Collaborator

@dlstadther dlstadther Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reversordered is not a word. Perhaps we reword something to the effect of the below:

Both config lists increase in priority (from low to high).

Loading


# IMPORTANT: don't inherit from `object`!
# ConfigParser have some troubles in this case.
class BaseParser:
Copy link
Collaborator

@dlstadther dlstadther Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is explained better on SO

Loading

if cls._instance is None:
cls._instance = cls(*args, **kwargs)
loaded = cls._instance.reload()
logging.getLogger('luigi-interface').info('Loaded %r', loaded)
Copy link
Collaborator

@dlstadther dlstadther Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was just raising an unrelated question - a topic for future possible PR (configurable logging names).

Nothing for you to do here.

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 25, 2018

All is fine, our pipeline works right with new configs. I'm planning to commit dlstadther's improvements tomorrow.

Future improvements for other pull requests:

1. Local configs

I want to have file for local settings for storing some config in env

2. Configure logging via main config

Config example:

[logging]
version = 1
disable_existing_loggers = true

[logging.formatters.simple]
format = "{levelname:8} {asctime} {module}:{lineno} {message}"
style = "{"
datefmt = "%Y-%m-%d %H:%M:%S"

[logging.handlers.console]
level = "DEBUG"
class = "logging.StreamHandler"
formatter = "simple"

[logging.loggers.parser]
handlers = ["console"]
level = "DEBUG"
disabled = false
propagate = false

How we use it in our projects:

import os.path
import logging
from logging.config import dictConfig
import toml

path = os.path.dirname(os.path.abspath(__file__))
CONFIG = toml.load(os.path.join(path, 'config.toml'))

dictConfig(CONFIG['logging'])
logger = logging.getLogger('parser')

Loading

@orsinium orsinium requested a review from ulzha as a code owner Jul 26, 2018
@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 26, 2018

Added all actual @dlstadther 's improvements and merged spotify/master

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 26, 2018

@ulzha, sorry for mentioning. This is from my branch actualization

Loading

Copy link
Collaborator

@dlstadther dlstadther left a comment

Just a couple comments and clarifying questions

Loading


in increasing order of preference. The order only matters in case of key conflicts (see docs for ConfigParser.read_). These files are meant for both the client and ``luigid``. If you decide to specify your own configuration you should make sure that both the client and ``luigid`` load it properly.
You can choose right parser via ``LUIGI_CONFIG_PARSER`` environment variable. For example, ``LUIGI_CONFIG_PARSER=toml``.
Copy link
Collaborator

@dlstadther dlstadther Jul 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add an improvement down the road to define a hierarchical level of config extensions. I.e. (.cfg, .toml, .yml, <etc.>). I'm fine with requiring an ENV variable for now

Loading

@@ -486,7 +486,7 @@ def _get_s3_config(self, key=None):
defaults = dict(configuration.get_config().defaults())
try:
config = dict(configuration.get_config().items('s3'))
except NoSectionError:
except (NoSectionError, KeyError):
Copy link
Collaborator

@dlstadther dlstadther Jul 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am i correct to assume that ConfigParser returns the NoSectionError and toml returns the KeyError when a config section heading is missing?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I think, this is incorrect raise from toml parser any exceptions from ConfigParser. Maybe, we can make custom common exception:

class SectionError(NoSectionError, KeyError):
    pass

Loading

Copy link
Collaborator

@dlstadther dlstadther Jul 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with it as it is currently written; I was just verifying that they raised different errors for the same problem.

Loading

setup.py Outdated
@@ -75,6 +76,9 @@ def get_static_files(path):
]
},
install_requires=install_requires,
extras_require={
'toml': ['toml'],
Copy link
Collaborator

@dlstadther dlstadther Jul 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are your thoughts on limiting this to the current milestone version? i.e. 'toml<1.0' (since the current is 0.9.4) I just don't want version 1.0.0 to break this. I'd rather tread with caution, even with an optional configurable feature.

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python toml package version related to toml spec version. TOML spec v1.0 will be backward compatible. Some quotes:

1.0.0 will be the first version to support TOML 1.0.0. TOML still has yet to hit TOML 1.0.0 so its release is postponed indefinitely.

As of version 0.5.0, TOML should be considered extremely stable. The goal is for version 1.0.0 to be backwards compatible (as much as humanly possible) with version 0.5.0. All implementations are strongly encouraged to become 0.5.0 compatible so that the transition to 1.0.0 will be simple when that happens.

Loading

Copy link
Collaborator

@dlstadther dlstadther Jul 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, why don't we put 'toml<2.0'?

Loading

Copy link
Contributor Author

@orsinium orsinium Jul 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. This don't break anything. This fix added. Thank you for review :)

Loading

tox.ini Outdated
@@ -35,6 +35,7 @@ deps=
hypothesis[datetime]
selenium==3.0.2
pymongo==3.4.0
toml
Copy link
Collaborator

@dlstadther dlstadther Jul 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we agree to a version requirement in setup.py, it'll need to be reflected here.

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 27, 2018

I think, this PR is ready to merge. What do you think? :)

Loading

Copy link
Collaborator

@dlstadther dlstadther left a comment

I'm good with this now

Loading

@Tarrasch Tarrasch merged commit 7d2c557 into spotify:master Jul 28, 2018
4 checks passed
Loading
@Tarrasch
Copy link
Collaborator

@Tarrasch Tarrasch commented Jul 28, 2018

Good then! Let's see if this gets well used by othres. 🙌

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Jul 28, 2018

Thank you! We've done great work :)

Loading

thisiscab added a commit to glossier/luigi that referenced this issue Aug 3, 2018
thisiscab added a commit to glossier/luigi that referenced this issue Aug 8, 2018
dlstadther added a commit to dlstadther/luigi that referenced this issue Aug 14, 2018
* upstream-master: (82 commits)
  S3 client refactor (spotify#2482)
  Rename to rpc_log_retries, and make it apply to all the logging involved
  Factor log_exceptions into a configuration parameter
  Fix attribute forwarding for tasks with dynamic dependencies (spotify#2478)
  Add a visiblity level for luigi.Parameters (spotify#2278)
  Add support for multiple requires and inherits arguments (spotify#2475)
  Add metadata columns to the RDBMS contrib (spotify#2440)
  Fix race condition in luigi.lock.acquire_for (spotify#2357) (spotify#2477)
  tests: Use RunOnceTask where possible (spotify#2476)
  Optional TOML configs support (spotify#2457)
  Added default port behaviour for Redshift (spotify#2474)
  Add codeowners file with default and specific example (spotify#2465)
  Add Data Revenue to the `blogged` list (spotify#2472)
  Fix Scheduler.add_task to overwrite accepts_messages attribute. (spotify#2469)
  Use task_id comparison in Task.__eq__. (spotify#2462)
  Add stale config
  Move github templates to .github dir
  Fix transfer config import (spotify#2458)
  Additions to provide support for the Load Sharing Facility (LSF) job scheduler (spotify#2373)
  Version 2.7.6
  ...
dlstadther added a commit to dlstadther/luigi that referenced this issue Aug 14, 2018
* upstream-master:
  S3 client refactor (spotify#2482)
  Rename to rpc_log_retries, and make it apply to all the logging involved
  Factor log_exceptions into a configuration parameter
  Fix attribute forwarding for tasks with dynamic dependencies (spotify#2478)
  Add a visiblity level for luigi.Parameters (spotify#2278)
  Add support for multiple requires and inherits arguments (spotify#2475)
  Add metadata columns to the RDBMS contrib (spotify#2440)
  Fix race condition in luigi.lock.acquire_for (spotify#2357) (spotify#2477)
  tests: Use RunOnceTask where possible (spotify#2476)
  Optional TOML configs support (spotify#2457)
  Added default port behaviour for Redshift (spotify#2474)
  Add codeowners file with default and specific example (spotify#2465)
  Add Data Revenue to the `blogged` list (spotify#2472)
dlstadther added a commit to dlstadther/luigi that referenced this issue Aug 16, 2018
* upstream-master:
  Remove long-deprecated scheduler config variable alternatives (spotify#2491)
  Bump tornado milestone version (spotify#2490)
  Update moto to 1.x milestone version (spotify#2471)
  Use passed password when create a redis connection (spotify#2489)
  S3 client refactor (spotify#2482)
  Rename to rpc_log_retries, and make it apply to all the logging involved
  Factor log_exceptions into a configuration parameter
  Fix attribute forwarding for tasks with dynamic dependencies (spotify#2478)
  Add a visiblity level for luigi.Parameters (spotify#2278)
  Add support for multiple requires and inherits arguments (spotify#2475)
  Add metadata columns to the RDBMS contrib (spotify#2440)
  Fix race condition in luigi.lock.acquire_for (spotify#2357) (spotify#2477)
  tests: Use RunOnceTask where possible (spotify#2476)
  Optional TOML configs support (spotify#2457)
  Added default port behaviour for Redshift (spotify#2474)
  Add codeowners file with default and specific example (spotify#2465)
  Add Data Revenue to the `blogged` list (spotify#2472)
  Fix Scheduler.add_task to overwrite accepts_messages attribute. (spotify#2469)
  Use task_id comparison in Task.__eq__. (spotify#2462)
try:
import toml
except ImportError:
toml = False
Copy link

@adamist521 adamist521 Aug 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orsinium
This will end up following error if you have toml library not installed...
Works great after toml installation!

File "/Users/*****/.pyenv/versions/miniconda-latest/envs/ddhr-rjb/lib/python3.6/site-packages/luigi/configuration/toml_parser.py", line 53, in read
    self.data = self._update_data(self.data, toml.load(path))
AttributeError: 'bool' object has no attribute 'load'

Loading

Copy link
Collaborator

@Tarrasch Tarrasch Aug 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh noes, @orsinium can you fix this asap?

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Aug 25, 2018

@adamist521, please, can you provide full traceback?

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Aug 25, 2018

@Tarrasch, don't panic. This is unusual usage or something like this, because get_config function checks reader's enabled option. Also this error possible when you install Luigi without toml support, but want to use toml parser.

Hmmmm... I need to add info about toml support in installation section :)

Loading

@honnix
Copy link
Contributor

@honnix honnix commented Sep 6, 2018

This PR effectively fails from luigi.configuration import NoSectionError. Was this intended?

Loading

@orsinium
Copy link
Contributor Author

@orsinium orsinium commented Sep 6, 2018

Yes. Please, import NoSectionError from configparser from stdlib:

from configparser import NoSectionError```

Loading

tanmay2893 added a commit to lexentbio/luigi that referenced this issue Oct 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants