DEBUG cannot be enabled by env var, even though it is said to be possible

[imrehg]

The docs say about settings, that

All settings that you can find in the bonobo.settings module. You can override those settings using environment variables.

I was trying to set DEBUG for a while, and was wondering why it was turning out false all the time, while other settings (such as PROFILE) were correctly picked up.

Tested it with

# debugtest.py
import bonobo
import logging

logger = logging.getLogger()


def extract():
    yield "test"


if __name__ == "__main__":
    print(bonobo.settings.DEBUG)
    print(bonobo.settings.PROFILE)
    parser = bonobo.get_argument_parser()
    with bonobo.parse_args(parser) as options:
        bonobo.run(bonobo.Graph(extract), services={})

and running:

$ bonobo run debugtest.py --env DEBUG=True --env PROFILE=True
<Setting DEBUG=False>
<Setting PROFILE=True>
 - extract in=1 out=1 [done] 
 `-> Memory: 31.80 Mb

On the other hand:

$ bonobo --debug run debugtest.py --env PROFILE=True
DEBG:0000:root: Command: run Arguments: {'command': 'run', 'file': 'debugtest.py', 'mod': None, 'default_env_file': None, 'default_env': None, 'env_file': None, 'env': ['PROFILE=True'], 'quiet': False, 'verbose': False, 'install': False}
<Setting DEBUG=True>
<Setting PROFILE=True>
DEBG:0000:root: Creating execution strategy 'threadpool'...
DEBG:0000:bonobo.execution.contexts.node: Node loop starts for <NodeExecutionContext(+extract)>.
DEBG:0000:bonobo.execution.contexts.node: Node loop ends for <NodeExecutionContext(-extract) in=1 out=1>.
 - extract(0) in=1 out=1 [done] 
 `-> Memory: 31.86 Mb

Looking at the code, I think this argument parsing:

bonobo/bonobo/commands/__init__.py

Line 23 in 274058d

parser.add_argument("--debug", "-D", action="store_true")

basically overrides that env var for DEBUG, and basically blocks

bonobo/bonobo/settings.py

Line 81 in 274058d

DEBUG = Setting("DEBUG", formatter=to_bool, default=False)

from working.

But this is my hunch of what's going on, either way, the env var way of setting debug does not seem to be effective.

Versions

• Bonobo version:

  bonobo v.0.7.0rc2

• Python version:

  CPython 3.8.2 (default, Mar 25 2020, 18:15:24)
[Clang 11.0.0 (clang-1100.0.33.16)]

• Platform:

  Darwin 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64 (MacOS 10.15.4)

[imrehg]

Digging a bit more, I guess this might be actually misunderstanding, as seems like I was using --env incorrectly:

If the env var is just applied as a normal env var, things work:

$ DEBUG=true bonobo run debugtest.py
DEBG:0000:stevedore.extension: found extension EntryPoint.parse('convert = bonobo.commands.convert:ConvertCommand')
DEBG:0000:stevedore.extension: found extension EntryPoint.parse('download = bonobo.commands.download:DownloadCommand')
DEBG:0000:stevedore.extension: found extension EntryPoint.parse('examples = bonobo.commands.examples:ExamplesCommand')
DEBG:0000:stevedore.extension: found extension EntryPoint.parse('init = bonobo.commands.init:InitCommand')
DEBG:0000:stevedore.extension: found extension EntryPoint.parse('inspect = bonobo.commands.inspect:InspectCommand')
DEBG:0000:stevedore.extension: found extension EntryPoint.parse('run = bonobo.commands.run:RunCommand')
DEBG:0000:stevedore.extension: found extension EntryPoint.parse('version = bonobo.commands.version:VersionCommand')
DEBG:0000:root: Command: run Arguments: {'command': 'run', 'file': 'debugtest.py', 'mod': None, 'default_env_file': None, 'default_env': None, 'env_file': None, 'env': None, 'quiet': False, 'verbose': False, 'install': False}
<Setting DEBUG=True>
<Setting PROFILE=False>
DEBG:0000:root: Creating execution strategy 'threadpool'...
DEBG:0000:bonobo.execution.contexts.node: Node loop starts for <NodeExecutionContext(+extract)>.
DEBG:0000:bonobo.execution.contexts.node: Node loop ends for <NodeExecutionContext(-extract) in=1 out=1>.
 - extract(0) in=1 out=1 [done] 

(not exactly the same --debug, actually it gives more logs)

The confusion was:

• for DEBUG, either DEBUG=True bonobo run .... or bonobo --debug run ... has to be used, --env DEBUG=True doesn't take effect
• for other variables, ENV=something bonobo ... or bonobo run --env ... (or other environment settings that are available) can be used, but --env shouldn't really be used for this, but env vars for the actual graph....

This might warrant some documentation clarifications, but does not feel like a bug anymore, sorry for the noise!

[hartym]

The issue you faced is because arguments are parsed by bonobo.parse_args(parser).

You should be able to see the values you wanted to see if you do print(bonobo.settings.DEBUG) within the with block. If this does not work, then it's a bug.

[imrehg]

Retried again, moving the test inside the with block:

# debugtest.py
import bonobo

def extract():
    yield "test"

if __name__ == "__main__":
    parser = bonobo.get_argument_parser()
    with bonobo.parse_args(parser) as options:
        print(bonobo.settings.DEBUG)
        bonobo.run(bonobo.Graph(extract), services={})

resulting in these outcomes of the 4 different cases of applying env vars, that I understood:

• DEBUG=true bonobo run debugtest.py, DEBUG env var is set ✅
• bonobo --debug run debugtest.py, DEBUG env var is set ✅
• bonobo run debugtest.py --env DEBUG=true, DEBUG env var is not set (stays default false) ❌
• bonobo run debugtest.py --env-file env (with env containing DEBUG=true) DEBUG env var is not set (stays default false) ❌