-W option cannot use non-standard categories

[remram44]

BPO	22543
Nosy	@pitrou, @Bluehorn, @gerritholl, @mitya57, @mmerickel, @remram44, @kernc, @ozars

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-10-02.21:32:00.695>
labels = ['3.7', 'type-bug', 'library', '3.9']
title = '-W option cannot use non-standard categories'
updated_at = <Date 2022-03-11.23:18:27.139>
user = 'https://github.com/remram44'

bugs.python.org fields:

activity = <Date 2022-03-11.23:18:27.139>
actor = 'mmerickel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2014-10-02.21:32:00.695>
creator = 'remram'
dependencies = []
files = []
hgrepos = []
issue_num = 22543
keywords = []
message_count = 8.0
messages = ['228261', '228286', '228287', '228326', '276416', '330700', '359316', '414945']
nosy_count = 10.0
nosy_names = ['pitrou', 'THRlWiTi', 'torsten', 'Gerrit.Holl', 'mitya57', 'mmerickel', 'remram', 'kernc', 'ozars', 'Feng Yu']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22543'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.9']

[remram44]

warnings._processoptions is called very early, before site-packages are enabled. Because of this, using a non-standard 'category' will almost certainly fail with the message:

Invalid -W option ignored: invalid module name: '...'

The -W option would be a lot more useful if it could actually match non-standard categories (it does, after all, pretend to support modulename.classname).

I don't see any easy way of fixing this, other than initializing the warnings module later or matching category names with the given string (and getting rid of the import).

[pitrou]

How would this work? Would it auto-import the module?

[remram44]

It already does auto-import, but it does it before site-packages are set up, meaning that it fails in any practical setup.

See _getcategory(), called by _processoptions(): https://hg.python.org/cpython/file/b15c5a66213f/Lib/warnings.py#l148

[brettcannon]

Doing the initialization later is not really an option as there can be warnings in the parser so it has to be resolved very early on.

I guess some form could be a possibility, but that's also tricky as subclass matching also still needs to work. I think we would need to see a patch on the proposed solution before considering its acceptance.

Another option would to parse it twice: once early on for Python internals and then again later once site-packages and various other things have been set up. Once again I would need to look at a patch before making a decision on whether it's an acceptable solution.

[Bluehorn]

Wow, this was news to me and I just ran into it in python 2.7. Checked in Python 3 and it's still there:

(py3)->torsten.landschoff@horatio:~$ python3 --version
Python 3.6.0a3+
(py3)->torsten.landschoff@horatio:~$ python3 -W error::sqlalchemy.exc.SAWarning -c "print()"
Invalid -W option ignored: invalid module name: 'sqlalchemy.exc'

I see no easy way to fix this. One way I thought about is to have hooks for importing of modules (called when a module is put into sys.modules, not sure if there is something like this already) and have the warning system trigger on that.

There is no way to raise an exception before it is imported anyway...

[gerritholl]

There is a thread on StackOverflow related to this problem:

https://stackoverflow.com/q/42495641/974555

The (currently only and accepted) answer proposes as a workaround to set the PYTHONPATH variable, as the contents of those apparently *are* already in sys.path at the time warnings._processoptions is called, unlike site-packages. So manually adding site-packages to the PYTHONPATH variable does circumvent this problem (tested in Python 3.7.1).

[FengYu]

Coming here from astropy/astropy#9832 (comment)

Importing the named module during warning initialization is somewhat like code injection. And it may introduce unintended side-effects -- for example if this causes some 3rd party libraries to initialize too early?

String matching on the class name of every class in the hierarchy is safer, as there is no 'expected' code execution.

[mmerickel]

Updated affected versions as I ran into this on 3.9.7.

[astrojuanlu]

This is still an issue in Python 3.11.

[jlumpe]

Any plans to fix this? It seems like string matching on the warning class' __module__ and __name__ attributes as suggested by @fengyu is a much better solution and very easy to implement.