BUG: Creating a pandas.DataFrame affects warnings filters

[anmyachev]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import warnings


def test():
    warnings.warn("Some message")

def test2():
    warnings.warn("Some message")
    pd.DataFrame()  # this call somehow affect 

print(f"First case")

for _ in range(5):
    test()

print(f"\nSecond case with constructing pandas.DataFrame")

for _ in range(5):
    test2()

Issue Description

Looks like it change default filter for UserWarning to always or something similar.

Output of the reproducer:

First case
...\test_warnings2.py:6: UserWarning: Some message
  warnings.warn("Some message")

Second case with constructing pandas.DataFrame
...\test_warnings2.py:9: UserWarning: Some message
  warnings.warn("Some message")
...\test_warnings2.py:9: UserWarning: Some message
  warnings.warn("Some message")
...\test_warnings2.py:9: UserWarning: Some message
  warnings.warn("Some message")
...\test_warnings2.py:9: UserWarning: Some message
  warnings.warn("Some message")
...\test_warnings2.py:9: UserWarning: Some message
  warnings.warn("Some message")

Expected Behavior

I would expect the following output:

First case
...\test_warnings2.py:6: UserWarning: Some message
  warnings.warn("Some message")

Second case with constructing pandas.DataFrame

Installed Versions

pandas 2.1.2
Python 3.9.17

[jrmylow]

This is interesting, the repeated warnings only crop up when test2 is called, and test is unaffected:

import pandas as pd
import warnings

def test():
    warnings.warn('test warning')

def test2():
    warnings.warn('test warning 2')
    pd.DataFrame()

print('First warning')
for _ in range(5):
    test()

print('Second warning')
for _ in range(5):
    test2()

print('Testing first warning again')
for _ in range(5):
    test()

Output:

First warning
...\t.py:5: UserWarning: test warning
  warnings.warn('test warning')
Second warning
...\t.py:8: UserWarning: test warning 2
  warnings.warn('test warning 2')
...\t.py:8: UserWarning: test warning 2
  warnings.warn('test warning 2')
...\t.py:8: UserWarning: test warning 2
  warnings.warn('test warning 2')
...\t.py:8: UserWarning: test warning 2
  warnings.warn('test warning 2')
...\t.py:8: UserWarning: test warning 2
  warnings.warn('test warning 2')
Testing first warning again
...:5: UserWarning: test warning
  warnings.warn('test warning')

[jrmylow]

After some bisecting, #51917 is the PR that introduced this warning in the first place

[jrmylow]

take

[jrmylow]

This bug exists in the standard library [0]. It has a very interesting history and is due to the way that the warnings module handles __warningregistry__, which gets flushed/cleared every time the filter context changes [1].

Two options:

• Patch pandas/core/dtypes/common.py to not use warnings.catch_warnings() (and possibly other areas where this context manager is used); or
• Patch cpython to update the behaviour (PR already in the works [2])

Would welcome discussion/comments on this

[0] Known issue in cpython: cpython/issues/73858
[1] The original issue where the code was raised: issue4180
[2] PR in progress that can resolve this in cpython: 8232

[anmyachev]

@jrmylow Thanks for the research!

The first option will require changing the code in many Python libraries (at least in all pandas dependencies). Let's try to find out first if there is any expected (nearest) time for solving this problem in python itself. I asked the author about this in python/cpython#8232.