Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pandas is no longer importable with -OO optimization #21071

Closed
shoyer opened this Issue May 16, 2018 · 10 comments

Comments

Projects
None yet
8 participants
@shoyer
Copy link
Member

shoyer commented May 16, 2018

Code Sample, a copy-pastable example if possible

In your shell:

$ python -OO -c 'import pandas'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/__init__.py", line 42, in <module>
    from pandas.core.api import *
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/api.py", line 10, in <module>
    from pandas.core.groupby.groupby import Grouper
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/groupby/__init__.py", line 2, in <module>
    from pandas.core.groupby.groupby import (
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 46, in <module>
    from pandas.core.index import (Index, MultiIndex,
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/index.py", line 2, in <module>
    from pandas.core.indexes.api import *
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/indexes/api.py", line 4, in <module>
    from pandas.core.indexes.base import (Index,
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 7, in <module>
    from pandas._libs import (lib, index as libindex, tslib as libts,
  File "pandas/_libs/index.pyx", line 28, in init pandas._libs.index
  File "pandas/_libs/tslibs/period.pyx", line 59, in init pandas._libs.tslibs.period
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/tseries/offsets.py", line 1092, in <module>
    class CustomBusinessMonthEnd(_CustomBusinessMonth):
  File "/Users/shoyer/miniconda3/envs/xarray-py36/lib/python3.6/site-packages/pandas/tseries/offsets.py", line 1093, in CustomBusinessMonthEnd
    __doc__ = _CustomBusinessMonth.__doc__.replace('[BEGIN/END]', 'end')
AttributeError: 'NoneType' object has no attribute 'replace'

Problem description

-OO optimization strips out docstrings, which may give a minor performance boost (I honestly don't know). Nonetheless, users requested that xarray import properly properly with the -OO flag (pydata/xarray#1706), so we added a regression test that caught this in the latest pandas release (pydata/xarray#1708).

Expected Output

Pandas should be imported without any errors.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: 3.5.0
pip: 9.0.1
setuptools: 39.0.1
Cython: None
numpy: 1.14.2
scipy: None
pyarrow: None
xarray: None
IPython: 6.3.0
sphinx: None
patsy: None
dateutil: 2.7.2
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@shoyer shoyer changed the title pandas is not longer importable with -OO pandas is no longer importable with -OO optimization May 16, 2018

@WillAyd

This comment has been minimized.

Copy link
Member

WillAyd commented May 16, 2018

Hmm OK. Just running locally it looks like it could be fixed adding a condition in 3 places that assume we have a docstring, but don't if you run with those flags.

Just out of curiosity do you only have this in your travis configuration or have you placed it somewhere in your unit testing? Think it would be easier to track if we had the latter but off the top of my head not sure how to do that - curious if you have any insight

@shoyer

This comment has been minimized.

Copy link
Member Author

shoyer commented May 16, 2018

You could probably test this with a Python subprocess, but I didn’t bother for xarray so it’s just in our Travis config.

@TomAugspurger TomAugspurger added this to the 0.23.1 milestone May 16, 2018

@WillAyd WillAyd referenced this issue May 16, 2018

Merged

Support for OO Optimization #21093

4 of 4 tasks complete

@jreback jreback added the Compat label May 16, 2018

@bryevdv

This comment has been minimized.

Copy link

bryevdv commented May 17, 2018

FWIW in case it's helpful: Bokeh had to define a format_docstring function to avoid direct access to docstrings:

https://github.com/bokeh/bokeh/blob/master/bokeh/util/string.py#L84-L100

and added a test to maintain working imports with -OO

https://github.com/bokeh/bokeh/blob/master/tests/test_python_execution_with_OO.py

@jorisvandenbossche

This comment has been minimized.

Copy link
Member

jorisvandenbossche commented May 17, 2018

@bryevdv thanks for the links. @mattpap pointed out the same test file in the PR, that is certainly useful.
We have some machinery (decorators) to deal with docstring manipulations, we should just be careful to always use it :-)

@Siecje

This comment has been minimized.

Copy link

Siecje commented Jun 7, 2018

I ran into this bug when packaging an application with cx_Freeze. I have reverted to pandas==0.22.0 for now.

@jorisvandenbossche

This comment has been minimized.

Copy link
Member

jorisvandenbossche commented Jul 8, 2018

@stonebig you removed your comment, does that mean it was false alarm?

@stonebig

This comment has been minimized.

Copy link
Contributor

stonebig commented Jul 8, 2018

my problem is still there, but I'm not so sure if this is THIS root cause. I can leave it again : pyviz/holoviews#2738

uploading typical problem in 40' at https://sourceforge.net/projects/winpython/files/WinPython_3.7/3.7.0.1/betas/WinPython64-3.7.0.1rc.exe

@stonebig

This comment has been minimized.

Copy link
Contributor

stonebig commented Jul 8, 2018

living outside miniconda brings original issues, sorry.

@jorisvandenbossche

This comment has been minimized.

Copy link
Member

jorisvandenbossche commented Jul 8, 2018

How is the issue that you posted above related to this one? You don't get the error as in the top post?

@stonebig

This comment has been minimized.

Copy link
Contributor

stonebig commented Jul 8, 2018

the relationship is that I followed the cause of bokeh not being tested against pandas-0.23.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.