Cython extensions built twice?

[jseabold]

It looks like the Cython extensions get built in both the build step and the install step. Need to look into this.

[jseabold]

This is due to our conditional checks for the C compiler. If has_c_compiler is True, then we call cython no matter what the command given to distutils is. We could have a check to see if the C file is stale and then build it if "build" is in sys.argv, but I'd rather get rid of this check entirely and fix this in a sane way for 0.5.0

[jseabold]

Yet another problem with the conditional Cython hacks. It doesn't look like we ever get the Cython version of things in Python 3 at least on Linux. At least with newest Cython versions. The try_compile line returns

*** ValueError: underlying buffer has been detached

I still want to get rid of all this cruft and just require a compiler. That way we can also close #266, which has been hanging around a bit long for a useful contribution.

[josef-pkt]

which cython version? TravisCI looks like it's doing fine with cython.

I still don't really like it because it make testing and switching branches and python versions more complicated for me. And I'm still the main test runner.

[jseabold]

There's no error because it's in a try/except loop. It just won't build Cython extension.

It doesn't change anything for you AFAICT. You just need to do the same stuff within a SDK CMD shell. Or you can modify the build scripts I've committed.

[josef-pkt]

The difference is that currently I'm not compiling at all when I just want to quickly switch versions or branches.

I will look into it.

[jseabold]

If you don't touch the pyx files, then you don't have to worry about it. Though I'm not certain, you'll likely have to recompile or keep different folders for Python 3 if you don't want to, though it doesn't add more than a few seconds to the build as of now.

[josef-pkt]

actually, python 3 got smarter they keep the byte code files version specific in parallel. (But I'm keeping python 3 versions in separate directories

Right now I'm running python 2.7 with compiled extensions, and python 2.6 with pure python, and sometimes python 2.5 with pure python.

But your right, switching branches with the same python doesn't need recompile, if I don't touch the extensions.

[rgommers]

@jseabold is this still an issue with latest numpy/cython? Do you happen to use Macports?

[rgommers]

Similar issue: luispedro/mahotas#25

[jseabold]

Yes this is still an issue, but I think it's avoidable when we remove the hacked compiler checks IIRC. I see it on Linux when doing the compiler check under 64-bit on Python 3.2. (To reproduce, you actually need to remove the try/except block that's swallowing the error). I do not know what's causing it yet. Was planning to dig into this if necessary when we can finally merge #266 and require a compiler to build.

[stromnov]

On OSX it's a problem with setuptools dist.py (in handle_display_option).

For some reason, this fails on OSX with py3.2:

import sys
sys.stdout.detach()

[rgommers]

Same for the mahotas issue, but then with sys.stdout.fileno(). Looks like a Python bug then. Not clear to me yet if it's Macports specific there; with python.org Python I haven't seen this.

[stromnov]

Yes, it looks like Python 3.x bug.

It's the complex bug: after sys.stdout.detach() exception just swallowed, and reraised on next sys.stdout access (in sys.stdout.fileno() in numpy's distutils extension).

As a draft workaround you can add this to setuptools/dist.py (line 660):

return _Distribution.handle_display_options(self, option_order)

[rgommers]

It seems my setuptools is different than yours. Can you give some context for your suggested workaround?

[rgommers]

This was reported before, but apparently core devs don't approve of using detach: https://groups.google.com/forum/?fromgroups=#!topic/comp.lang.python/QHyrZhN79D0

It's recommended in the Python docs though: http://docs.python.org/3.2/library/sys.html

[stromnov]

def handle_display_options(self, option_order):

    ...skipped...

    # Stdout may be StringIO (e.g. in tests)
    import io
    if not isinstance(sys.stdout, io.TextIOWrapper):
        return _Distribution.handle_display_options(self, option_order)

    return _Distribution.handle_display_options(self, option_order)

[josef-pkt]

I have problems following here.

We have statsmodels running on python 3.2 for a long time. I just checked again and it build from an sdist including compiling extensions with mingw32 on python 3.2.3 without problems.
I did all the python 3.2 conversion and testing with windows.

I'm using numpy 1.6.2 to install and cython 0.15.1 on py2.6 to build the sdist.

Did building extensions for python 3.2 never work for Mac and Linux, or are there new problems with versions of packages newer than what I use?

[cdeil]

What is the expected output of this code?

import sys
sys.stdout.detach()

On Ubuntu with Python 3.2 and on Mac with Macports Python 3.1, 3.2 and 3.3 I see the ValueError:

$ python3.2
Python 3.2.3 (default, Sep 14 2012, 09:29:22) 
[GCC 4.2.1 Compatible Apple Clang 4.0 ((tags/Apple/clang-421.0.60))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='US-ASCII'>
>>> sys.stdout.detach()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: underlying buffer has been detached
>>> sys.stdout
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: underlying buffer has been detached
>>> exit()
Exception ValueError: 'underlying buffer has been detached' in $

Is that OK or a bug?

For reference: mahotas and scikit-image can't be installed with Macports Python 3.2 because of this issue:
https://trac.macports.org/ticket/37022#comment:4

Since several scientific python packages are affected, maybe the discussion on this issue should be continued somewhere more central, e.g. the python or numpy tracker?

[josef-pkt]

I get the same ValueError on Windows with python 3.2.3.
But so far this hasn't prevented me from installing a sdist of statsmodels in python 3.2

[josef-pkt]

another question: what setuptools/dist.py are you talking about?

I have setuptools in distribute for python 3.2 and 3.2.3 but the dist.py file in there doesn't define handle_display_options, and I don't find anywhere a usage of detach

python\Lib\distutils.dist.py handle_display_options doesn't use stdout.

Unless all this modules are platform specific.
???

[rgommers]

@cdeil it's a Python bug, following up on the Python bug tracker would be great. There's issues being opened all over the place because of this.

[g2p]

Not a Python bug. See this comment.

[cdeil]

Like @josef-pkt in #204 (comment) I wonder what has changed in python or numpy or somewhere else so that these problems come up now several years after Python 3 came out. Does anyone know?

[rgommers]

This has the best explanation (change in distribute): pypa/virtualenv#359

[rgommers]

I take back the Python bug comment, I misunderstood. (happens to me sometimes if there's no docstrings.......)

[josef-pkt]

I'm glad this has been cleared up, and it's not a problem in our code.

Thanks for tracking this.

[josef-pkt]

do we need this? I didn't look at the details (yet)

pandas-dev/pandas#2365

[rgommers]

Testing against numpy master may show other failures, so better to first check that since this issue doesn't prevent building statsmodels.

[jseabold]

I've only followed this at a high level, but, IIUC, it will prevent building when we drop having duplicate python/cython code or merge in the cython code that doesn't have python back up (which I've wanted to do for 6 months now).

[rgommers]

Correct. If Cython becomes a build-time dependency, copy @y-p's workaround and live with a few failures I'd say.

[josef-pkt]

just cross-linking here (I haven't looked at any details)

Instead of building cython twice (if this is really the case), there is also a possible problem to get cython build at various stages (possibly not at all)
scipy/scipy@c012492#commitcomment-2869722

[jseabold]

While it's always good practice to double check me, this is and has been really the case...you can check the code as well. There's no distinction between build and install because I don't check sys.argv (and don't want to because this will, in turn, be fragile). I still want to get rid of this. From our INSTALL.txt.

Optional Dependencies
---------------------

cython >= 0.15.1

    http://cython.org/

    If Cython is available during installation from source faster functions 
    will be installed in some places provided you have a C compiler. It is 
    recommended to install from source with Cython available, as this will not
    be optional in the 0.5 release.

Current master. Clean checkout.

[~] 
|1 $ cd statsmodels/statsmodels
[~/statsmodels/statsmodels] (master)
|2 $ git status .
# On branch master
nothing to commit (working directory clean)
[~/statsmodels/statsmodels] (master)
|3 $ git log --oneline -n 1
d59bd37 Merge pull request #731 from jseabold/handle-wrapped
[~/statsmodels/statsmodels] (master)

Build step output

[~/statsmodels/statsmodels] (master)
|4 $ python setup.py build
compiling '_configtest.c':

/* This file is generated from statsmodels/tools/_build.py to */
void do_nothing(void);
int main(void) {
    do_nothing();
    return 0;
}

C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
success!
removing: _configtest.c _configtest.o
cython -o /home/skipper/statsmodels/statsmodels/statsmodels/nonparametric/fast_linbin.c /home/skipper/statsmodels/statsmodels/statsmodels/nonparametric/fast_linbin.pyx
Appending statsmodels.nonparametric configuration to statsmodels
Ignoring attempt to set 'name' (from 'statsmodels' to 'statsmodels.nonparametric')
compiling '_configtest.c':

/* This file is generated from statsmodels/tools/_build.py to */
void do_nothing(void);
int main(void) {
    do_nothing();
    return 0;
}                                                                               

C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC                                                           

compile options: '-c'
gcc: _configtest.c
success!
removing: _configtest.c _configtest.o
cython -o /home/skipper/statsmodels/statsmodels/statsmodels/tsa/kalmanf/kalman_loglike.c /home/skipper/statsmodels/statsmodels/statsmodels/tsa/kalmanf/kalman_loglike.pyx
Appending statsmodels.tsa.kalmanf configuration to statsmodels.tsa
Ignoring attempt to set 'name' (from 'statsmodels.tsa' to 'statsmodels.tsa.kalmanf')
Appending statsmodels.tsa configuration to statsmodels
Ignoring attempt to set 'name' (from 'statsmodels' to 'statsmodels.tsa')
non-existing path in 'statsmodels': 'sandbox/panel/test_data.txt'
Appending statsmodels configuration to 
Ignoring attempt to set 'name' (from '' to 'statsmodels')
non-existing path in '': 'docs/build/htmlhelp/statsmodelsdoc.chm'
non-existing path in '': 'statsmodels/statsmodelsdoc.chm'
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "statsmodels.nonparametric.fast_linbin" sources
building extension "statsmodels.tsa.kalmanf.kalman_loglike" sources
building data_files sources
build_src: building npy-pkg config files
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/statsmodels

<snip>

running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'statsmodels.nonparametric.fast_linbin' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/statsmodels
creating build/temp.linux-x86_64-2.7/statsmodels/nonparametric
compile options: '-I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gcc: statsmodels/nonparametric/fast_linbin.c
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1726:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/fast_linbin.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:26:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/fast_linbin.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__multiarray_api.h:1609:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ufuncobject.h:311:0,
                 from statsmodels/nonparametric/fast_linbin.c:258:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__ufunc_api.h:236:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/statsmodels/nonparametric/fast_linbin.o -o build/lib.linux-x86_64-2.7/statsmodels/nonparametric/fast_linbin.so
building 'statsmodels.tsa.kalmanf.kalman_loglike' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-2.7/statsmodels/tsa
creating build/temp.linux-x86_64-2.7/statsmodels/tsa/kalmanf
compile options: '-I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gcc: statsmodels/tsa/kalmanf/kalman_loglike.c
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1726:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:26:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__multiarray_api.h:1609:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ufuncobject.h:311:0,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:258:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__ufunc_api.h:236:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/statsmodels/tsa/kalmanf/kalman_loglike.o -o build/lib.linux-x86_64-2.7/statsmodels/tsa/kalmanf/kalman_loglike.so

Install step. Here we go again...

[~/statsmodels/statsmodels] (master)
|5 $ sudo python setup.py install
[sudo] password for skipper: 
compiling '_configtest.c':

/* This file is generated from statsmodels/tools/_build.py to */
void do_nothing(void);
int main(void) {
    do_nothing();
    return 0;
}

C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
success!
removing: _configtest.c _configtest.o
cython -o /home/skipper/statsmodels/statsmodels/statsmodels/nonparametric/fast_linbin.c /home/skipper/statsmodels/statsmodels/statsmodels/nonparametric/fast_linbin.pyx
Appending statsmodels.nonparametric configuration to statsmodels
Ignoring attempt to set 'name' (from 'statsmodels' to 'statsmodels.nonparametric')
compiling '_configtest.c':

/* This file is generated from statsmodels/tools/_build.py to */
void do_nothing(void);
int main(void) {
    do_nothing();
    return 0;
}

C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-c'
gcc: _configtest.c
success!
removing: _configtest.c _configtest.o
cython -o /home/skipper/statsmodels/statsmodels/statsmodels/tsa/kalmanf/kalman_loglike.c /home/skipper/statsmodels/statsmodels/statsmodels/tsa/kalmanf/kalman_loglike.pyx
Appending statsmodels.tsa.kalmanf configuration to statsmodels.tsa
Ignoring attempt to set 'name' (from 'statsmodels.tsa' to 'statsmodels.tsa.kalmanf')
Appending statsmodels.tsa configuration to statsmodels
Ignoring attempt to set 'name' (from 'statsmodels' to 'statsmodels.tsa')
non-existing path in 'statsmodels': 'sandbox/panel/test_data.txt'
Appending statsmodels configuration to 
Ignoring attempt to set 'name' (from '' to 'statsmodels')
non-existing path in '': 'docs/build/htmlhelp/statsmodelsdoc.chm'
non-existing path in '': 'statsmodels/statsmodelsdoc.chm'
running install
Checking .pth file support in /usr/local/lib/python2.7/dist-packages
/usr/bin/python -E -c pass
TEST PASSED: /usr/local/lib/python2.7/dist-packages appears to support .pth files
running bdist_egg
running egg_info
running build_src
build_src
building extension "statsmodels.nonparametric.fast_linbin" sources
building extension "statsmodels.tsa.kalmanf.kalman_loglike" sources
building data_files sources
build_src: building npy-pkg config files
creating statsmodels.egg-info
writing statsmodels.egg-info/PKG-INFO
writing top-level names to statsmodels.egg-info/top_level.txt
writing dependency_links to statsmodels.egg-info/dependency_links.txt
writing manifest file 'statsmodels.egg-info/SOURCES.txt'
reading manifest file 'statsmodels.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory 'dist'
warning: no previously-included files found matching 'docs/source/generated/*'
warning: no previously-included files matching '*' found under directory 'docs/build'                                                                           
warning: no previously-included files matching '*' found under directory 'docs/build/htmlhelp'                                                                  
warning: no files found matching 'statsmodels/statsmodelsdoc.chm'
warning: no files found matching 'statsmodels/sandbox/panel/test_data.txt'
warning: no previously-included files matching '*~' found anywhere in distribution                                                                              
warning: no previously-included files matching '*.swp' found anywhere in distribution                                                                           
warning: no previously-included files matching '*.pyc' found anywhere in distribution                                                                           
warning: no previously-included files matching '*.bak' found anywhere in distribution                                                                           
writing manifest file 'statsmodels.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying statsmodels/version.py -> build/lib.linux-x86_64-2.7/statsmodels
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
building 'statsmodels.nonparametric.fast_linbin' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gcc: statsmodels/nonparametric/fast_linbin.c
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1726:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/fast_linbin.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:26:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/fast_linbin.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__multiarray_api.h:1609:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ufuncobject.h:311:0,
                 from statsmodels/nonparametric/fast_linbin.c:258:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__ufunc_api.h:236:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/statsmodels/nonparametric/fast_linbin.o -o build/lib.linux-x86_64-2.7/statsmodels/nonparametric/fast_linbin.so
building 'statsmodels.tsa.kalmanf.kalman_loglike' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

compile options: '-I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gcc: statsmodels/tsa/kalmanf/kalman_loglike.c
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1726:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:26:0,
                 from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__multiarray_api.h:1609:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
In file included from /usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/ufuncobject.h:311:0,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:258:
/usr/local/lib/python2.7/dist-packages/numpy/core/include/numpy/__ufunc_api.h:236:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/statsmodels/tsa/kalmanf/kalman_loglike.o -o build/lib.linux-x86_64-2.7/statsmodels/tsa/kalmanf/kalman_loglike.so
creating build/bdist.linux-x86_64
<snip>

Installed /usr/local/lib/python2.7/dist-packages/statsmodels-0.5.0-py2.7-linux-x86_64.egg
Processing dependencies for statsmodels==0.5.0
Finished processing dependencies for statsmodels==0.5.0

[josef-pkt]

why do you do a separate build python setup.py build?

python setup.py install already does the build.

In the past, I always got a separate, second build with other packages, unless I specified --skip-build, but a quick look doesn't show it in the python documentation (for py 2.6 and 2.7) and install after build shouldn't rebuild.

[jseabold]

Because if you do a system-wide install with sudo, then you do not have permissions to remove the local build folder that is created in this step. With compiled extensions, you're "supposed" to do it in separate steps. It should at least not be broken to use this in the manner intended.

You shouldn't get a separate second build. If that were the case, numpy, scipy, and pandas users would be screaming bloody murder.

[jseabold]

This really shouldn't be so contentious. What if I go the next mile and offer to make every effort to add nightly windows binaries to my cron jobs, so that you can just grab a nightly build if some compiled extension changes and then you never have to install the SDK and follow the same steps from instead its shell instead of the CMD prompt (or use the build scripts already provided)?

[josef-pkt]

It's contentious because it's a big jump. Anyone who wants to install (and work on) master will need a full developement environment, or mess around with an installed version (that's what I do when I want to change anything in scipy).

Currently, it's possible to use the python version for development and a binary distribution (in Linux, Mac or Windows) for "production" and performance.

Once a compiler is required, we have to deal with the compilation issues, instead of avoiding them.

For my testing a binary distribution would be too complicated, since tox and pip cannot install them. However, since I had problems installing pandas on Windows 64 bit, I added a 32bit python 3.3 virtualenv for which I can compile with MingW.

Given the number of request for Pandas nightly binaries on the pydata mailing list, I assume that we need to provide updated binaries also. Nightly sounds a bit too much for me, but weekly or even monthly would be necessary.

What I would really like to know, is whether we actually have (m)any users/developers, besides me, that run the pure python version, because building the extensions (or creating the environment for it) is too much work.

[jseabold]

Maybe it was contentious a year ago when we discussed this, but now it shouldn't be I don't think. This was always the plan, but now I'm forcing the issue.

Setting up a development environment on Windows really isn't that bad, and I'm glad to support windows (and have so far), but I don't think it should hold back development or dictate development policy.

Dealing with compilation issues is fine. We haven't heard of many yet, and I haven't run into any other than maybe a stray "you need to upgrade Cython." It works now and has always worked on all platforms. There's not going to be much to do here for people, provided they read and follow the given instructions (which are much better than many packages provide IMO). At some point though users have to help themselves if they want to work with source. (See my recent two-day journey building OpenBlas/UMFPACK on the MLs). This is the price of being on the bleeding edge. People will help each other and figure things out.

That said, part of my plan for 0.6 is to implement this #372. Then people will not need to build statsmodels at all to test add-on models.

There are build scripts for every supported Python version 32-bit and 64-bit on windows. Surely there is a way to incorporate them into your testing process?

https://github.com/statsmodels/statsmodels/tree/master/tools

My worry is that we have many people inadvertently running the pure Python version because they didn't install Cython and then we get FUD numbers out there in blog posts doing comparisons for things like KDE and ARIMA.

[josef-pkt]

At some point though users have to help themselves if they want to work with source. (See my recent two-day journey building OpenBlas/UMFPACK on the MLs). This is the price of being on the bleeding edge. People will help each other and figure things out.

Or it might not happen, It took you two days on Linux
I gave up after trying to install umfpack on Windows after half a day and scikits.sparse and other packages are scary enough with there warnings that I don't even try.
I also tried to build matplotlib once for half a day before I gave up.
And if you try to ask for help with building on Windows, then it would be silence in the forest, except for Christoph Goelkhe, and David Cournapeau a few years ago.

Contributing to something where the bleeding edge makes you "bleed" might scare of some potential developers.
However, sympy and scikit-learn don't have Windows developers (from reputation), but they are successful in attracting developers.

statsmodels won't be so bad, since it's just cython and a c compiler, as long as we stay away from Fortran. (I haven't tried building pymc in a loong time.)

Windows users are not "Hackers" that like to spend their time on building issues. (Spending 2 days on tox and pip a few weeks ago, scared me enough that I have done more manual testing and easy_installing in virtualenvs, then fighting with tools that are build for Linux or pure python packages.)

My strong aversion is because I want to do statistics and econometrics instead of fighting with build issues.

[jseabold]

I understand the concern, and the want to focus on stats. But if you couldn't make an R package that contained Fortran or C, then R wouldn't be anywhere, because many estimators would be unusable. You don't see it because on Windows it just downloads the binaries, but if I install.packages in R, then it downloads the source and builds it in front of me seamlessly. There are very few packages that are pure R.