BUG/BLD 3.4 compatibility of cython c files #1311

Closed
holocronweaver opened this Issue Jan 11, 2014 · 45 comments

Projects

None yet

8 participants

@holocronweaver

cython generated files are not compatilbe with python 3.4

see comments below


original description

clarify that 2to3 is needed for Python 3 support

From what I can tell, this project requires 2to3 to create a Python 3 build, but this is not stated in the README or the INSTALL file. Please clarify this, preferably including an example coversion like:
2to3 --output-dir=statsmodels3 -W -n statsmodels
Thanks!

@holocronweaver

Sorry, it appears this is only a problem on Python 3.4. python3.3 setup.py build compiles just fine.

@josef-pkt
Member

@holocronweaver Do you get an error message that 2to3 is not automatically run in python 3.4?

Sometimes there were small adjustments to make for 2to3 to run correctly with a new python version.

@holocronweaver

@josef-pkt If I run python3.4 setup.py build I get the error

statsmodels/tsa/kalmanf/kalman_loglike.c: In function ‘__Pyx_TraceSetupAndCall’:
statsmodels/tsa/kalmanf/kalman_loglike.c:7131:17: error: ‘PyFrameObject’ has no member named ‘f_tstate’
         (*frame)->f_tstate = tstate;
                 ^
@josef-pkt
Member

That looks like the cython generated c files don't work for python 3.4

Hopefully this will be fixed by using a newer python. I think it has nothing to do with running 2to3, since that is shouldn't change .pyx files. AFAIU

Did you run cython on the statsmodels files? That is Did you install a statsmodels release distribution that includes the c file, or did you install from a git checkout?
In the latter case it would be useful if you can tell us your cython version.

>>> import Cython
>>> Cython.__version__
'0.17.4'

I reopen and edit the description.
We will need to test python 3.4 compatibility for the next release.

Thanks for testing.

@josef-pkt josef-pkt reopened this Jan 11, 2014
@holocronweaver

I am using the latest git checkout with

>>> Cython.__version__
'0.19.2'
@GongYiLiao

I confirmed that Cython 0.20.1 does not work.

@jseabold
Member
jseabold commented Mar 2, 2014

Is Cython compatible with Python 3.4? E.g., can you build pandas?

@jseabold
Member
jseabold commented Mar 2, 2014

Also, make sure you're not dealing with stale .c files. To be absolutely sure I usually run

git clean -xdf

Before building. You might need permissions to clean up the build/ directory, depending how you install. Alternatively, you can run

python setup.py clean

If you don't want to nuke everything not under version control.

@GongYiLiao

Oh, I forgot to clean up, after cleanup the old generated files, now statsimodels can be built with Python 3.4 (rc1) and Cython (0.20.1)

@jseabold
Member
jseabold commented Mar 2, 2014

Ok, thanks. I suspected that was the case here. I'm closing this. Feel free to re-open if there are further issues.

@jseabold jseabold closed this Mar 2, 2014
@sboehler

Hi

I have encountered the same error with Python 3.4, and I am unable to install statsmodels both via pip (from pypi) and from the git repo. Please reopen this issue.

Last few build lines:


/opt/local/bin/gcc-mp-4.8 -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names build/temp.macosx-10.9-x86_64-3.4/statsmodels/nonparametric/_smoothers_lowess.o -o build/lib.macosx-10.9-x86_64-3.4/statsmodels/nonparametric/_smoothers_lowess.so

skipping 'statsmodels/tsa/kalmanf/kalman_loglike.c' Cython extension (up-to-date)

building 'statsmodels.tsa.kalmanf.kalman_loglike' extension

creating build/temp.macosx-10.9-x86_64-3.4/statsmodels/tsa

creating build/temp.macosx-10.9-x86_64-3.4/statsmodels/tsa/kalmanf

/opt/local/bin/gcc-mp-4.8 -Wno-unused-result -Werror=declaration-after-statement -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -pipe -Os -I/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include -I/Users/sboehler/prod34/include -I/opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c statsmodels/tsa/kalmanf/kalman_loglike.c -o build/temp.macosx-10.9-x86_64-3.4/statsmodels/tsa/kalmanf/kalman_loglike.o

In file included from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ndarraytypes.h:1760:0,

                 from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,

                 from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/arrayobject.h:4,

                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:

/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

 #warning "Using deprecated NumPy API, disable it by " \

  ^

statsmodels/tsa/kalmanf/kalman_loglike.c: In function '__Pyx_TraceSetupAndCall':

statsmodels/tsa/kalmanf/kalman_loglike.c:7021:17: error: 'PyFrameObject' has no member named 'f_tstate'

         (*frame)->f_tstate = PyThreadState_GET();

                 ^

In file included from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:26:0,

                 from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/arrayobject.h:4,

                 from statsmodels/tsa/kalmanf/kalman_loglike.c:257:

statsmodels/tsa/kalmanf/kalman_loglike.c: At top level:

/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:1629:1: warning: '_import_array' defined but not used [-Wunused-function]

 _import_array(void)

 ^

In file included from /Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/ufuncobject.h:327:0,

                 from statsmodels/tsa/kalmanf/kalman_loglike.c:258:

/Users/sboehler/prod34/lib/python3.4/site-packages/numpy/core/include/numpy/__ufunc_api.h:241:1: warning: '_import_umath' defined but not used [-Wunused-function]

 _import_umath(void)

 ^

error: command '/opt/local/bin/gcc-mp-4.8' failed with exit status 1

----------------------------------------
Cleaning up...
Command /Users/sboehler/prod34/bin/python3.4 -c "import setuptools, tokenize;__file__='/Users/sboehler/prod34/build/statsmodels/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/z5/q6c60q2n00q75kvbysg2bfmr0000gn/T/pip-fneuzmt_-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/sboehler/prod34/include/site/python3.4 failed with error code 1 in /Users/sboehler/prod34/build/statsmodels
Storing debug log for failure in /Users/sboehler/.pip/pip.log

I am using Python 3.4 in a pyvenv on OS X with dependencies installed from macports. I have also cloned the git repo and clean everything according to the instructions above.

~/develop/python/statsmodels% ${CC} --version
gcc-mp-4.8 (MacPorts gcc48 4.8.2_0) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
~/develop/python/statsmodels% python --version
Python 3.4.0
~/develop/python/statsmodels% cython --version
Cython version 0.20.1

Let me know if you need any more info. Many thanks!

@jseabold
Member

The only thing that can cause this is stale files or files generated by Cython for a different version of Python. Please make sure your Cython is also using Python 3.4.

which cython

What is the first line there? Is this the Python you expect it to be using?

@sboehler

Yes, it says
/Users/sboehler/prod34/bin/cython
which is version 0.20.1 (edit: and whereas prod34 is my pyvenv).

How can I check for stale files? I thought that if I "pip install statsmodels" it will download, build and install from a new directory, or am I missing something.

@jseabold
Member

What is the first line of /Users/sboehler/prod34/bin/cython does it point to a different version of Python than the one you expect?

Ok, I see how this is happening now. You need to do 1 of two things.

  1. Install from a fresh github checkout. Assuming you're using the write Cython+Python combination. This should just work.

  2. If you download a source distribution from PyPi and it has the .c files already checked in for a certain version of Python. We include these in the release so people won't need Cython to install, but this isn't Python version independent, so we'll have to stop doing this or figure out Python 2.x and 3.x source tarballs.

So...

If you clone the source from github, you shouldn't run into this problem.

If you're using our tarballs of a source release (from pypi), then we'll have .c files in there that won't work on your system. You'll need to delete these. You can do that from within the source directory with a

python setup.py clean
@jseabold jseabold reopened this Mar 17, 2014
@sboehler

Thanks, I just retried using a fresh git checkout and it worked fine this time! So 1) was the way to go.

If I correctly understand 2), it is thus currently not possible to install statsmodels via pip? I'd appreciate if you could shed some light on this or if you see a way out while using pip, for the benefit of other users.

My personal trouble is solved, however, thank you very much for the quick help!

@jseabold jseabold modified the milestone: 0.5.1, 0.6 Mar 17, 2014
@jseabold
Member

On Python 3.4, no, I don't think so. @josef-pkt What do you think about uploading some source tarballs that are 0.5.0 but with the .c files removed on pypi? We'd now require Cython for all builds.

I changed the milestone for this to 0.5.1, so in the worst case, pip install will work for Python 3.4 on the next micro release sometime soon.

@jseabold
Member

I suppose there may be a way to run the CleanCommand in setup.py if the version is 3.4 and raise a Warning that you'll need Cython. Maybe that's the easiest patch.

@josef-pkt
Member

question first before removing c files which might make it more difficult for many users to install:

Does the latest cython generate c code that is compatible with python 2.6?
I think up to now, it was always possible to use the latest required cython and still generate code that is compatible with older versions of everything else.

Then, we just need new source distributions, tar and zip, with the updated c files.

(If we had thought about it, we could have tried a better timing with respect to the python 3.4 release.)

Very fast solution would be to upload sdist files for python 3.4, if pip checks python version names in sdist. (I doubt it.)

@jseabold
Member

Why is it more difficult without the C files? You still need a compiler, so presumably you'll have Cython. What I proposed only makes Cython necessary on 3.4 anyway, which is very bleeding edge as far as the Python world goes.

I don't think it has anything to do with the Cython version. I just think the code Cython outputs for Python 2.6-3.3 is different than that for Python 3.4. I'll ping their list though, maybe no one else has run into this yet.

@josef-pkt
Member

my guess is adding the sdist with new c files and version 0.5.0.1 should work.
But it might make easy_install ignore the Windows binaries because they have a lower version number (I don't know). We would have to reupload the same old windows binaries with a higher version number also, 0.5.0.1.

What I proposed only makes Cython necessary on 3.4 anyway.

Not if sdist are not interpreted as python specific. pip for python 3.3 and python 2.6 might pick the same tar balls. I don't know.
In this case: Users won't have cython installed and we will get for several weeks comments about why pip install doesn't work anymore.

@jseabold
Member

On Mon, Mar 17, 2014 at 8:33 PM, Josef Perktold notifications@github.comwrote:

my guess is adding the sdist with new c files and version 0.5.0.1 should
work.
But it might make easy_install ignore the Windows binaries because they
have a lower version number (I don't know). We would have to reupload the
same old windows binaries with a higher version number also, 0.5.0.1.

Let's just push to get this solved in 0.5.1 and get it out the door. I'm
traveling next weekend and may have a minute to do this. We should be in
pretty good shape already.

What I proposed only makes Cython necessary on 3.4 anyway.

Not if sdist are not interpreted as python specific. pip for python 3.3
and python 2.6 might pick the same tar balls. I don't know.
In this case: Users won't have cython installed and we will get for
several weeks comments about why pip install doesn't work anymore.

What I'm saying is that the tarballs will be the same. It doesn't have
anything to do with pip. We'd call CleanCommand in the setup.py and make
sure the .c files are nuked early on in the build process on 3.4 and that
they get rebuilt (would have to check that there's not a flag that controls
this). We should have a nice 'Cython must be installed' error message, if
it's not deteceted. Right now we're already getting several weeks comments
about why pip install doesn't work anymore. This should be a change that
only affects 3.4 and makes it work.

@josef-pkt
Member

Ok, I misunderstood the last part. It should work ok, if we change the setup.py.

But we should still check first if the cython developers managed to maintain compatibility across python versions, which would make any special casing of python 3.4 obsolete.

@jseabold
Member

Yeah, I pinged the list and gave the link here. I'll wait to hear back.

@jseabold
Member

Ok, heard back on cython-users. tl;dr we don't need to do anything. We just need to build the 0.5.1 release tarball with Cython 0.20.1 (or higher). Python 3.4 has been a moving target and they're still adapting to late changes.

> for our source distributions. So pip install ... fails with this kind
> of error on Python 3.4.
>
> statsmodels/tsa/kalmanf/kalman_loglike.c:7131:17: error:
> 'PyFrameObject' has no member named 'f_tstate'
>          (*frame)->f_tstate = tstate;

I remember that, it was a fairly late change in the Py3.4 release cycle,
which was a heavily moving target this time with a surprisingly high number
of this kind of late changes. It's only been officially released yesterday,
so I guess people will just have to live with third party tools needing a
moment to adapt (or bugs and regressions getting fixed in Py3.4 etc.).


> https://github.com/statsmodels/statsmodels/issues/1311
>
> Is this a known issue? Is it something we're doing wrong? Is there a
> way to compile .C files that work Python 2.6 through 3.4?

Sure. Cython 0.20 has adapted to this change, with further Py3.4 fixes in
0.20.1. There will soon be a 0.20.2 release, BTW, although that shouldn't
change anything w.r.t. Py3.4. So rebuilding the sources with 0.20.1 should
fix it for you.
@josef-pkt
Member

very good, that's what I was hoping for and expecting.
I don't remember any case where the generated c code was not compatible with all supported python versions. The cython developers have been very good in this respect.

@jseabold
Member

Leave this open as a reminder to bump the cython version in the docs (for 3.4). I'm juggling branches right now.

@jseabold jseabold added the FAQ label Mar 26, 2014
@jseabold jseabold closed this in #1527 Mar 28, 2014
@pentlander

I still haven't been able to get it working even with the suggestions in this post. When I try to install from source and try to run OLS:

File "stringsource", line 275, in init statsmodels.tsa.kalmanf.kalman_loglike (statsmodels/tsa/kalmanf/kalman_loglike.c:19787)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 1: invalid start byte

When I try using a nightly binary I get:

runtime error r6034

Running Python 3.4, tried both Cython 0.20.1 and 0.20.2, in Windows 64bit using the SDK 7.1 and .NET 4.0. Am I missing something?

@jseabold
Member

I have no idea what the first error could be. Does the test suite pass on the one you built from source? If so, can you post code to replicate this.

I think there's an issue with the Python 3.4 nightly builds. I need to update the code on the machine with what's on the repo. I'll fix it before the next build. I was kinda waiting for reports as a data point.

@pentlander

Thanks for the timely response. How do I run the test suite?

EDIT: I think i may have found the problem. Below I have posted the cmd output for the python setup.py build. When building, it appears to be using Anaconda's gcc.bat for compilation, instead of the Microsoft compiler. Is that it?

d64-3.3\statsmodels\discrete\tests\results
copying statsmodels\discrete\tests\results\predict_prob_poisson.csv -> build\lib
.win-amd64-3.3\statsmodels\discrete\tests\results
copying statsmodels\discrete\tests\results\ships.csv -> build\lib.win-amd64-3.3\
statsmodels\discrete\tests\results
copying statsmodels\discrete\tests\results\yhat_mnlogit.csv -> build\lib.win-amd
64-3.3\statsmodels\discrete\tests\results
copying statsmodels\discrete\tests\results\yhat_poisson.csv -> build\lib.win-amd
64-3.3\statsmodels\discrete\tests\results
copying statsmodels\discrete\tests\results\mn_logit_summary.txt -> build\lib.win
-amd64-3.3\statsmodels\discrete\tests\results
copying statsmodels\genmod\tests\results\epil.csv -> build\lib.win-amd64-3.3\sta
tsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\gee_linear_1.csv -> build\lib.win-amd64
-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\gee_logistic_1.csv -> build\lib.win-amd
64-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\gee_nested_linear_1.csv -> build\lib.wi
n-amd64-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\gee_nominal_1.csv -> build\lib.win-amd6
4-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\gee_ordinal_1.csv -> build\lib.win-amd6
4-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\gee_poisson_1.csv -> build\lib.win-amd6
4-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\igaussident_resids.csv -> build\lib.win
-amd64-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\inv_gaussian.csv -> build\lib.win-amd64
-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\iris.csv -> build\lib.win-amd64-3.3\sta
tsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\medparlogresids.csv -> build\lib.win-am
d64-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\stata_cancer_glm.csv -> build\lib.win-a
md64-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\stata_lbw_glm.csv -> build\lib.win-amd6
4-3.3\statsmodels\genmod\tests\results
copying statsmodels\genmod\tests\results\stata_medpar1_glm.csv -> build\lib.win-
amd64-3.3\statsmodels\genmod\tests\results
copying statsmodels\iolib\tests\results\data_missing.dta -> build\lib.win-amd64-
3.3\statsmodels\iolib\tests\results
copying statsmodels\iolib\tests\results\time_series_examples.dta -> build\lib.wi
n-amd64-3.3\statsmodels\iolib\tests\results
copying statsmodels\nonparametric\tests\results\results_kde.csv -> build\lib.win
-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\results_kde_fft.csv -> build\lib
.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\results_kde_univ_weights.csv ->
build\lib.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\results_kde_weights.csv -> build
\lib.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\results_kernel_regression.csv ->
 build\lib.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\test_lowess_delta.csv -> build\l
ib.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\test_lowess_frac.csv -> build\li
b.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\test_lowess_iter.csv -> build\li
b.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\nonparametric\tests\results\test_lowess_simple.csv -> build\
lib.win-amd64-3.3\statsmodels\nonparametric\tests\results
copying statsmodels\regression\tests\results\lme00.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme01.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme02.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme03.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme04.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme05.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme06.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme07.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme08.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme09.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme10.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\lme11.csv -> build\lib.win-amd64-3.
3\statsmodels\regression\tests\results
copying statsmodels\regression\tests\results\leverage_influence_ols_nostars.txt
-> build\lib.win-amd64-3.3\statsmodels\regression\tests\results
copying statsmodels\sandbox\regression\tests\griliches76.dta -> build\lib.win-am
d64-3.3\statsmodels\sandbox\regression\tests
copying statsmodels\sandbox\regression\tests\racd10data_with_transformed.csv ->
build\lib.win-amd64-3.3\statsmodels\sandbox\regression\tests
copying statsmodels\stats\libqsturng\CH.r -> build\lib.win-amd64-3.3\statsmodels
\stats\libqsturng
copying statsmodels\stats\libqsturng\LICENSE.txt -> build\lib.win-amd64-3.3\stat
smodels\stats\libqsturng
copying statsmodels\stats\tests\test_data.txt -> build\lib.win-amd64-3.3\statsmo
dels\stats\tests
copying statsmodels\stats\libqsturng\tests\bootleg.dat -> build\lib.win-amd64-3.
3\statsmodels\stats\libqsturng\tests
copying statsmodels\stats\tests\results\bootleg.csv -> build\lib.win-amd64-3.3\s
tatsmodels\stats\tests\results
copying statsmodels\stats\tests\results\influence_measures_bool_R.csv -> build\l
ib.win-amd64-3.3\statsmodels\stats\tests\results
copying statsmodels\stats\tests\results\influence_measures_R.csv -> build\lib.wi
n-amd64-3.3\statsmodels\stats\tests\results
copying statsmodels\stats\tests\results\influence_lsdiag_R.json -> build\lib.win
-amd64-3.3\statsmodels\stats\tests\results
copying statsmodels\tests\results\cancerdata.csv -> build\lib.win-amd64-3.3\stat
smodels\tests\results
copying statsmodels\tests\results\cancerident_resids.csv -> build\lib.win-amd64-
3.3\statsmodels\tests\results
copying statsmodels\tests\results\cancer_resids.csv -> build\lib.win-amd64-3.3\s
tatsmodels\tests\results
copying statsmodels\tsa\tests\results\arima111_forecasts.csv -> build\lib.win-am
d64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\arima212_forecast.csv -> build\lib.win-amd
64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\ARMLEConstantPredict.csv -> build\lib.win-
amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\AROLSConstantPredict.csv -> build\lib.win-
amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\AROLSNoConstantPredict.csv -> build\lib.wi
n-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\resids_css_c.csv -> build\lib.win-amd64-3.
3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\resids_css_nc.csv -> build\lib.win-amd64-3
.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\resids_exact_c.csv -> build\lib.win-amd64-
3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\resids_exact_nc.csv -> build\lib.win-amd64
-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arima_exog_forecasts_css.csv -> bu
ild\lib.win-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arima_exog_forecasts_mle.csv -> bu
ild\lib.win-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arima_forecasts.csv -> build\lib.w
in-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arima_forecasts_all_css.csv -> bui
ld\lib.win-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arima_forecasts_all_css_diff.csv -
> build\lib.win-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arima_forecasts_all_mle.csv -> bui
ld\lib.win-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arima_forecasts_all_mle_diff.csv -
> build\lib.win-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_arma_forecasts.csv -> build\lib.wi
n-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_ar_forecast_mle_dynamic.csv -> bui
ld\lib.win-amd64-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\results_corrgram.csv -> build\lib.win-amd6
4-3.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\yhat_css_c.csv -> build\lib.win-amd64-3.3\
statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\yhat_css_nc.csv -> build\lib.win-amd64-3.3
\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\yhat_exact_c.csv -> build\lib.win-amd64-3.
3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\yhat_exact_nc.csv -> build\lib.win-amd64-3
.3\statsmodels\tsa\tests\results
copying statsmodels\tsa\tests\results\y_arma_data.csv -> build\lib.win-amd64-3.3
\statsmodels\tsa\tests\results
copying statsmodels\tsa\vector_ar\tests\results\vars_results.npz -> build\lib.wi
n-amd64-3.3\statsmodels\tsa\vector_ar\tests\results
creating build\lib.win-amd64-3.3\statsmodels\tsa\vector_ar\data
copying statsmodels\tsa\vector_ar\data\e1.dat -> build\lib.win-amd64-3.3\statsmo
dels\tsa\vector_ar\data
copying statsmodels\tsa\vector_ar\data\e2.dat -> build\lib.win-amd64-3.3\statsmo
dels\tsa\vector_ar\data
copying statsmodels\tsa\vector_ar\data\e3.dat -> build\lib.win-amd64-3.3\statsmo
dels\tsa\vector_ar\data
copying statsmodels\tsa\vector_ar\data\e4.dat -> build\lib.win-amd64-3.3\statsmo
dels\tsa\vector_ar\data
copying statsmodels\tsa\vector_ar\data\e5.dat -> build\lib.win-amd64-3.3\statsmo
dels\tsa\vector_ar\data
copying statsmodels\tsa\vector_ar\data\e6.dat -> build\lib.win-amd64-3.3\statsmo
dels\tsa\vector_ar\data
running build_ext
building 'statsmodels.nonparametric.linbin' extension
creating build\temp.win-amd64-3.3
creating build\temp.win-amd64-3.3\Release
creating build\temp.win-amd64-3.3\Release\statsmodels
creating build\temp.win-amd64-3.3\Release\statsmodels\nonparametric
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\Scripts\gcc.bat -DMS_WIN64 -md
ll -O -Wall -IC:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\s
ite-packages\numpy\core\include -IC:\Users\pentla\AppData\Local\Continuum\Anacon
da3\envs\py3.3\include -IC:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\
py3.3\include -c statsmodels/nonparametric/linbin.c -o build\temp.win-amd64-3.3\
Release\statsmodels\nonparametric\linbin.o
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarraytypes.h:1761:0,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:17,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/linbin.c:346:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/npy_1_7_deprecated_api.h:12:9: note: #pragma message: C:
\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\num
py\core\include/numpy/npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecat
ed NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:26:0,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/linbin.c:346:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/__multiarray_api.h:1629:1: warning: '_import_array' defi
ned but not used [-Wunused-function]
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ufuncobject.h:327:0,
                 from statsmodels/nonparametric/linbin.c:347:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/__ufunc_api.h:241:1: warning: '_import_umath' defined bu
t not used [-Wunused-function]
writing build\temp.win-amd64-3.3\Release\statsmodels\nonparametric\linbin.def
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\Scripts\gcc.bat -DMS_WIN64 -sh
ared -s build\temp.win-amd64-3.3\Release\statsmodels\nonparametric\linbin.o buil
d\temp.win-amd64-3.3\Release\statsmodels\nonparametric\linbin.def -LC:\Users\pen
tla\AppData\Local\Continuum\Anaconda3\envs\py3.3\libs -LC:\Users\pentla\AppData\
Local\Continuum\Anaconda3\envs\py3.3\PCbuild\amd64 -lpython33 -lmsvcr100 -o buil
d\lib.win-amd64-3.3\statsmodels\nonparametric\linbin.pyd
building 'statsmodels.tsa.kalmanf.kalman_loglike' extension
creating build\temp.win-amd64-3.3\Release\statsmodels\tsa
creating build\temp.win-amd64-3.3\Release\statsmodels\tsa\kalmanf
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\Scripts\gcc.bat -DMS_WIN64 -md
ll -O -Wall -IC:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\s
ite-packages\numpy\core\include -IC:\Users\pentla\AppData\Local\Continuum\Anacon
da3\envs\py3.3\include -IC:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\
py3.3\include -c statsmodels/tsa/kalmanf/kalman_loglike.c -o build\temp.win-amd6
4-3.3\Release\statsmodels\tsa\kalmanf\kalman_loglike.o
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarraytypes.h:1761:0,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:17,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/arrayobject.h:4,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:346:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/npy_1_7_deprecated_api.h:12:9: note: #pragma message: C:
\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\num
py\core\include/numpy/npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecat
ed NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ufuncobject.h:327:0,
                 from statsmodels/tsa/kalmanf/kalman_loglike.c:347:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/__ufunc_api.h:241:1: warning: '_import_umath' defined bu
t not used [-Wunused-function]
writing build\temp.win-amd64-3.3\Release\statsmodels\tsa\kalmanf\kalman_loglike.
def
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\Scripts\gcc.bat -DMS_WIN64 -sh
ared -s build\temp.win-amd64-3.3\Release\statsmodels\tsa\kalmanf\kalman_loglike.
o build\temp.win-amd64-3.3\Release\statsmodels\tsa\kalmanf\kalman_loglike.def -L
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\libs -LC:\Users\pen
tla\AppData\Local\Continuum\Anaconda3\envs\py3.3\PCbuild\amd64 -lpython33 -lmsvc
r100 -o build\lib.win-amd64-3.3\statsmodels\tsa\kalmanf\kalman_loglike.pyd
building 'statsmodels.nonparametric._smoothers_lowess' extension
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\Scripts\gcc.bat -DMS_WIN64 -md
ll -O -Wall -IC:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\s
ite-packages\numpy\core\include -IC:\Users\pentla\AppData\Local\Continuum\Anacon
da3\envs\py3.3\include -IC:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\
py3.3\include -c statsmodels/nonparametric/_smoothers_lowess.c -o build\temp.win
-amd64-3.3\Release\statsmodels\nonparametric\_smoothers_lowess.o
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarraytypes.h:1761:0,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:17,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/_smoothers_lowess.c:346:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/npy_1_7_deprecated_api.h:12:9: note: #pragma message: C:
\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\num
py\core\include/numpy/npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecat
ed NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:26:0,
                 from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/arrayobject.h:4,
                 from statsmodels/nonparametric/_smoothers_lowess.c:346:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/__multiarray_api.h:1629:1: warning: '_import_array' defi
ned but not used [-Wunused-function]
In file included from C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3
.3\lib\site-packages\numpy\core\include/numpy/ufuncobject.h:327:0,
                 from statsmodels/nonparametric/_smoothers_lowess.c:347:
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\lib\site-packages\n
umpy\core\include/numpy/__ufunc_api.h:241:1: warning: '_import_umath' defined bu
t not used [-Wunused-function]
writing build\temp.win-amd64-3.3\Release\statsmodels\nonparametric\_smoothers_lo
wess.def
C:\Users\pentla\AppData\Local\Continuum\Anaconda3\Scripts\gcc.bat -DMS_WIN64 -sh
ared -s build\temp.win-amd64-3.3\Release\statsmodels\nonparametric\_smoothers_lo
wess.o build\temp.win-amd64-3.3\Release\statsmodels\nonparametric\_smoothers_low
ess.def -LC:\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\libs -LC:
\Users\pentla\AppData\Local\Continuum\Anaconda3\envs\py3.3\PCbuild\amd64 -lpytho
n33 -lmsvcr100 -o build\lib.win-amd64-3.3\statsmodels\nonparametric\_smoothers_l
owess.pyd
@jseabold
Member

That could be a problem. I don't know. You can run the tests by doing

import statsmodels.api as sm
sm.test()
@pentlander

I can't even test it. Trying to import statsmodels.api throws that error.

@jseabold
Member

Yes, the 3.4 binaries are broken. Does it throw an error for the ones you built?

@pentlander

To clarify, I'm not bothering with the binaries. When I build from the source and try to import the module, I get:

File "stringsource", line 275, in init statsmodels.tsa.kalmanf.kalman_loglike (statsmodels/tsa/kalmanf/kalman_loglike.c:19787)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 1: invalid start byte

I use the instructions exactly from the dev install page to build it with the Microsoft compiler. However, for some reason distutils seems to avoid using it. When I uninstall mingw, it tries to use the cygwin c compiler.

@josef-pkt
Member

@pentla Did you truncate your build output at the end? I don't see any error message.

You mentioned before that you are using python 3.4, but Anaconda is using python 3.3
The pyd files seem to be created without problems.

my guess is that mingw64 should work, but I never tried

@jseabold
Member

Are you doing this from the SDK command prompt? DISTUTILS_USE_SDK should make sure we pick up the correct compiler. This is the batch file I use to build them

https://github.com/statsmodels/binary_builds/blob/master/build_win_bdist64-py34.bat

@pentlander

@josef-pkt It builds just fine. That was when I was trying to switch over to use 3.3 instead of working throught these errors. The build log is exactly the same when I switch base to 3.4. The problem occurs when I try to import the module/use it, not when building it.

@jseabold Yeah I'm using the SDK command prompt. I don't know why it's still using the gcc even after setting the flag. I added --compiler=mcvs to the build command which is not the right compiler because that points to "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin" not the Microsoft SDK but it's something.

@jseabold
Member

Out of curiosity what windows version are you on?

@pentlander

Windows 7 Enterprise x64 SP 1

@jseabold
Member

You might try asking on the anaconda list and/or bumping the message on the cython list below. This appears to be a common problem and not necessarily anything we can or should do about it, especially since it works with the SDK and I'm not sure how to get distutils to use the SDK over anaconda gcc since I don't use it. In any event the binary builds put up tonight should be ok. I've just got to hook up a monitor to that machine and update the repo.

https://stackoverflow.com/questions/24210563/cython-in-win64-with-tdm-gcc-reports-utf-8-codec-cant-decode-byte-0x83-in-po
https://stackoverflow.com/questions/24109794/cython-memoryviews-on-windows
https://groups.google.com/forum/#!topic/cython-users/7uDpymocmSc

@pentlander

Great thanks! I'll try the binary tomorrow, for now I've switched to 3.3 to have a working statsmodel. I really appreciate your hard work.

@jseabold
Member

Hey, I finally had a chance to update the 3.4 binaries. Can you give them a whirl and let us know?

@pentlander

The ols function works perfectly, so I'm happy. I ran the tests too just for posterity and got one error:

         Current function value: 1.650960
         Iterations: 7
         Function evaluations: 8
         Gradient evaluations: 8

======================================================================
ERROR: Failure: AttributeError ('module' object has no attribute 'f_gen')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\pentla\AppData\Local\Continuum\Anaconda3\lib\site-packages\nose\failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "C:\Users\pentla\AppData\Local\Continuum\Anaconda3\lib\site-packages\nose\loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "C:\Users\pentla\AppData\Local\Continuum\Anaconda3\lib\site-packages\nose\importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "C:\Users\pentla\AppData\Local\Continuum\Anaconda3\lib\site-packages\nose\importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "C:\Users\pentla\AppData\Local\Continuum\Anaconda3\lib\imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "C:\Users\pentla\AppData\Local\Continuum\Anaconda3\lib\imp.py", line 171, in load_source
    module = methods.load()
  File "<frozen importlib._bootstrap>", line 1220, in load
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "C:\Users\pentla\AppData\Local\Continuum\Anaconda3\lib\site-packages\statsmodels\sandbox\distributions\tests\testtransf.py", line 48, in <module>
    stats.distributions.f_gen._stats = f_stats
AttributeError: 'module' object has no attribute 'f_gen'

----------------------------------------------------------------------
Ran 3010 tests in 769.055s

FAILED (SKIP=24, errors=1)
Out[2]: <nose.result.TextTestResult run=3010 errors=1 failures=0>
@cel4
cel4 commented Aug 23, 2014

Is there any update for this issue? At the moment the prebuilt c-files in statsmodels-0.5.0 won't work with python3.4. So any package that depends on statsmodels cannot support python3.4 either, until statsmodels-0.6.x is released. It looks as if rebuilding all .c files with the latest cython version fixes this error.
This issue suggested a 0.5.0.1 release with rebuilt .c files. However it seems that this idea didn't make it into a pypi release. Any reasons for this?

@fly-away

cel4, try this:

pip3 install cython
p3 install git+https://github.com/statsmodels/statsmodels.git

Find this solution here: http://web-salad.hateblo.jp/entry/2014/07/17/002439
Looks like its fixed in latest git but not released yet.

@cel4
cel4 commented Aug 31, 2014

thanks for your comment, @fly-away. This issue is indeed fixed in the master branch. Your suggestion works. However, my problem is that currently no stable version of statsmodels supports python3.4.

@cel4 cel4 referenced this issue in mwaskom/seaborn Sep 6, 2014
Merged

Fix py3k tests #279

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment