Switching Compilers (Intel Related) #4518

frenchwr · 2015-06-12T16:22:55Z

Hello all, I have an intel-compiled version of Python 2.7.8 that I'm trying to build and install Matplotlib into. When I run "python setup.py build" everything builds fine until:

icc -pthread -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DMPL_DEVNULL=/dev/null -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__qhull_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -Iextern -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/qhull_wrap.c -o build/temp.linux-x86_64-2.7/src/qhull_wrap.o
In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(106): warning #77: this declaration has no storage class or type specifier 
template <typename T> 
^

In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(106): error: expected a ";"
template <typename T>
         ^

In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(115): warning #12: parsing restarts here after previous syntax error
  void    qh_qhull(void);
                        ^

compilation aborted for src/qhull_wrap.c (code 2) 
error: command 'icc' failed with exit status 2

The issue here is that icc does not support templates. When I manually run this command with icpc rather than icc it compiles fine. Can someone advise me on how I can go about switching to icpc to compile qhull_wrap.c? I'm also happy to submit a pull request if this is something that can be generalized in the build process. Version info listed below.

[frenchwr@vmps11 matplotlib]$ uname -a
Linux vmps11 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[frenchwr@vmps11 matplotlib]$ lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.5 (Final)
Release:    6.5
Codename:   Final

[frenchwr@vmps11 matplotlib]$ python --version
Python 2.7.8

[frenchwr@vmps11 matplotlib]$ icc --version
icc (ICC) 14.0.2 20140120
Copyright (C) 1985-2014 Intel Corporation.  All rights reserved.

[frenchwr@vmps11 matplotlib]$ pip freeze
biopython==1.65
Cython==0.22
mock==1.0.1
nose==1.3.4
numpy==1.9.1
pandas==0.15.2
pyparsing==2.0.3
python-dateutil==2.4.0
pytz==2014.10
scikit-learn==0.15.2
scipy==0.15.1
seaborn==0.4.dev0
six==1.9.0
word-count==0.1.0

[frenchwr@vmps11 matplotlib]$ git describe
v1.4.3-1743-ga951b75

tacaswell · 2015-06-12T16:33:25Z

Taking a quick look at setupext.py it looks like we outsource this work to distutils (and maybe specifically distutils.sysconfig?). I suspect we are using it wrong and relying on gcc 'just working' if you throw c++ at it which we should fix, but that involves touching the build system which is a terrifying prospect.

I think you can get it to behave by doing

CC=icpc python setup.py install

which gives me output like

12:30 $ CC=icpc python setup.py develop
============================================================================
Edit setup.cfg to change the build options

BUILDING MATPLOTLIB
            matplotlib: yes [1.5.dev1]
                python: yes [3.4.3 |Continuum Analytics, Inc.| (default, Mar
                        6 2015, 12:03:53)  [GCC 4.4.7 20120313 (Red Hat
                        4.4.7-1)]]
              platform: yes [linux]

REQUIRED DEPENDENCIES AND EXTENSIONS
                 numpy: yes [version 1.9.2]
                   six: yes [using six version 1.9.0]
              dateutil: yes [using dateutil version 2.4.2]
                  pytz: yes [using pytz version 2015.4]
               tornado: yes [using tornado version 4.1]
             pyparsing: yes [using pyparsing version 2.0.3]
                libagg: yes [pkg-config information for 'libagg' could not
                        be found. Using local copy.]
              freetype: yes [version 2.5.2]
                   png: yes [version 1.5.13]
                 qhull: yes [pkg-config information for 'qhull' could not be
                        found. Using local copy.]

OPTIONAL SUBPACKAGES
           sample_data: yes [installing]
              toolkits: yes [installing]
                 tests: yes [using nose version 1.3.4 / using unittest.mock]
        toolkits_tests: yes [using nose version 1.3.4 / using unittest.mock]

OPTIONAL BACKEND EXTENSIONS
                macosx: no  [Mac OS-X only]
                qt5agg: no  [PyQt5 not found]
                qt4agg: yes [installing, Qt: 4.8.6, PyQt: 4.8.6; PySide not
                        found]
               gtk3agg: no  [Requires pygobject to be installed.]
             gtk3cairo: no  [Requires cairocffi or pycairo to be installed.]
                gtkagg: no  [Requires pygtk]
                 tkagg: yes [installing, version not identified]
                 wxagg: no  [requires wxPython]
                   gtk: no  [Requires pygtk]
                   agg: yes [installing]
                 cairo: no  [cairocffi or pycairo not found]
             windowing: no  [Microsoft Windows only]

OPTIONAL LATEX DEPENDENCIES
                dvipng: yes [version 1.14]
           ghostscript: yes [version 9.16]
                 latex: yes [version 3.14159265]
               pdftops: yes [version 0.33.0]

running develop
running egg_info
writing lib/matplotlib.egg-info/PKG-INFO
writing requirements to lib/matplotlib.egg-info/requires.txt
writing dependency_links to lib/matplotlib.egg-info/dependency_links.txt
writing namespace_packages to lib/matplotlib.egg-info/namespace_packages.txt
writing top-level names to lib/matplotlib.egg-info/top_level.txt
reading manifest file 'lib/matplotlib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'lib/matplotlib.egg-info/SOURCES.txt'
running build_ext
building 'matplotlib.ft2font' extension
icpc -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib_ft2font_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/home/tcaswell/.virtualenvs/dd_3k/lib/python3.4/site-packages/numpy/core/include -I/home/tcaswell/.virtualenvs/dd_3k/include/freetype2 -I/usr/local/include -I/usr/include -I. -I/home/tcaswell/.virtualenvs/dd_3k/include/python3.4m -c src/ft2font.cpp -o build/temp.linux-x86_64-3.4/src/ft2font.o
unable to execute 'icpc': No such file or directory
error: command 'icpc' failed with exit status 1

which correctly fails as I don't have the intel compilers installed.

Hope some of that is helpful!

WeatherGod · 2015-06-12T16:34:16Z

I think you can do python setup.py build --compiler icpc or something to
the effect?

On Fri, Jun 12, 2015 at 12:22 PM, Will French notifications@github.com
wrote:

Hello all, I have an intel-compiled version of Python 2.7.8 that I'm
trying to build and install Matplotlib into. When I run "python setup.py
build" everything builds fine until:

icc -pthread -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DMPL_DEVNULL=/dev/null -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__qhull_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -Iextern -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/qhull_wrap.c -o build/temp.linux-x86_64-2.7/src/qhull_wrap.o
In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(106): warning #77: this declaration has no storage class or type specifier
template
^

In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(106): error: expected a ";"
template
^

In file included from src/qhull_wrap.c(10):
extern/qhull/qhull_a.h(115): warning #12: parsing restarts here after previous syntax error
void qh_qhull(void);
^

compilation aborted for src/qhull_wrap.c (code 2)
error: command 'icc' failed with exit status 2

The issue here is that icc does not support templates. When I manually run
this command with icpc rather than icc it compiles fine. Can someone advise
me on how I can go about switching to icpc to compile qhull_wrap.c? I'm
also happy to submit a pull request if this is something that can be
generalized in the build process. Version info listed below.

[frenchwr@vmps11 matplotlib]$ uname -a
Linux vmps11 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[frenchwr@vmps11 matplotlib]$ lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final

[frenchwr@vmps11 matplotlib]$ python --version
Python 2.7.8

[frenchwr@vmps11 matplotlib]$ icc --version
icc (ICC) 14.0.2 20140120
Copyright (C) 1985-2014 Intel Corporation. All rights reserved.

[frenchwr@vmps11 matplotlib]$ pip freeze
biopython==1.65
Cython==0.22
mock==1.0.1
nose==1.3.4
numpy==1.9.1
pandas==0.15.2
pyparsing==2.0.3
python-dateutil==2.4.0
pytz==2014.10
scikit-learn==0.15.2
scipy==0.15.1
seaborn==0.4.dev0
six==1.9.0
word-count==0.1.0

[frenchwr@vmps11 matplotlib]$ git describe
v1.4.3-1743-ga951b75

—
Reply to this email directly or view it on GitHub
#4518.

frenchwr · 2015-06-12T19:37:11Z

When I use CC=icpc python setup.py build the build makes a bit more progress but eventually bombs out with:

building 'matplotlib._cntr' extension
icpc -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__cntr_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/cntr.c -o build/temp.linux-x86_64-2.7/src/cntr.o
src/cntr.c(1846): error: a value of type "void *" cannot be assigned to an entity of type "char *"
      if (mpa) mask = PyArray_DATA(mpa);
                    ^

src/cntr.c(1904): error: a value of type "void *" cannot be assigned to an entity of type "char *"
      data = PyArray_DATA(Cdata);
           ^

compilation aborted for src/cntr.c (code 2)
error: command 'icpc' failed with exit status 2

This looks like the opposite problem as before, i.e. the C++ compiler is having trouble with C code. I then tried CC=icc CXX=icpc python setup.py build which triggers the original error. It does appear that in this case both icc and icpc are invoked, but it looks somewhat random...C files are occasionally compiled with icpc, C++ files occasionally compiled with icc, so I don't know what to make of that.

I suspect there is a flag I can pass either icc to give it better C++ support, or icpc to give it better C support. I'll keep poking around.

tacaswell · 2015-06-12T19:41:25Z

https://mail.python.org/pipermail/python-dev/2012-March/117219.html <- is
that helpful at all? I suspect you may get better help upstream talking to
either the numpy/scipy folks or the distutils folks.

Do you have similar issues compiling numpy/scipy/scikit-image ?

On Fri, Jun 12, 2015 at 3:37 PM Will French notifications@github.com
wrote:

When I use CC=icpc python setup.py build the build makes a bit more
progress but eventually bombs out with:

building 'matplotlib._cntr' extension
icpc -fno-strict-aliasing -O3 -fPIC -fp-model strict -fomit-frame-pointer -DNDEBUG -g -O3 -Wall -fPIC -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__cntr_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/include -I. -I/usr/local/python2/2.7.8/x86_64/intel14/nonet/include/python2.7 -c src/cntr.c -o build/temp.linux-x86_64-2.7/src/cntr.o
src/cntr.c(1846): error: a value of type "void *" cannot be assigned to an entity of type "char *"
if (mpa) mask = PyArray_DATA(mpa);
^

src/cntr.c(1904): error: a value of type "void *" cannot be assigned to an entity of type "char *"
data = PyArray_DATA(Cdata);
^

compilation aborted for src/cntr.c (code 2)
error: command 'icpc' failed with exit status 2

This looks like the opposite problem as before, i.e. the C++ compiler is
having trouble with C code. I then tried CC=icc CXX=icpc python setup.py
build which triggers the original error. It does appear that in this case
both icc and icpc are invoked, but it looks somewhat random...C files are
occasionally compiled with icpc, C++ files occasionally compiled with icc,
so I don't know what to make of that.

I suspect there is a flag I can pass either icc to give it better C++
support, or icpc to give it better C support. I'll keep poking around.

—
Reply to this email directly or view it on GitHub
#4518 (comment)
.

frenchwr · 2015-06-12T23:00:37Z

Yes I battled with numpy for quite some time before getting it to build and pass all tests, but what I learned there does not help much in this case. I explored the use of different compiler flag options for both icc and icpc but I could not manage to get the matplotlib build to succeed.

In the end, I decided to modify a few lines of code to remove Intel preprocessors and the template definition that triggered the original error. The build succeeded and no additional unit test errors or failures were introduced.

See PR here: #4519

When pulling data from from PyArrray_DATA explicitly cast to (char *). Addresses issues raised in matplotlib#4518

tacaswell · 2015-06-14T20:24:03Z

Reading this again, it looks like the errors you are getting from the icpc are finding sloppyness in our c code. See #4524

tacaswell · 2015-06-19T21:54:37Z

This has been fixed two different ways, closing.

tacaswell added the Build label Jun 12, 2015

frenchwr mentioned this issue Jun 12, 2015

Removing intel preprocessors from qhull_a.h #4519

Merged

tacaswell added a commit to tacaswell/matplotlib that referenced this issue Jun 14, 2015

CLN: explicitly cast (void *) -> (char *)

e0da0b1

When pulling data from from PyArrray_DATA explicitly cast to (char *). Addresses issues raised in matplotlib#4518

tacaswell mentioned this issue Jun 14, 2015

CLN: explicitly cast (void *) -> (char *) #4524

Merged

tacaswell closed this as completed Jun 19, 2015

pramodskumbhar mentioned this issue Sep 9, 2017

Fix matplotlib build issue with intel compiler spack/spack#5321

Merged

This was referenced May 29, 2018

py-matplotlib fails with intel compiler spack/spack#8303

Closed

matplotlib fails to install from source with intel compiler #11340

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switching Compilers (Intel Related) #4518

Switching Compilers (Intel Related) #4518

frenchwr commented Jun 12, 2015

tacaswell commented Jun 12, 2015

WeatherGod commented Jun 12, 2015

frenchwr commented Jun 12, 2015

tacaswell commented Jun 12, 2015

frenchwr commented Jun 12, 2015

tacaswell commented Jun 14, 2015

tacaswell commented Jun 19, 2015

Switching Compilers (Intel Related) #4518

Switching Compilers (Intel Related) #4518

Comments

frenchwr commented Jun 12, 2015

tacaswell commented Jun 12, 2015

WeatherGod commented Jun 12, 2015

frenchwr commented Jun 12, 2015

tacaswell commented Jun 12, 2015

frenchwr commented Jun 12, 2015

tacaswell commented Jun 14, 2015

tacaswell commented Jun 19, 2015