Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build issue while compiling cpp files in AIX #79379

Closed
Ayappan mannequin opened this issue Nov 9, 2018 · 15 comments
Closed

Build issue while compiling cpp files in AIX #79379

Ayappan mannequin opened this issue Nov 9, 2018 · 15 comments
Labels
3.7 (EOL) end of life 3.8 (EOL) end of life build The build process and cross-build

Comments

@Ayappan
Copy link
Mannequin

Ayappan mannequin commented Nov 9, 2018

BPO 35198
Nosy @vstinner, @aixtools, @miss-islington, @kadler
PRs
  • bpo-35198 Fix C++ extension compilation on AIX #10437
  • [3.7] bpo-35198 Fix C++ extension compilation on AIX (GH-10437) #12162
  • Files
  • pip3_build_pandas_0001.text: complete pip3 build pandas output
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-03-04.15:25:03.009>
    created_at = <Date 2018-11-09.15:20:45.102>
    labels = ['3.8', 'build', '3.7']
    title = 'Build issue while compiling cpp files in AIX'
    updated_at = <Date 2019-03-04.15:25:03.009>
    user = 'https://bugs.python.org/Ayappan'

    bugs.python.org fields:

    activity = <Date 2019-03-04.15:25:03.009>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-03-04.15:25:03.009>
    closer = 'vstinner'
    components = ['Build']
    creation = <Date 2018-11-09.15:20:45.102>
    creator = 'Ayappan'
    dependencies = []
    files = ['47919']
    hgrepos = []
    issue_num = 35198
    keywords = ['patch']
    message_count = 15.0
    messages = ['329522', '329622', '329625', '329684', '329722', '332961', '332971', '332984', '332989', '332991', '333231', '333342', '337123', '337128', '337130']
    nosy_count = 6.0
    nosy_names = ['vstinner', 'David.Edelsohn', 'Michael.Felt', 'Ayappan', 'miss-islington', 'kadler']
    pr_nums = ['10437', '12162']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue35198'
    versions = ['Python 3.7', 'Python 3.8']

    @Ayappan
    Copy link
    Mannequin Author

    Ayappan mannequin commented Nov 9, 2018

    I am trying to build pandas-0.23.4 using python3 in AIX.
    Everything goes fine till it encounters a cpp file. Below is the compile error output.
    building 'pandas._libs.window' extension
    gcc -fPIC -maix64 -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -I/usr/include -I/opt/freeware/include -I/opt/freeware/include/ncurses -DAIX_GENUINE_CPLUSCPLUS -D_LINUX_SOURCE_COMPAT -Wl,-brtl -O2 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/freeware/lib64/python3.6/site-packages/numpy/core/include -I/opt/freeware/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.aix-7.2-3.6/pandas/_libs/window.o -Wno-unused-function
    In file included from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1821,
    from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
    from /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
    from pandas/_libs/window.cpp:587:
    /opt/freeware/lib64/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
    #warning "Using deprecated NumPy API, disable it by " \
    ^~~~~~~
    g++ gcc -maix64 -pthread -bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib -L. -L/usr/include -L/opt/freeware/include -L/usr/lib/threads -L/opt/freeware/lib64 -L/opt/freeware/lib -L/usr/lib64 -L/usr/lib build/temp.aix-7.2-3.6/pandas/_libs/window.o -L/opt/freeware/lib64 -o build/lib.aix-7.2-3.6/pandas/_libs/window.so
    g++: error: gcc: No such file or directory
    g++: error: unrecognized command line option '-bI:/opt/freeware/lib/python3.6/config-3.6m/python.exp'
    error: command 'g++' failed with exit status 1

    Seems like somewhere setting up the proper compile options for compiling cpp files is not proper. Any hint will be really useful.

    @Ayappan Ayappan mannequin added 3.7 (EOL) end of life build The build process and cross-build labels Nov 9, 2018
    @aixtools
    Copy link
    Contributor

    There are, perhaps, other issues as well.

    After a build of "master"

    a) pip3 install pandas

    • failed to find/download and build numpy
    • after "manual" pip3 install numpy and got to " six, python-dateutil, pytz, pandas"

    Then:
    Running setup.py install for pandas ... error
    Complete output from command /opt/bin/python3.8 -u -c "import setuptools, tokenize;file='/tmp/pip-install-f5rvzoc0/pandas/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kmi2p9sd/install-record.txt --single-version-externally-managed --compile:
    ...
    UPDATING build/lib.aix-6.1-3.8-pydebug/pandas/_version.py
    set build/lib.aix-6.1-3.8-pydebug/pandas/_version.py to '0.23.4'
    running build_ext
    building 'pandas._libs.algos' extension
    creating build/temp.aix-6.1-3.8-pydebug
    creating build/temp.aix-6.1-3.8-pydebug/pandas
    creating build/temp.aix-6.1-3.8-pydebug/pandas/_libs
    xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/algos.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/algos.o -Wno-unused-function
    xlc_r: 1501-210 (S) command option Wno-unused-function contains an incorrect subargument
    error: command 'xlc_r' failed with exit status 40

    \----------------------------------------
    

    Command "/opt/bin/python3.8 -u -c "import setuptools, tokenize;file='/tmp/pip-install-f5rvzoc0/pandas/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kmi2p9sd/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-f5rvzoc0/pandas/

    So, it looks like, using xlc - I may not have even gotten as far window.gcc due to issues in pandas (the incorrect flag -Wno-used-function - have to find where that came from!)

    @aixtools
    Copy link
    Contributor

    Getting farther - after "hacking" pandas setup.py to not force a gcc flag.

    Still not getting as far, I think, as the initial poster.

    ...
    /opt/include/python3.8dm -c pandas/_libs/parsers.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/parsers.o
    xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/src/parser/tokenizer.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/src/parser/tokenizer.o
    "/opt/include/python3.8dm/pyconfig.h", line 1538.9: 1506-236 (W) Macro name _POSIX_C_SOURCE has been redefined.
    "/opt/include/python3.8dm/pyconfig.h", line 1538.9: 1506-358 (I) "_POSIX_C_SOURCE" is defined on line 160 of /usr/include/standards.h.
    "/opt/include/python3.8dm/pyconfig.h", line 1553.9: 1506-236 (W) Macro name _XOPEN_SOURCE has been redefined.
    "/opt/include/python3.8dm/pyconfig.h", line 1553.9: 1506-358 (I) "_XOPEN_SOURCE" is defined on line 156 of /usr/include/standards.h.
    "/usr/include/unistd.h", line 171.17: 1506-343 (S) Redeclaration of lseek64 differs from previous declaration on line 169 of "/usr/include/unistd.h".
    "/usr/include/unistd.h", line 171.17: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long".
    "/usr/include/unistd.h", line 171.17: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
    "/usr/include/sys/lockf.h", line 64.20: 1506-343 (S) Redeclaration of lockf64 differs from previous declaration on line 62 of "/usr/include/sys/lockf.h".
    "/usr/include/sys/lockf.h", line 64.20: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type "long".
    "/usr/include/unistd.h", line 809.33: 1506-343 (S) Redeclaration of ftruncate64 differs from previous declaration on line 807 of "/usr/include/unistd.h".
    "/usr/include/unistd.h", line 809.33: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
    "/usr/include/unistd.h", line 845.33: 1506-343 (S) Redeclaration of truncate64 differs from previous declaration on line 843 of "/usr/include/unistd.h".
    "/usr/include/unistd.h", line 845.33: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
    "/usr/include/unistd.h", line 862.33: 1506-343 (S) Redeclaration of pread64 differs from previous declaration on line 859 of "/usr/include/unistd.h".
    "/usr/include/unistd.h", line 862.33: 1506-377 (I) The type "long long" of parameter 4 differs from the previous type "long".
    "/usr/include/unistd.h", line 863.33: 1506-343 (S) Redeclaration of pwrite64 differs from previous declaration on line 860 of "/usr/include/unistd.h".
    "/usr/include/unistd.h", line 863.33: 1506-377 (I) The type "long long" of parameter 4 differs from the previous type "long".
    "/usr/include/unistd.h", line 942.25: 1506-343 (S) Redeclaration of fclear64 differs from previous declaration on line 939 of "/usr/include/unistd.h".
    "/usr/include/unistd.h", line 942.25: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the previous return type "long".
    "/usr/include/unistd.h", line 942.25: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type "long".
    "/usr/include/unistd.h", line 943.25: 1506-343 (S) Redeclaration of fsync_range64 differs from previous declaration on line 940 of "/usr/include/unistd.h".
    "/usr/include/unistd.h", line 943.25: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type "long".
    "pandas/_libs/src/parser/tokenizer.c", line 265.40: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
    "pandas/_libs/src/parser/tokenizer.c", line 292.30: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
    "pandas/_libs/src/parser/tokenizer.c", line 323.28: 1506-280 (W) Function argument assignment between types "long long*" and "unsigned long*" is not allowed.
    error: command 'xlc_r' failed with exit status 1
    root@x066:[/data/prj/python/pandas-0.23.4]set | grep OBJ
    root@x066:[/data/prj/python/pandas-0.23.4]python3
    Python 3.8.0a0 (heads/master-dirty:b9498e2367, Nov 10 2018, 13:51:41) [C] on aix6
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.maxsize()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'int' object is not callable
    >>> sys.maxsize
    2147483647

    +++++
    Will uninstall and build a 64-bit python3 and see if that helps.
    +++++

    @aixtools aixtools added the 3.8 (EOL) end of life label Nov 10, 2018
    @aixtools
    Copy link
    Contributor

    Still getting the same errors, even with 64-bit build, so still not close to testing the actual problem.
    FYI: last time I build pandas the version was 0.19.0 - then it was 'simple'.

    Anyway, short of a simple (test) python module that includes c++ code, I'll look at changing the pandas build files to force any c++ file to the front.

    More - (asap, but still when I have time)

    @Ayappan
    Copy link
    Mannequin Author

    Ayappan mannequin commented Nov 12, 2018

    The PR 10437 worked !!!.
    Now i am able to compile pandas-0.23.4

    Michael Felt,

    About the re-declaration issues you are facing, that is because of the "#define _LARGE_FILES 1" in pyconfig-ppc64.h file.
    For 64bit build we actually don't need it.
    For 32bit build, we need it and also we have to pass the flag "-D_LARGE_FILES" through the _sysconfigdata_m_aix6_.py , otherwise one will get this re-declaration error.

    @kadler
    Copy link
    Mannequin

    kadler mannequin commented Jan 4, 2019

    Just a friendly ping that there's a PR for this bug waiting to be reviewed.

    @aixtools
    Copy link
    Contributor

    aixtools commented Jan 4, 2019

    While the PR probably solves this - there is a 'bug' in pandas (I expect) that prevents me from completing the test - as, I expect LONG before the .cpp source is to be compiled - there is a error because a wrong flag is passed to the compiler (-Wno-unused-function) and the build stops.

    Have not had the time to disect pandas so that I can get a .cpp source compiled.

    +++++
    Successfully installed pip-18.1
    root@x066:[/data/prj/python/git/kadler]pip list
    /opt/lib/python3.8/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWdeprecated, and in 3.8 it will stop working
    from collections import Mapping
    Package Version
    ---------- -------
    pip 18.1
    setuptools 39.0.1
    root@x066:[/data/prj/python/git/kadler]pip install numpy
    /opt/lib/python3.8/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWdeprecated, and in 3.8 it will stop working
    from collections import Mapping
    Collecting numpy
    /opt/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py:220: PendingDepreca
    warnings.warn(
    Using cached https://files.pythonhosted.org/packages/2d/80/1809de155bad674b494248b
    Installing collected packages: numpy
    Running setup.py install for numpy ... done
    Successfully installed numpy-1.15.4

    ...
    Installing collected packages: six, python-dateutil, pytz, pandas
    ...
    copying pandas/io/formats/templates/html.tpl -> build/lib.aix-6.1-3.8-pydebug/pandas/io/formats/templates
    UPDATING build/lib.aix-6.1-3.8-pydebug/pandas/_version.py
    set build/lib.aix-6.1-3.8-pydebug/pandas/_version.py to '0.23.4'
    running build_ext
    building 'pandas._libs.algos' extension
    creating build/temp.aix-6.1-3.8-pydebug
    creating build/temp.aix-6.1-3.8-pydebug/pandas
    creating build/temp.aix-6.1-3.8-pydebug/pandas/_libs
    xlc_r -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/algos.c -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/algos.o -Wno-unused-function
    xlc_r: 1501-210 (S) command option Wno-unused-function contains an incorrect subargument
    error: command 'xlc_r' failed with exit status 40

    \----------------------------------------
    

    Command "/opt/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-xnscgehv/pandas/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-0w46p413/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-xnscgehv/pandas/

    @kadler
    Copy link
    Mannequin

    kadler mannequin commented Jan 4, 2019

    Ah. We always compile with GCC, so would not have hit that particular problem.

    @aixtools
    Copy link
    Contributor

    aixtools commented Jan 4, 2019

    On 04/01/2019 17:08, Kevin wrote:

    Kevin <kadler@us.ibm.com> added the comment:

    Ah. We always compile with GCC, so would not have hit that particular problem.

    ----------


    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue35198\>


    FYI. I found a 'hack' in ./setup.py to skip adding the argument.
    Progressing.

    @aixtools
    Copy link
    Contributor

    aixtools commented Jan 4, 2019

    Further along - however, I never get to the "link" routine.

    Again, this is likely a pandas coding issue - currently python is calling xlc_r ..., but when I manually modify it to xlC_r I get the same error.

    xlc_r -D_LARGE_FILES -O -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/lib/python3.8/site-packages/numpy/core/include -I/opt/include/python3.8dm -c pandas/_libs/window.cpp -o build/temp.aix-6.1-3.8-pydebug/pandas/_libs/window.o
    "pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
    "pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
    "pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.
    error: command 'xlc_r' failed with exit status 1
    root@x066:[/data/prj/aixtools/git/pandas-master]ebug/pandas/_libs/window.o <
    "pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
    "pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
    "pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.

    root@x066:[/data/prj/aixtools/git/pandas-master]xlC_r -D_LARGE_FILES -O -I/opt/include -O2 -qm>
    "pandas/_libs/window.cpp", line 9943.23: 1540-0063 (S) The text "(" is unexpected.
    "pandas/_libs/window.cpp", line 10030.23: 1540-0063 (S) The text "(" is unexpected.
    "pandas/_libs/window.cpp", line 11165.21: 1540-0063 (S) The text "(" is unexpected.

    FYI. Will look more tomorrow, but if you have an idea, like the -D_LARGE_FILES fix, I am much obliged.

    @Ayappan
    Copy link
    Mannequin Author

    Ayappan mannequin commented Jan 8, 2019

    Not sure what went wrong here.
    I used gcc & g++ and didn't hit this issue.

    @aixtools
    Copy link
    Contributor

    aixtools commented Jan 9, 2019

    On 08/01/2019 15:40, Ayappan wrote:

    Ayappan <ayappanec@gmail.com> added the comment:

    Not sure what went wrong here.
    I used gcc & g++ and didn't hit this issue.

    ----------


    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue35198\>


    Well, it is probably an issue with xlc as I have vac.C and
    vacpp.cmp.core installed (aka xlc and xlC). These days it seems only gcc
    environment gets tested in depth and portability issues with "posix"
    compilers get missed.

    GCC is a fine compiler - just not the one I am using for a number of
    "logistical" reasons.

    Thanks for the update!

    Michael

    @vstinner
    Copy link
    Member

    vstinner commented Mar 4, 2019

    New changeset 800d5cd by Victor Stinner (Kevin Adler) in branch 'master':
    bpo-35198 Fix C++ extension compilation on AIX (GH-10437)
    800d5cd

    @miss-islington
    Copy link
    Contributor

    New changeset 06e9953 by Miss Islington (bot) in branch '3.7':
    bpo-35198 Fix C++ extension compilation on AIX (GH-10437)
    06e9953

    @vstinner
    Copy link
    Member

    vstinner commented Mar 4, 2019

    Thanks Kevin Adler.

    @vstinner vstinner closed this as completed Mar 4, 2019
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 (EOL) end of life build The build process and cross-build
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants