Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Fortran bindings #28

Merged
merged 15 commits into from Sep 19, 2016
Merged

Add Fortran bindings #28

merged 15 commits into from Sep 19, 2016

Conversation

astrofrog
Copy link
Contributor

@astrofrog astrofrog commented Jul 19, 2016

Closes #11

Continued version of #14

@conda-forge-linter
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@@ -32,22 +32,25 @@ requirements:
- python # [win]
- cmake >=3.1
- zlib 1.2*
- gcc # [not win]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we only needed gcc for Linux and not OSX. Is this right @jakirkham ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not when it comes to Fortran. That is the only way we have to build Fortran on OS X. We will need to make sure the clang stuff stays entact. We don't want this to start shipping libgcc on OS X if we can avoid it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes agree about libfortran but gcc should have the linux selector instead? Is that right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh you mean the line below.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I misread. Though we do have to do something about that line too.

Unfortunately we need gcc on OS X too if we want to have access to gfortran. Mac doesn't provide a gfortran compiler. I believe they use to, but that was on 10.4. Fortunately, the work we did getting clang to be the compiler and libc++ used should actually help us here. As a result, we should only need to ship libgfortran with this on OS X. I just went through a similar ordeal with scipy. So, am pretty confident that this should work. We still want to review this carefully nonetheless.

@jakirkham
Copy link
Member

Yeah, actually, I'm going to propose the exact same thing we did with scipy (which has worked out quite well. Please add this yum_requirements.txt to the recipe directory. Then make these other changes that follow.

@@ -34,22 +34,24 @@ requirements:
- libtool # [unix]
- cmake >=3.1
- zlib 1.2*
- gcc # [not win]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change # [not win] -> # [osx].

@jakirkham
Copy link
Member

jakirkham commented Jul 20, 2016

These changes were needed for SciPy to ensure that we did not need to ship libstdc++. The same story exists here where we build support for C++ bindings and don't want to ship libstdc++ on Linux. It is a bit hacky, but has been found to be effective.

@@ -11,13 +11,17 @@ source activate "${CONDA_DEFAULT_ENV}"
if [ "$(uname)" == "Darwin" ]
then
export CXX="${CXX} -stdlib=libc++"
export DYLD_FALLBACK_LIBRARY_PATH=$PREFIX/lib
export LIBRARY_PATH="${PREFIX}/lib"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh this last line should be for both platforms. In particular, we need this for Linux.

@jakirkham
Copy link
Member

Also forgot to mention please do this locally in the top level of this repo.

  • conda install -n root -c conda-forge conda-smithy=0.10.5.
  • conda smithy rerender
  • git add .
  • git commit -m "Re-rendered with conda-smithy 0.10.5."
  • git push

This should let us use the yum_requirements.txt.

@astrofrog
Copy link
Contributor Author

@jakirkham - thanks for the tips! Strangely, re-rendering led to the Python 3 builds being dropped from AppVeyor, so I did not include that change in the latest commit. Should I have done?

@@ -26,7 +26,7 @@ install:

conda config --set show_channel_urls true
conda update --yes conda
conda install --yes conda-build jinja2 anaconda-client
conda install --yes conda-build=1.20.0 jinja2 anaconda-client
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it generated this, but we should not include it. See this issue ( #23 ) for background as to why we removed it.

@jakirkham
Copy link
Member

@jakirkham - thanks for the tips!

Of course.

Strangely, re-rendering led to the Python 3 builds being dropped from AppVeyor, so I did not include that change in the latest commit. Should I have done?

Nope, you were right not to. I think you are running into this issue ( conda-tools/conda-build-all#49 ). If you are able to create a reproducer and add it to that issue, that would be amazing. I've had such trouble trying to get anything that would be reproducible.

@astrofrog
Copy link
Contributor Author

Looks like everything is passing! :)

@astrofrog astrofrog changed the title WIP: Add Fortran bindings Add Fortran bindings Jul 20, 2016
@jakirkham
Copy link
Member

Thanks @astrofrog. Could I ask that we add some test code here? I would like to see a couple test programs (it could be copied from HDF5's examples for all I care). Something for Fortran, C, and C++. I expect this will work, but I want to be very sure that it does. These tests can be Unix only. Building in the test phase for Windows has proved very problematic and we have not changed that build AFAICT.

@astrofrog
Copy link
Contributor Author

@jakirkham - no problem, I'll add some test code today!

@astrofrog
Copy link
Contributor Author

I've added some test cases that I copied from the HDF5 examples - I picked one that includes compression etc. to really test things. I checked and included the HDF5 license to make sure this is acceptable.

@astrofrog astrofrog force-pushed the fortran branch 2 times, most recently from 002d312 to ec9317e Compare July 20, 2016 16:08
- python 2.7.* # [win and py27]
- python 3.4.* # [win and py34]
- python 3.5.* # [win and py35]
- python {{ environ['PY_VER'] + '*' }} # [win]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add gcc for OS X only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought this was already the case:

    - gcc  # [osx]

I think GitHub messed up the lines for the comments, @jakirkham can you clarify which line you were referring to?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, but I'm suggesting we make it a test requirement. Travis CI is complaining we don't have gfortran. So this should fix it.

@jakirkham
Copy link
Member

Once we get this one worked out, I'd like to test it out some locally. I really want to be sure we don't accidentally break hdf5 or end up with some weird linking mess. Unfortunately, I don't think the the CI test will be sufficient alone to do this. Though the tests that you added, @astrofrog, should be very helpful in this process. I hope everyone can understand my desire to take this slow.

@astrofrog
Copy link
Contributor Author

@jakirkham - absolutely! I wonder if it would be worth pushing my branch to a prerelease branch in the feedstock so that we can all test out the builds by going to a particular label?

@astrofrog
Copy link
Contributor Author

Ok, so the MacOS X build is now failing with:

dyld: Library not loaded: @rpath/./libgfortran.3.dylib
  Referenced from: /Users/travis/miniconda3/conda-bld/test-tmp_dir/./h5_cmprss
  Reason: image not found
/Users/travis/miniconda3/conda-bld/test-tmp_dir/run_test.sh: line 17: 77628 Trace/BPT trap: 5       ./h5_cmprss

I'm assuming this should not be fixed with the fallback path, because we can't expect users to actually do this too. Does anyone know why the libgfortran library wouldn't get picked up at this point? (I'm pretty sure it's installed since it's in the runtime dependencies).

@astrofrog
Copy link
Contributor Author

(just rebased)

@jakirkham
Copy link
Member

What is the path forward to go ahead and merge this? Are there some specific packages for which we want to try updating to the version in the test label?

Trying to use this with h5py, pytables, and possibly some other HDF5 libraries in conda-forge with this package would be good. The easiest thing would be to run their test suites, but if you have any creative ideas, I would be all ears. I'd also like to know if we are going to need to rebuild things.

@astrofrog
Copy link
Contributor Author

I did some tests with HDF5 from the test label on MacOS X. The environment I am using has:

hdf5                      1.8.17                        3    conda-forge/label/test
ca-certificates           2016.8.31                     0    conda-forge
certifi                   2016.8.31                py35_0    conda-forge
h5py                      2.6.0               np111py35_6    conda-forge
libgfortran               3.0.0                         0    conda-forge
ncurses                   5.9                           9    conda-forge
numexpr                   2.6.1               np111py35_0    conda-forge
openssl                   1.0.2h                        2    conda-forge
pip                       8.1.2                    py35_0    conda-forge
pytables                  3.3.0               np111py35_0    conda-forge
python                    3.5.2                         3    conda-forge
readline                  6.2                           0    conda-forge
setuptools                26.1.1                   py35_0    conda-forge
six                       1.10.0                   py35_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
tk                        8.5.19                        0    conda-forge
wheel                     0.29.0                   py35_0    conda-forge
xz                        5.2.2                         0    conda-forge
zlib                      1.2.8                         3    conda-forge
mkl                       11.3.3                        0  
numpy                     1.11.1                   py35_0  

The h5py tests pass:

In [3]: h5py.tests.run_tests()
....................................................x.......................s.......................................x....................................s........ss..............................................................................................ssssss.................................................................x....x..x..................x.....x......................................ssssssssss............
----------------------------------------------------------------------
Ran 423 tests in 1.839s

OK (skipped=20, expected failures=7)
Out[3]: <unittest.runner.TextTestResult run=423 errors=0 failures=0>

and likewise the pytables tests pass:

In [1]: import tables
tables.
In [2]: tables.test()
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PyTables version:    3.3.0
HDF5 version:        1.8.17
NumPy version:       1.11.1
Numexpr version:     2.6.1 (not using Intel's VML/MKL)
Zlib version:        1.2.8 (in Python interpreter)
BZIP2 version:       1.0.6 (6-Sept-2010)
Blosc version:       1.11.1 (2016-09-03)
Blosc compressors:   blosclz (1.0.5), lz4 (1.7.2), lz4hc (1.7.2), snappy (1.1.1), zlib (1.2.8), zstd (1.0.0)
Blosc filters:       shuffle, bitshuffle
Python version:      3.5.2 | packaged by conda-forge | (default, Sep  8 2016, 14:36:38) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)]
Platform:            Darwin-15.6.0-x86_64-i386-64bit
Byte-ordering:       little
Detected cores:      2
Default encoding:    utf-8
Default FS encoding: utf-8
Default locale:      (en_US, UTF-8)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Performing only a light (yet comprehensive) subset of the test suite.
If you want a more complete test, try passing the --heavy flag to this script
(or set the 'heavy' parameter in case you are using tables.test() call).
The whole suite will take more than 4 hours to complete on a relatively
modern CPU and around 512 MB of main memory.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/filters.py:330: FiltersWarning: compression library ``lzo`` is not available; using ``zlib`` instead
  % (complib, default_complib), FiltersWarning)
version: 1.11.1
./Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: 'a:b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '/a/b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '.'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
..................../Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: 'a:b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '/a/b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '.'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
..................../Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: 'a:b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '/a/b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '.'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
..................../Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: 'a:b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '/a/b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '.'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
..................../Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: 'a:b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '/a/b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '.'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
..................../Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: 'a:b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '/a/b'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/path.py:112: NaturalNameWarning: object name is not a valid Python identifier: '.'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; you will not be able to use natural naming to access this object; using ``getattr()`` will still work, though
  NaturalNameWarning)
....................................................................................................ss..................................................................................................................................................................................................................................................................ssss...........................s.........................................................................................................................sssss.................................................................................................................................................................ssssssssssssssssssssssssssss............................................................................................................................................................................................................................................................................................................................................................................ssssssssssssssssssssssssssssssssssssssssssssssssssssssss.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ssssss...................................................................................................................................................................................................................................................................................................................................................................................................................................................sssssss........................................................ssssssssssssss...................................................................................................................................................................................................ssssssssss...........................................................................ssssssssss.................................................................sssss...............sssss......................................................................................................................................................................................ssssss..............................................................................................................................................................................................................s...............................................................................................................................................................................................................................................................................................s...s.......................................................................................................................................................................................................................................................................................................................................................<string>:1: RuntimeWarning: invalid value encountered in arcsin
.........................<string>:1: RuntimeWarning: invalid value encountered in arccos
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ssssssssssssssssssss............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................/Users/tom/miniconda3/envs/test-hdf5/lib/python3.5/site-packages/tables/nodes/filenode.py:170: UserWarning: host PyTables file is already closed!
  warnings.warn("host PyTables file is already closed!")
......
----------------------------------------------------------------------
Ran 6173 tests in 629.514s

OK (skipped=182)

On Linux, the h5py tests pass:

In [1]: import h5py

In [2]: h5py.tests.run_tests()
....................................................x.......................s.......................................x....................................s........ss..............................................................................................ssssss.................................................................x....x..x..................x.....x......................................ssssssssss............
----------------------------------------------------------------------
Ran 423 tests in 0.610s

OK (skipped=20, expected failures=7)
Out[2]: <unittest.runner.TextTestResult run=423 errors=0 failures=0>

and I'm currently running the 'heavy' pytables test suite.

@astrofrog
Copy link
Contributor Author

The full test suite for pytables ran (in 3 hours!) with no issues:

Ran 67114 tests in 11490.133s

OK (skipped=43)

These packages didn't need re-building or anything, I just used the latest conda-forge versions.

@jakirkham @pelson - any other tests you would like me to do before we can merge this?

@pelson
Copy link
Member

pelson commented Sep 14, 2016

From my perspective, I see no blockers, but I haven't been that close to the discussion. I think @astrofrog's testing looks pretty extensive - @jakirkham, do you have any remaining concerns that @astrofrog can address, or are you OK to let this fly?

@jakirkham
Copy link
Member

So was PyTables tested on Linux too, @astrofrog? Am a little unclear. Not worried about the heavy test suite on both, but just confirming that the standard test suite works on both OS X and Linux.

@astrofrog
Copy link
Contributor Author

astrofrog commented Sep 14, 2016

@jakirkham - yes, I ran the normal pytables suite on Mac and the heavy one on Linux, and the h5py test suite on both Mac and Linux.

@astrofrog
Copy link
Contributor Author

@jakirkham - should I increase the build number btw? or does that happen automatically?

@jakirkham
Copy link
Member

Yep, please bump the build number and then will merge.

@astrofrog
Copy link
Contributor Author

@jakirkham - I think this is ready to go - thanks!

@jakirkham jakirkham merged commit 267fd43 into conda-forge:master Sep 19, 2016
@jakirkham
Copy link
Member

Thanks @astrofrog.

@jakirkham
Copy link
Member

They changed the website in the last couple hours so the source download is failing. 😒

When you have a chance, could you please add a PR to correct this, @astrofrog?

@astrofrog
Copy link
Contributor Author

@jakirkham - done in #42

@jakirkham
Copy link
Member

Thanks for doing that, @astrofrog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants