Windows build

[ocefpaf]

We should build should build nco from source on Windows.

See http://nco.sourceforge.net/nco_qt_msvc.shtml

[ocefpaf]

Ping @czender. If you have any hints for us here it would really help us get the Windows builds going.

[czender]

the expert on windows builds of NCO is @pedro-vicente

[ocefpaf]

Obrigado Pedro! I'll take give it a try. We have almost all of those in
conda-forge. I need to work on antlr though.
Em 13/06/2016 23:06, "Pedro Vicente" notifications@github.com escreveu:

@ocefpaf https://github.com/ocefpaf

Building NCO from source is a time consuming task that for a first timer
could take several frustrating days to finish. On the NCO main page you
will see a list of library dependencies

http://nco.sourceforge.net/

on the link you posted above there are several environment variables listed

LIB_NETCDF
LIB_HDF5
LIB_HDF5_HL
LIB_ZLIB
LIB_SZIP
LIB_ANTLR
LIB_GSL
LIB_CURL
LIB_UDUNITS
LIB_EXPAT

Additionnaly, the following environment variable must also be defined for
the location of the netCDF, UDUnits, GSL and Antlr header files:
HEADER_NETCDF
HEADER_UDUNITS
HEADER_GSL
HEADER_ANTLR

By their name it's obvious to which library they refer. So what you have
to do is:

1. Obtain the source code for all the libraries above. (ANTLR must be
   version 2.7)
   LIB_UDUNITS
   LIB_EXPAT
   these 2 it's better to send an email to Ward Fisher on the netcdf mailing
   list because I think the netcdf group ported a semi-official port of these
   for Windows (that by now might be official)

2. Build the libraries, and define the above environment variables

Boa sorte !

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AA6BL3SeUf5OC7kd4VWgXT1_LjiXIIgKks5qLgy2gaJpZM4H5hSE
.

[rsignell-usgs]

@ocefpaf, seems like recent progress allowing antlr window builds
antlr/antlr4#1268

[ocefpaf]

@ocefpaf, seems like recent progress allowing antlr window builds
antlr/antlr4#1268

See my comment in conda-forge/pynco-feedstock#2 (comment)

I never got time to look at the Windows builds but it should be easier now that conda brings msys2 in the default channel.

[czender]

that's good news but beware that NCO requires antlr version 2.7 not antlr version 4.x. not sure whether the antlr4 package will cut the mustard for NCO.

[kwilcox]

@ocefpaf I have a windows laptop and an afternoon... if you want to give me a 10 minute brief I can hack away at it.

We can start with a non-ncap2 build for Windows and make incremental improvements as needed if its blocked upstream somewhere.

[ocefpaf]

@kwilcox I am a little bit swamped but you can start by setting your laptop up with conda and a MSVC.

https://github.com/conda/conda-recipes/wiki/Building-from-Source-on-Windows-32-bit-and-64-bit

Once that is done we need to check if we have all the dependencies already in conda and compiled on Windows.

• LIB_NETCDF, HEADER_NETCDF
• LIB_HDF5
• LIB_HDF5_HL # not sure I cofirmed and it is present in conda-forge's packages
• LIB_ZLIB
• LIB_SZIP # absent but I believe it is not mandatory (alternative to zlib)
• LIB_ANTLR, HEADER_ANTLR # absent
• LIB_GSL, HEADER_GSL # absent but I believe it is not mandatory (ncap2?)
• LIB_CURL
• LIB_UDUNITS, HEADER_UDUNITS
• LIB_EXPAT

[czender]

you are correct that antlr and gsl are only required for ncap2, and that szip is not mandatory (NCO does not require it, though netCDF4 may require it depending on how it is built). so building NCO with --disable-ncap2 might be the easiest way to start.

[rsignell-usgs]

@pedro-vicente, any chance you could take a stab at getting the settings right for a conda build on windows without ncap2?

I might be able to help you address conda issues...

[ocefpaf]

Based on http://nco.sourceforge.net/nco_qt_msvc.shtml we need Qt's qmake. @ccordoba12 do you know if that is possible with anaconda's Qt? Any experience with qmake?

@czender would it be complicated to switch to cmake?

@pedro-vicente if you want to try this please feel free to bug me with any conda/conda-build question.

[czender]

@ocefpaf A switch from autotools to cmake might be conceptually simple, yet I've never written cmake build scripts and my time is limited. If someone else provided a working cmake build system then I would be happy to include it in NCO with the intent of maintaining both cmake and autotools build systems so long as each has significant usage.

[ccordoba12]

qmake comes with all our Qt builds :-) I don't have much experience with it though.

[ocefpaf]

qmake comes with all our Qt builds :-) I don't have much experience with it though.

Thanks @ccordoba12!

I never worked with conda, and my time is limited to learn yet another build system.
But if you post here or email me some instructions of where to start, I can give it a try, but no promises about time.

I started a PR to show you what we can do so far. See #22

Conda is not that complicated as it is a layer on top of other build systems.

On my post above from June 13, there are the steps for building with the supplied NCO Qt build:
what you have to do is, with Visual Studio, build the NCO dependencies in some way.

We have "cmake" builds for all but ANTLR and GSL.

https://github.com/conda-forge/hdf5-feedstock/blob/master/recipe/bld.bat
https://github.com/conda-forge/libnetcdf-feedstock/blob/master/recipe/bld.bat
https://github.com/conda-forge/zlib-feedstock/blob/master/recipe/bld.bat
https://github.com/conda-forge/curl-feedstock/blob/master/recipe/bld.bat
https://github.com/conda-forge/udunits2-feedstock/blob/master/recipe/bld.bat
https://github.com/conda-forge/expat-feedstock/blob/master/recipe/bld.bat

not sure how well those will play with nco but we can make adjustments to those recipes if necessary.

[ocefpaf]

Do the above bld.bat files get the libraries from some "official" location (like github) or do you host them somewhere?

All official sources. In some rare cases we patch them to enable the build.

I believe there is no "official" cmake script for Windows for ANTLR 2.7 and GSL that you could do the same.

That is in the background for now. We are focused on getting a build without GSL and ANTLR.

I have Visual Studio 2013 solutions to build ANTLR 2.7 and GSL, but I don't have any way to make these available, like a public machine for download.

You can send it via e-mail or upload right here on the GitHub issues. (Drag-and-drop should work for most file formats.) BTW people discussing the GSL Windows build at conda-forge/gsl-feedstock#7 will be quite happy with that 😄

[pedro-vicente]

@ocefpaf
@czender
I volunteer to write a CMake build.

[czender]

Much appreciated, @pedro-vicente

[ocefpaf]

I volunteer to write a CMake build.

That is awesome @pedro-vicente! Please feel free to bug me for any help you need.

[pedro-vicente]

The Cmake build is only for after NCO 4.6.2

[ocefpaf]

The Cmake build is only for after NCO 4.6.2

Was that released? If not we can open the PR with a pre-release just to get the ball rolling.

[pedro-vicente]

NCO release dates are here
http://nco.sourceforge.net/

[czender]

Latest version is 4.6.2-beta03. Final 4.6.2 probably this coming week.

[pedro-vicente]

I pushed to github a prototype for a CMake build. The file /doc/index.shtml contains build instructions, which are

CMake build. To build with CMake, do:

mkdir build
cd build
cmake ..
make

The CMake script tries to find header and dependency libraries on standard system locations. If these are not found, they can be set with (the locations in full path are examples)

cmake .. -DNETCDF_INCLUDE:PATH=/my/netcdf/include/path -DNETCDF_LIBRARY=/my/netcdf/library/file -DHDF5_LIBRARY=/my/hdf5/library/file -DHDF5_HL_LIBRARY=/my/hdf5 high level/library/file -DSZIP_LIBRARY=/my/szip/library/file -DZLIB_LIBRARY=/my/zlib/library/file -DCURL_LIBRARY=/my/curl/library/file

Right now only the minimal dependencies are detected: netcdf header file, netcdf, HDF5, HDF5_HL, zlib, szip, curl libraries

[czender]

Where does this issue stand? @pedro-vicente have you produced a working feedstock that builds a conda-installable NCO (including basic ncap2) on Windows?

[pedro-vicente]

@czender The cmake build status is the same as the above post.
Binaries are built for these NCO operators with just the minimum dependencies (netCDF, HDF5, szip, zlib)
ncks
ncbo
ncecat
ncflint
ncpdq
ncra
ncrename

Regarding the conda build, can anyone please remind me what is needed?
Is it a NCO cmake build, including library dependencies?
Like I said before, I can spend some time on the NCO cmake build but I don't have time to learn conda.

[czender]

That's a great start. I see the CMakelists.txt file in the top level directory. I think @ocefpaf or someone else familiar with the conda Windows environment ( @rsignell-usgs ? ) needs to give that a try and work it from the feedstock end, and tell us what mods they need in the upstream. @pedro-vicente please install conda on your windows machine so you can iterate/verify that the install works.

[ocefpaf]

That's a great start. I see the CMakelists.txt file in the top level directory.

Sorry. I am a bit lost. You mean a custom CMakelists.txt file somewhere or are you referring to the one shipped with the source code? (Sorry I never looked into that part of the code so maybe this is something obvious.)

I think @ocefpaf or someone else familiar with the conda Windows environment ( @rsignell-usgs ? ) needs to give that a try and work it from the feedstock end, and tell us what mods they need in the upstream.

I am in the processing of moving to another city and I won't be able to try that for at least 2 weeks (sorry).

@pedro-vicente please install conda on your windows machine so you can iterate/verify that the install works.

@pedro-vicente conda is not that complicated. I will get a simple PR that tries to build this so we can iterate together.

[czender]

@ocefpaf what makes you think ANTLR is "found anyway"? It's not requested in bld.bat, and the AppVeyor build log quoted above shows that it is not found. And because it is not found, building ncap2 is not attempted. At least that's how I read the log. Here is the CMake example file, nco/cmake/build.bat, distributed with NCO:

cmake .. ^
-DNETCDF_INCLUDE:PATH=I:/grafic_project/netcdf/include ^
-DNETCDF_LIBRARY:FILE=I:/grafic_project/Debug/netcdf.lib ^
-DHDF5_LIBRARY:FILE=I:/grafic_project/Debug/hdf5.1.8.5.patch1.lib ^
-DHDF5_HL_LIBRARY:FILE=I:/grafic_project/Debug/hdf5_hl.1.8.5.patch1.lib ^
-DZLIB_LIBRARY:FILE=I:/grafic_project/Debug/zlib.lib ^
-DSZIP_LIBRARY:FILE=I:/grafic_project/Debug/szlib.lib ^
-DCURL_LIBRARY:FILE=I:/grafic_project/Debug/libcurld.lib ^
-DANTLR_INCLUDE:PATH=I:/antlr-2.7.7/lib/cpp ^
-DANTLR_LIBRARY:FILE=I:/antlr-2.7.7/lib/cpp/debug/antlr.lib ^
-DUDUNITS2_INCLUDE:PATH=I:/udunits2/lib ^
-DUDUNITS2_LIBRARY:FILE=I:/udunits2/build/lib/Debug/udunits2.lib ^
-DEXPAT_LIBRARY:FILE=I:/expat/expat/win32/bin/Debug/libexpatMT.lib ^
-DGSL_INCLUDE:PATH=I:/gsl-1.8/src ^
-DGSL_LIBRARY:FILE=I:/gsl-1.8/src/gsl/1.8/gsl-1.8/VC8/libgsl/Debug-StaticLib/libgsl_d.lib ^
-DGSL_CBLAS_LIBRARY:FILE=I:/gsl-1.8/src/gsl/1.8/gsl-1.8/VC8/libgslcblas/Debug-StaticLib/libgslcblas_d.lib

I suggest adding the equivalent paths for ANTLR to bld.bat that will work on AppVeyor.

[ocefpaf]

@ocefpaf what makes you think ANTLR is "found anyway"?

A typo. I meant not found 😄 (because we don't have that for windows.)

[pedro-vicente]

@ocefpaf I have a working version of ANTLR for Windows. I'm not sure about the best way to make this public. There should be ANTLR version 2.7.7 source code available somewhere

[ocefpaf]

There should be ANTLR version 2.7.7 source code available somewhere

The source is available and we do build it for OS X and Linux. We would love to add Windows builds we do lack the know-how there.

I'm not sure about the best way to make this public.

Using a pre-compiled binary blob may be an option, but I would like to avoid that if possible.
If we cannot build it the alternative would be to re-package your binary in the conda-forge channel.

[pedro-vicente]

@ocefpaf if you have the source, then all is needed are the Visual Studio projects. I'll attach those somewhere here later. I'll give it a try your bld.bat
Using a pre-compiled binary blob is a bad idea

[pedro-vicente]

here it is the Visual Studio project
this should located in
/lib/cpp
on your ANTLR source
I remember there was a compiling error using the Linux only official source, if you have that, post here, but it was a trivial thing
antlr.zip

[ocefpaf]

Thanks. I'll give it a try over the weekend.

[pedro-vicente]

@ocefpaf I tried on the master branch of nco-feedstock

K:\nco-feedstock>conda install nco
Fetching package metadata ...............

PackageNotFoundError: Packages missing in current channels:

  - nco

We have searched for the packages in the following channels:

  - https://conda.anaconda.org/conda-forge/win-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.continuum.io/pkgs/main/win-64
  - https://repo.continuum.io/pkgs/main/noarch
  - https://repo.continuum.io/pkgs/free/win-64
  - https://repo.continuum.io/pkgs/free/noarch
  - https://repo.continuum.io/pkgs/r/win-64
  - https://repo.continuum.io/pkgs/r/noarch
  - https://repo.continuum.io/pkgs/pro/win-64
  - https://repo.continuum.io/pkgs/pro/noarch
  - https://repo.continuum.io/pkgs/msys2/win-64
  - https://repo.continuum.io/pkgs/msys2/noarch

[ocefpaf]

@ocefpaf I tried on the master branch of nco-feedstock

Not sure what you mean by that. The PR is in #48, not master.

Also, there won't be a package in the channel until we merge that PR.
In the PR we have all the information for the test build and the logs.

[pedro-vicente]

@ocefpaf I tried the try_win branch, I get the same error

[ocefpaf]

@ocefpaf I tried the try_win branch, I get the same error

Same error as the AppVeyor logs? However, if you are referring the the package not found error above I guess we still have a confusion.

The built package, like I said in #6 (comment), will be available in the channel for installation, via conda install nco, only after we merge the PR that submits it, like #48.

That means, we need to make #48 work first. One can test the PR locally by forking/cloning the repo, getting that particular PR, and issue the build commands locally.

[pedro-vicente]

@ocefpaf

what I did

git clone https://github.com/conda-forge/nco-feedstock
git checkout try_win

I installed Anaconda3, opened a Anaconda3 command prompt
then did what it says here

https://github.com/conda-forge/nco-feedstock

conda config --add channels conda-forge
conda install nco

error is the one above

PackageNotFoundError: Packages missing in current channels:

  - nco

[pedro-vicente]

@ocefpaf

The built package, like I said in #6 (comment), will be available in the channel for installation, via conda install nco, only after we merge the PR that submits it, like #48.

That means, we need to make #48 work first. One can test the PR locally by forking/cloning the repo, getting that particular PR, and issue the build commands locally.

It would be interesting to find out why that ncks -M test fails, but that shouldn't prevent you from release the nco-feedstock as it is, after adding ANTLR.

If all nco-feedstock builds that should be fine. We NCO developers can then try to find the cause of that error. So, I would just remove that test, add ANTLR and release it

[ocefpaf]

what I did

git clone https://github.com/conda-forge/nco-feedstock
git checkout try_win

OK you got the code to build the package, but the instructions only refer to built packages. nco in that PR is not built, it is "building" 😉
Only after the merge it will be available.

It would be interesting to find out why that ncks -M test fails, but that shouldn't prevent you from release the nco-feedstock as it is, after adding ANTLR.

Indeed. If you want to try it locally you are close. Now that you have the code for #48 you can issue:

conda build recipe

from the code main directory. Building on Windows is kind of annoying and that probably won't succeed in the first try. You may need to configure your channels to get the dependencies from conda-forge (I guess that was already done above when you tried to install nco) . There may be also some trouble configuring MSVC. If you get a "vcvarsall.bat not found" message you may need to call the problem .bat manually b/c conda was unable to find it.

So, I would just remove that test, add ANTLR and release it

The only test passing is ncks --help, so I don't think that is advisable. I want to do at least a local file test before merging that PR. Also, I'll merge that only after #46 and #47.

If all nco-feedstock builds that should be fine. We NCO developers can then try to find the cause of that error.

Sure. If you need more help setting a conda environment for building and testing it locally I am glad to help. Although Windows is not my forte 😬 (I rely mostly on AppVeyor for that.)

[pedro-vicente]

@ocefpaf

The only test passing is ncks --help, so I don't think that is advisable. I want to do at least a local file test before merging that PR.

agreed. @czender probably is more qualified to choose a test.
You happened to choose a problematic/difficult to debug test, because it relies on DAP, curl, the internet. Basically all NCO tests do some change to a local file, and the read the result to check if it is what expected.

[pedro-vicente]

@ocefpaf

Sure. If you need more help setting a conda environment for building and testing it locally I am glad to help. Although Windows is not my forte

ok

I did

conda build recipe
Skipped: K:\nco-feedstock\recipe defines build/skip for this configuration.

and nothing seemed to happen

How can I use AppVeyor?

[ocefpaf]

If you got K:\nco-feedstock\recipe defines build/skip for this configuration. that means you are not using the try_win branch. The main branch skips Windows.

AppVeyor will build automatically when pushing a PR, so nothing different from what we have now.

You happened to choose a problematic/difficult to debug test, because it relies on DAP, curl, the internet.

Yep. Package testing is like integration tests, not really meant for debugging, only to check if the expected features are working.

[pedro-vicente]

@ocefpaf I am totally sure I am using try_win 😃

can I change that setting in one of the files?

also, do I need to change this ?

CONDA_INSTALL_LOCN: C:\Miniconda36-x64

[ocefpaf]

@ocefpaf I am totally sure I am using try_win 😃

Are you on a Windows 32 machine or Python <=35? If the former I don't think the recipe will work. If the latter than you can probably just remove that line b/c it has no baring outside of conda-forge (Python is attached to the msvc and both Python 3.5 and 3.6 use vc14, so we need to build only once and we choose to build with conda on Python 3.6).

[czender]

Regarding question above about a simple local file test, this may suffice and does not require ncap2:

fin=data/in.nc
ncks -H --trd -v one $fin
# Success: output to stdout is "one = 1" and zero return code
# Failure: non-zero return code and lots of error messages

[ocefpaf]

Thanks @czender. I'll add it tomorrow. #47 is almost ready and it s time to 😴 here.

PS: most of the delay is Travis-CI backlog to build the OS X version 😒

[pedro-vicente]

@ocefpaf

Are you on a Windows 32 machine or Python <=35?

I have Windows 10 64 bit and I installed Anaconda 32 bit with Python 3.6. Should I try with Anaconda 64 bit.?

[pedro-vicente]

@ocefpaf
that was it, with Anaconda 64 bit it builds

[pedro-vicente]

I found the cause of the error, that turns out to be a NCO bug

@ocefpaf
if you want to reproduce this or any other bug do

1. change the cmake target to be

cmake -G "Visual Studio 14 Win64"

so that it generates a Visual Studio solution for debugging

2. open the solution in Visual Studio and edit a project (say ncks) Properties->Debugging->Environment

add the PATH where the .DLLs for dependent libraries are located , e.g

PATH=%PATH%;C:\Users\pedro\Anaconda3\conda-bld\nco_1510360320825\_h_env\Library\bin;C:\Users\pedro\Anaconda3\conda-bld\nco_1510360320825\_h_env\Scripts

by the way is it possible to have static linking (.lib instead of .dll) of all the dependency libraries?

[pedro-vicente]

@ocefpaf
Is it possible to use the current NCO git code into the nco-feedstock? the current code should fix the Win64 issue that made your test fail.

[ocefpaf]

@ocefpaf
Is it possible to use the current NCO git code into the nco-feedstock? the current code should fix the Win64 issue that made your test fail.

It is possible. See cdbcba9

[pedro-vicente]

there is a build error because the file AST.hpp is missing from

 -D ANTLR_INCLUDE:PATH=C:\Users\pedro\Anaconda3\conda-bld\nco_1513179527208\_h_env\Library\include\antlr  


[ 79%] Building CXX object CMakeFiles/ncap2.dir/src/nco++/Invoke.cc.obj
Invoke.cc
C:\Users\pedro\Anaconda3\conda-bld\nco_1513179527208\work\src\nco++\Invoke.cc(11): fatal error C1083: Cannot open include file: 'antlr/AST.hpp': No such file or directory

[ocefpaf]

Done in #48