Building with CMAKE on Windows

[rsignell-usgs]

@semmerson and @WardF I'm trying to build UDUNITS using CMAKE with Visual Studio 2008 (Visual Studio 9). From the Visual Studio 2008 Command Prompt, I typed:

cd c:\programs\udunits-2.2.16
mkdir build
cd build
../cmake

and I get back

c:\programs\udunits-2.2.16\build>cmake ..
DEFAULT_UDUNITS2_XML_PATH = "C:\Program Files\udunits\share\udunits\udunits2.xml
"
-- Could NOT find EXPAT (missing:  EXPAT_LIBRARY EXPAT_INCLUDE_DIR)
CMake Error at CMakeLists.txt:128 (MESSAGE):
  Unable to find an EXPAT library.

-- Configuring incomplete, errors occurred!

I tried installing expat for windows but that didn't help. Do I need to build it with CMAKE & same compiler?

[semmerson]

Hi Rich,

Apparently, the "FindEXPAT" plugin of CMake is unable to locate the EXPAT
library. Did you install it in the default location?

Regards,
Steve Emmerson

On Thu, Jul 17, 2014 at 2:28 PM, Rich Signell notifications@github.com
wrote:

@semmerson https://github.com/semmerson and @WardF
https://github.com/wardf I'm trying to build UDUNITS using CMAKE with
Visual Studio 2008 (Visual Studio 9). From the Visual Studio 2008 Command
Prompt, I typed:

cd c:\programs\udunits-2.2.15
mkdir build
cd build
../cmake

and I get back

c:\programs\udunits-2.2.16\build>cmake ..
DEFAULT_UDUNITS2_XML_PATH = "C:\Program Files\udunits\share\udunits\udunits2.xml
"
-- Could NOT find EXPAT (missing: EXPAT_LIBRARY EXPAT_INCLUDE_DIR)
CMake Error at CMakeLists.txt:128 (MESSAGE):
Unable to find an EXPAT library.

-- Configuring incomplete, errors occurred!

I tried installing expat for windows but that didn't help. Do I need to
build it with CMAKE & same compiler?

—
Reply to this email directly or view it on GitHub
#16.

[rsignell-usgs]

I downloaded the expat-win32bin-2.1.0.exe from http://sourceforge.net/projects/expat/?source=dlp and installed it in the default location:
C:\Program Files (x86)\Expat 2.1.0

Do I need to set some environment variables for Cmake to find it?

[rsignell-usgs]

@aashish24, any ideas?

[aashish24]

@rsignell-usgs looks like for some reason its looking in the Program Files(x86). you can set these manually and see if that works? These should be visible in the ccmake or cmake-gui

EXPAT_INCLUDE_DIR
EXPAT_LIBRARY

[rsignell-usgs]

cmake was installed at C:\Program Files (x86)\CMake but I moved it to c:\programs\cmake where I have write permissions and it seems happier. Looks like I need to add those params, but I think I'm making progress:

[aashish24]

the problem with expat is not related to CMAKE_INSTALL_PREFIX. You may have to click on the Advanced check option to see EXPAT variable.

[rsignell-usgs]

I'm setting the EXPAT_LIBRARY, but when I click Generate the variable disappears and it can't find the library. @semmerson , how did you get this to work?

[aashish24]

EXPAT_LIBRARY should be set tot he libexpat.dll

[rsignell-usgs]

Okay, and what about EXPAT_INCLUDE_DIR?
I thought it should be the directory that contains expat.h, but that doesn't seem to work.

[rsignell-usgs]

I had a typo. Okay, this is working. The advanced tab was key.

[rsignell-usgs]

Okay, NOW I have it. EXPAT is found!! But not sure what to do next...

I clicked on ALL_BUILD and clicked Build, but get lots of errors:

[aashish24]

Seems like you only have one error? libgen,h is not found?

[aashish24]

Looks like you are using cygwin shell and that may be the problem source.

[rsignell-usgs]

I was using the Visual Studio 2008 Command Prompt. Do I want something else?

[aashish24]

Right. but looks like you are using the command prompt in a cygwin environment which I think made udunits thinks it needs libegen.h which I don't think exist in windows environment. Are you using git bash-shell of git via cygwin?

[WardF]

Hi, just chiming in since I've only now caught up on the messages. I don't
know if it's related, but Visual Studio usually balks at linking against
.dll files directly; instead, they require you to use an import library,
generally with the suffix '.lib'. The .dll is used at run-time.

-Ward

On Fri, Jul 18, 2014 at 9:59 AM, Rich Signell notifications@github.com
wrote:

I was using the Visual Studio 2008 Command Prompt. Do I want something
else?

—
Reply to this email directly or view it on GitHub
#16 (comment).

[semmerson]

Thanks Aashish.

What was the underlying problem with the EXPAT plug-in and can we get
feedback to its developer?

Regards,
Steve Emmerson

On Fri, Jul 18, 2014 at 9:27 AM, Rich Signell notifications@github.com
wrote:

I had a typo. Okay, this is working. The advanced tab was key.
[image: 7-18-2014 10-46-28 am]
https://cloud.githubusercontent.com/assets/1872600/3628043/0cb53124-0e90-11e4-8fe2-cac610481c25.png

—
Reply to this email directly or view it on GitHub
#16 (comment).

[rsignell-usgs]

Yeah, there is a problem finding libgen.h and other files. Googling on libgen.h and udunits turned up this page where the Vapor guys hacked a version of the udunits2 code to work on Windows. On this page:
http://sourceforge.net/p/vapor/git/ci/master/tree/lib/udunits2/
they say:

Note:  The udunits2 code has been ported to Windows for use in VAPOR.  However, whenever udunits2 is updated, the code will again need to be ported.  This file documents the conversion process that should be performed whenever the udunits source code that is used by VAPOR is updated.

Build the latest version (currently 2.1.24) of udunits on Mac (run configure, then make then make install)
Copy over the following files into lib/udunits2:

unitcore.c, formatter.c, parser.c, converter.c, idToUnitMap.c, udunits-1.c, unitToIdMap.c, xml.c, scanner.c, ut_free_system.c, unitAndId.c, systemMap.c, status.c, prefix.c, error.c

parser.y, scanner.l

udunits.h, unitToIdMap.h, unitAndId.h, converter.h, idToUnitMap.h,systemMap.h

add udunits2.h to include/vapor directory

Add all these files (except for parser.y and scanner.l and scanner.c) to the VS2010 DLL project udunits2.
Add the include directory where expat.h is located.

In converter.c, and formatter.c and scanner.l define _USE_MATH_DEFINES before including math.h 
Also at the start of parser.c

Replace snprintf with _snprintf (in converter.c, formatter.c, xml.c)

In formatter.c, make size_t arguments of asciiPrintProduct, latin1PrintProduct, and utf8PrintProduct not const.

In idToUnitMap.c, parser.c, unitcore.c, and xml.c, remove #include <strings.h>
In scanner.c and xml.c, remove #include <unistd.h>

Replace strcasecmp with stricmp in idToUnitMap.c, parser.c, xml.c

In status.c replace <udunits2.h> with "vapor/udunits2.h"

In unitcore.c, remove "include inttypes.h"
Insert "#define int32_t __int32"

In xml.c remove "include libgen.h"
#define _POSIX_PATH_MAX 256
comment out
#define DEFAULT_UDUNITS2_XML_PATH 
Where DEFAULT_UDUNITS2_XML_PATH is used, replace it with
getenv(VAPOR_HOME)\share\udunits2\udunits2.xml

add tsearch.c and tsearch.h to project
change <search.h> to "tsearch.h" in include statements in 6 files.

Add libexpat.lib to project linker inputs, and its directory to linker additional library directives

In xml.c replace close with _close, open with _open, read with _read.  Insert #include <io.h> at top.  Instead of using dirname() as an argument to memmove(), do the following:
char dir[_POSIX_PATH_MAX];
char drv[_POSIX_PATH_MAX];
_splitpath(base,drv,dir,NULL,NULL);
strcat(dir,drv);
memmove(base,drv,sizeof(base));

In common.h insert:
#ifdef UDUNITS2_EXPORTS
#define UDUNITS2_API __declspec(dllexport)
#else
#define UDUNITS2_API __declspec(dllimport)
#endif

then put UDUNITS2_API in front of all the methods defined in udunits2.h
as well as in front of cv_free in converter.h 

insert #include "vapor/udunits2.h" in converter.h and converter.c
replace "udunits2.h" with "vapor/udunits2.h" wherever it occurs in #includes
insert #include "vapor/common.h" in udunits2.h

Yipes. Do we need to that kind of hacking to get this to work?

[semmerson]

Building the UDUNITS-2 package via CMake has only been tested using the
MinGW environment: it has not be tested using either CYGWIN or Visual
Studio.

Regards,
Steve Emmerson

On Fri, Jul 18, 2014 at 12:24 PM, Rich Signell notifications@github.com
wrote:

Yeah, there is a problem finding libgen.h and other files. Googling on
libgen.h and udunits turned up this page where the Vapor guys hacked a
version of the udunits2 code to work on Windows. On this page:
http://sourceforge.net/p/vapor/git/ci/master/tree/lib/udunits2/
they say:

Note: The udunits2 code has been ported to Windows for use in VAPOR. However, whenever udunits2 is updated, the code will again need to be ported. This file documents the conversion process that should be performed whenever the udunits source code that is used by VAPOR is updated.

Build the latest version (currently 2.1.24) of udunits on Mac (run configure, then make then make install)
Copy over the following files into lib/udunits2:

unitcore.c, formatter.c, parser.c, converter.c, idToUnitMap.c, udunits-1.c, unitToIdMap.c, xml.c, scanner.c, ut_free_system.c, unitAndId.c, systemMap.c, status.c, prefix.c, error.c

parser.y, scanner.l

udunits.h, unitToIdMap.h, unitAndId.h, converter.h, idToUnitMap.h,systemMap.h

add udunits2.h to include/vapor directory

Add all these files (except for parser.y and scanner.l and scanner.c) to the VS2010 DLL project udunits2.
Add the include directory where expat.h is located.

In converter.c, and formatter.c and scanner.l define _USE_MATH_DEFINES before including math.h
Also at the start of parser.c

Replace snprintf with _snprintf (in converter.c, formatter.c, xml.c)

In formatter.c, make size_t arguments of asciiPrintProduct, latin1PrintProduct, and utf8PrintProduct not const.

In idToUnitMap.c, parser.c, unitcore.c, and xml.c, remove #include <strings.h>
In scanner.c and xml.c, remove #include <unistd.h>

Replace strcasecmp with stricmp in idToUnitMap.c, parser.c, xml.c

In status.c replace <udunits2.h> with "vapor/udunits2.h"

In unitcore.c, remove "include inttypes.h"
Insert "#define int32_t __int32"

In xml.c remove "include libgen.h"
#define _POSIX_PATH_MAX 256
comment out
#define DEFAULT_UDUNITS2_XML_PATH
Where DEFAULT_UDUNITS2_XML_PATH is used, replace it with
getenv(VAPOR_HOME)\share\udunits2\udunits2.xml

add tsearch.c and tsearch.h to project
change <search.h> to "tsearch.h" in include statements in 6 files.

Add libexpat.lib to project linker inputs, and its directory to linker additional library directives

In xml.c replace close with _close, open with _open, read with _read. Insert #include <io.h> at top. Instead of using dirname() as an argument to memmove(), do the following:
char dir[_POSIX_PATH_MAX];
char drv[_POSIX_PATH_MAX];
_splitpath(base,drv,dir,NULL,NULL);
strcat(dir,drv);
memmove(base,drv,sizeof(base));

In common.h insert:
#ifdef UDUNITS2_EXPORTS
#define UDUNITS2_API __declspec(dllexport)
#else
#define UDUNITS2_API __declspec(dllimport)
#endif

then put UDUNITS2_API in front of all the methods defined in udunits2.h
as well as in front of cv_free in converter.h

insert #include "vapor/udunits2.h" in converter.h and converter.c
replace "udunits2.h" with "vapor/udunits2.h" wherever it occurs in #includes
insert #include "vapor/common.h" in udunits2.h

Yipes. Do we need to that kind of hacking to get this to work?

—
Reply to this email directly or view it on GitHub
#16 (comment).

[WardF]

I had no problem compiling libexpat via cmake on Windows. I'm poking around with UDUNITS-2 right now. I'm not sure I'll have time to incorporate all of the changes suggested by vapor's port today, but as it stands it looks fairly straightforward.

[rsignell-usgs]

@WardF, I was actually hoping you would try the Native Windows port PR by sdebionne: #4
Looks like he did a really nice job!

You can grab the zip from here:
https://github.com/sdebionne/UDUNITS-2/archive/winport.zip

and just build the solution in .\win\build-vs11 directory.

[WardF]

Great, I'll give that a try.

[WardF]

Does not work right out of the box with Visual Studio 10, unfortunately. It appears that it needs some of the changes made by VAPOR (conditional inclusion of strings.h, for example). I'll poke around with it some more; I've forked UDUNITS-2 to https://github.com/WardF/UDUNITS-2, so that I can avoid polluting the main branch.

[rsignell-usgs]

@WardF, I'm really hoping we can get UDUNITS working with MS Visual Studio 2008 so that we can build conda python modules that depend on it. In our Integrated Ocean Observing System (IOOS) work, lack of UDUNITS on PC is a blocker for our CF and ACDD compliance checker (https://github.com/ioos/compliance-checker) and for the British Met Office Iris Project (https://github.com/SciTools/iris), which we depend on for reading CF compliant datasets in Python.

Sounds like the VAPOR team would be appreciative too!

Do you have a few cycles to spend on this?

[WardF]

@rsignell-usgs I don't today, but I will find some time tomorrow to look at it. My feeling is it will either A) Be very straight forward and I can have something that works with VS put together in a few hours, or B) There are deep hooks into Unix/POSIX that will be very difficult to work around.

Given the preponderance of extant Windows/UDUNITS-2 projects, I'm hopeful that it will be the former instead of the latter. I'm currently working on some netcdf-fortran issues, as well as putting together an abstract for AMS, but I will try to find some cycles to see what the process is going to look like. I'll post a follow-up here :).

[aashish24]

Lets us know if we can help. Although we can not help officially (as we don't have any direct funding for this) we can try to help as much as possible over the emails.

[WardF]

Here is what I've found. The following things would have to be implemented in order for me to move forward towards compiling UDUNITS-2 with Visual Studio 2010:

1. Implement cross-platform snprintf and dirname. This looks doable, although not in the time I have available to me right now.
2. Implement cross-platform getopt. I did this for netcdf-c, so wouldn't be much more than cut-and-paste.
3. Resolve the following functions:
   • tsearch
   • tfind
   • tdelete
   • All other functions provided by search.h.

On Windows using MSVC, search.h uses btree functions. Unfortunately the method signatures differ, so using #define tsearch bsearch, etc, doesn't work. From a technical standpoint, this could be resolved. While I'd be happy to help with this as best I could, I would be setting myself up to fail if I were to commit to doing this in the short-middle term.

What work I've done so far lives at http://github.com/WardF/UDUNITS-2. CMake is used to generate Visual Studio project files, and I have been focusing on getting libudunits2 to compile; after that works, I was planning on moving on to the 'udunits2' target.

So! That is where we're at currently. I am open to further suggestions, if solutions to the above occur to anybody.

[rsignell-usgs]

@WardF , thanks so much for scoping the problem.

Now that the issues are known, it would be great if could move into a fast tracked "official" project instead of a "doing a favor for Rich" project. ;-)

Would the best way to get this work on the official schedule to be to speak with Ethan or Mohan, or should I just wait? Please advise.

I'm sure the IOOS system-test developers and British Met Office developers would be happy to pile on with their endorsement of this work as being immensely useful and would be used right away.

[WardF]

@rsignell-usgs Rich, no problem. I'm disappointed that it wasn't something I could just make happen.

I would start with @semmerson, as UDUNITS is his project. I've briefed him on what I found, in addition to my comment above. He will have a better perspective on moving forward with this in whatever fashion, "official" or otherwise. As always, I'm available and happy to help to the extent that my big-picture workload will let me.

[WardF]

Additionally, I don't know if these are issues which are corrected in later versions of Visual Studio; I see there is a Visual Studio 2012 express that I can try to build with. It's a long shot, but perhaps these functions are implemented in the newer tools. @aashish24, do you know if newer VS provides tsearch, tfind, tdelete, etc?

[rsignell-usgs]

@WardF, actually I belive we will need to use Visual Studio 2008 for compatibility with python 2.7. @aashish24, can you confirm that as well?

[WardF]

Ah, good to know; scrolling back I see that now. Once it is working with
Visual Studio 2010, I will see if there are any major changes needed to get
it working with VS2008.

On Wed, Jul 23, 2014 at 2:18 PM, Rich Signell notifications@github.com
wrote:

@WardF https://github.com/wardf, actually I belive we will need to use
Visual Studio 2008 for compatibility with python 2.7. @aashish24
https://github.com/aashish24, can you confirm that as well?

—
Reply to this email directly or view it on GitHub
#16 (comment).

[rsignell-usgs]

okay, hopefully with @semmerson knowing the UDUNITS codebase, @WardF knowing the Windows stuff, and @aashish24 knowing the CMAKE stuff, we can bring this wild pony into the stable!

[rsignell-usgs]

Reading @WardF's comments a bit closer, it seems that the search.h stuff might be the toughest hurdle to get over. @aashish24, do you guys have a cross-platform solution?

[rsignell-usgs]

I see that in the approach the Vapor gang used, they did this:

add tsearch.c and tsearch.h to project
change <search.h> to "tsearch.h" in include statements in 6 files.

Does this work?

[WardF]

I'm on my way to the vast wilderness that is the middle of nowhere right now (camping in Wyoming) but will take a closer look at tsearch.h Monday. I had missed that in the README that Vapor provided. Have a good weekend!

Sent from my iPhone

On Jul 25, 2014, at 12:29 PM, Rich Signell notifications@github.com wrote:

I see that in the approach the Vapor gang used, they did this:

add tsearch.c and tsearch.h to project
change <search.h> to "tsearch.h" in include statements in 6 files.
Does this work?

—
Reply to this email directly or view it on GitHub.

[aashish24]

@rsignell-usgs I don't have any replacement (cross platform search) for search.h. Seems like you can replace it with tsearch.h as you found out.

[WardF]

I have some more time to look at this and adopting tsearch.c and tsearch.h got through that issue, thanks very much for your input @rsignell-usgs and @aashish24! I've tackled the next issue which came up, the lack of snprintf in MSVC. Now I'm getting a handful of unresolved external symbols errors, which I will investigate and then follow up here.

[WardF]

I'm able to get libudunits2 to build, now, using Visual Studio 2010. My working repository is:

• http://github.com/WardF/UDUNITS-2

Note that it remains untested because my Windows VM is not configured to run cunit-based unit tests!

Next I should be able to find some time in the next day or so to try to get udunits2 itself going. At a glance, it appears that perhaps it only needs to have a replacement getopt dropped in. If this is the case, then it should be fairly straightforward.

@semmerson, does UDUNITS-2 require large file support? I don't believe it would but I'm not certain. If it does, then there is work to be done getting that wired in. It took more effort than anticipated to wire in LFS for netcdf on Windows. So, it's doable, but not something I'm going to even think about unless I know for sure it is required.

Also note there are a number of warnings thrown by the compiler. I'm not worrying too much about those, but want people to be aware that if they compile this, it will be a noisy compile.

[rsignell-usgs]

@WardF. This is exciting! Go Ward Go! 👏

[semmerson]

@WardF UDUNITS doesn't require support for large files. The only files the UDUNITS library deals with are the XML files that contain the units database -- and they're relatively small.

[WardF]

I got a custom getopt wired in, as well as working around the lack of basename and dirname in Visual Studio. The udunits2 program is building fine, but failing to link because libudunits is only generating a DLL and not an export file (.LIB). I remember having to address this issue in the Windows port of netcdf, but I don't remember how. I'll dig into it and see if I can get it sorted out!

On another front, my first-pass research into "CUnit with Visual Studio" suggests that they do not work together, or if so with great difficulty. As a result, out of the box testing of udunits on Windows may need to be by hand, or downstream in the projects which use it. I'll leave that to those more knowledgable about udunits2 than I :)

[semmerson]

Ward,

Excellent progress!

Too bad about CUnit support on Windows. Those tests are fairly
comprehensive.

--Steve

[WardF]

As of commit dad48f3, the UDUNITS-2 fork at http://github.com/WardF/UDUNITS-2 compiles. I don't know if it works, but it compiles. I have to switch gears for a while so any feedback anybody else can contribute would be great, otherwise I will turn my attention back to it when I can later this week :).

[WardF]

Addenum to previous comment: it may be next week before I can look back at this. If anybody wants to try building it, here are the steps I take to build on Windows, using VS2010, with libexpat installed in c:\expat64 or c:\expat32 for 64-and-32-bit builds, respectively.

Forgive the 'linux'-style command line stuff, it's easier for me to type that than windows-style command prompt stuff :).

Compile from shell

Assuming that we're starting in the base UDUNITS-2 directory.

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_PREFIX_PATH=c:\expat64 -G"Visual Studio 10 Win64"
$ cmake --build . --config Release --target libudunits2
$ cmake --build . --config Release --target udunits2

I have not tried compiling any of the other directories, or 'ALL' because I don't know if it will break on other directories. The output of the commands above are the shared .DLL and import .LIB in build/lib/Release/, and udunits2.exe in build/prog/Release/.

Future Work

As mentioned above, I will try to take a look at it later this week, or more likely next week, to see if it actually works or not. Once a naive command-line test works, I will submit a merge to the main project and then withdraw back to my own little world, emerging to provide further assistance as best I can but eschewing an active role in the project due to other commitments. :)

[WardF]

Executive Summary

Things are working. I'll discuss merging my changes into the root project with Steve, but am going to be at a symposium for the next two days. Until then, the project at http://github.com/WardF/UDUNITS-2 contains my progress and can be used or at least tested against. @rsignell-usgs, it would be helpful to know if this actually works for your purposes or if there are problems.

Building

Ok, I've spent some more time with this and here's where we're at. I'm able to compile libudunits and udunits.exe using Visual Studio 2010, 32-and-64-bit versions, using CMake to generate Visual Studio configuration files. I'm also able to build static and shared versions.

Note: Compile in RELEASE mode. Otherwise, you will get any number of warnings or other diagnostic messages when you actually try to use the library.

Running

I'm able to run udunits2.exe from the command line, running through the examples found at http://www.unidata.ucar.edu/software/udunits/udunits-2.0.4/udunits2prog.html. In order to make this work, I had to take the following additional steps.

• Ensure libexpat.dll and libudunits2.dll are on my system path when I run the executable.
• Copy the xml files from UDUNITS-2/lib into C:\Program Files\udunits\share\udunits\

Once I took these steps, I was able to run the executable without any issue.

These changes live at http://github.com/WardF/UDUNITS-2. I will talk with Steve about issuing a pull request into the main project, but I am going to be out of the office at the FRC HPC symposium for the next two days.

If anybody finds any problems please let me know, otherwise I'll consider my involvement here done post-merge and assume that everything works :).

[WardF]

Issued a pull request, #17, which can be automatically merged post-review. The changes affect a number of files but are individually very modest.

[semmerson]

Ward's modification for Visual Studio have been merged into the master branch. If all goes well, a new distribution should be available very soon.

[rsignell-usgs]

Fixed in 77a8494
Woo Hoo!