Modify install-cmake.py to build CMake with SSL (#220)

[bkochuna]

For CMake to be able to submit to a drop site via HTTPS
the cmake/ctest executables must have been built with OpenSSL
support. This commit modifies the install-cmake.py script to:

• Check if CMake is installed on the system
• Configure CMake with CMake and OpenSSL support

The CMakeLists.txt for configuring CMake with CMake with is smart
enough to allow you to still build CMake even if you request OpenSSL
and it is not installed. The only requirement is that CMake be
configured with CMake instead of autotools. If CMake is not available
then install-cmake.py defaults to its previous behavior.

[bkochuna]

Hi Ross,

Summarizing changes here:

• I added an if/else block to the doConfigure method.
  • The if/else branches depending on whether or not a cmake executable is found.
• The configure command was updated for CMake if the executable is found.
  • This sets CMAKE_USE_OPENSSL=ON and the CMAKE_INSTALL_PREFIX for the install-dir passed to the script.
  • The parallel option is not included, since I don't believe CMake does a parallel configure.
• For finding the cmake executable, I added a which method based on this code at stack overflow.
  • https://stackoverflow.com/questions/377017/test-if-executable-exists-in-python
    • Just realized this may be supported intrinsically in python with distutils.spawn.find_executable
  • I kept it localized since I'm not sure what else might need it/want it, but if you think this method should be implemented elsewhere like GeneralScriptSupport or InstallProgramDriver let me know.
  • I expect if you have feedback it'll probably be about changing this.

You had also mentioned probably not implementing a test for this in the issue #220. If you have an idea about this (I'm not sure how to do it) I can try to implement a test.

I'm going to post some follow-on comments with output from my manual testing.

[bkochuna]

Below is the output from a successful configure & install of CMake with CMake and OpenSSL libraries installed.
Short (grepped) Output:

$ grep -i openssl cmake-3.3.2-install.log
Running: /usr/local/bin/cmake ../cmake-3.3.2  -DCMAKE_USE_OPENSSL=ON -DCMAKE_INSTALL_PREFIX=/home/bkochuna/tmp/tmpTribitsDVInstall/common_tools/cmake-3.3.2
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.0.1")
-- Looking for openssl/crypto.h
-- Looking for openssl/crypto.h - found
-- Looking for openssl/engine.h
-- Looking for openssl/engine.h - found
-- Looking for openssl/err.h
-- Looking for openssl/err.h - found
-- Looking for openssl/pem.h
-- Looking for openssl/pem.h - found
-- Looking for openssl/pkcs12.h
-- Looking for openssl/pkcs12.h - found
-- Looking for openssl/rsa.h
-- Looking for openssl/rsa.h - found
-- Looking for openssl/ssl.h
-- Looking for openssl/ssl.h - found
-- Looking for openssl/x509.h
-- Looking for openssl/x509.h - found
-- Looking for openssl/rand.h
-- Looking for openssl/rand.h - found
-- Checking support for ARCHIVE_CRYPTO_MD5_OPENSSL
-- Checking support for ARCHIVE_CRYPTO_MD5_OPENSSL -- found
-- Checking support for ARCHIVE_CRYPTO_RMD160_OPENSSL
-- Checking support for ARCHIVE_CRYPTO_RMD160_OPENSSL -- found
-- Checking support for ARCHIVE_CRYPTO_SHA1_OPENSSL
-- Checking support for ARCHIVE_CRYPTO_SHA1_OPENSSL -- found
-- Checking support for ARCHIVE_CRYPTO_SHA256_OPENSSL
-- Checking support for ARCHIVE_CRYPTO_SHA256_OPENSSL -- found
-- Checking support for ARCHIVE_CRYPTO_SHA384_OPENSSL
-- Checking support for ARCHIVE_CRYPTO_SHA384_OPENSSL -- found
-- Checking support for ARCHIVE_CRYPTO_SHA512_OPENSSL
-- Checking support for ARCHIVE_CRYPTO_SHA512_OPENSSL -- found
[ 60%] Building C object Utilities/cmcurl/lib/CMakeFiles/cmcurl.dir/vtls/openssl.c.o
-- Installing: /home/bkochuna/tmp/tmpTribitsDVInstall/common_tools/cmake-3.3.2/share/cmake-3.3/Help/module/FindOpenSSL.rst
-- Installing: /home/bkochuna/tmp/tmpTribitsDVInstall/common_tools/cmake-3.3.2/share/cmake-3.3/Modules/FindOpenSSL.cmake

[bkochuna]

For a configure without OpenSSL installed the output is:

...

Running: /usr/bin/cmake ../cmake-3.3.2  -DCMAKE_USE_OPENSSL=ON -DCMAKE_INSTALL_PREFIX=/home/bkochuna/tmpTribitsDVInstall/common_tools/cmake-3.3.2
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing:  OPENSSL_LIBRARIES OPENSSL_INCLUDE_DIR)
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing:  OPENSSL_LIBRARIES OPENSSL_INCLUDE_DIR)

...

-- Build files have been written to: /home/bkochuna/tribits_devtools_test/cmake-3.3.2-base/cmake-build

D) Build cmake-3.3.2 ...


Changing current directory to '/home/bkochuna/tribits_devtools_test/cmake-3.3.2-base/cmake-build'

Current directory is '/home/bkochuna/tribits_devtools_test/cmake-3.3.2-base/cmake-build'


Running: make  -j8

...

E) Install cmake-3.3.2 ...


Changing current directory to '/home/bkochuna/tribits_devtools_test/cmake-3.3.2-base/cmake-build'

Current directory is '/home/bkochuna/tribits_devtools_test/cmake-3.3.2-base/cmake-build'


Running: make  install

[  1%] Built target cmIML_test
[  4%] Built target cmsys
[  4%] Built target cmsysTestDynload
[  5%] Built target cmsys_c
[  5%] Built target cmsysTestProcess
[  5%] Built target cmsysTestSharedForward
[  6%] Built target cmsysTestsC
[  8%] Built target cmsysTestsCxx
[ 10%] Built target cmzlib
[ 30%] Built target cmcurl
[ 30%] Built target LIBCURL
[ 30%] Built target cmcompress
[ 32%] Built target cmbzip2
[ 44%] Built target cmliblzma
[ 62%] Built target cmlibarchive
[ 62%] Built target cmexpat
[ 63%] Built target cmjsoncpp
[ 84%] Built target CMakeLib
[ 87%] Built target CPackLib
[ 96%] Built target CTestLib
[ 96%] Built target cmake
[ 96%] Built target cpack
[ 96%] Built target ctest
[ 97%] Built target CMakeLibTests
[ 97%] Built target runcompilecommands
[ 97%] Built target memcheck_fail
[ 97%] Built target pseudo_BC
[ 97%] Built target pseudo_purify
[ 98%] Built target pseudo_valgrind
[ 98%] Built target pseudonl_BC
[ 98%] Built target pseudonl_purify
[ 98%] Built target pseudonl_valgrind
[ 98%] Built target pseudo_emulator
[ 98%] Built target pseudo_iwyu
[100%] Built target foo
Install the project...

So we see that the install procedure still finished without incident.

[bartlettroscoe]

For a configure without OpenSSL installed

So when Open SSL can't be found, the CMake configure of CMake will just exclude it and keep going? Does the produced CMake work correctly?

[bkochuna]

So when Open SSL can't be found, the CMake configure of CMake will just exclude it and keep going?

Correct.

Does the produced CMake work correctly?

I did not test the CMake executables, but I assume so. I can do this (although probably won't have a chance till tomorrow).

[bartlettroscoe]

I think, as a fail safe, we might consider adding the option --use-native-cmake-config=[on,off,auto] with the default being auto (which means try to find a local CMake if you can and use it). But a user could pass in --use-native-cmake-config=on to force a native CMake to be used or --use-native-cmake-config=off to force usage of configure script. What worries me is that there might be systems where an existing CMake was badly installed and is not working and then it will stop us from installing a working version. The default should always be --use-native-cmake-config=auto (including when run from the install_devtools.py script) since that gives us the greatest features from the installed CMake (for now OpenSSL support for https submits).

[bkochuna]

Hi Ross,

I've added an option called --user-native-cmake-config with the options [on,off,auto]. Let me know if there's anything else.

[bartlettroscoe]

@bkochuna, the updated script looks great. I will give it a try myself and see how it goes. If that all checks out, then I will go ahead and merge the PR branch.

Thanks!

[bartlettroscoe]

@bkochuna, I have posted a new PR #306 that merged in this branch and makes a few tweaks to make this robust out of the box:

• By default, must use bootstrap and not a native cmake, even if one is available (e.g. CMake 2.8.11 is the default CMake on RHEL7 but that can't be used to build CMake 3.16). Also note that you can pass extra CMake options to bootstrap so I don't see any advantage in not allowing CMake to booststrap itself.
• By default, OpenSSL support is disabled because it did not work out of the box on any of the machines tried. To do this, I added the option --use-openssl which is off by default.

Sorry it took me so long to get this this. I was hoping that Spack would make install-cmake.py obsolute. But if you just clone the Spack GitHub repo and they run sparck install cmake@X.Y.Z, it will try to download and install Perl, Python, and a who bunch of other packages that may not build on your machine. The default Spack install of CMake is not robust :-(