Improve FindMagic_EP to support system installs #3270

Shelnutt2 · 2022-06-12T16:02:19Z

This improves the support for detection of system installations with libmagic. We remove checking for the find_package and default to always looking for the library and header files for simplicity.

TYPE: IMPROVEMENT
DESC: Improve FindMagic_EP to support system installs of libmagic

shortcut-integration · 2022-06-12T16:02:22Z

This pull request has been linked to Shortcut Story #17869: Generalise cmake setup for libmagic to allow existing libraries and binary files to be used.

eddelbuettel · 2022-06-12T16:31:04Z

Builds cleanly (and more quickly, no libmagic.a recompilation) for me. The console talks about bz2 though:

[...]
-- Starting TileDB superbuild. 
-- Found ccache: /usr/bin/ccache 
-- Found Bzip2: /usr/lib/x86_64-linux-gnu/libbz2.so
-- Found LZ4: /usr/lib/x86_64-linux-gnu/liblz4.so    
-- Looking for pthread.h     
-- Looking for pthread.h - found 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Found Zlib, adding imported target: /usr/lib/x86_64-linux-gnu/libz.so  
-- Found Zstd: /usr/lib/x86_64-linux-gnu/libzstd.so   
-- Found libmagic: /usr/lib/x86_64-linux-gnu/libbz2.so 
-- Found Magic, adding imported target: /usr/lib/x86_64-linux-gnu/libbz2.so
-- Could NOT find CapnProto (missing: CapnProto_DIR) 
-- Cap'n Proto was not found
-- Adding Capnp as an external project
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found suitable version "3.0.2", minimum required is "1.1.0") 
[...]
Install the project...
-- Found Zstd: /usr/lib/x86_64-linux-gnu/libzstd.so 
-- Found libmagic: /usr/lib/x86_64-linux-gnu/libbz2.so 
-- Found Magic, adding imported target: /usr/lib/x86_64-linux-gnu/libbz2.so   
[...]
-- Found libmagic: /usr/lib/x86_64-linux-gnu/libbz2.so     
-- Found Magic, adding imported target: /usr/lib/x86_64-linux-gnu/libbz2.so

and trying to use the library (from R) then fails in

** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘tiledb’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/usr/local/lib/R/site-library/00LOCK-tiledb-r/00new/tiledb/libs/tiledb.so':
  /usr/local/lib/libtiledb.so.2.11: undefined symbol: magic_load_buffers
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/usr/local/lib/R/site-library/tiledb’

eddelbuettel · 2022-06-12T16:31:51Z

cmake/Modules/FindMagic_EP.cmake

-    REQUIRED_VARS libmagic_LIBRARIES libmagic_INCLUDE_DIR
-  )
+          NAMES
+          bz2 libbz2


Should this be magic libmagic ?

Apparently. If I make the change the core library still builds, emits the correct message, and is loadable from R. I'll make the change to the branch and we'll see what CI does.

Console log from the build:

-- Found libmagic: /usr/lib/x86_64-linux-gnu/libmagic.so -- Found Magic, adding imported target: /usr/lib/x86_64-linux-gnu/libmagic.so

eddelbuettel

The suggested change in a commit I made earlier (and which was now force-pushed over) is needed to actually pick the identified system library up:

modified   cmake/Modules/FindMagic_EP.cmake
@@ -57,7 +57,7 @@ elseif(NOT TILEDB_FORCE_ALL_DEPS)
   # Static EP not found, search in system paths.
   find_library(libmagic_LIBRARIES
           NAMES
-          bz2 libbz2
+          magic libmagic
           PATH_SUFFIXES lib bin
           ${TILEDB_DEPS_NO_DEFAULT_PATH}
           )

dhoke4tdb

Attempt local windows build yielded error

CMake Error at cmake/Modules/FindMagic_EP.cmake:132 (message):
  Unable to find libmagic dictionary
Call Stack (most recent call first):
  cmake/TileDB-Superbuild.cmake:89 (include)
  CMakeLists.txt:187 (include)

and I was wondering, but can't get far enuf because of above, if the new location of 'bin' has to be provided to the unit_mgc_dict target items as well...

dhoke4tdb · 2022-06-13T01:01:50Z

cmake/Modules/FindMagic_EP.cmake

+        libmagic${CMAKE_STATIC_LIBRARY_SUFFIX}
+        libmagic${CMAKE_STATIC_LIBRARY_SUFFIX}


Does having this twice do something special?

This improves the support for detection of system installations with libmagic. We remove checking for the find_package and default to always looking for the library and header files for simplicity.

dhoke4tdb · 2022-06-13T10:29:24Z

failed again after pull of recent changes,
tho got further

  No patch step for 'tiledb'
  Performing configure step for 'tiledb'
  -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19044.
  -- The C compiler identification is MSVC 19.32.31329.0
  -- The CXX compiler identification is MSVC 19.32.31329.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.32.31326/bin/Hostx64/x64/cl.exe - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.32.31326/bin/Hostx64/x64/cl.exe - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Starting TileDB regular build.
  -- Performing Test HAVE_AVX2
  -- Performing Test HAVE_AVX2 - Success
  -- Looking for pthread.h
  -- Looking for pthread.h - not found
  -- Found Threads: TRUE
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- Found Catch2: D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src/ep_catch/single_include/catch2
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- Found Bzip2: D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/install/lib/libbz2static.lib
  -- Found LZ4: D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/install/lib/lz4.lib
  -- Found Zlib: D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/install/lib/zlibstatic.lib
  -- Found Zlib, adding imported target: D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/install/lib/zlibstatic.lib
  -- Found Zstd: D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/install/lib/zstd_static.lib
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- Found Win32 lib bcrypt: C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/um/x64/bcrypt.lib
  -- Found Win32 lib shlwapi: C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/um/x64/ShLwApi.Lib
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- searching for catch in D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/src
  -- Found Win32 lib bcrypt: C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/um/x64/RpcRT4.Lib
  -- Found libmagic: D:/dev/tiledb/gh.sethshelnutt.sc-17869.generalise-cmake-setup-for-libmagic-to-allow.git/bld.vs19.A/externals/install/lib/libmagic.lib
  CMake Error at cmake/Modules/FindMagic_EP.cmake:132 (message):
    Unable to find libmagic dictionary
  Call Stack (most recent call first):
    tiledb/sm/misc/CMakeLists.txt:125 (find_package)

Shelnutt2 added the backport release-2.10 label Jun 12, 2022

Shelnutt2 requested review from ihnorton, eddelbuettel and dhoke4tdb June 12, 2022 16:02

Shelnutt2 force-pushed the sethshelnutt/sc-17869/generalise-cmake-setup-for-libmagic-to-allow branch from 782f922 to 3a34665 Compare June 12, 2022 16:24

eddelbuettel reviewed Jun 12, 2022

View reviewed changes

Shelnutt2 force-pushed the sethshelnutt/sc-17869/generalise-cmake-setup-for-libmagic-to-allow branch from 09c0112 to aa9afd3 Compare June 12, 2022 16:51

eddelbuettel requested changes Jun 12, 2022

View reviewed changes

Shelnutt2 force-pushed the sethshelnutt/sc-17869/generalise-cmake-setup-for-libmagic-to-allow branch 2 times, most recently from 86cd63c to 877bbda Compare June 12, 2022 17:48

dhoke4tdb suggested changes Jun 13, 2022

View reviewed changes

Shelnutt2 force-pushed the sethshelnutt/sc-17869/generalise-cmake-setup-for-libmagic-to-allow branch from 877bbda to cd771c7 Compare June 13, 2022 02:38

Improve FindMagic_EP to support system installs

bc235a6

This improves the support for detection of system installations with libmagic. We remove checking for the find_package and default to always looking for the library and header files for simplicity.

Shelnutt2 force-pushed the sethshelnutt/sc-17869/generalise-cmake-setup-for-libmagic-to-allow branch from cd771c7 to bc235a6 Compare June 13, 2022 02:51

Shelnutt2 mentioned this pull request Aug 8, 2022

Bugfix for preparation of magic.mgc data (#3430) #3431

Merged

Shelnutt2 added backport release-2.11 and removed backport release-2.10 labels Aug 9, 2022

ihnorton closed this Sep 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve FindMagic_EP to support system installs #3270

Improve FindMagic_EP to support system installs #3270

Shelnutt2 commented Jun 12, 2022

shortcut-integration bot commented Jun 12, 2022

eddelbuettel commented Jun 12, 2022 •

edited

Loading

eddelbuettel Jun 12, 2022

eddelbuettel Jun 12, 2022

eddelbuettel Jun 12, 2022 •

edited

Loading

eddelbuettel left a comment

dhoke4tdb left a comment

dhoke4tdb Jun 13, 2022

dhoke4tdb commented Jun 13, 2022

		libmagic${CMAKE_STATIC_LIBRARY_SUFFIX}
		libmagic${CMAKE_STATIC_LIBRARY_SUFFIX}

Improve FindMagic_EP to support system installs #3270

Improve FindMagic_EP to support system installs #3270

Conversation

Shelnutt2 commented Jun 12, 2022

shortcut-integration bot commented Jun 12, 2022

eddelbuettel commented Jun 12, 2022 • edited Loading

eddelbuettel Jun 12, 2022

Choose a reason for hiding this comment

eddelbuettel Jun 12, 2022

Choose a reason for hiding this comment

eddelbuettel Jun 12, 2022 • edited Loading

Choose a reason for hiding this comment

eddelbuettel left a comment

Choose a reason for hiding this comment

dhoke4tdb left a comment

Choose a reason for hiding this comment

dhoke4tdb Jun 13, 2022

Choose a reason for hiding this comment

dhoke4tdb commented Jun 13, 2022

eddelbuettel commented Jun 12, 2022 •

edited

Loading

eddelbuettel Jun 12, 2022 •

edited

Loading