Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot install BPCells on Mac #6

Closed
longmanz opened this issue Mar 15, 2023 · 26 comments
Closed

Cannot install BPCells on Mac #6

longmanz opened this issue Mar 15, 2023 · 26 comments

Comments

@longmanz
Copy link

Hi @bnprks,
Thank you for developing BPCells.

I still cannot install BPCells on Mac, even following your suggestions at #3 (comment).

I have installed clang, hdf5, using Conda. I also generated an R Makevars file as below:
Screen Shot 2023-03-15 at 1 33 17 PM

However, when installing BPCells, I encounter the following issue:
Screen Shot 2023-03-15 at 2 08 05 PM

I am using macOS 12.4 (M1 chip). R is 4.2.2.

Do you have any idea what is going on here? Many thanks in advance.

@bnprks
Copy link
Owner

bnprks commented Mar 15, 2023

It looks like your Makevars file accidentally assigns both clang-cpp and clang-14 to CXX, whereas I believe clang-14 should be assigned to CC instead. (Here I'm assuming that your clang-cpp binary is still from the clang version 14 package). Although clang supports both C and C++, you need to call the correct binary or else it will error.

In any case, I'd keep working on getting your pkgbuild::check_build_tools to pass, as BPCells can't even start trying to install until that's set.

@longmanz
Copy link
Author

Dear @bnprks ,
Thanks for your quick response. Yes, that was indeed an error I made. Sorry about that. However, after setting the CC and CXX to the correct ones, I still could not get pkgbuild::check_build_tools() to pass.

I then realized that my Mac is with ARM 64 M1 chip. Therefore, I did the following things to fix the issue:
0. create a new Conda environment.

  1. install a ARM64 version of clang: conda install -c conda-forge clang_osx-arm64
  2. install a ARM64 gfortran: conda install -c conda-forge gfortran_osx-arm64
  3. install binutils: conda install -c conda-forge binutils
  4. install R in conda: conda install -c conda-forge r-base
  5. install hdf5: conda install -c conda-forge hdf5
  6. install a bunch of R packages with conda: devtools, ‘dplyr’, ‘tidyr’, ‘ggrepel’, ‘RcppEigen’ GenomicRanges, IRanges
  7. Re-generate the ~/.R/Makevars and use the correct clang and gfortran:
    CC=/Users/uqljian5/opt/anaconda3/envs/clang/bin/clang
    CXX=/Users/uqljian5/opt/anaconda3/envs/clang/bin/clang-cpp
    FC=/Users/uqljian5/opt/anaconda3/envs/clang/bin/gfortran

After the above steps, I am able to get pkgbuild::check_build_tools() to pass!
Screen Shot 2023-03-16 at 6 20 17 PM

However, the installation of BPCells is still not successful (please see log + Session-Info in the attached file).
Log_install_BPCells_2023Mar16.txt

Do you have any idea what is the problem here based on the log file?

In the meantime, I will try to install BPCells using an Intel-chip Mac tonight, and see if this is an issue specific to M1-chip Mac or not.

Many thanks in advance.

@bnprks
Copy link
Owner

bnprks commented Mar 17, 2023

The third line of your log file is: Platform: x86_64-apple-darwin13.4.0 (64-bit). This indicates to me that your R version is running through rosetta 2 in x86 mode. I do not have an M1 mac immediately available to test on, but I would worry that your setup could be compiling BPCells as ARM code while trying to run from an x86 R version. Unfortunately I won't be able to provide much more help given that I'm unfamiliar with getting things set up on the ARM macs.

If you are able to resolve this yourself, please do leave a comment to help out future M1 users. In the mean time I'll see if I can find an M1 computer to check that the installation works in my hands. For what it's worth, I have successfully installed BPCells on ARM linux computers, so it should be compatible with the CPU architecture

@longmanz
Copy link
Author

Hi @bnprks ,
Yes, that makes sense. The R seems to be a wrong version. I will keep testing and once I have a solution, I will post it here. Thank you for your help on this!

@bnprks
Copy link
Owner

bnprks commented Mar 20, 2023

Hi longmanz,

I just got my hands on an M1 mac to test. I pushed a couple code changes that seemed to have bugs specific to compilation with the MacOS version of clang, so that may also make a difference for you.

To clarify, the suggestions in #3 were specific to old versions of MacOS that lacked full support for C++17 features. It should be fine to use the Apple-provided compilers on all the versions of MacOS that the M1 macs come with.

The installation steps that worked for me in my test were:

  • try running /usr/bin/gcc --version, make sure it runs okay
    • If not run xcode-select --install to install command line tools. You may need to run rm -rf /Library/Developer/CommandLineTools first if you have an x86 version of the command line tools leftover from transferring an x86 backup
  • Install R as normal (preferrably not through conda -- I have had many problems over the years mixing R with conda)
  • Install hdf5 (optionally through conda, or homebrew, etc.)
    • If installing through conda, use this https://stackoverflow.com/a/65432861 to make sure you are installing the arm version of hdf5
    • CONDA_SUBDIR=osx-arm64 conda create -n osx-arm hdf5
    • conda activate osx-arm
  • Run your non-conda R (/usr/local/bin/R) (while the conda env is active if hdf5 installed through conda)
    • ensure the lines at the top include Platform: aarch64-apple-darwin...
  • run devtools::install_github("bnprks/BPCells")

And the version information I was running with:

(osx-arm) bparks@DN52em4l bpcells-install-test % clang++ --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
(osx-arm) bparks@DN52em4l bpcells-install-test % which clang++
/usr/bin/clang++
(osx-arm) bparks@DN52em4l bpcells-install-test % /usr/local/bin/R --version
R version 4.1.3 (2022-03-10) -- "One Push-Up"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: aarch64-apple-darwin20 (64-bit)

Hope that helps -- in general if you get a compiler error that happens while BPCells is actively installing (e.g. after you see the first command printed starting with g++ or clang++), then that is an error I can likely help debug. For other general issues caused by your specific computer's configuration you'll probably have better luck getting help from someone in your lab. I think this case might have been a mix of both, but I think the BPCells side of things should be solved now

@bnprks bnprks closed this as completed Mar 20, 2023
@longmanz
Copy link
Author

longmanz commented Mar 21, 2023

Dear bnprks,
Thank you so much for your help! I have successfully installed BPCells following your suggestions, without any problem.

To briefly summarize @bnprks 's suggestions for installing BPCells on Mac:

  1. If you are using a Mac with Intel CPU, please follow the suggestions at Compilation issues on macOS #3 (comment)
  2. If you are using a Mac with M1 (Macs after 2020), please follow the procedures above (Cannot install BPCells on Mac #6 (comment)).

In addition, if anyone with M1 chip is following 2, please make sure to remove the ~/.R/Makevars if you have previously created/altered it, so that R will not use the wrong compilers.

Again, many thanks for bnprks's help on this.

@bnprks
Copy link
Owner

bnprks commented Mar 24, 2023

One slight clarification -- the special instructions you linked for Intel Macs should only be necessary if you're running OS X 10.14 Mojave (2018) and older. The Apple-provided compilers work fine in the newer OS versions.

@ccnawrocki
Copy link

Hi, I am having trouble downloading BPCells:

Downloading GitHub repo bnprks/BPCells@HEAD
── R CMD build ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file ‘/private/var/folders/_6/qhytbgh10k5cp65dy9zhhcym0000gp/T/RtmpMWAGDv/remotes3d352b4464d1/bnprks-BPCells-9d1a4ca/DESCRIPTION’ ...
─  preparing ‘BPCells’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building ‘BPCells_0.1.0.tar.gz’
   
* installing *source* package ‘BPCells’ ...
** using staged installation
Testing hdf5 by compiling example program...


Retrying with pkg-config flags...
Error running 'pkg-config hdf5 --cflags --libs'


Searching for config information with h5cc -showconfig and h5cc -show...
Failure running either 'h5cc -show' or: 'h5cc -showconfig | awk -F: '/FLAGS|Extra libraries:/ {printf(%s , $2)}'


Searching for hdf5 in a conda env...
no conda environment found in '$CONDA_PREFIX'


Unable to locate libhdf5. Please install manually or edit compiler flags.
ERROR: configuration failed for package ‘BPCells’
* removing ‘/Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library/BPCells’
Warning: installation of package ‘/var/folders/_6/qhytbgh10k5cp65dy9zhhcym0000gp/T//RtmpMWAGDv/file3d351736eb88/BPCells_0.1.0.tar.gz’ had non-zero exit status

Any suggestions?

@bnprks
Copy link
Owner

bnprks commented Jun 8, 2023

The error message is stating it's having trouble locating the hdf5 dependency. I believe on Macs the most reliable way to install hdf5 for BPCells is via homebrew: brew install hdf5. I'd start there. If you still have trouble and are on an ARM mac, make sure that you are running an ARM version of R and not an x86 version via Rosetta 2.

If all that fails, this comment thread gives some information on how to enable additional debug messages to help diagnose the issue. If you do need to include a long debug message, consider following these instructions to put it in a collapsible section.

@ccnawrocki
Copy link

Installing with homebrew worked. Much appreciated.

@mbm2224
Copy link

mbm2224 commented Jun 26, 2023

Hi, I'm wondering if it is possible to install BPCells with R for x86 (not ARM64) on a Mac M1 because many bioconductor R packages are not compatible with ARM64, so I've been using x86 via rosetta, and I'd like to be able to continue using these.

The full error message is in the collapsed section and attached.

error from (remotes::install_github("bnprks/BPCells"))
R version 4.2.0 (2022-04-22) -- "Vigorous Calisthenics"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> pkgbuild::check_build_tools()
Your system is ready to build packages!
> devtools::install_github("bnprks/BPCells")
Downloading GitHub repo bnprks/BPCells@HEAD
   checking for file ‘/private/var/folders/ch/bjcxq2_97_sdqwd6vxxsj48m0000gn/T/R✔  checking for file ‘/private/var/folders/ch/bjcxq2_97_sdqwd6vxxsj48m0000gn/T/RtmpLsZqox/remotesc0dd7ba3cbd0/bnprks-BPCells-9d1a4ca/DESCRIPTION’
─  preparing ‘BPCells’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building ‘BPCells_0.1.0.tar.gz’
   
Installing package into ‘/Users/madelinemariani/Library/R/x86_64/4.2/library’
(as ‘lib’ is unspecified)
* installing *source* package ‘BPCells’ ...
** using staged installation
+ ENABLE_INSTALL_COUNTING=yes
+ '[' -n yes ']'
+ curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CC
+ CC='clang -mmacosx-version-min=10.13'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXX
+ CXX='clang++ -mmacosx-version-min=10.13 -std=gnu++14'
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CFLAGS
+ CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.1/include
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXXFLAGS
+ CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.1/include
++ /Library/Frameworks/R.framework/Resources/bin/R CMD config LDFLAGS
+ LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.1/lib
+ echo 'Testing hdf5 by compiling example program...'
Testing hdf5 by compiling example program...
+ HDF5_CFLAGS=
+ HDF5_LIBS=-lhdf5
+ HDF5_OK=
+ clang -mmacosx-version-min=10.13 tools/h5write.c -I/opt/homebrew/Cellar/hdf5/1.14.1/include -L/opt/homebrew/Cellar/hdf5/1.14.1/lib -lhdf5 -o tools/h5write
ld: warning: ignoring file /opt/homebrew/Cellar/hdf5/1.14.1/lib/libhdf5.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
  "_H5Dclose", referenced from:
      _main in h5write-a11140.o
  "_H5Dcreate2", referenced from:
      _main in h5write-a11140.o
  "_H5Dwrite", referenced from:
      _main in h5write-a11140.o
  "_H5Fclose", referenced from:
      _main in h5write-a11140.o
  "_H5Fcreate", referenced from:
      _main in h5write-a11140.o
  "_H5Sclose", referenced from:
      _main in h5write-a11140.o
  "_H5Screate_simple", referenced from:
      _main in h5write-a11140.o
  "_H5T_NATIVE_INT_g", referenced from:
      _main in h5write-a11140.o
  "_H5Tclose", referenced from:
      _main in h5write-a11140.o
  "_H5Tcopy", referenced from:
      _main in h5write-a11140.o
  "_H5Tset_order", referenced from:
      _main in h5write-a11140.o
  "_H5check_version", referenced from:
      _main in h5write-a11140.o
  "_H5open", referenced from:
      _main in h5write-a11140.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
+ '[' -z ']'
+ printf '\n\nRetrying with pkg-config flags...\n'


Retrying with pkg-config flags...
++ pkg-config hdf5 --cflags
+ HDF5_CFLAGS='Package hdf5 was not found in the pkg-config search path.
Perhaps you should add the directory containing `hdf5.pc'\''
to the PKG_CONFIG_PATH environment variable
No package '\''hdf5'\'' found'
+ echo 'Error running '\''pkg-config hdf5 --cflags --libs'\'''
Error running 'pkg-config hdf5 --cflags --libs'
+ '[' -z ']'
+ printf '\n\nSearching for config information with h5cc -showconfig and h5cc -show...\n'


Searching for config information with h5cc -showconfig and h5cc -show...
++ h5cc -showconfig
+ H5CC_CONFIG='	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.14.1-2
                  Configured on: Fri May 12 14:02:44 UTC 2023
                  Configured by: brew@Ventura-arm64.local
                    Host system: aarch64-apple-darwin22.4.0
              Uname information: Darwin Ventura-arm64.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:55:35 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_VMAPPLE arm64
                       Byte sex: little-endian
             Installation point: /opt/homebrew/Cellar/hdf5/1.14.1

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables: 
                        LDFLAGS: 
                     H5_LDFLAGS:  -Wl,-commons,use_dylibs
                     AM_LDFLAGS:  -L/opt/homebrew/opt/libaec/lib
                Extra libraries: -lsz -lz -ldl -lm 
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: clang ( Apple clang version 14.0.3 )
                       CPPFLAGS: 
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling
                    AM_CPPFLAGS:  -I/opt/homebrew/opt/libaec/include
                        C Flags: 
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3
                     AM C Flags: 
               Shared C Library: yes
               Static C Library: yes


                        Fortran: yes
               Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.1.0) 13.1.0)
                  Fortran Flags: 
               H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -Wno-array-temporaries -O3
               AM Fortran Flags: 
         Shared Fortran Library: yes
         Static Fortran Library: yes
               Module Directory: ${includedir}

                            C++: yes
                   C++ Compiler: clang++ ( Apple clang version 14.0.3 )
                      C++ Flags: 
                   H5 C++ Flags:   -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                   AM C++ Flags: 
             Shared C++ Library: yes
             Static C++ Library: yes

                           Java: no


Features:
---------
                     Parallel HDF5: no
  Parallel Filtered Dataset Writes: no
                Large Parallel I/O: no
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: no
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib),szip(encoder)
                     Map (H5M) API: no
                        Direct VFD: no
                        Mirror VFD: no
                     Subfiling VFD: no
                (Read-Only) S3 VFD: no
              (Read-Only) HDFS VFD: no
    Packages w/ extra debug output: none
                       API tracing: no
              Using memory checker: no
            Function stack tracing: no
                  Use file locking: best-effort
         Strict file format checks: no
      Optimization instrumentation: no'
++ echo '	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.14.1-2
                  Configured on: Fri May 12 14:02:44 UTC 2023
                  Configured by: brew@Ventura-arm64.local
                    Host system: aarch64-apple-darwin22.4.0
              Uname information: Darwin Ventura-arm64.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:55:35 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_VMAPPLE arm64
                       Byte sex: little-endian
             Installation point: /opt/homebrew/Cellar/hdf5/1.14.1

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables: 
                        LDFLAGS: 
                     H5_LDFLAGS:  -Wl,-commons,use_dylibs
                     AM_LDFLAGS:  -L/opt/homebrew/opt/libaec/lib
                Extra libraries: -lsz -lz -ldl -lm 
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: clang ( Apple clang version 14.0.3 )
                       CPPFLAGS: 
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling
                    AM_CPPFLAGS:  -I/opt/homebrew/opt/libaec/include
                        C Flags: 
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3
                     AM C Flags: 
               Shared C Library: yes
               Static C Library: yes


                        Fortran: yes
               Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.1.0) 13.1.0)
                  Fortran Flags: 
               H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -Wno-array-temporaries -O3
               AM Fortran Flags: 
         Shared Fortran Library: yes
         Static Fortran Library: yes
               Module Directory: ${includedir}

                            C++: yes
                   C++ Compiler: clang++ ( Apple clang version 14.0.3 )
                      C++ Flags: 
                   H5 C++ Flags:   -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                   AM C++ Flags: 
             Shared C++ Library: yes
             Static C++ Library: yes

                           Java: no


Features:
---------
                     Parallel HDF5: no
  Parallel Filtered Dataset Writes: no
                Large Parallel I/O: no
                High-level library: yes
Dimension scales w/ new references: no
                  Build HDF5 Tests: yes
                  Build HDF5 Tools: yes
                   Build GIF Tools: no
                      Threadsafety: no
               Default API mapping: v114
    With deprecated public symbols: yes
            I/O filters (external): deflate(zlib),szip(encoder)
                     Map (H5M) API: no
                        Direct VFD: no
                        Mirror VFD: no
                     Subfiling VFD: no
                (Read-Only) S3 VFD: no
              (Read-Only) HDFS VFD: no
    Packages w/ extra debug output: none
                       API tracing: no
              Using memory checker: no
            Function stack tracing: no
                  Use file locking: best-effort
         Strict file format checks: no
      Optimization instrumentation: no'
++ awk -F: '/FLAGS|Extra libraries:/ {printf("%s ", $2)}'
+ H5CC_CONFIG='    -Wl,-commons,use_dylibs   -L/opt/homebrew/opt/libaec/lib  -lsz -lz -ldl -lm   cr      -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling   -I/opt/homebrew/opt/libaec/include '
++ h5cc -show
+ H5CC_CONFIG2='clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5/1.14.1/lib /opt/homebrew/Cellar/hdf5/1.14.1/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5/1.14.1/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
+ HDF5_CFLAGS=
+ HDF5_LIBS=
+ HDF5_RPATH_LAST_ARG=false
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr false = true
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr '' : '.*-L/opt/homebrew/opt/libaec/lib'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib' : '.*-lsz'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz' : '.*-lz'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz' : '.*-ldl'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl' : '.*-lm'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr '' : '.*-I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling'
+ HDF5_CFLAGS=' -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling' : '.*-I/opt/homebrew/opt/libaec/include'
+ HDF5_CFLAGS=' -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include' : '.*-I/opt/homebrew/opt/libaec/include'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm' : '.*-L/opt/homebrew/Cellar/hdf5/1.14.1/lib'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib' : '.*-L/opt/homebrew/opt/libaec/lib'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib' : '.*-lsz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib' : '.*-lz'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib' : '.*-ldl'
+ for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
+ case "$arg" in
+ expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib' : '.*-lm'
+ HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib -lhdf5'
+ clang -mmacosx-version-min=10.13 tools/h5write.c -I/opt/homebrew/Cellar/hdf5/1.14.1/include -L/opt/homebrew/Cellar/hdf5/1.14.1/lib -I/private/tmp/hdf5-20230512-77393-d1q2kf/hdf5-1.14.1-2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.1/lib -lhdf5 -o tools/h5write
ld: warning: ignoring file /opt/homebrew/opt/libaec/lib/libsz.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
ld: warning: ignoring file /opt/homebrew/Cellar/hdf5/1.14.1/lib/libhdf5.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
  "_H5Dclose", referenced from:
      _main in h5write-5876a6.o
  "_H5Dcreate2", referenced from:
      _main in h5write-5876a6.o
  "_H5Dwrite", referenced from:
      _main in h5write-5876a6.o
  "_H5Fclose", referenced from:
      _main in h5write-5876a6.o
  "_H5Fcreate", referenced from:
      _main in h5write-5876a6.o
  "_H5Sclose", referenced from:
      _main in h5write-5876a6.o
  "_H5Screate_simple", referenced from:
      _main in h5write-5876a6.o
  "_H5T_NATIVE_INT_g", referenced from:
      _main in h5write-5876a6.o
  "_H5Tclose", referenced from:
      _main in h5write-5876a6.o
  "_H5Tcopy", referenced from:
      _main in h5write-5876a6.o
  "_H5Tset_order", referenced from:
      _main in h5write-5876a6.o
  "_H5check_version", referenced from:
      _main in h5write-5876a6.o
  "_H5open", referenced from:
      _main in h5write-5876a6.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
+ '[' -z ']'
+ printf '\n\nSearching for hdf5 in a conda env...\n'


Searching for hdf5 in a conda env...
+ '[' -z '' ']'
+ echo 'no conda environment found in '\''$CONDA_PREFIX'\'''
no conda environment found in '$CONDA_PREFIX'
+ '[' -z ']'
+ printf '\n\nUnable to locate libhdf5. Please install manually or edit compiler flags.\n'


Unable to locate libhdf5. Please install manually or edit compiler flags.
+ exit 1
ERROR: configuration failed for package ‘BPCells’
* removing ‘/Users/madelinemariani/Library/R/x86_64/4.2/library/BPCells’
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/ch/bjcxq2_97_sdqwd6vxxsj48m0000gn/T//RtmpLsZqox/filec0dd3e9a041a/BPCells_0.1.0.tar.gz’ had non-zero exit status
> sessionInfo()
R version 4.2.0 (2022-04-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur/Monterey 10.16

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] magrittr_2.0.3    usethis_2.1.5     devtools_2.4.3    pkgload_1.2.4    
 [5] R6_2.5.1          rlang_1.1.1       fastmap_1.1.1     tools_4.2.0      
 [9] pkgbuild_1.3.1    sessioninfo_1.2.2 cli_3.6.1         withr_2.5.0      
[13] ellipsis_0.3.2    remotes_2.4.2     rprojroot_2.0.3   lifecycle_1.0.3  
[17] crayon_1.5.1      brio_1.1.3        processx_3.5.3    purrr_1.0.1      
[21] callr_3.7.0       vctrs_0.6.3       fs_1.5.2          ps_1.7.0         
[25] curl_4.3.2        testthat_3.1.4    memoise_2.0.1     glue_1.6.2       
[29] cachem_1.0.8      compiler_4.2.0    desc_1.4.1        prettyunits_1.1.1
> q()

~/.R/Makevars is

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.1/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.1/include
LDFLAGS=-L/opt/homebrew/Cellar/hdf5/1.14.1/lib

Any help with this would be appreciated! Thank you in advance for your time.

bpcells_installation_error.odt

@bnprks
Copy link
Owner

bnprks commented Jun 26, 2023

I'm not aware of an easy and reliable option yet to get an x86 installation on an M1 Mac. The main difficulty is the installation of the hdf5 library which BPCells depends on. Homebrew appears to be a reliable source on Mac, but it defaults to ARM as you've noticed. For whatever reason, installing hdf5 through conda on an ARM mac seems to be very finicky and cause compiler issues I don't know how to fix.

The next option I might suggest trying would be to try getting hdf5 from the x86 version of homebrew (instructions). This would probably involve uninstalling hdf5 from your existing homebrew install, then installing under the x86 version of homebrew. The other possibility is that the hdf5 website seems to offer downloads with the creation of a free account.

Unfortunately I don't have an M1 mac to test on, so I can't tell you for sure if these options will work. But I think they are the next logical direction to try, and hopefully you can give an update if something works (or doesn't) for you.

@bnprks bnprks reopened this Jun 26, 2023
@mbm2224
Copy link

mbm2224 commented Jun 27, 2023

Success! Thank you so much for your quick response!

What I did:

  1. Uninstalled hdf5
  2. Updated rosetta
    softwareupdate --install-rosetta
  3. This is where it gets complicated. I was using this tutorial. It's a bit outdated, so it wanted me to duplicate terminal, but I can no longer do that on mac M1. So, following this tutorial, to the end of ~/.zshrc file I added:
alias arm="env /usr/bin/arch -arm64 /bin/zsh --login"

alias intel="env /usr/bin/arch -x86_64 /bin/zsh --login" 
  1. Now, when I open a new terminal window, type 'intel' to login to x86 architecture. I can also type 'arm' for ARM, but this is the default. To check which I am using, I use 'arch' which returns 'arm64' for ARM and 'i386' for x86.
  2. Open a new terminal window and type 'intel'
  3. run brew install hdf5 llvm
  4. Check that these are installed in /usr/local using brew info
  5. Then (based on steps outlined in this issue thread), I checked locations of hdf5.h and llvm and updated ~/.R/Makevars based on those lcoations:
CC=/usr/local/Cellar/llvm/16.0.2/bin/clang
CXX=/usr/local/Cellar/llvm/16.0.2/bin/clang++
CFLAGS=-I/usr/local/include
CXXFLAGS=-I/usr/local/include

For x86, brew installs llvm and brew in /usr/local, so they must be there.
9. Then start an R session and run devtools::install_github("bnprks/BPCells"). Now I can call BPCells in an R session or in RStudio

Thanks again for your help!!

@bnprks
Copy link
Owner

bnprks commented Jun 28, 2023

Glad it's working for you, and thanks for providing the detailed steps of what you did to make it work

@bnprks bnprks closed this as completed Jun 28, 2023
@bhayete-empress
Copy link

bhayete-empress commented Aug 16, 2023

I installed hdf5 using brew, but R still fails to install BPCells:
Testing hdf5 by compiling example program...

Retrying with pkg-config flags...
Error running 'pkg-config hdf5 --cflags --libs'

Any suggestions?

P.S. I've removed ~/.R/Makevars and restarted RStudio after installing hdf5

@bnprks
Copy link
Owner

bnprks commented Aug 19, 2023

Hi @bhayete-empress, to get more complete debugging information, I'd suggest you try running Sys.setenv(BPCELLS_DEBUG_INSTALL="true") prior to remotes::install_github("bnprks/BPCells"). Please use a collapsible section if you include that output in a future comment.

For M1/M2 Mac users, having mismatched x86 and ARM binaries is a frequent source of errors, e.g. homebrew installs ARM-based hdf5 by default, but if you have x86-based R I think this will generate errors. If you're on an M1/M2 mac, could you also confirm that your R is running on ARM (check the "Platform" section when running sessionInfo())?

Those tips plus some googling might be enough to get you fixed, but feel free to post back the results of both the BPCELLS_DEBUG_INSTALL log and checking the ARM / x86 binaries if you're still having trouble getting it to work.

@antholee
Copy link

Hi I am having issues with installing BPCells. I verified both hdf5 and R are ARM. I installed hdf5 via homebrew.

Looks like unable to find hdf5. any suggestions? Thank you in advance!

Parallel HDF5: no
Parallel Filtered Dataset Writes: no
Large Parallel I/O: no
High-level library: yes
Dimension scales w/ new references: no
Build HDF5 Tests: yes
Build HDF5 Tools: yes
Build GIF Tools: no
Threadsafety: no
Default API mapping: v114
With deprecated public symbols: yes
I/O filters (external): deflate(zlib),szip(encoder)
Map (H5M) API: no
Direct VFD: no
Mirror VFD: no
Subfiling VFD: no
(Read-Only) S3 VFD: no
(Read-Only) HDFS VFD: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: no
Function stack tracing: no
Use file locking: best-effort
Strict file format checks: no
Optimization instrumentation: no'

  • H5CC_CONFIG=' -Wl,-commons,use_dylibs -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm cr -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include '
    ++ h5cc -show
  • H5CC_CONFIG2='clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5/1.14.2/lib /opt/homebrew/Cellar/hdf5/1.14.2/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5/1.14.2/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
  • HDF5_CFLAGS=
  • HDF5_LIBS=
  • HDF5_RPATH_LAST_ARG=false
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr false = true
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr '' : '.*-L/opt/homebrew/opt/libaec/lib'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib' : '.*-lsz'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz' : '.*-lz'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz' : '.*-ldl'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl' : '.*-lm'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr '' : '.*-I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling'
  • HDF5_CFLAGS=' -I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling' : '.*-I/opt/homebrew/opt/libaec/include'
  • HDF5_CFLAGS=' -I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include' : '.*-I/opt/homebrew/opt/libaec/include'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm' : '.*-L/opt/homebrew/Cellar/hdf5/1.14.2/lib'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib' : '.*-L/opt/homebrew/opt/libaec/lib'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib' : '.*-lsz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib' : '.*-lz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib' : '.*-ldl'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib' : '.*-lm'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib -lhdf5'
  • clang -arch arm64 tools/h5write.c -falign-functions=64 -Wall -g -O2 -L/opt/R/arm64/lib -I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib -lhdf5 -o tools/h5write
    tools/h5write.c:19:10: fatal error: 'hdf5.h' file not found
    #include "hdf5.h"
    ^~~~~~~~
    1 error generated.
  • '[' -z ']'
  • printf '\n\nSearching for hdf5 in a conda env...\n'

Searching for hdf5 in a conda env...

  • '[' -z '' ']'
  • echo 'no conda environment found in '''$CONDA_PREFIX''''
    no conda environment found in '$CONDA_PREFIX'
  • '[' -z ']'

Unable to locate libhdf5. Please install manually or edit compiler flags.

  • printf '\n\nUnable to locate libhdf5. Please install manually or edit compiler flags.\n'
  • exit 1
    ERROR: configuration failed for package ‘BPCells’
  • removing ‘/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/BPCells’
    Warning message:
    In i.p(...) :
    installation of package ‘/var/folders/83/l9c67pp53b11t0gt3vrp_dgw0000gn/T//Rtmpm2Loqj/file43db63e11/BPCells_0.1.0.tar.gz’ had non-zero exit status

@bnprks
Copy link
Owner

bnprks commented Aug 31, 2023

Thanks @antholee for reaching out and providing the detailed error log. The core of the problem is due to not finding the hdf5 header files at compile time (visible from this line):

clang -arch arm64 tools/h5write.c -falign-functions=64 -Wall -g -O2 -L/opt/R/arm64/lib -I/private/tmp/hdf5-20230824-74768-1107cac/hdf5-1.14.2/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.2/lib -lhdf5 -o tools/h5write
tools/h5write.c:19:10: fatal error: 'hdf5.h' file not found
#include "hdf5.h"
^~~~~~~~
1 error generated.

I'm not exactly sure why this happened for you, but it should be possible to circumvent with a little manual intervention. I'd try the following options in order to see if one of them works for you:

  1. Add the following to your .zshrc (or .bashrc if you use bash), as per this stackoverflow suggestion Then restart your terminal and try installing again.

    export CPATH=/opt/homebrew/include
    export LIBRARY_PATH=/opt/homebrew/lib

    After restarting your shell, you can confirm everything was set correctly by running echo $CPATH which should print /opt/homebrew/include

  2. Create/edit the file ~/.R/Makevars and include the following lines:

    CFLAGS=-I/opt/homebrew/include
    CXXFLAGS=-I/opt/homebrew/include
    LDFLAGS=-L/opt/homebrew/lib

    [EDIT 11/28/2023: added line for CXXFLAGS]

  3. Try (2), but instead put CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.2/include (here I'm assuming the folder /opt/homebrew/Cellar/hdf5/1.14.2/include will contain a file called hdf5.h)

All these options are aiming to tell the compiler where to find the hdf5 libraries + headers, and are given in order of most likely to improve all things you compile to most specific to just the homebrew-installed hdf5 libraries.

Let me know if any of them work for you, and also which version of MacOS you're running so I can update the installation instructions for users with your setup.

@kkelley85
Copy link

Hi @bnprks , I was having similar issues as antholee with hdf5 on M2 Mac and your option (3) above seemed to work however now I'm getting the below error. Thank you.

Error output * checking for file ‘/private/var/folders/rx/0rmp3h8x26gghc7lh0p70gxc0000gq/T/RtmpTXNVqQ/remotes2ee07ae60f0/bnprks-BPCells-ac4376d/DESCRIPTION’ ... OK * preparing ‘BPCells’: * checking DESCRIPTION meta-information ... OK * cleaning src * running ‘cleanup’ * checking for LF line-endings in source and make files and shell scripts * checking for empty or unneeded directories * building ‘BPCells_0.1.0.tar.gz’ Testing hdf5 by compiling example program... Found working hdf5 HDF5_CFLAGS='' HDF5_LIBS='-lhdf5'

Testing architecture flag support support...'-march=native' succeeded
ARCH_FLAG='-march=native'
clang++ -arch arm64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include' -I/opt/R/arm64/include -march=native -std=c++17 -DRCPP_EIGEN -DEIGEN_PERMANENTLY_DISABLE_STUPID_WARNINGS -Wno-ignored-attributes -Wno-unknown-pragmas -fPIC -falign-functions=64 -Wall -g -O2 -c bitpacking_io.cpp -o bitpacking_io.o
Warning message:
In i.p(...) :
installation of package ‘/var/folders/rx/0rmp3h8x26gghc7lh0p70gxc0000gq/T//RtmpTXNVqQ/file2ee05c5d089c/BPCells_0.1.0.tar.gz’ had non-zero exit status

sessionInfo() R version 4.3.1 (2023-06-16) Platform: aarch64-apple-darwin20 (64-bit) Running under: macOS Sonoma 14.0

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] compiler_4.3.1 tools_4.3.1 curl_5.1.0 remotes_2.4.2.1

@bnprks
Copy link
Owner

bnprks commented Oct 19, 2023

Hi @kkelley85, a couple notes from your helpfully attached logs:

  1. Based on your log, it looks like the config script likely ran successfully and found a working HDF5
  2. I'd suspect the error would have happened when it began compilation (running the line starting with clang++ -arch arm64)
  3. It's not clear to me where the actual error during your compilation happened, as there's no compiler error message printed out, just the message about "non-zero exit status". Is this the full extent of error messages printed out during the failed install?

Could you double-check that there is not any other error output produced during compilation? (Perhaps if you are running in RStudio some errors might be in a different output tab).

Otherwise, could you include the contents of the file ~/.R/Makevars if you have one, and maybe re-run with Sys.setenv(BPCELLS_DEBUG_INSTALL="true") prior to remotes::install_github("bnprks/BPCells") in case that generates more informative error messages?

@kkelley85
Copy link

Hi @bnprks, thanks for responding to figure this out. For some reason, when I installed your package by accessing R via the Mac terminal the installation worked. But I kept getting errors when trying to install using the R GUI. Anyways, thanks again.

@DevBioCode
Copy link

Hi @bnprks, I'm seeing an issue similar to the commenters above. I added CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.3/include to my Makevars file and got past the part where it couldn't find hdf5.h, but now it can't find H5Ipublic.h even though it's in the same folder:

Error

  • installing source package ‘BPCells’ ...
    ** using staged installation
  • ENABLE_INSTALL_COUNTING=yes
  • '[' -n yes ']'
  • curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure
    ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CC
  • CC='clang -arch arm64'
    ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXX
  • CXX='clang++ -arch arm64 -std=gnu++17'
    ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CFLAGS
  • CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.3/include
    ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config CXXFLAGS
  • CXXFLAGS='-falign-functions=64 -Wall -g -O2'
    ++ /Library/Frameworks/R.framework/Resources/bin/R CMD config LDFLAGS
  • LDFLAGS=-L/opt/R/arm64/lib
  • echo 'Testing hdf5 by compiling example program...'
  • HDF5_CFLAGS=
  • HDF5_LIBS=-lhdf5
  • HDF5_OK=
    Testing hdf5 by compiling example program...
  • clang -arch arm64 tools/h5write.c -I/opt/homebrew/Cellar/hdf5/1.14.3/include -L/opt/R/arm64/lib -lhdf5 -o tools/h5write
    ld: library 'hdf5' not found
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
  • '[' -z ']'
  • printf '\n\nRetrying with pkg-config flags...\n'

Retrying with pkg-config flags...
++ pkg-config hdf5 --cflags

  • HDF5_CFLAGS='./configure: line 42: pkg-config: command not found'
  • echo 'Error running '''pkg-config hdf5 --cflags --libs''''
    Error running 'pkg-config hdf5 --cflags --libs'
  • '[' -z ']'
  • printf '\n\nSearching for config information with h5cc -showconfig and h5cc -show...\n'

Searching for config information with h5cc -showconfig and h5cc -show...
++ h5cc -showconfig

  • H5CC_CONFIG=' SUMMARY OF THE HDF5 CONFIGURATION
    =================================

General Information:

               HDF5 Version: 1.14.3
              Configured on: Wed Nov  8 05:32:45 UTC 2023
              Configured by: brew@Ventura-arm64.local
                Host system: aarch64-apple-darwin22.6.0
          Uname information: Darwin Ventura-arm64.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:19 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_VMAPPLE arm64
                   Byte sex: little-endian
         Installation point: /opt/homebrew/Cellar/hdf5/1.14.3

Compiling Options:

                 Build Mode: production
          Debugging Symbols: no
                    Asserts: no
                  Profiling: no
         Optimization Level: high

Linking Options:

                  Libraries: static, shared

Statically Linked Executables:
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS: -L/opt/homebrew/opt/libaec/lib
Extra libraries: -lsz -lz -ldl -lm
Archiver: ar
AR_FLAGS: cr
Ranlib: ranlib

Languages:

                          C: yes
                 C Compiler: clang ( Apple clang version 14.0.3 )
                   CPPFLAGS: 
                H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling
                AM_CPPFLAGS:  -I/opt/homebrew/opt/libaec/include
                    C Flags: 
                 H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3
                 AM C Flags: 
           Shared C Library: yes
           Static C Library: yes


                    Fortran: yes
           Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.2.0) 13.2.0)
              Fortran Flags: 
           H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -Wno-array-temporaries -O3
           AM Fortran Flags: 
     Shared Fortran Library: yes
     Static Fortran Library: yes
           Module Directory: ${includedir}

                        C++: yes
               C++ Compiler: clang++ ( Apple clang version 14.0.3 )
                  C++ Flags: 
               H5 C++ Flags:   -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
               AM C++ Flags: 
         Shared C++ Library: yes
         Static C++ Library: yes

                       Java: no

Features:

                 Parallel HDF5: no

Parallel Filtered Dataset Writes: no
Large Parallel I/O: no
High-level library: yes
Dimension scales w/ new references: no
Build HDF5 Tests: yes
Build HDF5 Tools: yes
Build GIF Tools: no
Threadsafety: no
Default API mapping: v114
With deprecated public symbols: yes
I/O filters (external): deflate(zlib),szip(encoder)
Map (H5M) API: no
Direct VFD: no
Mirror VFD: no
Subfiling VFD: no
(Read-Only) S3 VFD: no
(Read-Only) HDFS VFD: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: no
Function stack tracing: no
Use file locking: best-effort
Strict file format checks: no
Optimization instrumentation: no'
++ awk -F: '/FLAGS|Extra libraries:/ {printf("%s ", $2)}'
++ echo ' SUMMARY OF THE HDF5 CONFIGURATION
=================================

General Information:

               HDF5 Version: 1.14.3
              Configured on: Wed Nov  8 05:32:45 UTC 2023
              Configured by: brew@Ventura-arm64.local
                Host system: aarch64-apple-darwin22.6.0
          Uname information: Darwin Ventura-arm64.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:19 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_VMAPPLE arm64
                   Byte sex: little-endian
         Installation point: /opt/homebrew/Cellar/hdf5/1.14.3

Compiling Options:

                 Build Mode: production
          Debugging Symbols: no
                    Asserts: no
                  Profiling: no
         Optimization Level: high

Linking Options:

                  Libraries: static, shared

Statically Linked Executables:
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS: -L/opt/homebrew/opt/libaec/lib
Extra libraries: -lsz -lz -ldl -lm
Archiver: ar
AR_FLAGS: cr
Ranlib: ranlib

Languages:

                          C: yes
                 C Compiler: clang ( Apple clang version 14.0.3 )
                   CPPFLAGS: 
                H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling
                AM_CPPFLAGS:  -I/opt/homebrew/opt/libaec/include
                    C Flags: 
                 H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -Wno-missing-noreturn -O3
                 AM C Flags: 
           Shared C Library: yes
           Static C Library: yes


                    Fortran: yes
           Fortran Compiler: /opt/homebrew/opt/gcc/bin/gfortran ( GNU Fortran (Homebrew GCC 13.2.0) 13.2.0)
              Fortran Flags: 
           H5 Fortran Flags:  -std=f2008  -Waliasing -Wall -Wcharacter-truncation -Wextra -Wimplicit-interface -Wsurprising -Wunderflow -pedantic -Wintrinsics-std -Wimplicit-procedure -Wreal-q-constant -Wfunction-elimination -Wrealloc-lhs -Wrealloc-lhs-all -Wno-c-binding-type -Winteger-division -Wfrontend-loop-interchange  -fdiagnostics-urls=never -fno-diagnostics-color -s  -Wno-unused-dummy-argument -Wno-array-temporaries -O3
           AM Fortran Flags: 
     Shared Fortran Library: yes
     Static Fortran Library: yes
           Module Directory: ${includedir}

                        C++: yes
               C++ Compiler: clang++ ( Apple clang version 14.0.3 )
                  C++ Flags: 
               H5 C++ Flags:   -std=c++11 -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat     -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
               AM C++ Flags: 
         Shared C++ Library: yes
         Static C++ Library: yes

                       Java: no

Features:

                 Parallel HDF5: no

Parallel Filtered Dataset Writes: no
Large Parallel I/O: no
High-level library: yes
Dimension scales w/ new references: no
Build HDF5 Tests: yes
Build HDF5 Tools: yes
Build GIF Tools: no
Threadsafety: no
Default API mapping: v114
With deprecated public symbols: yes
I/O filters (external): deflate(zlib),szip(encoder)
Map (H5M) API: no
Direct VFD: no
Mirror VFD: no
Subfiling VFD: no
(Read-Only) S3 VFD: no
(Read-Only) HDFS VFD: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: no
Function stack tracing: no
Use file locking: best-effort
Strict file format checks: no
Optimization instrumentation: no'

  • H5CC_CONFIG=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm cr -DNDEBUG -UH5_DEBUG_API -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include '
    ++ h5cc -show
  • H5CC_CONFIG2='clang -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/Cellar/hdf5/1.14.3/lib /opt/homebrew/Cellar/hdf5/1.14.3/lib/libhdf5_hl.a /opt/homebrew/Cellar/hdf5/1.14.3/lib/libhdf5.a -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
  • HDF5_CFLAGS=
  • HDF5_LIBS=
  • HDF5_RPATH_LAST_ARG=false
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr '' : '.*-L/opt/homebrew/opt/libaec/lib'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib' : '.*-lsz'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz' : '.*-lz'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz' : '.*-ldl'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl' : '.*-lm'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr '' : '.*-I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling'
  • HDF5_CFLAGS=' -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling' : '.*-I/opt/homebrew/opt/libaec/include'
  • HDF5_CFLAGS=' -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include' : '.*-I/opt/homebrew/opt/libaec/include'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm' : '.*-L/opt/homebrew/Cellar/hdf5/1.14.3/lib'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib' : '.*-L/opt/homebrew/opt/libaec/lib'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib' : '.*-lsz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib' : '.*-lz'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib' : '.*-ldl'
  • for arg in '$H5CC_CONFIG' '$H5CC_CONFIG2'
  • case "$arg" in
  • expr ' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib' : '.*-lm'
  • HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib -lhdf5'
  • clang -arch arm64 tools/h5write.c -I/opt/homebrew/Cellar/hdf5/1.14.3/include -L/opt/R/arm64/lib -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib -lhdf5 -o tools/h5write
  • HDF5_OK=yes
  • '[' -z yes ']'
  • '[' -z yes ']'
  • echo 'Found working hdf5'
    Found working hdf5
  • echo 'HDF5_CFLAGS=''' -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include''''
    HDF5_CFLAGS=' -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include'
  • echo 'HDF5_LIBS=''' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib -lhdf5''''
    HDF5_LIBS=' -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib -lhdf5'
  • ARCH_FLAG=-march=native
  • printf '\nTesting architecture flag support support...'

Testing architecture flag support support...+ for ARCH_FLAG in '"-march=native"' '""'

  • clang -arch arm64 tools/h5write.c -march=native -I/opt/homebrew/Cellar/hdf5/1.14.3/include -L/opt/R/arm64/lib -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib -lhdf5 -o tools/h5write
    '-march=native' succeeded
  • printf ''''%s''' succeeded\n' -march=native
  • break
  • echo 'ARCH_FLAG='''-march=native''''
  • CXX17_OK=
    ARCH_FLAG='-march=native'
  • CXX_FS_FLAG=
  • clang++ -arch arm64 -std=gnu++17 tools/cxx17_filesystem.cc -falign-functions=64 -Wall -g -O2 -L/opt/R/arm64/lib -std=c++17 -o tools/cxx17_filesystem
  • CXX17_OK=yes
  • '[' -z yes ']'
  • '[' -z yes ']'
  • '[' -z yes ']'
  • sed -e 's|%HDF5_CFLAGS%| -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include|g' -e 's|%HDF5_LIBS%| -L/opt/homebrew/opt/libaec/lib -lsz -lz -ldl -lm -L/opt/homebrew/Cellar/hdf5/1.14.3/lib -lhdf5|g' -e 's|%ARCH_FLAG%|-march=native|g' -e 's|%CXX_FS_FLAG%||g' src/Makevars.in
  • '[' -n yes ']'
  • curl --silent https://plausible.benparks.net/flask-plausible/bpcells-configure-success
    ** libs
    using C++ compiler: ‘Apple clang version 15.0.0 (clang-1500.0.40.1)’
    using SDK: ‘MacOSX14.0.sdk’
    clang++ -arch arm64 -std=gnu++17 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/RcppEigen/include' -I/opt/R/arm64/include -I/private/tmp/hdf5-20231108-71041-7ly7w8/hdf5-1.14.3/src/H5FDsubfiling -I/opt/homebrew/opt/libaec/include -march=native -std=c++17 -DRCPP_EIGEN -DEIGEN_PERMANENTLY_DISABLE_STUPID_WARNINGS -Wno-ignored-attributes -Wno-unknown-pragmas -fPIC -falign-functions=64 -Wall -g -O2 -c bitpacking_io.cpp -o bitpacking_io.o
    In file included from bitpacking_io.cpp:7:
    In file included from ./arrayIO/hdf5.h:6:
    In file included from ./arrayIO/../lib/highfive/H5DataSet.hpp:14:
    In file included from ./arrayIO/../lib/highfive/H5DataSpace.hpp:18:
    ./arrayIO/../lib/highfive/H5Object.hpp:14:10: fatal error: 'H5Ipublic.h' file not found
    #include <H5Ipublic.h>
    ^~~~~~~~~~~~~
    1 error generated.
    make: *** [bitpacking_io.o] Error 1
    ERROR: compilation failed for package ‘BPCells’
  • removing ‘/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library/BPCells’
    Warning message:
    In i.p(...) :
    installation of package ‘/var/folders/87/mm01w5x93fz2gdw_y0w3c7d80000gp/T//RtmpQxG01c/file10cd3884ff92/BPCells_0.1.0.tar.gz’ had non-zero exit status

sessionInfo()

R version 4.3.2 (2023-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.5.2

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Los_Angeles
tzcode source: internal

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] compiler_4.3.2 tools_4.3.2 curl_5.1.0 remotes_2.4.2.1

I see the same error in both Terminal and RStudio. The CFLAGS line above is the sole content of Makevars. Appreciate the help!

@bnprks
Copy link
Owner

bnprks commented Nov 15, 2023

Hi @DevBioCode, I think you might also need to add CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.3/include to your Makevars file, and possibly also LDFLAGS=-L/opt/homebrew/lib (assuming an hdf5 .dylib or .so file is in that folder)

Details on what I think was happening in your log (thanks for including) It looks like the initial attempt at finding the hdf5 library failed (hence my LDFLAGS suggestion), so it fell back to running `h5cc` to detect compilation flags for hdf5 (worked during config, but not later during package compilation). Then, I see at the end of your log the command that hit the error (starting with `clang++ -arch arm64`) didn't actually include the compiler flag you tried to add via CFLAGS. I might have mixed this up and only CXXFLAGS is important here.

Could you test if adding CXXFLAGS to Makevars fixes your issue? I'd appreciate if you can report back if that works so we can update the Mac instructions

@DevBioCode
Copy link

Hi @bnprks, it's up and running! Having only CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.3/include in Makevars was not sufficient, as it could not find hdf5.h again. But having both:

CFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.3/include
CXXFLAGS=-I/opt/homebrew/Cellar/hdf5/1.14.3/include

in Makevars led to successful installation, and I've checked that it's running fine too. Setting LDFLAGS was not necessary.

Thanks again for your help, and for making possible these analyses on personal machines!

@mainyanghr
Copy link

I have to see it is very frustrating to install this BPcell package. I hope someone could improve it. My Mac consitantly tell me Hdf4 is not able to find, though I did lots of troubleshooting.

Very sad!!!!!

@bnprks
Copy link
Owner

bnprks commented Nov 29, 2023

Hi @mainyanghr, I'm usually happy to volunteer my time to help troubleshoot people's installation issues, as you can see from the above thread. If you're having trouble, feel free to chime in with what you've tried and what errors you're getting, or open a new dedicated issue. If you're not comfortable with the command line, you might be able to install a pre-built binary from my r-universe account with this line: install.packages('BPCells', repos = c('https://bnprks.r-universe.dev', 'https://cloud.r-project.org')). I don't own an M1 mac so I can't test if it will work for you, but it's another option to try.

If figuring out installation is too much of a challenge for you, then you'll probably prefer to wait until BPCells is available through CRAN.

If you do want to post more details on your issue, I will have to ask that you try to keep a polite and professional tone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants