MacOS build linker warning

[afrubin]

I get the following warning on Mac OS. It hasn't interfered with loading the package or running the beginning of the Introduction vignette so far. If there is a specific function or set of functions that rely on gfortran that I can test please let me know.

ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin15/6.1.0'
ld: warning: directory not found for option '-L/usr/local/gfortran/lib'

[mikldk]

Thank you for reporting this. What version of R and Mac OSX do you run?

I think it is more related to your local installation than to the package. Maybe you can try the suggestion at https://stackoverflow.com/a/36001449?

[afrubin]

Yes, this does seem like a local installation issue. I'll try installing the homebrew gcc and updating the R files. This should be mentioned in the documentation as a workaround.

As an aside, if a separated installation of gcc and modification of .R/Makevars is required, then this means that -fopenmp can be re-added to the compiler flags if it improves performance, see #9.

[mikldk]

Yes, added at the user's Makevars?

Where do you suggest this is added? Readme? And then a new release? Or is that not necessary?

[afrubin]

My Makevars contained:

CC=clang
CXX=clang++

and I have updated it to contain:

CC=gcc
CXX=g++

I suggest adding a sub-heading under Installation called "For Mac OS users" or similar with a brief explanation:

Building this package requires gfortran, so a local installation of gcc is required.

and a brief set of instructions for novice users:

The easiest way is to install using homebrew with the command brew install gcc. Your ~/.R/Makevars will need to be updated to use gcc instead of clang (insert what Makevars should be changed to).

A new release is not necessary for this. I recommend waiting until all pre-publication changes are completed and then releasing a 0.0.3.1 (or even a 0.1 to celebrate acceptance!). The manuscript can be updated to match the correct version and DOI at that point.

[mikldk]

Thanks. I use clang myself on Ubuntu so I think that it would be wrong to claim that gcc is necessary. But I don't know MacOSX and it may be correct there. Do you know that? Or else I might formulate it a bit.

[afrubin]

The version of clang that Apple provides on Mac OS lacks certain features, so the common workaround is to install gcc via homebrew.

[afrubin]

Ok, I have tried several different ways to solve this problem and I have managed to build successfully without linker errors. Unfortunately, I now get a segfault when running the tests. A detailed description of what I've done as well as output from the segfault follows.

From what I can find online, it appears that the way to get the gfortran libraries is to install a local version of gcc using homebrew. The current version of gcc in homebrew is 8.1.0 so that is what I have installed.

After that is installed, the ~/.R/Makevars file needs to be set up to use those libraries. I am also using g++ as the C++ compiler. I modified my ~/.R/Makevars based on the documentation in the official R Installation and Administration manual.

BREWPATH=/path/to/homebrew/
CCVER=8
CC=${BREWPATH}/bin/gcc-${CCVER}
CXX=${BREWPATH}/bin/g++-${CCVER}
CXX11=${BREWPATH}/bin/g++-${CCVER}
F77=${BREWPATH}/bin/gfortran-${CCVER}
FC=${BREWPATH}/bin/gfortran-${CCVER}
FLIBS=-L${BREWPATH}/lib/gcc/${CCVER} -lgfortran -lquadmath -lm

The build also completes successfully with only the CXX11 and FLIBS lines set, so I'm not sure if any of the other lines are required.

To try to make sure that there wasn't a strange interaction with a dependency that was already built, I also removed all user installed packages using this script.

I can build and load the package, but when I try to run the tests I get a segfault. I have successfully built the package at least five times with various permutations of the ~/.R/Makevars and I consistently get the same output (with some slight variations in memory address) every time.

Do you have a suggestion for how to address this? Clearly it's something specific to my environment but I'm now well outside my area of expertise.

> library('malan')
> library('testthat')
> test_package('malan')

 *** caught segfault ***
address 0x40, cause 'memory not mapped'

Traceback:
 1: .Call("_malan_estimate_theta_subpops_individuals", PACKAGE = "malan",     subpops, subpops_sizes)
 2: estimate_theta_subpops_individuals(l_nonindv, l_nonindv_sizes)
 3: eval_bare(get_expr(quo), get_env(quo))
 4: doTryCatch(return(expr), name, parentenv, handler)
 5: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6: tryCatchList(expr, classes, parentenv, handlers)
 7: tryCatch({    code    NULL}, error = function(e) e)
 8: capture(act$val <- eval_bare(get_expr(quo), get_env(quo)))
 9: quasi_capture(enquo(object), capture_error, label = label)
10: expect_error(estimate_theta_subpops_individuals(l_nonindv, l_nonindv_sizes))
11: eval(code, test_env)
12: eval(code, test_env)
13: withCallingHandlers({    eval(code, test_env)    if (!handled && !is.null(test)) {        skip_empty()    }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning,     message = handle_message, error = handle_error)
14: doTryCatch(return(expr), name, parentenv, handler)
15: tryCatchOne(expr, names, parentenv, handlers[[1L]])
16: tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
17: doTryCatch(return(expr), name, parentenv, handler)
18: tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]),     names[nh], parentenv, handlers[[nh]])
19: tryCatchList(expr, classes, parentenv, handlers)
20: tryCatch(withCallingHandlers({    eval(code, test_env)    if (!handled && !is.null(test)) {        skip_empty()    }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning,     message = handle_message, error = handle_error), error = handle_fatal,     skip = function(e) {    })
21: test_code(desc, code, env = parent.frame())
22: test_that("estimate_theta_subpops_individuals not list of list of individuals",     {        expect_error(estimate_theta_subpops_individuals(l_nonindv,             l_nonindv_sizes))    })
23: eval(code, test_env)
24: eval(code, test_env)
25: withCallingHandlers({    eval(code, test_env)    if (!handled && !is.null(test)) {        skip_empty()    }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning,     message = handle_message, error = handle_error)
26: doTryCatch(return(expr), name, parentenv, handler)
27: tryCatchOne(expr, names, parentenv, handlers[[1L]])
28: tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
29: doTryCatch(return(expr), name, parentenv, handler)
30: tryCatchOne(tryCatchList(expr, names[-nh], parentenv, handlers[-nh]),     names[nh], parentenv, handlers[[nh]])
31: tryCatchList(expr, classes, parentenv, handlers)
32: tryCatch(withCallingHandlers({    eval(code, test_env)    if (!handled && !is.null(test)) {        skip_empty()    }}, expectation = handle_expectation, skip = handle_skip, warning = handle_warning,     message = handle_message, error = handle_error), error = handle_fatal,     skip = function(e) {    })
33: test_code(NULL, exprs, env)
34: source_file(path, new.env(parent = env), chdir = TRUE, wrap = wrap)
35: force(code)
36: with_reporter(reporter = reporter, start_end_reporter = start_end_reporter,     {        lister$start_file(basename(path))        source_file(path, new.env(parent = env), chdir = TRUE,             wrap = wrap)        end_context()    })
37: FUN(X[[i]], ...)
38: lapply(paths, test_file, env = env, reporter = current_reporter,     start_end_reporter = FALSE, load_helpers = FALSE, wrap = wrap)
39: force(code)
40: with_reporter(reporter = current_reporter, results <- lapply(paths,     test_file, env = env, reporter = current_reporter, start_end_reporter = FALSE,     load_helpers = FALSE, wrap = wrap))
41: test_files(paths, reporter = reporter, env = env, stop_on_failure = stop_on_failure,     stop_on_warning = stop_on_warning, wrap = wrap)
42: test_dir(path = test_path, reporter = reporter, env = env, filter = filter,     ..., stop_on_failure = stop_on_failure, stop_on_warning = stop_on_warning,     wrap = wrap)
43: test_package_dir(package = package, test_path = test_path, filter = filter,     reporter = reporter, ..., stop_on_failure = stop_on_failure,     stop_on_warning = stop_on_warning)
44: test_package("malan")

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 1
R is aborting now ...
Segmentation fault: 11

[mikldk]

Thank you very much for this great effort.

First of all. It runs on Mac OSX on Travis: https://travis-ci.org/mikldk/malan. But that's of course just one particular environment. So there seem to exist at least one Mac OSX environment where it works.

For the segfault, can you try putting those commands in file.R and then run R -d valgrind -f file.R? It does require valgrind but should give more details on where the error happens.

[mikldk]

I have added instructions in e913b1c.

I am not entirely sure why gfortran is needed. I don't use it, so probably some of the dependencies does.

[afrubin]

I have run valgrind and there are a large number of warnings about object files that can't be found when malan is loaded. I think this is due to some kind of build issue specific to my setup that probably relates to switching toolchains. The full output from running R -d valgrind -f file.R &> valgrind.txt is attached.

Having done that, I decided to return to the conda install. While it still can't find the vignettes, I was able to get the tests running using the instructions as written and all tests completed successfully.

It is now my recommendation that Mac users who need a workaround should consider installing malan using an R installation managed by conda rather than messing about with gcc and .R/Makevars.

Regardless of which workaround you suggest in the documentation, the instructions in e913b1c are not sufficient since they don't address the fortran library linking issue. I made that suggestion before I fully understood the nature of the problem. Sorry about that!

[afrubin]

I should add that I'm now satisfied that I can get the tests to run locally, and I'm not overly concerned with figuring out how I broke my installation.

Once I've tested and verified the updated Mac OS workaround instructions, we can call it good.

[mikldk]

Thanks. What about replacing

Building this package requires gfortran. The easiest way to obtain this is to install gcc using homebrew with the command brew install gcc. Your ~/.R/Makevars will need to be updated to use gcc instead of clang:

CC=gcc
CXX=g++

with

Some Mac OSX configurations may have installation problems (for example missing gfortran). In such cases, it may help to use Conda package management system. See also the discussion in issue #14.

?

It is very vague, but I have no idea why you are having problems. The Travis Mac OSX runs fine (https://travis-ci.org/mikldk/malan/jobs/375073756).

Do you have any suggestions on how I could narrow it down or debug it?

[afrubin]

I would change "(for example missing gfortran)" to "(for example missing gfortran libraries)" to clarify that it's a linker issue. Other than that it looks good!

From what I can guess, there are some old object files left over from when I tried to compile the malan dependencies with clang and these object files (or references to the object files) have not been removed properly by remove.packages(). I honestly don't think it's worth trying to debug directly. The right answer is to probably just nuke and reinstall my non-conda R environment, and it doesn't make sense to let this delay your progress.