[R] configure script for R fails to set PKG_CFLAGS #35140
Comments
atsuik
added a commit
to atsuik/arrow
that referenced
this issue
Apr 14, 2023
|
I wonder if the root of your problem is that you have Arrow installed via homebrew which is getting picked up by pkg-config Is there a reason you need to install the package from source? The binary install from CRAN is the preferred way to install on MacOS since it already contains all the components noted. |
If I recall, the point of the change you identified was that we're using the generated cmake configuration to understand the features of the C++ library. So I would guess then that the Homebrew package isn't including these cmake files, and it probably should. |
|
The homebrew package does, but the pkg-config name for that is That's a problem wherever a system install of Arrow exists (e.g., #31989 ) |
|
But this is about cmake and not pkg-config. It turns out the cmake files are present in the homebrew build: I misread the original report, it's about
|
nealrichardson
added a commit
to nealrichardson/arrow
that referenced
this issue
Apr 14, 2023
1 task
|
conda-forge/r-arrow-feedstock#56 (comment) may be another manifestation of this (or may not, I haven't investigated) |
nealrichardson
added a commit
to nealrichardson/arrow
that referenced
this issue
Apr 25, 2023
nealrichardson
added a commit
to nealrichardson/arrow
that referenced
this issue
Apr 25, 2023
…rowOptions.cmake" This reverts commit a4bc2c1.
|
In conda-forge/r-arrow-feedstock#63 we found a workaround for this for 11.0.0: in your case, set the env var |
|
FYI: We can use |
|
Oh yes, of course. I wasn't trying to suggest a robust solution since this will be no longer useful after the 12.0.0 release closes. |
nealrichardson
added a commit
to nealrichardson/arrow
that referenced
this issue
Apr 30, 2023
nealrichardson
added a commit
to nealrichardson/arrow
that referenced
this issue
Apr 30, 2023
…rowOptions.cmake" This reverts commit a4bc2c1.
nealrichardson
added a commit
that referenced
this issue
May 3, 2023
…ched libarrow (#35147) I've significantly rewritten `r/configure` to make it easier to reason about and harder for issues like #34229 and #35140 to happen. I've also added a version check to make sure that we don't obviously try to use a system C++ library that doesn't match the R package version. Making sure this was applied in all of the right places and handling what to do if the versions didn't match was the impetus for the whole refactor. `configure` has been broken up into some functions, and the flow of the script is, as is documented at the top of the file: ``` # * Find libarrow on the system. If it is present, make sure # that its version is compatible with the R package. # * If no suitable libarrow is found, download it (where allowed) # or build it from source. # * Determine what features this libarrow has and what other # flags it requires, and set them in src/Makevars for use when # compiling the bindings. # * Run a test program to confirm that arrow headers are found ``` All of the detection of CFLAGS and `-L` dirs etc. happen in one place now, and they all prefer using `pkg-config` to read from the libarrow build what libraries and flags it requires, rather than hard-coding. (autobrew is the only remaining exception, but I didn't feel like messing with that today.) This should make the builds more future proof, should make it so more build configurations work (e.g. I suspect that a static build in ARROW_HOME wouldn't have gotten picked up correctly because it didn't add `-larrow_bundled_dependencies` to the libs, but now it will), and it may eliminate the redundant `-l` and `-D` setting I've observed in some builds (not harmful but definitely sloppy). Version checking has been added in an R script for ease of testing (and for easier handling of arithmetic), and there is an accompanying `test-check-versions.R` added. These are run on all the builds that use `ci/scripts/r_test.sh`. ### Behavior changes * If libarrow is found on the system (via ARROW_HOME, pkg-config, or brew), but the version does not match, it will not be used, and we will try a bundled build. This should mean that users installing a released version will never have libarrow version problems. * If both the found C++ library and R package are on matching dev versions (i.e. not identical given the x.y.z.9000 vs x+1.y.z-SNAPSHOT difference), it will proceed with a warning that you may need to rebuild if there are issues. This means that regular developers will see an extra message in the build output. * autobrew is only used on a release version unless you set FORCE_AUTOBREW=true. This eliminates another source of version mismatches (C++ release version, R dev version). * The path where you could set `LIB_DIR` and `INCLUDE_DIR` env vars has been removed. Use `ARROW_HOME` instead. * Closes: #35140 * Closes: #31989 Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
liujiacheng777
pushed a commit
to LoongArch-Python/arrow
that referenced
this issue
May 11, 2023
…mismatched libarrow (apache#35147) I've significantly rewritten `r/configure` to make it easier to reason about and harder for issues like apache#34229 and apache#35140 to happen. I've also added a version check to make sure that we don't obviously try to use a system C++ library that doesn't match the R package version. Making sure this was applied in all of the right places and handling what to do if the versions didn't match was the impetus for the whole refactor. `configure` has been broken up into some functions, and the flow of the script is, as is documented at the top of the file: ``` # * Find libarrow on the system. If it is present, make sure # that its version is compatible with the R package. # * If no suitable libarrow is found, download it (where allowed) # or build it from source. # * Determine what features this libarrow has and what other # flags it requires, and set them in src/Makevars for use when # compiling the bindings. # * Run a test program to confirm that arrow headers are found ``` All of the detection of CFLAGS and `-L` dirs etc. happen in one place now, and they all prefer using `pkg-config` to read from the libarrow build what libraries and flags it requires, rather than hard-coding. (autobrew is the only remaining exception, but I didn't feel like messing with that today.) This should make the builds more future proof, should make it so more build configurations work (e.g. I suspect that a static build in ARROW_HOME wouldn't have gotten picked up correctly because it didn't add `-larrow_bundled_dependencies` to the libs, but now it will), and it may eliminate the redundant `-l` and `-D` setting I've observed in some builds (not harmful but definitely sloppy). Version checking has been added in an R script for ease of testing (and for easier handling of arithmetic), and there is an accompanying `test-check-versions.R` added. These are run on all the builds that use `ci/scripts/r_test.sh`. ### Behavior changes * If libarrow is found on the system (via ARROW_HOME, pkg-config, or brew), but the version does not match, it will not be used, and we will try a bundled build. This should mean that users installing a released version will never have libarrow version problems. * If both the found C++ library and R package are on matching dev versions (i.e. not identical given the x.y.z.9000 vs x+1.y.z-SNAPSHOT difference), it will proceed with a warning that you may need to rebuild if there are issues. This means that regular developers will see an extra message in the build output. * autobrew is only used on a release version unless you set FORCE_AUTOBREW=true. This eliminates another source of version mismatches (C++ release version, R dev version). * The path where you could set `LIB_DIR` and `INCLUDE_DIR` env vars has been removed. Use `ARROW_HOME` instead. * Closes: apache#35140 * Closes: apache#31989 Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
ArgusLi
pushed a commit
to Bit-Quill/arrow
that referenced
this issue
May 15, 2023
…mismatched libarrow (apache#35147) I've significantly rewritten `r/configure` to make it easier to reason about and harder for issues like apache#34229 and apache#35140 to happen. I've also added a version check to make sure that we don't obviously try to use a system C++ library that doesn't match the R package version. Making sure this was applied in all of the right places and handling what to do if the versions didn't match was the impetus for the whole refactor. `configure` has been broken up into some functions, and the flow of the script is, as is documented at the top of the file: ``` # * Find libarrow on the system. If it is present, make sure # that its version is compatible with the R package. # * If no suitable libarrow is found, download it (where allowed) # or build it from source. # * Determine what features this libarrow has and what other # flags it requires, and set them in src/Makevars for use when # compiling the bindings. # * Run a test program to confirm that arrow headers are found ``` All of the detection of CFLAGS and `-L` dirs etc. happen in one place now, and they all prefer using `pkg-config` to read from the libarrow build what libraries and flags it requires, rather than hard-coding. (autobrew is the only remaining exception, but I didn't feel like messing with that today.) This should make the builds more future proof, should make it so more build configurations work (e.g. I suspect that a static build in ARROW_HOME wouldn't have gotten picked up correctly because it didn't add `-larrow_bundled_dependencies` to the libs, but now it will), and it may eliminate the redundant `-l` and `-D` setting I've observed in some builds (not harmful but definitely sloppy). Version checking has been added in an R script for ease of testing (and for easier handling of arithmetic), and there is an accompanying `test-check-versions.R` added. These are run on all the builds that use `ci/scripts/r_test.sh`. ### Behavior changes * If libarrow is found on the system (via ARROW_HOME, pkg-config, or brew), but the version does not match, it will not be used, and we will try a bundled build. This should mean that users installing a released version will never have libarrow version problems. * If both the found C++ library and R package are on matching dev versions (i.e. not identical given the x.y.z.9000 vs x+1.y.z-SNAPSHOT difference), it will proceed with a warning that you may need to rebuild if there are issues. This means that regular developers will see an extra message in the build output. * autobrew is only used on a release version unless you set FORCE_AUTOBREW=true. This eliminates another source of version mismatches (C++ release version, R dev version). * The path where you could set `LIB_DIR` and `INCLUDE_DIR` env vars has been removed. Use `ARROW_HOME` instead. * Closes: apache#35140 * Closes: apache#31989 Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
rtpsw
pushed a commit
to rtpsw/arrow
that referenced
this issue
May 16, 2023
…mismatched libarrow (apache#35147) I've significantly rewritten `r/configure` to make it easier to reason about and harder for issues like apache#34229 and apache#35140 to happen. I've also added a version check to make sure that we don't obviously try to use a system C++ library that doesn't match the R package version. Making sure this was applied in all of the right places and handling what to do if the versions didn't match was the impetus for the whole refactor. `configure` has been broken up into some functions, and the flow of the script is, as is documented at the top of the file: ``` # * Find libarrow on the system. If it is present, make sure # that its version is compatible with the R package. # * If no suitable libarrow is found, download it (where allowed) # or build it from source. # * Determine what features this libarrow has and what other # flags it requires, and set them in src/Makevars for use when # compiling the bindings. # * Run a test program to confirm that arrow headers are found ``` All of the detection of CFLAGS and `-L` dirs etc. happen in one place now, and they all prefer using `pkg-config` to read from the libarrow build what libraries and flags it requires, rather than hard-coding. (autobrew is the only remaining exception, but I didn't feel like messing with that today.) This should make the builds more future proof, should make it so more build configurations work (e.g. I suspect that a static build in ARROW_HOME wouldn't have gotten picked up correctly because it didn't add `-larrow_bundled_dependencies` to the libs, but now it will), and it may eliminate the redundant `-l` and `-D` setting I've observed in some builds (not harmful but definitely sloppy). Version checking has been added in an R script for ease of testing (and for easier handling of arithmetic), and there is an accompanying `test-check-versions.R` added. These are run on all the builds that use `ci/scripts/r_test.sh`. ### Behavior changes * If libarrow is found on the system (via ARROW_HOME, pkg-config, or brew), but the version does not match, it will not be used, and we will try a bundled build. This should mean that users installing a released version will never have libarrow version problems. * If both the found C++ library and R package are on matching dev versions (i.e. not identical given the x.y.z.9000 vs x+1.y.z-SNAPSHOT difference), it will proceed with a warning that you may need to rebuild if there are issues. This means that regular developers will see an extra message in the build output. * autobrew is only used on a release version unless you set FORCE_AUTOBREW=true. This eliminates another source of version mismatches (C++ release version, R dev version). * The path where you could set `LIB_DIR` and `INCLUDE_DIR` env vars has been removed. Use `ARROW_HOME` instead. * Closes: apache#35140 * Closes: apache#31989 Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
|
@nealrichardson just to make this clear: is it included into |
|
@nealrichardson Thanks for the info, do you think you could provide any more accurate ETA for |
No, but it will be discussed on the dev@arrow.apache.org mailing list, so you can follow the discussion there.
I suggested a workaround above, does that work for you? |
Apologies, I decided to put this on hold and wait for the official release. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug, including details regarding any error messages, version, and platform.
arrow version: 11.0.0.3
R version: 4.2.3
platform: arm64
I found that current
r/configurefails to set PKG_CFLAGS parsing ARROW_OPTS_CMAKE file.This is a log I tried to install arrow to my m1 mac.
The log shows
PKG_CFLAGSis empty.This causes that capabilities, such as dataset, parquet, etc..., are FALSE.
This is caused by
$LIB_DIRis empty at this linearrow/r/configure
Line 255 in 5c73973
I found that the deletion of line at this pull request
https://github.com/apache/arrow/pull/14235/files#diff-089697faebdb7820ca629a2bb316b878cc0ba18a5bfb0b60996f8dbcd1fa11e7L234
causes this problem.
I have already checked that to recover this line resolves this problem.
This is the log.
Component(s)
R
The text was updated successfully, but these errors were encountered: