fix(build): Update src/Makevars.in for Ubuntu 20.04 #528

brownag · 2023-11-20T14:04:20Z

Fix installation in ubuntu fails with undefined symbol: shm_unlink #514
Adds ubuntu-20.04 to matrix build in release-lib.yml

Background:

I got successful installation in Ubuntu 20.04 docker container by adding "-lrt" to PKG_LIBS in src/Makevars.in. As in this diff:

diff --git a/src/Makevars.in b/src/Makevars.in
index b44a238..97dd8a8 100644
--- a/src/Makevars.in
+++ b/src/Makevars.in
@@ -7,7 +7,7 @@ LIBNAME = libr_polars.a
 TARGET_DIR = $(CURDIR)/rust/target
 LIBDIR = $(TARGET_DIR)/$(TARGET)/$(RPOLARS_PROFILE)
 STATLIB = $(LIBDIR)/$(LIBNAME)
-PKG_LIBS = -L$(LIBDIR) -lr_polars
+PKG_LIBS = -L$(LIBDIR) -lr_polars -lrt
 RPOLARS_RUST_SOURCE ?= ./rust
 
 rpolars_ext_binary = $(RPOLARS_RUST_SOURCE)/target/$(TARGET)/$(RPOLARS_PROFILE)/$(LIBNAME)

root@aed99dc1418b:/workspace/r-polars# Rscript -e "remotes::install_local(f=T)"
Running `R CMD build`...
* checking for file '/tmp/RtmpuwoEcc/fileae4d3a2946e0/r-polars/DESCRIPTION' ... OK
* preparing 'polars':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* running 'cleanup'
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building 'polars_0.10.1.9000.tar.gz'
Installing package into '/usr/local/lib/R/site-library'
(as 'lib' is unspecified)
* installing *source* package 'polars' ...
** using staged installation

--------------------------- [RUST FOUND] ---------------------------
cargo 1.74.0 (ecb9851af 2023-10-18)

rustc 1.74.0 (79e9716c9 2023-11-13)
binary: rustc
commit-hash: 79e9716c980570bfd1f666e3b16ac583f0168962
commit-date: 2023-11-13
host: x86_64-unknown-linux-gnu
release: 1.74.0
LLVM version: 17.0.4
--------------------------------------------------------------------

** libs
using C compiler: 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0'
rm -Rf polars.so /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a entrypoint.o
gcc -I"/usr/share/R/include" -DNDEBUG       -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-DdVjkr/r-base-4.3.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -c entrypoint.c -o entrypoint.o
if [ -f "/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/../tools/libr_polars.a" ]; then \
	mkdir -p "/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release" ; \
	mv "/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/../tools/libr_polars.a" "/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a" ; \
	exit 0; \
fi && \
if [ "true" != "true" ]; then \
	export CARGO_HOME=/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/.cargo; \
fi && \
export PATH="/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.cargo/bin" && \
if [ "" = "true" ]; then \
	cargo build --target=x86_64-unknown-linux-gnu --lib --profile release --manifest-path="./rust/Cargo.toml" --features "full_features"; \
else \
	cargo build --target=x86_64-unknown-linux-gnu --lib --profile release --manifest-path="./rust/Cargo.toml"; \
fi
   Compiling libc v0.2.149
   Compiling autocfg v1.1.0
   Compiling proc-macro2 v1.0.69
   Compiling unicode-ident v1.0.12
   Compiling cfg-if v1.0.0
   Compiling version_check v0.9.4
   Compiling serde v1.0.190
   Compiling scopeguard v1.2.0
   Compiling crossbeam-utils v0.8.16
   Compiling libm v0.2.8
   Compiling once_cell v1.18.0
   Compiling ahash v0.8.6
   Compiling num-traits v0.2.17
   Compiling memoffset v0.9.0
   Compiling crossbeam-epoch v0.9.15
   Compiling quote v1.0.33
   Compiling syn v2.0.38
   Compiling jobserver v0.1.27
   Compiling zerocopy v0.7.21
   Compiling rayon-core v1.12.0
   Compiling cc v1.0.83
   Compiling getrandom v0.2.10
   Compiling crossbeam-deque v0.8.3
   Compiling futures-core v0.3.29
   Compiling either v1.9.0
   Compiling static_assertions v1.1.0
   Compiling memchr v2.6.4
   Compiling slab v0.4.9
   Compiling siphasher v0.3.11
   Compiling allocator-api2 v0.2.16
   Compiling phf_shared v0.11.2
   Compiling rayon v1.8.0
   Compiling itoa v1.0.9
   Compiling regex-syntax v0.8.2
   Compiling rand_core v0.6.4
   Compiling equivalent v1.0.1
   Compiling futures-sink v0.3.29
   Compiling pin-project-lite v0.2.13
   Compiling serde_json v1.0.108
   Compiling crc32fast v1.3.2
   Compiling ryu v1.0.15
   Compiling rand v0.8.5
   Compiling futures-channel v0.3.29
   Compiling futures-task v0.3.29
   Compiling pkg-config v0.3.27
   Compiling target-features v0.1.5
   Compiling phf_generator v0.11.2
   Compiling rle-decode-fast v1.0.3
   Compiling snap v1.1.0
   Compiling futures-util v0.3.29
   Compiling regex-automata v0.4.3
   Compiling zstd-sys v2.0.9+zstd.1.5.5
   Compiling libflate_lz77 v1.2.0
   Compiling phf_codegen v0.11.2
   Compiling aho-corasick v1.1.2
   Compiling phf v0.11.2
   Compiling array-init-cursor v0.2.0
   Compiling thiserror v1.0.50
   Compiling fallible-streaming-iterator v0.1.9
   Compiling futures-io v0.3.29
   Compiling syn v1.0.109
   Compiling crc-catalog v1.1.1
   Compiling pin-utils v0.1.0
   Compiling adler32 v1.2.0
   Compiling libflate v1.4.0
   Compiling crc v2.1.0
   Compiling planus v0.3.1
   Compiling hashbrown v0.14.2
   Compiling regex v1.10.2
   Compiling parse-zoneinfo v0.3.0
   Compiling lz4-sys v1.9.4
   Compiling smartstring v1.0.1
   Compiling chrono-tz-build v0.2.1
   Compiling lock_api v0.4.11
   Compiling simdutf8 v0.1.4
   Compiling zstd-safe v7.0.0
   Compiling iana-time-zone v0.1.58
   Compiling chrono-tz v0.8.4
   Compiling lexical-util v0.8.5
   Compiling polars-utils v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling serde_derive v1.0.190
   Compiling futures-macro v0.3.29
   Compiling thiserror-impl v1.0.50
   Compiling bytemuck_derive v1.5.0
   Compiling bytemuck v1.14.0
   Compiling sysinfo v0.29.10
   Compiling parking_lot_core v0.9.9
   Compiling atoi v2.0.0
   Compiling rustversion v1.0.14
   Compiling ethnum v1.5.0
   Compiling futures-executor v0.3.29
   Compiling futures v0.3.29
   Compiling foreign_vec v0.1.0
   Compiling streaming-iterator v0.1.9
   Compiling dyn-clone v1.0.14
   Compiling atoi_simd v0.15.3
   Compiling strength_reduce v0.2.4
   Compiling ppv-lite86 v0.2.17
   Compiling fast-float v0.2.0
   Compiling smallvec v1.11.1
   Compiling rand_chacha v0.3.1
   Compiling lexical-parse-integer v0.8.6
   Compiling lexical-write-integer v0.8.5
   Compiling multiversion-macros v0.7.3
   Compiling hashbrown v0.13.2
   Compiling matrixmultiply v0.3.8
   Compiling num-integer v0.1.45
   Compiling ref-cast v1.0.20
   Compiling lexical-write-float v0.8.5
   Compiling parking_lot v0.12.1
   Compiling lexical-parse-float v0.8.5
   Compiling ref-cast-impl v1.0.20
   Compiling indexmap v2.1.0
   Compiling arrow-format v0.8.1
   Compiling chrono v0.4.31
   Compiling bitflags v2.4.1
   Compiling multiversion v0.7.3
   Compiling halfbrown v0.2.4
   Compiling float-cmp v0.9.0
   Compiling cmake v0.1.50
   Compiling heck v0.4.1
   Compiling log v0.4.20
   Compiling rawpointer v0.2.1
   Compiling libz-ng-sys v1.1.12
   Compiling value-trait v0.8.0
   Compiling strum_macros v0.25.3
   Compiling crossterm v0.27.0
   Compiling avro-schema v0.3.0
   Compiling lexical-core v0.8.5
   Compiling polars-error v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling num-complex v0.4.4
   Compiling polars-core v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling strum v0.25.0
   Compiling async-trait v0.1.74
   Compiling alloc-no-stdlib v2.0.4
   Compiling unicode-width v0.1.11
   Compiling comfy-table v7.1.0
   Compiling alloc-stdlib v0.2.2
   Compiling ndarray v0.15.6
   Compiling simd-json v0.13.4
   Compiling rand_distr v0.4.3
   Compiling polars-ops v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling base64 v0.21.5
   Compiling xxhash-rust v0.8.7
   Compiling itoap v1.0.1
   Compiling adler v1.0.2
   Compiling miniz_oxide v0.7.1
   Compiling jsonpath_lib v0.3.0 (https://github.com/ritchie46/jsonpath?branch=improve_compiled#24eaf0b4)
   Compiling brotli-decompressor v2.5.1
   Compiling async-stream-impl v0.3.5
   Compiling argminmax v0.6.1
   Compiling num_cpus v1.16.0
   Compiling mio v0.8.9
   Compiling socket2 v0.5.5
   Compiling bytes v1.5.0
   Compiling hex v0.4.3
   Compiling async-stream v0.3.5
   Compiling parquet-format-safe v0.2.4
   Compiling brotli v3.4.0
   Compiling tokio v1.33.0
   Compiling now v0.1.3
   Compiling streaming-decompression v0.1.2
   Compiling seq-macro v0.3.5
   Compiling memmap2 v0.7.1
   Compiling polars-plan v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling crossbeam-queue v0.3.8
   Compiling percent-encoding v2.3.0
   Compiling home v0.5.5
   Compiling crossbeam-channel v0.5.8
   Compiling polars-pipe v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling rustix v0.38.21
   Compiling libR-sys v0.6.0
   Compiling enum_dispatch v0.3.12
   Compiling jemalloc-sys v0.5.4+5.3.0-patched
   Compiling polars-lazy v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling linux-raw-sys v0.4.10
   Compiling tokio-util v0.7.10
   Compiling cfg-if v0.1.10
   Compiling paste v1.0.14
   Compiling net2 v0.2.39
   Compiling iovec v0.1.4
   Compiling polars v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling glob v0.3.1
   Compiling extendr-api v0.6.0 (https://github.com/extendr/extendr?rev=0ac58377c50174818e02de54b1032c5f4cb75e4f#0ac58377)
   Compiling fastrand v2.0.1
   Compiling tempfile v3.8.1
   Compiling mio v0.6.23
   Compiling bincode v1.3.3
   Compiling flate2 v1.0.28
   Compiling spin v0.9.8
   Compiling extendr-macros v0.6.0 (https://github.com/extendr/extendr?rev=0ac58377c50174818e02de54b1032c5f4cb75e4f#0ac58377)
   Compiling uuid v1.5.0
   Compiling nanorand v0.7.0
   Compiling fnv v1.0.7
   Compiling lazy_static v1.4.0
   Compiling ipc-channel v0.17.0
   Compiling flume v0.11.0
   Compiling indenter v0.3.3
   Compiling state v0.6.0
   Compiling lz4 v1.24.0
   Compiling zstd v0.13.0
   Compiling polars-arrow v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling polars-row v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling polars-json v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling polars-parquet v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling jemallocator v0.5.4
   Compiling polars-time v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling polars-io v0.35.2 (https://github.com/pola-rs/polars.git?rev=b13afbecac039205dacbaca766ecca4bf441b347#b13afbec)
   Compiling r-polars v0.35.0 (/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust)
    Finished release [optimized] target(s) in 15m 24s
if [ "true" != "true" ]; then \
	rm -Rf /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/.cargo && \
	rm -Rf /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/build; \
fi
if [ -f "/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a" ]; then \
	echo "file is there: "; \
elif [ -f "./rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a" ]; then \
	echo "file is './rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a'"; \
	mkdir -p /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release ; \
	echo "trying to symlink in ./rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a"; \
	ln -s ./rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a ; \
fi
file is there: 
if [ "" = "true" ]; then \
	echo "cleanup!!" ; \
	mv /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/../temp_binary.a; \
	rm -rf /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release; \
	mkdir /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release; \
	mv /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/../temp_binary.a /tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release/libr_polars.a; \
	rm -rf ./src/.cargo; \
else \
	echo "hands off!!" ; \
fi
hands off!!
gcc -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o polars.so entrypoint.o -L/tmp/Rtmp5iYZ7r/R.INSTALLae923eb2241a/polars/src/rust/target/x86_64-unknown-linux-gnu/release -lr_polars -lrt -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-polars/00new/polars/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (polars)

This is my first PR in r-polars, so please let me know if you need anything else.

eitsupi

Thanks for your contribution!
Could you change the CI setting?

.github/workflows/release-lib.yaml

…into fix-makevars-514

eitsupi · 2023-11-20T14:47:03Z

src/Makevars.in

@@ -7,7 +7,7 @@ LIBNAME = libr_polars.a
 TARGET_DIR = $(CURDIR)/rust/target
 LIBDIR = $(TARGET_DIR)/$(TARGET)/$(RPOLARS_PROFILE)
 STATLIB = $(LIBDIR)/$(LIBNAME)
-PKG_LIBS = -L$(LIBDIR) -lr_polars
+PKG_LIBS = -L$(LIBDIR) -lr_polars -lrt


It seems that we should pass -lrt to Linux only.
https://github.com/cran/arrow/blob/61e0a7ade8ec52e2d1463c982632c3038c1a51bb/tools/cpp/CMakeLists.txt#L976-L979

Could you update the configure script and change it to not pass anything to macOS? (If it's too difficult, I can do it myself)

OK, I gave it a shot in fd67334

brownag · 2023-11-20T14:52:18Z

Looks like linking to librt fails on macOS. I suppose the additional lib could be added conditionally for non-macOS platforms?

eitsupi · 2023-11-20T15:02:14Z

I suppose the additional lib could be added conditionally for non-macOS platforms?

I think it can be configured within the configure script (note: must be sh, not bash) to determine if that machine is darwin and pass something like s|@ADDITIONAL_PKG_LIBS_FLAG@|| to sed -e in these lines, or something like s|@ADDITIONAL_PKG_LIBS_FLAG@|-lrt| other cases.

r-polars/configure

Line 70 in a4d3772

sed -e "s|@RUST_TARGET@||" src/Makevars.in >src/Makevars

r-polars/configure

Line 89 in a4d3772

sed -e "s|@RUST_TARGET@||" src/Makevars.in >src/Makevars

r-polars/configure

Line 104 in a4d3772

    
           sed -e "s|@RUST_TARGET@|$(rustc -vV | grep host | cut -d' ' -f2)|" src/Makevars.in >src/Makevars

eitsupi

Seems working fine, thanks!

brownag and others added 3 commits November 20, 2023 05:52

Add "-lrt" to PKG_LIBS Makevars.in for pola-rs#514

7409884

Add ubuntu-20.04 to release-lib.yaml

3f44bb9

Merge branch 'main' into fix-makevars-514

733961b

eitsupi requested changes Nov 20, 2023

View reviewed changes

.github/workflows/release-lib.yaml Outdated Show resolved Hide resolved

brownag added 2 commits November 20, 2023 06:25

Use ubuntu-20.04 instead of ubuntu-22.04 in release-lib.yaml

7e529d1

Merge branch 'fix-makevars-514' of ssh://github.com/brownag/r-polars …

314e73d

…into fix-makevars-514

eitsupi reviewed Nov 20, 2023

View reviewed changes

brownag added 3 commits November 20, 2023 07:46

Add conditional ADDITIONAL_PKG_LIBS_FLAG

fd67334

cleanup

12839f9

fill missing cases, check darwin first

57eb38e

eitsupi approved these changes Nov 20, 2023

View reviewed changes

eitsupi changed the title ~~Update src/Makevars.in for Ubuntu 20.04~~ fix(build): Update src/Makevars.in for Ubuntu 20.04 Nov 20, 2023

eitsupi merged commit 52df4c4 into pola-rs:main Nov 20, 2023
33 checks passed

brownag deleted the fix-makevars-514 branch November 20, 2023 23:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(build): Update src/Makevars.in for Ubuntu 20.04 #528

fix(build): Update src/Makevars.in for Ubuntu 20.04 #528

brownag commented Nov 20, 2023 •

edited by eitsupi

Loading

eitsupi left a comment

eitsupi Nov 20, 2023

brownag Nov 20, 2023

brownag commented Nov 20, 2023

eitsupi commented Nov 20, 2023 •

edited

Loading

eitsupi left a comment

fix(build): Update src/Makevars.in for Ubuntu 20.04 #528

fix(build): Update src/Makevars.in for Ubuntu 20.04 #528

Conversation

brownag commented Nov 20, 2023 • edited by eitsupi Loading

Background:

eitsupi left a comment

Choose a reason for hiding this comment

eitsupi Nov 20, 2023

Choose a reason for hiding this comment

brownag Nov 20, 2023

Choose a reason for hiding this comment

brownag commented Nov 20, 2023

eitsupi commented Nov 20, 2023 • edited Loading

eitsupi left a comment

Choose a reason for hiding this comment

brownag commented Nov 20, 2023 •

edited by eitsupi

Loading

eitsupi commented Nov 20, 2023 •

edited

Loading