Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: allow "R source package with Rust library binary" installation #450

Merged
merged 12 commits into from
Oct 29, 2023

Conversation

eitsupi
Copy link
Collaborator

@eitsupi eitsupi commented Oct 29, 2023

Close #408

Allows us to download binaries from GitHub Releases and skip the cargo build step using the mechanism copied from prqlr.

Like this:

> Sys.setenv(NOT_CRAN="true")
> remotes::install_github("pola-rs/r-polars#450")
Downloading GitHub repo pola-rs/r-polars@bin-lib-install
Running `R CMD build`...
* checking for file/tmp/Rtmpghk68j/remotes3e78001b4b/pola-rs-r-polars-9ece233/DESCRIPTION... OK
* preparingpolars:
* checking DESCRIPTION meta-information ... OK
* cleaning src
* runningcleanup* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* buildingpolars_0.8.1.9000.tar.gzInstalling package into/usr/local/lib/R/site-library’
(aslibis unspecified)
* installing *source* packagepolars...
** using staged installation

--------------- [ TRY TO DOWNLOAD PRE-BUILT BINARY ] ---------------
Found pre-built binary at <https://github.com/pola-rs/r-polars/releases/download/lib-v0.33.0/libr_polars-0.33.0-x86_64-unknown-linux-gnu.tar.gz>.
Downloading...
Checking SHA256 for </tmp/RtmpcX8QCj/file17b333984b8.tar.gz>...
SHA256 matches for </tmp/RtmpcX8QCj/file17b333984b8.tar.gz>.
Extracted pre-built binary to </tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/tools> directory.
--------------------------------------------------------------------


---------------------- [LIBRARY BINARY FOUND] ----------------------
The library was found at </tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/tools/libr_polars.a>. No need to build it.
--------------------------------------------------------------------

** libs
using C compiler: ‘gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0rm -Rf polars.so /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/libr_polars.a entrypoint.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include    -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c entrypoint.c -o entrypoint.o
if [ -f "/tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/../tools/libr_polars.a" ]; then \
        mkdir -p "/tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release" ; \
        mv "/tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/../tools/libr_polars.a" "/tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/libr_polars.a" ; \
        exit 0; \
fi && \
if [ "true" != "true" ]; then \
        export CARGO_HOME=/tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/.cargo; \
fi && \
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.cargo/bin" && \
if [ "" = "true" ]; then \
        cargo build --target= --lib --profile release --manifest-path="./rust/Cargo.toml" --features "full_features"; \
else \
        cargo build --target= --lib --profile release --manifest-path="./rust/Cargo.toml"; \
fi
if [ "true" != "true" ]; then \
        rm -Rf /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/.cargo && \
        rm -Rf /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/build; \
fi
if [ -f "/tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/libr_polars.a" ]; then \
        echo "file is there: "; \
elif [ -f "./rust/target//release/libr_polars.a" ]; then \
        echo "file is './rust/target//release/libr_polars.a'"; \
        mkdir -p /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release ; \
        echo "trying to symlink in ./rust/target//release/libr_polars.a"; \
        ln -s ./rust/target//release/libr_polars.a /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/libr_polars.a ; \
fi
file is there:
if [ "" = "true" ]; then \
        echo "cleanup!!" ; \
        mv /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/libr_polars.a /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/../temp_binary.a; \
        rm -rf /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release; \
        mkdir /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release; \
        mv /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/../temp_binary.a /tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release/libr_polars.a; \
        rm -rf ./src/.cargo; \
else \
        echo "hands off!!" ; \
fi
hands off!!
gcc -shared -L/usr/local/lib/R/lib -L/usr/local/lib -o polars.so entrypoint.o -L/tmp/RtmpGSDwc8/R.INSTALLb33d8fe552/polars/src/rust/target//release -lr_polars -L/usr/local/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-polars/00new/polars/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (polars)

@etiennebacher
Copy link
Collaborator

I must admit I'm a bit lost on the changes in CI and in the procedure to build/install polars so it would be great to have a summary of the changes somewhere in the docs when all of this is done 😄 (it doesn't have to be in this PR though)

@eitsupi
Copy link
Collaborator Author

eitsupi commented Oct 29, 2023

I must admit I'm a bit lost on the changes in CI and in the procedure to build/install polars so it would be great to have a summary of the changes somewhere in the docs when all of this is done 😄 (it doesn't have to be in this PR though)

Definitely. I was imagining something like DEVELOPMENT.md

@eitsupi eitsupi marked this pull request as ready for review October 29, 2023 11:21
@eitsupi
Copy link
Collaborator Author

eitsupi commented Oct 29, 2023

After merging this, builds on R-universe should use the pre-built binaries and SIMD should be enabled even when installing from R-universe.
I would like to update the documentation after checking it.

Comment on lines +43 to +47
if [ -z "${NOT_CRAN}" ] && [ -n "${MY_UNIVERSE}" ]; then
echo "It seems that this is on R-universe <${MY_UNIVERSE}>."
echo "Trying to download pre-built binary."
LIBR_POLARS_BUILD="false"
fi
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step detects R-universe and enable install with pre-built binary.

Copy link
Collaborator

@etiennebacher etiennebacher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've no clue about this, so just merge whenever you think it's ready

@eitsupi eitsupi merged commit ed11c51 into main Oct 29, 2023
25 checks passed
@eitsupi eitsupi deleted the bin-lib-install branch October 29, 2023 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

arrow like "R source package with Rust library binary" installation
2 participants