CRAN Proposal

[CGMossa]

Hello everyone!

We have been thinking on the current state of publishing Rust powered R-packages on CRAN,
and we have a few things,
we would like to propose, on behalf of the greater extendr community. I hope this can be an opportunity to gather feedback, and present a case that works for everyone.

First and foremost, I would like to state, that we share the same concerns as CRAN, and we are very conscientious about insuring all concerns are addressed sufficiently.

R CMD check a la Rust

We would like to have a more close dialogue with CRAN team and the checks that are done on Rust-powered R-packages. As part of {rextendr}, we have added various functions to aid R package authors to set these checks up

• extendr packages can adhere to the CRAN's policy viause_cran_defaults()
• Vendoring of rust dependencies can happen via vendor_pkgs()
• Check the minimal supported Rust version with use_msrv()

Also, a community member has already submitted a PR to checks, see r-devel/r-svn#182.

Rust version

CRAN machines have different versions of Rust tooling installed, and these rarely get updated.
We would like to suggest that CRAN maintainers use rustup, which is already available in all CRAN machines,
in order to install various versions of rustc. All versions of rustc work with cargo, so this will not require further configuration of the CRAN maintainers.

The calls to rustup that we suggest is done once:

rustup toolchain install 1.70.0
rustup toolchain install 1.71.1
rustup toolchain install 1.72.1
rustup toolchain install 1.73.0
rustup toolchain install 1.74.1
rustup toolchain install 1.75.0
rustup toolchain install 1.76.0
rustup toolchain install 1.77.2
rustup toolchain install 1.78.0
rustup toolchain install 1.79.0

Then package authors are then supposed to choose either of these versions, as their chosen compiler
and amend their Makevars files, where cargo-commands are substituted, e.g, cargo +1.74.1 build.

rustup ensures that these compilers are installed, and they do not conflict with each other, ensuring broad
compatability on CRAN machines.

End users

We'll make configure script that would tell the user,
to add the appropriate rustc version (by invoking one of the calls above) to their system.

Most Rust users are likely to use latest stable version, thus they usually invoke rustup update,
and a substantial number of Rust users are on nightly version, i.e. the daily development version.
In the Annual Rust Survey 2023, 31.1% of the 10490 respondents said that they use "latest nightly".

Also, the issue is only when an old Rust compiler is used to compile, things that require the a new rust compiler.
Rust is backwards compatible, thus newer compiler versions would compile older versions, without any problem.

Minimal supported version of rustc on CRAN

We also recommend that CRAN adopt cargo version 1.70 as the minimum supported version,
as this version is the one where cargo no longer uses GitHub.com for anything at all, not storage, nor indexing of packages. The Rust Foundation is the legal entity tasked with maintaining
and hosting the storage of the contents for rustup downloads, and more importantly the crates.io content.
This is a similar setup to the R Consortium.

This is to address a concern about availability of dependencies on crates.io.

Vendoring of Rust packages

We would propose to CRAN to revise their policy on vendoring Rust crates. While this policy seems harmless,
it does affect reverse dependencies authors, in that it takes away, from their notoriety, as that is in part determined via download numbers of their crates. Take https://crates.io/crates/arrow as an example, where the download numbers are visible at the bottom.

[JosiahParry]

I think we can also discuss what we are doing and willing to do to help improve this story. To boot, we have put a lot of effort in creating developer tools that work meet CRAN's requirements in {rextendr}.

• use_cran_defaults()
• vendor_pkgs()
• use_msrv()

Our goal with rextendr is to create a developer package like usethis but for extendr-powered R packages. We also acknowledge that CRAN is important to the R community and want to ensure that developers can publish there without adding undo strain on CRAN maintainers.

[JosiahParry]

Note that a community member has already attempted to improve the checks r-devel/r-svn#182

[CGMossa]

Okay. Updated original post.

[kbvernon]

Would it be worth adding some detail about how these changes will or won't affect the standard of reproducibility that they are trying to maintain?

Also, once you all have all the content worked out, I'm happy to take a pass on the text (style, grammar, formatting, etc.).

[kbvernon]

Wow, I guess ignore the question re: reproducibility.

TIL the word 'reproducibility' does not occur anywhere in the CRAN policies.

[JosiahParry]

Another package was archived https://github.com/schochastics/timeless/ without informing the maintainer.

For some additional context, my package {arcpbf} was archived in August without informing me. I resubmitted {arcpbf} on September 5th. On Sept. 12th I received the following message

Do you really need rustc >= 1.70.0? This is somewhat recent ...

-k

I responded

Thank you for the message!

This package uses the rust crate prost which has a minimum supported version of Rust of 1.70 which I cannot change, unfortunately. Rust 1.70 was released June 1, 2023 which was 15 months ago. The oldest version of rust on CRAN check machines is 1.69 which is found on the Fedora gcc and clang flavors. It is also worth noting that this was the release that disconnected rust from indexing on GitHub which I know is a concern of the CRAN team. Below you'll find the output of cargo msrv confirming the MSRV.

Perhaps we can get some of the folks from the R-Rust community—i.e. from extendr and savvy—and folks from the CRAN team to discuss how we can make supporting Rust in R packages amenable for both parties?

There are a handful of us in the extendr community who would be excited at the opportunity to help write additional checks.

The same day Dr. Brian Ripley messaged me that 4 additional packages will be removed from CRAN by Sept. 26th if the issues are not fixed. These packages require the same fix that is provided in {arcpbf} which is still in pending. It does not make sense to submit a fix that will not be accepted. I have sent the following email today

Hi team, it has been 11 days that this package has been in "pending."

Since then Dr. Brian Ripley has emailed me informing me that {arcgisutils}, {arcgisgeocode}, {arcgisplaces}, and {b64} will be removed from CRAN on Sept 26th, 2024.

I will be using the same fix in those packages as I have used in {arcpbf}. I do not want to overwhelm the CRAN volunteer team with extra submissions if this will not be successful.

Please let me know what I can do to get {arcpbf} out of the pending status.

[JosiahParry]

hellorust was also removed from CRAN https://cran-archive.r-project.org/web/checks/2024/2024-09-11_check_results_hellorust.html

[CGMossa]

Thanks for the comments so far! I believe we will write-up a proposal for change of CRAN policy in October/November. So please, give your thoughts then, so they can be included in the process.

I'm happy that so many people have engaged with this post. To me, it is important that CRAN does not fall behind on Rust, as
it would be to the detriment of the larger R community, to not make use of such a popular language. Other languages seem to be advancing rather rapidly in integrating Rust, so there is some urgency involved here, I would say.

[JosiahParry]

Thanks for your work organizing this, Mossa. We definitely will need to categorize the proposal and make sure that whatever we are asking for is actionable.

I think there are 3 persona's we need to keep in mind:

• Package developers
• CRAN maintainers
• Package users

We want to make sure it is easy to use Rust in R packages for all three categories of users. Roughly my thoughts are

Package Developers

• Make the package requirements / constraints clear and known
  • offline package build requirement (vendoring packages)
  • MSRV policy & relationship to configure
  • how to handle vendoring large dependency trees (important for anyone wanting to use arrow-rs)
    • we will need input from CRAN on this. Dewey D. has provided guidance in the past for their work in arrow & geoarrow
• provide R package testing that mimics CRAN (this can be in the form of GHA templates)
• provide configure & Makevars templates that adhere to CRAN policies
  • ideally these would also be flexible for development too —e.g. not require --offline flag when is on CRAN

CRAN Maintainers

We need to think about what we'd like them to have them implement / change /enhance / deprecate

• Automated checks for Rust in R packages that do not require additional manual checks
  • we'd like to help here as well community is already engaging (Improves check for Rust r-devel/r-svn#182)
• Improved MSRV policy and tooling—we should recommend to use rustup and install multiple toolchains

R users

Most importantly here, I want to make sure that R package users can install binaries easily. If they cannot, they also are informed of how to install a package from source if they need to. This is a show stopper for a lot of people. They just give up if a package fails to compile. Can we aid them in the configure script? Provide a utility in rextendr for installing rustup? At present rust_sitrep() is very good but it won't actually install anything. I quite like reticulate::install_miniconda() and install_python() because it is so simple and handles things for you. Ought we do that to lower the barrior to entry?

What else should we consider here?

[bencardoen]

Perhaps it can be of interest to note that Julia and Rust are distinct from say Python in that:

• They move fast, but ensure the toolchain is complete (do not really on distribution dependencies)
• Distribution maintained versions are not up to date enough

So while this can seem are breaking reproducibility from someone who is used to having R and G++ and related tools robustly available with dependencies by distributions, doing this for Rust and e.g. Julia would in fact break things more easily.

Rust documentation does not even mention distribution installers:
https://forge.rust-lang.org/infra/other-installation-methods.html

Same for Julia, who even explicitly discourage it
https://julialang.org/downloads/

I'm writing this as someone who swears by reroducibility, and I know this seems initially counterintuitive, but in practice X-up for these two cases is the better way.

Note that neither require elevated privileges, which are needed for package managers (dnf/apt), and package manager based installation can lead to subtle breakage because of e.g. incompatible libraries (e.g. BLAS), which can be highly specialized for these use cases.

[CGMossa]

Thanks for your thoughts and input. Could I ask you to expand a little bit? I think, that I can understand what you're saying, but my English is not good enough here.

[bencardoen]

In Fedora, to install Python or R, it is indeed recommended to use the package manager, as the provided versions are up to date. Indeed, Python is used as a core of the distribution, so it is almost always close to upstream.

But languages such as Rust and Julia move very fast, and provide their own toolchain to make sure they remain reproducible across versions.
In those two cases, it is common to not use the package manager (dnf in case of Fedora), but juliaup and rustup instead.

I hope this helps. If my comments are not applicable to the discussion, or do not apply, feel free to ignore them as such.

[CGMossa]

Sorry, I'm "back now".

I think your comments and opinions are same as mine. I'm just struggling to express them clearly.
This helps me try formulate that there is a difference in how Rust/Julia provide different compiler versions,
vs. C/C++/Fortran?

[bencardoen]

For C++ (I don't have experience with C/Fortran), you would rely on the system provided compiler (or build your own as user and install that + runtime), because Rust/Julia leverage LLVM and need control of their dependencies in a way that is not tractable with package managers, they provide their own.
For example, the juliaup package is provided by some Linux distros, and would pull in different versions as needed. Install julia would restrict to a single, often outdated and unsupported version.
It's in some sense similar to conda/virtualenv on Python, but the line is blurred there because a lot of distros heavily use Python for user and system space, so will be very close to upstream.

[Robinlovelace]

+1 to this, support for Rust in the R ecosystem, especially support from CRAN, which gives a level of confidence and indication of maturity, would be hugely useful for my work and others at the University of Leeds and in the Civil Service.