Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update some scripts to install R packages from source, on arm64 images #195

Merged
merged 3 commits into from
Jul 14, 2021

Conversation

eitsupi
Copy link
Member

@eitsupi eitsupi commented Jul 11, 2021

Related to #144

The RSPM CRAN mirrors does not provide arm64 binary packages for ubuntu at this time.
So, in this PR, I will modify the rocker script so that the R package will do source installations in the arm64 images.

  • install_R.sh: Change the default repository to be written to Rprofile.site to source in arm64 images.
  • install_tidyverse.sh and install_verse.sh: Remove the reference to the CRAN env var from the options of the install2.r command for to use the default repository. (In other scripts originally did not reference the CRAN env var from the install2.r -r option.)

With these fixes, I have run install_R.sh on the ubuntu:forcal arm64 image, and install_tidyverse.sh on it, and confirmed that I can use R normally.

@eitsupi eitsupi changed the title Fixed some scripts for arm64 build Update some scripts for arm64 build Jul 11, 2021
@eitsupi eitsupi changed the title Update some scripts for arm64 build Update some scripts to install R packages from source, on arm64 images Jul 11, 2021
@eddelbuettel
Copy link
Member

Out of curiousity do you know if anybody has said anything about such arm64 binaries?

@eitsupi
Copy link
Member Author

eitsupi commented Jul 12, 2021

Out of curiousity do you know if anybody has said anything about such arm64 binaries?

Sorry, I don't know. I looked into the architectures supported by RSPM and couldn't find any mention of arm64.

As you told me in #153 (comment), there is currently no RStudio or even R binary in arm64 Ubuntu, so I'm convinced that's why RSPM doesn't provide arm64 binaries.

@cboettig
Copy link
Member

@eitsupi nice work, this approach makes sense. Thanks for testing; I don't have access to any arm64 architectures and am not really up to speed on the details of multi-arch Docker -- is there a convenient way to have these images pre-build the arm64 versions, e.g. an arm64-based GH-Actions runner?

@eitsupi
Copy link
Member Author

eitsupi commented Jul 13, 2021

@cboettig Multi-arch images can be built using the docker buildx build command to emulate multiple architectures on a single machine. You should be able to use it by setting it to "experimental": "enabled" in ~/.docker/config.json.
https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

If you want to build a multi-arch image with GitHubActions, it is easy to use docker/build-push-action. The docker buildx build command is used in this action.
https://github.com/docker/build-push-action/blob/master/docs/advanced/multi-platform.md

For example, here is a multi-arch image that I ran install_R.sh included in b5469fc using docker/build-push-action.

ghcr.io/eitsupi/r-ver@sha256:22daf53a18f0335b7a391ef1c21cc61a8a4430a6c08baeda3643f8d418e2fa0e
image
image

As described above, it is not that difficult to build a multi-arch image nowadays, but there are two disadvantages compared to the current amd64-only build.

  1. It takes about two hours to run install_R.sh on GitHub Actions, which is more than ten times slower than the 10 minutes it would take to build only amd64.
  2. It is not possible to build multiple images with dependencies with a single command like docker-compose build. This may be resolved in the future with improvements to the docker buildx bake command, but for now, one command per image is required to build.

I submitted this PR because I figured out how to support arm64 images, but I don't think there is much need to do multi-arch builds of rocker/r-ver in this repository until the arm64 build of RStudio is released.
Of course, if you do decide to build it, I'd be happy to help.

By the way, I am testing the arm64 image on Ubuntu installed on a Raspberry Pi 4😂

@cboettig
Copy link
Member

Excellent, thanks for this. Right, certainly the builds from source are quite slow, but then that's why it would be nice to have pre-built images available to users. Meanwhile, seems like we can go ahead an merge this anyway.

Yup, hopefully buildx will support the compose-like depends-on option which would fit better into the current workflow, though it wouldn't be too difficult to work around that with a Makefile or similar.

@cboettig cboettig merged commit cd7ab3d into rocker-org:master Jul 14, 2021
@benz0li
Copy link
Contributor

benz0li commented Jul 14, 2021

By the way, I am testing the arm64 image on Ubuntu installed on a Raspberry Pi 4😂

@eitsupi Why not use a Mac mini M1 😉. There are also [less invasive] alternatives to GitHub for CI/CD.
ℹ️ https://benz0li.b-data.io/posts/2021/03/host-linux/arm64-gitlab-runners-on-your-mac-mini-m1/

@eitsupi eitsupi deleted the fix-cran-for-arm branch July 14, 2021 10:10
@eitsupi
Copy link
Member Author

eitsupi commented Jul 14, 2021

@cboettig Thank you for merging.

Yup, hopefully buildx will support the compose-like depends-on option which would fit better into the current workflow, though it wouldn't be too difficult to work around that with a Makefile or similar.

Yes, moving to using buildx seems to be a good idea to consider. The ability to set multiple tags is also very attractive.

For example, I heard that if you build the following docker-bake.json with the --push option in the docker buildx bake command, we can build and push an multi-arch image with 4 tags set with one command (I haven't tried it yet!)

{
    "target": {
        "r-ver": {
            "context": "../",
            "dockerfile": "dockerfiles/Dockerfile_r-ver_4.1.0",
            "tags": [
                "docker.io/rocker/r-ver:4.1.0",
                "docker.io/rocker/r-ver:4.1",
                "docker.io/rocker/r-ver:4",
                "docker.io/rocker/r-ver:latest"
            ],
            "platforms": [
                "linux/amd64",
                "linux/arm64"
            ]
        }
    }
}

It seems to make sense to proceed with the creation of a script to generate docker-bake.json from the stack files, and the Makefile and other files to repeat bake.

@benz0li Thanks for the information! The performance of the M1 chip is really amazing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants