New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package a session with specific external libraries #46

Open
nuest opened this Issue Jan 31, 2017 · 7 comments

Comments

2 participants
@nuest
Member

nuest commented Jan 31, 2017

As becomes clear in the discussion on geospatial libraries in Rocker, the versions of linked external libraries matter.

Can we support packaging explicit version of linked libraries?

> extSoftVersion()
                     zlib                     bzlib                        xz 
                  "1.2.8"      "1.0.6, 6-Sept-2010"              "5.1.0alpha" 
                     PCRE                       ICU                       TRE 
        "8.38 2015-11-23"                        "" "TRE 0.8.0 R_fixes (BSD)" 
                    iconv                  readline 
             "glibc 2.23"                     "6.3"

> library(sf)
Linking to GEOS 3.5.0, GDAL 2.1.2, proj.4 4.9.2
> sf::sf_extSoftVersion()
   GEOS    GDAL  proj.4 
"3.5.1" "2.1.2" "4.9.2" 

This information could be accessed by a funtion <pkgname_extSoftVersion>, see extSoftVersion and (sf_extSoftVersion()](https://github.com/edzer/sfr/blob/5c3dfea395af81bf352b4007d16c6a7d419883c2/R/init.R#L59)

@nuest nuest added this to Current Sprint (Jan 2017) in containerit package development Jan 31, 2017

@MatthiasHinz MatthiasHinz moved this from Sprint Feb 2017 to In Progress in containerit package development Mar 21, 2017

@MatthiasHinz

This comment has been minimized.

Show comment
Hide comment
@MatthiasHinz

MatthiasHinz Mar 21, 2017

Collaborator

This would be very tough, I think. First of all, there is not generic solution we could use. We would probably have to add individual support for every single version.

Via APT, it is possible to install different versions of one package, but not probable that we can make use of that: To find out, which versions of a package could be installed via apt-get install «pkg»=«version», you can use a command of this form: apt-cache show libreadline6-dev | grep Version. The main problem is, that the repositories mostly provide the most recent version of a package only. Even if there were repositories with historic packages, we would still have to match libraries with package names and map between version-tags, which may vary depending on the platform and architecture. If you want to read a bit more, their are plenty of discussion on how to 'downgrade' a package, e.g. https://askubuntu.com/questions/138284/how-to-downgrade-a-package-via-apt-get/138327.

It seems a little bit more doable to install versioned packages from source. This is slow, and there is also not a generic solution here. Neither is there a central source code repository that we could use (to my knowledge), but for most of the libraries, we can find some repository that includes older versions. For instance:

We already have examples for installing proj and gdal from source here. We could use such commands as templates with different version numbers. This will work for some cases , but of course not for all.

# install latest GDAL and PROJ, based on https://github.com/edzer/sfr/blob/master/.travis.yml
WORKDIR /tmp/gdal
RUN wget http://download.osgeo.org/gdal/2.1.0/gdal-2.1.0.tar.gz \
	&& tar zxf gdal-2.1.0.tar.gz \
	&& cd gdal-2.1.0 \
	&& ./configure \
	&& make \
	&& make install
WORKDIR /tmp/proj
RUN wget http://download.osgeo.org/proj/proj-4.9.3.tar.gz \
	&& tar zxvf proj-4.9.3.tar.gz \
	&& cd proj-4.9.3 \
	&& ./configure \
	&& make \
	&& make install \
	&& ldconfig
RUN rm -r /tmp/gdal /tmp/proj
Collaborator

MatthiasHinz commented Mar 21, 2017

This would be very tough, I think. First of all, there is not generic solution we could use. We would probably have to add individual support for every single version.

Via APT, it is possible to install different versions of one package, but not probable that we can make use of that: To find out, which versions of a package could be installed via apt-get install «pkg»=«version», you can use a command of this form: apt-cache show libreadline6-dev | grep Version. The main problem is, that the repositories mostly provide the most recent version of a package only. Even if there were repositories with historic packages, we would still have to match libraries with package names and map between version-tags, which may vary depending on the platform and architecture. If you want to read a bit more, their are plenty of discussion on how to 'downgrade' a package, e.g. https://askubuntu.com/questions/138284/how-to-downgrade-a-package-via-apt-get/138327.

It seems a little bit more doable to install versioned packages from source. This is slow, and there is also not a generic solution here. Neither is there a central source code repository that we could use (to my knowledge), but for most of the libraries, we can find some repository that includes older versions. For instance:

We already have examples for installing proj and gdal from source here. We could use such commands as templates with different version numbers. This will work for some cases , but of course not for all.

# install latest GDAL and PROJ, based on https://github.com/edzer/sfr/blob/master/.travis.yml
WORKDIR /tmp/gdal
RUN wget http://download.osgeo.org/gdal/2.1.0/gdal-2.1.0.tar.gz \
	&& tar zxf gdal-2.1.0.tar.gz \
	&& cd gdal-2.1.0 \
	&& ./configure \
	&& make \
	&& make install
WORKDIR /tmp/proj
RUN wget http://download.osgeo.org/proj/proj-4.9.3.tar.gz \
	&& tar zxvf proj-4.9.3.tar.gz \
	&& cd proj-4.9.3 \
	&& ./configure \
	&& make \
	&& make install \
	&& ldconfig
RUN rm -r /tmp/gdal /tmp/proj
@MatthiasHinz

This comment has been minimized.

Show comment
Hide comment
@MatthiasHinz

MatthiasHinz Mar 21, 2017

Collaborator

Any comments on this? I don't think that either of the solutions are easy to implement right now, but we could discuss. @nuest @edzer @MarkusKonk

Collaborator

MatthiasHinz commented Mar 21, 2017

Any comments on this? I don't think that either of the solutions are easy to implement right now, but we could discuss. @nuest @edzer @MarkusKonk

@nuest

This comment has been minimized.

Show comment
Hide comment
@nuest

nuest Mar 22, 2017

Member

If we can demonstrate that it is doable for "most" or "many" cases, that would already be quite something!

That there is no central repository for this means that the containerit package becomes central and handles the specifics, e.g. generating the install command from a specific GDAL version.

That image building is slow is not an issue for now.

Member

nuest commented Mar 22, 2017

If we can demonstrate that it is doable for "most" or "many" cases, that would already be quite something!

That there is no central repository for this means that the containerit package becomes central and handles the specifics, e.g. generating the install command from a specific GDAL version.

That image building is slow is not an issue for now.

@nuest

This comment has been minimized.

Show comment
Hide comment
@nuest

nuest Mar 27, 2017

Member

@MatthiasHinz plase put in here a few (!) sentences of the current state of the implementation.

Member

nuest commented Mar 27, 2017

@MatthiasHinz plase put in here a few (!) sentences of the current state of the implementation.

@MatthiasHinz

This comment has been minimized.

Show comment
Hide comment
@MatthiasHinz

MatthiasHinz Mar 27, 2017

Collaborator

I implemented a parameter versioned_libs (TRUE/FALSE) that triggers dockerfile() to match the linked external libraries.

Currently, the package only reads sf::sf_extSoftVersion() creates RUN Instructions for installing PROJ and GDAL from source.

Example:

> df <- dockerfile(expression(library(sf)),versioned_libs = TRUE)
INFO [2017-03-27 12:00:13] Creating an R session with the following arguments:
	 R  --silent --vanilla -e "library(sf)" -e "info <- sessionInfo()" -e "save(list = \"info\", file = \"/tmp/Rtmp69G1ZA/rdata-sessioninfo19c73ba050a\")"
> library(sf)
Linking to GEOS 3.5.0, GDAL 2.1.3, proj.4 4.9.3
> info <- sessionInfo()
> save(list = "info", file = "/tmp/Rtmp69G1ZA/rdata-sessioninfo19c73ba050a")
> 
> 
Loading required namespace: sf
WARN [2017-03-27 12:00:15] No explicit for support for the version 3.5.0 of the linked external software GEOS
WARN [2017-03-27 12:00:15] No explicit for support for the version NA of the linked external software lwgeom
INFO [2017-03-27 12:00:15] Trying to determine system requirements for the package(s) 'sf,magrittr,DBI,units,Rcpp,udunits2' from sysreq online DB
INFO [2017-03-27 12:00:16] Created Dockerfile-Object based on expression

Dockerfile:

FROM rocker/r-ver:3.3.3
LABEL maintainer="matthiashinz"
RUN export DEBIAN_FRONTEND=noninteractive; apt-get -y update \
 && apt-get install -y gdal-bin \
	libgeos-dev \
	libudunits2-dev \
	make \
	wget
WORKDIR /tmp/gdal
RUN wget http://download.osgeo.org/gdal/2.1.3/gdal-2.1.3.tar.gz \
 && tar zxf gdal-2.1.3.tar.gz \
 && cd gdal-2.1.3 \
 && ./configure \
 && make \
 && make install \
 && ldconfig \
 && rm -r /tmp/gdal
WORKDIR /tmp/proj
RUN wget http://download.osgeo.org/proj/proj-4.9.3.tar.gz \
 && tar zxf proj-4.9.3.tar.gz \
 && cd proj-4.9.3 \
 && ./configure \
 && make \
 && make install \
 && ldconfig \
 && rm -r /tmp/proj
RUN ["install2.r", "-r 'https://cloud.r-project.org'", "sf", "magrittr", "DBI", "units", "Rcpp", "udunits2"]
CMD ["R"]
Collaborator

MatthiasHinz commented Mar 27, 2017

I implemented a parameter versioned_libs (TRUE/FALSE) that triggers dockerfile() to match the linked external libraries.

Currently, the package only reads sf::sf_extSoftVersion() creates RUN Instructions for installing PROJ and GDAL from source.

Example:

> df <- dockerfile(expression(library(sf)),versioned_libs = TRUE)
INFO [2017-03-27 12:00:13] Creating an R session with the following arguments:
	 R  --silent --vanilla -e "library(sf)" -e "info <- sessionInfo()" -e "save(list = \"info\", file = \"/tmp/Rtmp69G1ZA/rdata-sessioninfo19c73ba050a\")"
> library(sf)
Linking to GEOS 3.5.0, GDAL 2.1.3, proj.4 4.9.3
> info <- sessionInfo()
> save(list = "info", file = "/tmp/Rtmp69G1ZA/rdata-sessioninfo19c73ba050a")
> 
> 
Loading required namespace: sf
WARN [2017-03-27 12:00:15] No explicit for support for the version 3.5.0 of the linked external software GEOS
WARN [2017-03-27 12:00:15] No explicit for support for the version NA of the linked external software lwgeom
INFO [2017-03-27 12:00:15] Trying to determine system requirements for the package(s) 'sf,magrittr,DBI,units,Rcpp,udunits2' from sysreq online DB
INFO [2017-03-27 12:00:16] Created Dockerfile-Object based on expression

Dockerfile:

FROM rocker/r-ver:3.3.3
LABEL maintainer="matthiashinz"
RUN export DEBIAN_FRONTEND=noninteractive; apt-get -y update \
 && apt-get install -y gdal-bin \
	libgeos-dev \
	libudunits2-dev \
	make \
	wget
WORKDIR /tmp/gdal
RUN wget http://download.osgeo.org/gdal/2.1.3/gdal-2.1.3.tar.gz \
 && tar zxf gdal-2.1.3.tar.gz \
 && cd gdal-2.1.3 \
 && ./configure \
 && make \
 && make install \
 && ldconfig \
 && rm -r /tmp/gdal
WORKDIR /tmp/proj
RUN wget http://download.osgeo.org/proj/proj-4.9.3.tar.gz \
 && tar zxf proj-4.9.3.tar.gz \
 && cd proj-4.9.3 \
 && ./configure \
 && make \
 && make install \
 && ldconfig \
 && rm -r /tmp/proj
RUN ["install2.r", "-r 'https://cloud.r-project.org'", "sf", "magrittr", "DBI", "units", "Rcpp", "udunits2"]
CMD ["R"]
@MatthiasHinz

This comment has been minimized.

Show comment
Hide comment
@MatthiasHinz

MatthiasHinz Mar 27, 2017

Collaborator

The information to install versioned libs is based on a config file that you can find here:

https://github.com/MatthiasHinz/containerit/blob/master/inst/containeRit_config.json

The following R script provides some helper methods for creating such a file, for reading/writing and initializing, and getter methods for the information of interest (I want to re-write the first part into a test later):
https://github.com/MatthiasHinz/containerit/blob/master/R/containeRit-config.R

The code that makes use of the config and creates the corresponding RUN commands is created here:
https://github.com/MatthiasHinz/containerit/blob/master/R/package-installation-methods.R#L82

Collaborator

MatthiasHinz commented Mar 27, 2017

The information to install versioned libs is based on a config file that you can find here:

https://github.com/MatthiasHinz/containerit/blob/master/inst/containeRit_config.json

The following R script provides some helper methods for creating such a file, for reading/writing and initializing, and getter methods for the information of interest (I want to re-write the first part into a test later):
https://github.com/MatthiasHinz/containerit/blob/master/R/containeRit-config.R

The code that makes use of the config and creates the corresponding RUN commands is created here:
https://github.com/MatthiasHinz/containerit/blob/master/R/package-installation-methods.R#L82

@MatthiasHinz

This comment has been minimized.

Show comment
Hide comment
@MatthiasHinz

MatthiasHinz Mar 27, 2017

Collaborator

In order to support versioned GEOS and lwgeom, you would only have to create corresponding entries in the json file.

If we want to support basse::extSoftVersion(), the implementation should be very similar.

Collaborator

MatthiasHinz commented Mar 27, 2017

In order to support versioned GEOS and lwgeom, you would only have to create corresponding entries in the json file.

If we want to support basse::extSoftVersion(), the implementation should be very similar.

@nuest nuest moved this from In Progress to Backlog in containerit package development Mar 30, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment