-
Notifications
You must be signed in to change notification settings - Fork 38
Builds for different versions of GDAL, PROJ, GEOS #31
Comments
Thanks @Robinlovelace , this is definitely something we want to support in rocker-versioned2. For us, I think the key piece is just knowing what versions/combinations would be useful to publish tags for, and what those tags should be. (i.e. in a You can get a sense of how the new system will specify a rocker stack along with specific versions of everything like so: https://github.com/rocker-org/rocker-versioned2/blob/master/versions-cuda.json |
That is the key question for me too but I don't feel qualified to answer. A starter for 10 could be:
There are many people who are well qualified to make informed suggestions, @edzer being the first person who comes to mind (sorry for the nudge, hope this will be useful for r-spatial). |
Looking at how you can seemingly use partial docker images, this may also be of use: |
Quickfire question for @cboettig (having already checked-in with Dirk). Please could you take a scan of this and let me know of anything you'd flag as unhelpful or in need of improvement before it goes out (plan is for Monday afternoon): https://github.com/geocompr/geocompr.github.io/blob/source/content/post/2020/installing-r-spatial-packages-linux.md Mentions rocker/geospatial and alludes to this very issue. Thank you! |
apologies for not getting to this on time but great blog post! |
Hope it helps with your efforts. Any update on that (I regard any work done in the current context as a bonus ; ) ? |
it's in my queue! remote teaching my students and homeschooling my kids has taken a toll on my rocker development! |
Heads-up @cboettig, another variation in build matrices should be RStudio set-up options. I have demonstrated the feasibility of this, building on the new shiny See the readme of that repo for the rather manual way I'm approaching this, inspired by your impressive work in https://github.com/rocker-org/rocker-versioned2 Let me know if I can be of any help, very glad to see stable docker images with many tags building from Ubuntu on the production line! |
@Robinlovelace Okay, finally getting back to this, apologies. So far we have these flavors:
Unfortunately, I'm hitting a problem on the upcoming release ( We may end up creating both |
@Robinlovelace nevermind, looks like we do not need So building on Ubuntu 20.04 we get:
So for 4.0.0, I think we will build the following tags for
How does this sound? |
That sounds ideal, and will simplify the set-up for https://hub.docker.com/r/geocompr/geocompr Looking forward to testing geographic packages on R 4.0.0 and, looking at https://hub.docker.com/r/rockerdev/geospatial/tags it seems they're working 🎉 |
Debating whether we keep alive the older The builds are fine and all, but mostly trying to avoid confusion with our tagging conventions -- i.e. do need to have tags that explicitly refer to both Thoughts? |
Sounds like a good plan to me. |
Update on this @cboettig, I see that PROJ 7.0 is currently only available on 18.04 currently. I'm wondering when they will support it in I think we need a way to get the latest version of PROJ and having Ubuntu 18.04 with Seems Focal may be a bit close to the cutting edge but I think you guys made the right decision by migrating early. |
Care to expand? It is a full-blown LTS release with full supported and a first-class release status. |
Sure: Source: https://launchpad.net/~ubuntugis/+archive/ubuntu/ubuntugis-unstable?field.series_filter=focal Vs https://launchpad.net/~ubuntugis/+archive/ubuntu/ubuntugis-unstable?field.series_filter=bionic Hopefully soon now that the point release is almost released they will update the UbuntuGIS PPA with full support for Focal which would make this true for geographic libraries (currently the release version of PROJ is 6.3 which has known issues). I think that it will have full support within a month or so, not a moment too soon! |
I see. I missed that the comment was about what ubuntugis has, or has not, offered so far. They may well be behind. Focal (and when left unqualified meaning the actual distro) is actually fine and stable, at least on my machines but also from what one can read more broadly online. We were just focussing on different sets here. |
Yeah, i still need to prep a focal-based unbuntugis:unstable... on the todo
list
On Mon, Jul 27, 2020 at 9:59 AM Dirk Eddelbuettel ***@***.***> wrote:
I see. I missed that the comment was about what ubuntugis has, or has not,
offered so far. They may well be behind.
Focal (and when left unqualified meaning the actual distro) is actually
fine and stable.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#31 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABWK6WXLHADWJUHRKF4D73R5WW7TANCNFSM4LKWG3GQ>
.
--
---
Carl Boettiger
http://carlboettiger.info/
|
May as well wait until there is actually anything (more than 3 packages) in the ubuntugis-unstable repo. I agree with @eddelbuettel that Focal as a distro is solid and that they are behind. Personally I'm waiting for the point release before I consider Focal an LTS and before I upgrade my work computers and I imagine the team may be thinking likewise. One way to elicit what they are thinking is by asking them here (I'm tempted, could ask on Rocker's behalf). |
This raises the parallel question: what is the status of Bionic-based Rocker images, will they be maintained for the next year or so? |
@Robinlovelace sorry I've been mostly away from my computer and didn't read your thread carefully on my phone -- now I see what you mean about the Yeah, I think keeping 18.04 alive is not off the table, we are finding a few other edge cases (some weirdness with CyVerse internals for instance) that also have issues on I do think it would be a very good idea to make contact with the Ubuntu-gis PPA maintainers and hear there thoughts on the matter. It seems a bit weird that |
I say lets revisit this after 6th August after the point release is out. I'm in favour of contacting the ppa maintainers if there are no big changes by early September and it may be worth reaching out before then. About tags... If it's good enough for Nvidia it's good enough for Rocker is my thinking. Many thanks for getting this stuff up and running, wrinkles are inevitable and Ubuntu is solid (although I do wonder in a grass-is-greener way what a Rocker container based on Clear Linux base image would look like - even more time consuming to set-up and maintain I imagine!). |
@Robinlovelace haven't forgotten this thread. Have you had a chance to reach out to the ubuntugis folks? Looks like they link a mailing list https://lists.osgeo.org/mailman/listinfo/ubuntu, on their PPA, I haven't subscribed yet. Looks like they might end up pointing us to the debian-gis group, https://debian-gis-team.pages.debian.net/, from which most of their stuff is derived anyway and may be more active right now? Apologies I don't have enough bandwidth to reach out those folks now; but I do think it would be preferable to have a the |
Hey @cboettig I did send and email to the list but it bounced. Can try again with the osgeo/debian-gis lists you link to and let you know if I have any more luck, but probably won't be until September... I would be surprised if UbuntuGIS didn't fully support Focal now that it's officially the LTS and imagine they're just waiting to update the PPA, in which case reviving the 18.04 wouldn't be necessary. |
I think that is worth doing anyway and reduces dependencies on other people. |
@Robinlovelace I took a quick look at doing this with https://github.com/rocker-org/rocker-versioned2/blob/master/scripts/install_geospatial_unstable.sh. Doesn't seem to build on GEOS 3.8.0 builds for me if I build this script on top of There's probably a well-known solution to this but might take a bit more futzing around or messaging some mailing lists. Also, just wanted to flag that regardless of how we get the newer libs installed, it will raise issues with the current default of installing binary R packages, since they will be compiled against the standard ubuntu versions. I'm currently trying to work around that on an ad-hoc basis by installing from source only for those R packages which I know bind these 3 libraries, but maybe might be safer to switch the default to install all R packages from source for the geospatial-unstable image? Other thoughts:
|
Many thanks for delving further into this Carl. The most simple and future-proof suggestion seems to be to add the bionic PPA to image, assuming that works. Regarding your question
Yes I think it would be safer to install all packages on the geospatial-ubuntu images from source. Those are just my first thoughts and I'm a bit out of my depth. Worth asking developers of geographic R packages who use Ubuntu. Apologies for the tag but what are your thoughts on this @edzer? Would a 20.04 image pointing to the 18.04 version of the UbuntuGIS unstable PPA be useful? These supported images, with different versions of OSGeo libraries, could help with testing of I imagine the best solution from an R developer's perspective would be to be able to test against the latest versions shipped by OSGeo. That could also be the best solution from a performance perspective. OSGeo has pretty good documentation on building from source I think and has Docker images with a range of configurations for PROJ and GDAL: https://hub.docker.com/u/osgeo So my inclination would be ambitious in terms of building from source (in addition to supporting the UbuntuGIS PPA somehow) but that's easy for me to say as I'm not volunteering to do the skilled work of getting these images working, although am very happy to provide further ideas/feedback. Important to get strong foundations on which many other things can build. |
Update, I tried adding these lines into
It updated PROJ which seemed to have worked by sf is still linked to PROJ 6.3.1, possibly due to previous binary installation... |
images are now live, e.g.
|
Great, plan to take a look tomorrow. Many thanks for amazing work Carl! I'm happy that this is 'good enough' to solve the purposes identified in the original post. However I will keep it open in case there is a possibility of getting dev versions of OSGeo packages installed from source. Looking at r-spatial/sf#1510 I think such a |
Thanks @Robinlovelace . My adaption of the OSGEO recipe is in the Now I think I did it right, but for reasons I haven't figured out,
gives me the error:
So it's successfully detecting maybe @edzer can spot what I've done wrong in linking |
Great reproducible example @cboettig it may be worth opening an issue on the sf tracker on this, should be able to find dev versions of these packages, these 'dev' containers will be very useful for debugging I think. Apologies for more unsolicited tagging but heads-up @rsbivand - this should help efforts to build against the latest versions of OSGeo packages so any suggestions welcome. Thanks. |
Heads-up @cboettig I'm testing this roughly as follows (reproducible example from imperfect memory):
It's taking ages, building GDAL from source looks like a monster operation best done upstream in docker image builds so confident this is well suited to dockerisation. Still compiling / doing what it needs to do... Does that approach to debugging this GDAL dev issue seem reasonable to you? Once that is up-and-running and the |
Heads-up @cboettig I've put in a PR that should fix the issue: rocker-org/rocker-versioned2#92 |
Looks very similar to r-spatial/sf#1268 (comment) where the problem seemed related to proj, not libsqlite3-dev. |
Hi @edzer thanks for the comment + link but I think I've fixed it in the PR above. |
Will follow-up in the other issues though to join things up, great someone's on the same track! |
Any luck getting this image built @cboettig ? I think it works locally... Disclosure of interest: want to use it as the basis of new Dockerfiles in the geocompr/docker repo. |
@Robinlovelace yup, I think it's working. just waiting for the main build to re-construct the stacks now (since all the |
Heads-up @cboettig I just took a look here and cannot see any 'osgeo-dev' or similar tagged images: https://hub.docker.com/r/rocker/geospatial/tags |
yeah apologies, still not up yet. students got a bit ambitious yesterday and crashed our server that does the builds... hopefully up soon, I'll ping |
Good to hear of ambitious students! Good on them for pushing the boundaries. Keep me posted, sorry for the nudging : ) |
@Robinlovelace actually looks like I'm still having the same issue in being unable to install |
Originally posted by @Robinlovelace in r-spatial/sf#1518 (comment) Hi @Robinlovelace, I continue here the discussion as suggested. Probably this was due to an error occurred running |
Thanks @ranghetti I'm trying this again in the latest version of
|
Heads-up @ranghetti, inspired by your work, I've also created a placeholder that builds on the |
Heads-up @cboettig this looks relevant: r-spatial/sf#1268 (comment) |
Unfortunately in my case this did not help: cd /tmp
git clone git@github.com:rocker-org/rocker-versioned2
cd rocker-versioned2
docker run --rm -ti -v $(pwd):/home/rstudio/data rocker/verse /bin/bash
rm -R /rocker_scripts/
ln -s /home/rstudio/data/scripts/ /rocker_scripts
export PKG_CONFIG_PATH=/path/to/proj/lib/pkgconfig/
bash /home/rstudio/data/scripts/install_gdal_source.sh
Rscript -e 'remotes::install_github("r-spatial/sf", configure.args=("--with-proj-lib=/usr/local/lib --with-proj-include=/usr/local/include --with-proj-share=/usr/local/share/proj --with-proj-data=/usr/local/share/proj"))' An error interrupted the installation before reaching step
I will deepen that as soon as I will have time (the step |
@Robinlovelace and co: Using @edzer 's Dockerfile as a base, I believe we now have a working
Importantly, this image should allow GEOS_VERSION, GDAL_VERSION, and PROJ_VERSION to be set as environmental variables. because Compare this to
and also
|
Great work @cboettig. Shouldn't the |
Good call. fixed, now
and thus provides the latest current releases now I think, more recent than and now versions are set explicitly in the stackfile. Good to close this out? |
👍 great work, epic issue! |
As discussed in r-spatial/discuss#35 it's useful to be able to control the upstream spatial libraries on which R's geographic infrastructure builds, for reproducibility and development work. I have created an issue in our geocompr book project to support this: geocompx/geocompr#476
To minimise re-inventing the wheel, I wonder if it would be possible to build from version-tags of rocker/geospatial e.g.
The text was updated successfully, but these errors were encountered: