Release ggplot2 3.1.0

[clauswilke]

Issues to resolve before release:

• summarize_layout does not return final range values #2895 (brushing in shiny doesn't work for coord_trans())

PRs to complete before release:

• Update documentation to meet CII badging requirements #2792
• Make graticule labeling configurable #2849 (needs add explicit parsing option to labels in coord_sf() #2881 first)
• Improve error messages about conflicted parameters #2854
• Implement sec.axis for date, time, and datetime scales #2806
• Force arguments that are used in closures #2807
• Document now unused argument #2808
• add explicit parsing option to labels in coord_sf() #2881
• Fix back-transformation of ranges in coords #2821 (fixes summarize_layout does not return final range values #2895, has been rewritten to not introduce breaking changes)

Prepare for release:

• devtools::check_win_devel()
• rhub::check_for_cran()
• revdepcheck::revdep_check(num_workers = 4)
• Polish NEWS
• If new failures, update email.yml then revdepcheck::revdep_email_maintainers()

Perform release:

• Bump version (in DESCRIPTION and NEWS)
• devtools::check_win_devel() (again!)
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• pkgdown::build_site()
• Tag release
• Merge RC back to master branch
• Bump dev version
• Write blog post
• Tweet
• Add link to blog post in pkgdown news menu

Template from r-lib/usethis#338

[clauswilke]

@topepo Could you run revdep_check() on the current ggplot2 master branch, please? All the pending patches have been merged and we are ready to proceed.

@hadley Could you run devtools::check_win_devel()? It looks like only the package maintainer can do it. And could you also run rhub::check_for_cran()? I got some strange error messages about a missing sf package and about long file names, and I'm not sure how to interpret them.

[clauswilke]

Here are the build logs from my check_for_cran() run:
https://builder.r-hub.io/status/ggplot2_3.0.0.9000.tar.gz-e0789aadf85149dea1cdf0e69ef0b65d
https://builder.r-hub.io/status/ggplot2_3.0.0.9000.tar.gz-a1b23d0e97de472f96fbacfcd8a15538
https://builder.r-hub.io/status/ggplot2_3.0.0.9000.tar.gz-4ef752110db84da59ff24e7b12f239a9

I'll rerun with _R_CHECK_FORCE_SUGGESTS_=false and see what happens.

[clauswilke]

@hadley Still getting rhub errors even with _R_CHECK_FORCE_SUGGESTS_=false:

* checking package dependencies ... ERROR
Package required but not available: 'viridisLite'

https://builder.r-hub.io/status/ggplot2_3.0.0.9000.tar.gz-a168b584c5be475a94a8ff7e8163c249
https://builder.r-hub.io/status/ggplot2_3.0.0.9000.tar.gz-b9766fe583ad49f69a8fdf0599a8fcc6
https://builder.r-hub.io/status/ggplot2_3.0.0.9000.tar.gz-fd4aa9d2c01d493a9bc7ea8383ca33cf

[yutannihilation]

It seems dependency installation fails. I see errors around line 509, though it's not clear which package is wrong.

Error in desc[1L, "Type"] : subscript out of bounds
Calls: ... force -> i.p -> .install.winbinary -> unpackPkgZip -> %in%
In addition: Warning messages:
1: In unzip(zipname, exdir = dest) :
write error in extracting from zip file
2: In unzip(zipname, exdir = dest) :
write error in extracting from zip file
Execution halted

This seems an error from here: https://github.com/wch/r-source/blob/521c90a175d67475b9f1b43d7ae68bc48062d8e6/src/library/utils/R/windows/install.packages.R#L58

[clauswilke]

15 packages are newly broken in revdeps. That's not too bad. I suspect most of them will be due to the aesthetic name standardization that we introduced. That's what's causing the problems in my ggridges package. I used an aesthetic point_color and it is now internally converted to point_colour and thus some parts of the code can't find it. I'll fix ggridges and see if I can write recommendations for best practice.

@hadley @batpigandme @thomasp85 @karawoo @dpseidel @yutannihilation If you have time to look at any of the other broken packages to see what the issues are that would be great.

[batpigandme]

I pulled out the Newly Brokens and popped 'em in a gist here:
https://gist.github.com/batpigandme/b59dca03bfb53b6e9a52332e306ff736

[yutannihilation]

ggmap's installation error seems due to #2741 and this is already fixed in the dev version of ggmap: dkahle/ggmap@86ccd74

Error in get(".all_aesthetics", envir = asNamespace("ggplot2")) : 
  object '.all_aesthetics' not found

[batpigandme]

Spreadsheet so we can keep track of what's been done (à la ggplot2 3.0.0 release) https://docs.google.com/spreadsheets/d/1T0t3dsF_6TPqa-dEztPiM51h-F9FhUWzVIsVeb3sl5A/edit#gid=0

[batpigandme]

Shoot, if you don't have an rstudio email, shoot me an email from wherever to my rstudio email (or request access from the googlesheet), and I can add editing powers for you

[yutannihilation]

Compared to @batpigandme's gist, @topepo's summary omits some packages. Here's the packages and reasons (seems ignorable except for ggtern):

• ggmap: installation error due to User-defined theme elements #2741 (and is already fixed)
• ggtern: installation error due to User-defined theme elements #2741 (needs to be informed to the maintener)
• derfinderPlot: LaTeX errors when creating PDF version.
• gifski: LaTeX errors when creating PDF version.
• PTXQC: R CMD check timed out

@batpigandme
Sorry, I couldn't find your email address... So, I just clicked "REQUEST EDIT ACCESS."

library(tidyverse)

res <- httr::GET("https://gist.githubusercontent.com/batpigandme/b59dca03bfb53b6e9a52332e306ff736/raw/bb05bfe98588d7a0194fd1fe59677d75efef3822/newlybroken_revdeps.md")

httr::content(res) %>% 
  stringr::str_split("\n") %>%
  unlist() %>%
  stringr::str_subset("^# ") %>%
  cat(sep = "\n")
#> # ActisoftR
#> # AneuFinder
#> # BloodCancerMultiOmics2017
#> # dartR
#> # derfinderPlot
#> # GenomicDataCommons
#> # ggmap
#> # ggridges
#> # ggtern
#> # gifski
#> # GOexpress
#> # MCbiclust
#> # mglR
#> # PTXQC
#> # RITAN
#> # scDD
#> # seqsetvis
#> # shadowtext
#> # tricolore
#> # tsbox

^{Created on 2018-09-26 by the reprex package (v0.2.1)}

[clauswilke]

For ggtern, maybe the best way forward to maintain backwards compatibility with 3.0.0 is to reintroduce a variable called .element_tree and then assign the contents of that to ggplot_global$element_tree. In a later release, we can then remove .element_tree.

@nhamilton1980 Could you chime in?

[clauswilke]

And maybe the same should be done with .all_aesthetics, though I really don't know why people access that.

[yutannihilation]

For shadowtext, it's very strange... I'm sure the error occurs in this vignette: https://github.com/GuangchuangYu/shadowtext/blob/master/vignettes/shadowtext.Rmd and this acutally fails in and after devtools::check(), but I cannot reproduce the error in the fresh session...

reprex here: https://gist.github.com/yutannihilation/4f33a397a4fbedb8bf00dca9c6a0d12a

[clauswilke]

I suspect shadowtext is the same problem as ggridges. It uses an aesthetic called bgcolor.

[yutannihilation]

Ah, you are right. The error is basically this (though I'm not sure what happens when devtools::check() yet...):

scales::alpha(NULL, c(NA, NA))
#> Error in col[, rep(1, length(alpha)), drop = FALSE]: subscript out of bounds

^{Created on 2018-09-27 by the reprex package (v0.2.1)}

[yutannihilation]

You are right. I got it; Since GeomShadowText's default_aes is determined on built, if I install the pre-compiled shadowtext binary via CRAN, bg.color stays as is. But, if it is built with ggplot2 v3.0.1, the default_aes no longer has bg.color since it is renamed to bg.colour. Accordingly, data$bg.color will be NULL, which cause the error above.

[clauswilke]

For the packages that fail on manual_scale(), they seem to be using code such as + scale_fill_manual(name = "abcd"):
https://github.com/SWRC/ActisoftR/blob/5df2f4255dcdf91935b90674709f591b347310eb/R/plot_Darwent.r#L134

I'm not sure what the purpose of such code is, but a workaround would be to set values = NULL:

library(ggplot2) # github master

ggplot(iris, aes(Sepal.Length)) + 
  geom_density() + scale_fill_manual(name = "abc")
#> Error in manual_scale(aesthetics, values, ...): argument "values" is missing, with no default

ggplot(iris, aes(Sepal.Length)) + 
  geom_density() + scale_fill_manual(name = "abc", values = NULL)

Created on 2018-09-26 by the reprex package (v0.2.0).

[clauswilke]

I've made a PR to fix ggtern: #2913 This should also fix BloodCancerMultiOmics2017 and tricolore.

I still need to investigate ggridges some more. Some things aren't working yet as expected. Once I've figured things out, I think it would be a good idea to write a blog post about best practices for non-standard aesthetics, so that we don't end up with a mess where everybody does something different. I'm glad that so far very few package developers seem to have tried to do this.

[clauswilke]

I have fixed ggridges and it is on its way to CRAN. In the mean time, here are notes on porting non-standard aesthetics from 3.0.0 to 3.0.1 and best practices for using non-standard aesthetics going forward.

Notes: ggplot2 has always supported both British and American spelling for the word "colour". However, there was no systematic support for alternative types of spelling in the ggplot2 internals. With the release of ggplot2 3.0.1, ggplot2 always uses British spelling internally, and any aesthetics that contain the word "color" are automatically translated into strings containing "colour". (For example, point_color is converted to point_colour.) This may affect package developers who have invented non-standard aesthetics. This code may behave differently in version 3.0.1 versus earlier versions.

First, if you have written a geom that defines British and American versions of your aesthetic, you may get warnings about duplicated aesthetics in ggplot2 3.0.1. Going forward, you must only list the British spelling in default_aes():

default_aes <- aes(point_colour = "red")

(If you list the American spelling, it is internally converted to British spelling.) For backwards compatibility with 3.0.0 or earlier, you can provide the American spelling as an optional aes:

optional_aes <- c("point_color")

Second, in the drawing code of your geom, the data frame will now hold a column named using the British spelling. However, for backwards compatibility with 3.0.0, you can write something lilke:

grid.gpar(col = data$point_color %||% data$point_colour)

This will use the American spelling if provided and the British spelling otherwise.
The operator %||% is defined in the rlang package.

The same construct can be used to make your geom use the standard colour aesthetic unless point_colour is provided:

grid.gpar(col = data$point_color %||% data$point_colour %||% data$colour)

For this to work, the default for point_colour needs to be NULL:

default_aes <- aes(point_colour = NULL)

Finally, it is a good idea to define default scales for any new aesthetics you define. For ggplot2 3.0.1 or later, you only need to define a scale for point_colour with the setting aesthetics = "point_colour", e.g.:

scale_point_colour_discrete <- function(...) scale_colour_hue(aesthetics = "point_colour", l = 40, ...)

For ggplot2 3.0.0 or earlier, you additionally need to define scale_point_color_discrete with aesthetics = "point_color", e.g.:

scale_point_color_discrete <- function(...) scale_colour_hue(aesthetics = "point_color", l = 40, ...)

[clauswilke]

Already sent an email to the maintainer yesterday, so that’s done.

[hadley]

Ok, let's plan for a release on Oct 10?

[clauswilke]

Works for me.

[hadley]

check_win_devel() revealed one NOTE:

Non-standard file/directory found at top level:
  'GOVERNANCE.md'

Would someone mind putting together a PR that adds it to .Rbuildignore?

[hadley]

Sorry for missing the release goal by two weeks. Before I re-kick off the process tonight/tomorrow, is there anything I should know?

[clauswilke]

From my side, there's only PR #2942 that you could consider merging. You suggested that we may want to do this before the release, to close #2938.

And, apparently there are now Travis issues (#2957) that haven't been resolved, due to changes in vdiffr and devtools. Not sure whether they need to be resolved before a release, but until they're resolved any future merge will fail the Travis checks.

[hadley]

Thanks @clauswilke — I've merged those PRs and added a GITHUB_PAT so the travis failures should hopefully be fixed soon.

I notice one missing piece is that we don't have an updated cran-comments.md. Since I emailed all the maintainers on Oct 1, I think I can just mention that, but in future we should probably be a bit more systematic. But just in case @topepo is going to re-run the revdeps, and I'll re-email everyone tomorrow.

[topepo]

Can you add me to the project so I can start a new branch?

I'll rerun today with a longer timeout (see below for prelim results and DB is at https://www.dropbox.com/s/llilyu97detb5xv/data.sqlite?dl=0).

Spoiler alert:

We checked 2338 reverse dependencies (1971 from CRAN + 367 from BioConductor), comparing R CMD check results across CRAN and dev versions of this package.

• We saw 3 new problems
• We failed to check 53 packages

Issues with CRAN packages are summarised below.

New problems

(This reports the first line of each new failure)

• FLightR
  checking examples ... ERROR

• lime
  checking tests ...

• postal
  checking tests ...

Failed to check

• ALA4R (check timed out)
• av (failed to install)
• BACCT (failed to install)
• bamdit (failed to install)
• BayesRS (failed to install)
• BMSC (failed to install)
• BNSP (failed to install)
• bsam (failed to install)
• BTSPAS (failed to install)
• choroplethr (failed to install)
• classify (failed to install)
• CoordinateCleaner (failed to install)
• crmPack (failed to install)
• DeLorean (failed to install)
• dynr (failed to install)
• ewoc (failed to install)
• fdq (failed to install)
• Fgmutils (failed to install)
• fieldRS (failed to install)
• fingerPro (failed to install)
• gastempt (failed to install)
• GENEAsphere (failed to install)
• geofacet (failed to install)
• ggstatsplot (check timed out)
• GmAMisc (failed to install)
• HTSSIP (check timed out)
• ITGM (failed to install)
• jarbes (failed to install)
• JointAI (failed to install)
• MetaStan (failed to install)
• milr (check timed out)
• morse (failed to install)
• mwaved (failed to install)
• onemap (check timed out)
• pcaPA (failed to install)
• phase1RMD (failed to install)
• phylosim (check timed out)
• pmc (check timed out)
• ppcSpatial (failed to install)
• RcmdrPlugin.FuzzyClust (check timed out)
• rpanel (failed to install)
• rstan (check timed out)
• rstanarm (check timed out)
• rsvg (failed to install)
• seewave (failed to install)
• SeqFeatR (failed to install)
• sf (failed to install)
• simmr (failed to install)
• simulator (check timed out)
• TeachingDemos (check timed out)
• trialr (failed to install)
• walker (check timed out)
• zooaRchGUI (failed to install)

[batpigandme]

Added you, @topepo! ✔️

[hadley]

@topepo don't worry about running with a longer time out — given that >2000 packages didn't have problems I think we can assume we didn't accidentally introduce any regressions.

@thomasp85 can you please look into the lime issue?

Would someone else mind looking into the postal and FLightR issues?

[clauswilke]

@topepo Do we have the updated problems.md file?

@hadley The packages lime and FLightR had issues the last time and I looked into it then. For lime, it looked like a network error. The relevant code needs to download data to run, and it worked just fine on my machine with ggplot2 master. Not sure why it reliably breaks in revdepcheck, though. For FLightR, the problem was with the changes to the google map API, specifically the new authentication requirement they added. Again, not sure why it shows up as a new problem in revdepcheck, it should break for old ggplot2 also.

[topepo]

Do we have the updated problems.md file?

Just checked in.

[hadley]

@clauswilke that's perfect - thanks!

[clauswilke]

Ok, now that I have the new problems.md file, I looked more closely at lime and FLightR.

lime is the same version as in the previous revdepcheck, and the error looks similar as before (network error, fails to download a dataset it needs).

FLightR actually has released a new version that has addressed the ggmap issues. The problem there now also seems to be a network error (Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE, : cannot open the connection). The examples work fine for me.

[thomasp85]

You can certainly ignore lime

[hadley]

It was just accepted by the CRAN bot! Thanks everyone for your hard work on this 🎉

[hadley]

Any volunteers to work on the blog post? I'm currently attempting to get travis to build the docs automatically.

[clauswilke]

Is there some convenient mechanism to work on it collaboratively? At a minimum I could provide some materials related to the parts I worked on.

[batpigandme]

Oh, let me PR as far as I've gotten, and you can modify as you like

[clauswilke]

I was just wondering whether we can close this issue and I noticed that we haven't done the last item of the check list: Add link to blog post in pkgdown news menu

@batpigandme Is this something you could do?

[batpigandme]

@clauswilke Yup — not quite sure that's done, though, so I'm not gonna tick it off just yet #3033

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/