New dodging algorithm for box plots #2196

karawoo · 2017-07-06T21:19:24Z

Fixes #2143. This creates a new ggproto object for box plot positions, PositionBoxdodge, and a function pos_boxdodge() that can handle regular and variable-width boxes. collide() gets split into several functions. collide_setup() does the initial setup of the data and gets called by both collide() (which behaves the same as it has been) or collide_box().

@hadley this is slightly different than the algorithm we talked about because it doesn't use a for loop to find the non-overlapping groups. The x value in the data already seems to capture that information, so I think we can avoid the loop, but maybe I'm missing something...

With these changes, box plots look like this:

library("ggplot2")

ggplot(data = iris, aes(Species, Sepal.Length)) +
  geom_boxplot(aes(colour = Sepal.Width < 3.2))

ggplot(data = iris, aes(Species, Sepal.Length)) +
  geom_boxplot(aes(colour = Sepal.Width < 3.2), varwidth = TRUE)

hadley · 2017-07-06T21:29:31Z

Can you include an example with continuous x? I think that's what I was thinking of for the for loop.

hadley · 2017-07-06T21:30:02Z

It also seems like we need a little padding between the individual elements. Could we make that a parameter?

karawoo · 2017-07-06T21:57:16Z

Ohhh continuous x. Yeah you're right, this doesn't work well for that.

library("ggplot2")

## Without varwidth = TRUE it's ok aside from the padding issue
ggplot(diamonds, aes(y = price)) +
  geom_boxplot(aes(x = cut_width(carat, 0.5), fill = depth < 65))

## With the changes from this PR and varwidth = TRUE
ggplot(diamonds, aes(y = price)) +
  geom_boxplot(aes(x = cut_width(carat, 0.5), fill = depth < 65), varwidth = TRUE)

Here's how the second plot looks with the current dev version of ggplot.

ggplot(diamonds, aes(y = price)) +
  geom_boxplot(aes(x = cut_width(carat, 0.5), fill = depth < 65), varwidth = TRUE)
#> Warning: position_dodge requires non-overlapping x intervals

for loop it is, then!

karawoo · 2017-07-09T20:15:34Z

Oh, and there is a larger problem with this: each pair of boxes is getting scaled separately to the group width, which makes the varying widths from varwidth meaningless.

mcol · 2017-07-10T13:23:54Z

Note that the order of the bars in the second example of the OP is different from the rest (I've confirmed it with a checkout of your current position-dodge branch).

karawoo · 2017-07-11T22:42:41Z

Thanks for pointing that out, @mcol. Right now the boxes get placed in the order in which they appear in the data passed to pos_boxdodge() and I need to change that to enforce consistency across groups of boxes.

karawoo · 2017-07-11T22:44:20Z

Also, my example above doesn't really have a continuous x variable since cut_width() converts it to a factor. Here is a better example for testing with (that is still broken):

set.seed(582)
dat <- data.frame(x = sample(4:5, size = 20, replace = TRUE),
                  y = rnorm(20),
                  class = sample(c("a", "b"), size = 20, replace = TRUE))

ggplot(dat, aes(x = x, y = y)) +
  geom_boxplot(aes(group = interaction(x, class), fill = class), varwidth = TRUE)

This is still wrong though

…be dodged from one another i.e. if there are 2 boxes per group all widths get divided by 2

karawoo · 2017-07-14T01:52:05Z

Alright, here is the new box plot algorithm. Below are a bunch of examples of how it looks in practice with continuous and discrete x variables and various combinations of varwidth = TRUE, varwidth = FALSE, preserve = "single", and preserve = "total". Right now preserve = "single" is the default for all box plots, but I could change that so that it's only the default when varwidth = TRUE. This definitely needs some more tests, which I will add soon.

library("ggplot2")

## From the original example:
ggplot(data = iris, aes(Species, Sepal.Length)) +
  geom_boxplot(aes(colour = Sepal.Width < 3.2))

ggplot(data = iris, aes(Species, Sepal.Length)) +
  geom_boxplot(aes(colour = Sepal.Width < 3.2), varwidth = TRUE)

## Diamonds data -- padding between boxes is less noticeable
ggplot(diamonds, aes(y = price)) +
  geom_boxplot(aes(x = cut_width(carat, 0.5), fill = depth < 65), varwidth = TRUE)

 
## A plot with truly continuous x
set.seed(582)
dat <- data.frame(x = sample(4:5, size = 20, replace = TRUE),
                  y = rnorm(20),
                  class = sample(c("a", "b"), size = 20, replace = TRUE))

ggplot(dat, aes(x = x, y = y)) +
  geom_boxplot(aes(group = interaction(x, class), fill = class))

## Truly continuous x and varwidth = TRUE
ggplot(dat, aes(x = x, y = y)) +
  geom_boxplot(aes(group = interaction(x, class), fill = class), varwidth = TRUE)

## Preserve total width
ggplot(mtcars, aes(factor(cyl), y = mpg, fill = factor(vs))) +
  geom_boxplot(position = position_boxdodge(preserve = "total"))

## Preserve individual width with varwidth = TRUE
ggplot(mtcars, aes(factor(cyl), y = mpg, fill = factor(vs))) +
  geom_boxplot(position = position_boxdodge(preserve = "single"), varwidth = TRUE)

## preserve = "total" is incompatible with varwidth = TRUE
ggplot(mtcars, aes(factor(cyl), y = mpg, fill = factor(vs))) +
  geom_boxplot(position = position_boxdodge(preserve = "total"), varwidth = TRUE)
#> Warning: Can't preserve total widths when varwidth = TRUE.

hadley

Results look great!

hadley · 2017-07-14T12:39:02Z

R/position-boxdodge.r

@@ -0,0 +1,116 @@
+#' Position dodge for box plots


I'd like to give this a name that reflects that it works for any geom with variable widths - i.e. it would also work for geom_rect().

position_vardodge()?

Or maybe position_flexdodge(), since it's a more flexible dodge that can handle variable width boxes and arbitrary rectangles?

hadley · 2017-07-14T12:39:32Z

R/position-boxdodge.r

+#' @export
+#' @examples
+#' ggplot(data = iris, aes(Species, Sepal.Length)) +
+#' geom_boxplot(aes(colour = Sepal.Width < 3.2))


Missing indent

hadley · 2017-07-14T12:41:50Z

R/position-boxdodge.r

+  }
+
+  # xid represents groups of boxes that share the same position
+  df$xid <- match(df$x, sort(unique(df$x)))


Oh this is why it won't work with geom_rect(). It's only a few lines of code, so I think it's worth using the for-loop that I originally proposed

hadley · 2017-07-14T12:42:59Z

R/position-boxdodge.r

+  df$xmax <- df$x + (df$new_width / 2)
+
+  # Find the total width of each group of boxes
+  group_sizes <- plyr::ddply(df, "xid", plyr::summarize, size = sum(new_width))


Could this be replaced by a tapply()? I'd prefer to not use plyr in new code, since one day I'd like to eliminate the dependency.

Oh yeah sure. aggregate() might be even better since it'll return a data frame.

hadley · 2017-07-14T12:43:45Z

R/position-boxdodge.r

+  }
+
+  # x values get moved to between xmin and xmax
+  df$x <- rowMeans(df[, c("xmin", "xmax")])


I think df$x <- (df$xmin + df$xmax) / 2 would be clearer

hadley · 2017-07-14T12:45:06Z

tests/testthat/test-geom-boxplot.R

+  overlaps <- vector(length = nrow(d) - 1)  
+  for (i in 2:nrow(d)) {
+    if (d$xmin[i] < d$xmax[i - 1]) {
+      print(i)


Need to remove this print()

hadley · 2017-07-14T12:45:35Z

tests/testthat/test-geom-boxplot.R

+
+  overlaps <- vector(length = nrow(d) - 1)  
+  for (i in 2:nrow(d)) {
+    if (d$xmin[i] < d$xmax[i - 1]) {


This for loop could become (with a small modification) a separate group_overlapping function, and you could use above.

hadley · 2017-07-14T12:48:28Z

The default padding looks slightly too big to my eyes — maybe try 0.05.

It would also be handy to show a couple of examples using with other geoms (i.e. geom_bar())

karawoo · 2017-07-20T23:57:44Z

I did some more research and it'll take a bit more work before position_dodge2() can be used in all the places dodge is currently used. Right now it doesn't work well with dot plots, violins, or error bars.

hadley · 2017-07-21T08:28:07Z

Ok. Let's make it the default for boxplots and bars and then take a look at the others later.

karawoo · 2017-07-25T00:18:25Z

I've made some slight changes to hopefully make this as easy to use for both boxes and bars as possible. position_dodge2() now defaults to having 0 padding, making it work nicely with bars -- you just have to add position = "dodge2" to override the default stacking position. geom_boxplot() uses a default position of position_dodge2(padding = 0.05), so boxes still get padding by default.

Something has gone wrong again with the ordering of the boxes/bars compared to the legend, I'll track that down and fix it.

ggplot(data = iris, aes(Species, Sepal.Length)) +
  geom_boxplot(aes(colour = Sepal.Width < 3.2), varwidth = TRUE)

ggplot(mtcars, aes(factor(cyl), fill = factor(vs))) +
  geom_bar(position = "dodge2")

hadley · 2017-07-25T15:45:35Z

This is looking great! (Although now padding of 0.05 looks too small. Maybe switch back to 0.10?)

…order to match the legend order

karawoo · 2017-07-25T18:49:28Z

Soooo close. I am rethinking the defaults that I changed above. Right now this adds padding between the boxes because geom_boxplot's default position is position_dodge2(padding = 0.1):

ggplot(mtcars, aes(factor(cyl), y = mpg, fill = factor(vs))) +
  geom_boxplot()

But if someone wants to change the preserve argument, they're likely going to do:

ggplot(mtcars, aes(factor(cyl), y = mpg, fill = factor(vs))) +
  geom_boxplot(position = position_dodge2(preserve = "total"))

which results in no padding (they'd need to also add padding = 0.1). So maybe it's better to have position_dodge2() default to adding padding, which will be consistent for box plots even if other position parameters get changed. And for bars, people will have to override that default if they don't want space between them.

hadley · 2017-07-26T07:39:47Z

Yeah, I think that's a good idea. I rather like the padding between bars too.

hadley

Feel free to merge once you've made the news and doc tweaks.

hadley · 2017-07-27T21:44:57Z

NEWS.md

@@ -1,5 +1,8 @@
 # ggplot2 2.2.1.9000

+* Box plot position is now controlled by `position_dodge2()` (@karawoo,
+  #2143).


And bar. And you should add a brief descrption.

hadley · 2017-07-27T21:45:48Z

R/position-dodge2.r

@@ -0,0 +1,154 @@
+#' Alternate method for dodging overlapping objects


Maybe it's worth documenting together with position_dodge()?

lock · 2019-01-18T11:29:15Z

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/

karawoo requested a review from hadley July 6, 2017 21:19

karawoo added 13 commits July 13, 2017 18:27

New position calculations for box plots

512fb33

Update documentation

4be63d9

Add test to ensure that variable width boxes don't overlap

54c1a2a

Don't warn about overlapping intervals for box plots

1f4a984

Scale boxes all at once, rather than by group

698ced7

This is still wrong though

Scale boxes across all the data based on the max number that need to …

10942d1

…be dodged from one another i.e. if there are 2 boxes per group all widths get divided by 2

Dodge boxes when there's a truly continuous x

8e058b9

Make sure boxes are ordered consistently

6c5dd86

Don't overwrite the n argument passed to pos_boxdodge

133ed71

Ensure proper behavior when preserve = "total".

1e08220

Modify test for overlapping boxes

502ad20

Add padding between boxes that occupy the same position

4707fc3

Add note to NEWS.md

e6e6f00

karawoo force-pushed the position-dodge branch from fcedb8f to e6e6f00 Compare July 14, 2017 01:35

hadley reviewed Jul 14, 2017

View reviewed changes

karawoo added 5 commits July 14, 2017 09:45

Merge branch 'master' into position-dodge

e2b1ded

Remove print statement in test 😳

59c014e

Indent code in examples

327597f

Replace rowMeans with (df$xmin + df$xmax) / 2

d169dc9

Replace plyr code

64c3688

karawoo added 3 commits July 20, 2017 13:29

Rename position_boxdodge to position_dodge2

9f7dcbc

Merge branch 'master' into position-dodge

d05a7df

Update geom-bar documentation to mention position_dodge2()

dbdc9a9

karawoo added 2 commits July 24, 2017 17:08

Don't dodge if current xmin is *equal* to previous xmax

79aedfb

Set default padding to 0 for position_dodge2, but override for boxes

7c9aec8

karawoo added 5 commits July 25, 2017 09:54

Change default box plot padding back to 0.1

3706b49

Merge branch 'master' into position-dodge

10e8616

Update position_dodge2 documentation

b059e39

collide_box() does need to reorder the differently than collide() in …

4e2052f

…order to match the legend order

Rename collide_box() to collide2() to match position_dodge2()

09ee427

Return to default padding of 0.1 for position_dodge2()

04666e7

hadley reviewed Jul 27, 2017

View reviewed changes

karawoo added 7 commits July 28, 2017 14:08

Document position_dodge2 together with position_dodge

8398e7e

Merge branch 'master' into position-dodge

b7553ac

Add description of position_dodge2 to NEWS.md

34002fa

Revert the order of preserve arguments for dodge2

b437848

Update dodge examples

a05c3e0

Add stats:: before aggregate()

1cdcc09

Merge branch 'master' into position-dodge

9cea3ef

karawoo merged commit 6c91c1d into tidyverse:master Jul 28, 2017

karawoo mentioned this pull request Aug 1, 2017

position dodge fails to dodge all overlapping values #2106

Closed

batpigandme mentioned this pull request Jan 15, 2018

Default width inconsistent between geom_boxplot and others #2404

Closed

lock bot locked and limited conversation to collaborators Jan 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New dodging algorithm for box plots #2196

New dodging algorithm for box plots #2196

karawoo commented Jul 6, 2017

hadley commented Jul 6, 2017

hadley commented Jul 6, 2017

karawoo commented Jul 6, 2017

karawoo commented Jul 9, 2017

mcol commented Jul 10, 2017

karawoo commented Jul 11, 2017

karawoo commented Jul 11, 2017

karawoo commented Jul 14, 2017

hadley left a comment

hadley Jul 14, 2017

karawoo Jul 14, 2017

karawoo Jul 17, 2017

hadley Jul 14, 2017

hadley Jul 14, 2017

hadley Jul 14, 2017

karawoo Jul 14, 2017

hadley Jul 14, 2017

hadley Jul 14, 2017

hadley Jul 14, 2017

hadley commented Jul 14, 2017

karawoo commented Jul 20, 2017

hadley commented Jul 21, 2017

karawoo commented Jul 25, 2017

hadley commented Jul 25, 2017

karawoo commented Jul 25, 2017

hadley commented Jul 26, 2017

hadley left a comment

hadley Jul 27, 2017

hadley Jul 27, 2017

lock bot commented Jan 18, 2019

		@@ -0,0 +1,154 @@
		#' Alternate method for dodging overlapping objects

New dodging algorithm for box plots #2196

New dodging algorithm for box plots #2196

Conversation

karawoo commented Jul 6, 2017

hadley commented Jul 6, 2017

hadley commented Jul 6, 2017

karawoo commented Jul 6, 2017

karawoo commented Jul 9, 2017

mcol commented Jul 10, 2017

karawoo commented Jul 11, 2017

karawoo commented Jul 11, 2017

karawoo commented Jul 14, 2017

hadley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hadley commented Jul 14, 2017

karawoo commented Jul 20, 2017

hadley commented Jul 21, 2017

karawoo commented Jul 25, 2017

hadley commented Jul 25, 2017

karawoo commented Jul 25, 2017

hadley commented Jul 26, 2017

hadley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lock bot commented Jan 18, 2019