scale_fill_manual values not respected

[tdhock]

Hi, I noticed that this code works in 0.8.x but not 0.9. Basically, I want to specify custom fill colors for geom_rect, but sometimes one of the colors disappears.

pro = data.frame(position=c(10,20),logratio=rnorm(2))
library(ggplot2)
sc = scale_fill_manual(values=c(
                      "1breakpoint"="#ff7d7d",
                      "normal"="#f6f4bf",
                      ">0breakpoints"="#a445ee"
                      ))
makeplot = function(ann){
ggplot()+
  geom_rect(aes(xmin=min,xmax=max,fill=annotation),
            ymax=2,ymin=-2,data=ann)+
  geom_point(aes(position,logratio),pch=1,colour="black",data=pro)
}
(p = makeplot(data.frame(min=c(10,13,18),max=c(12,15,20),
  annotation=c(">0breakpoints","1breakpoint","normal")))) #ok
p+sc ##ok
(p =makeplot(data.frame(min=c(10,13,18),max=c(12,15,20),
  annotation=c("1breakpoint",">0breakpoints",">0breakpoints")))) #ok
p+sc ## >0breakpoints color not what I specified in scale_fill_manual!

[wch]

Very strange - whether or not the scale works seems to depend on the number and order of items.

# A data frame with two levels of 'g'
dat2 <- data.frame(x=factor(1:3),
                   y=3,
                   g=c("B","A","A"))

p2 <- ggplot(dat2, aes(x=x,y=y,fill=g)) + geom_bar()

# OK with no scale added
p2

# OK: Scale with three values, order ABC
p2 + scale_fill_manual(values=c("A"="red", "B"="green", "C"="blue"))

# Not OK: Scale with three values, order BCA
p2 + scale_fill_manual(values=c("B"="red", "C"="green", "A"="blue"))

# OK: Scale with two values, order AB
p2 + scale_fill_manual(values=c("A"="green", "B"="red"))

# OK: Scale with two values, order BA
p2 + scale_fill_manual(values=c("B"="red", "A"="green"))

This may be related to #264.

@tdhock This isn't a solution to the underlying problem, but it should work if you change the order of items in your scale to match the factor level order -- that is, put ">0breakpoints" first in the scale.

[wch]

I ran this bisection script: https://gist.github.com/1740215. It says 7bcccb6 is the first bad commit, and that's the commit that fixed #264.

The output of the script can be viewed (without doing all the git bisect stuff) by just running it from the command line.

The colors of the bars in the graph should be lightblue, blue, blue, going from left to right. In 0.8.9, they're in that order. At some point, they become blue, darkblue, darkblue, indicating that the names for the values are not being respected -- that was bug 264. After the fix for 264, the colors are blank, blue, blue.

[briandk]

@wch I'm having similar issues.

I'm starting to develop a working hypothesis. Whatever is causing the problem seems to behave like this:

• Compute the number of unique fill values (n) in the data. In your example above, n = 2, since the only unique values for g are "B" and "A". Let's call the collection of unique data values dataValues.
• Read the first n items in the values argument to scale_fill_manual(). We'll call those first n items scaleValues. In your first "Not OK" example above, scaleValues = c("B"="red", "C"="green")
• Match any values in dataValues with those in scaleValues.

Note that in your examples above, the first "Not OK" one is where the definition for a data's fill value ("A" = "blue") is the third element of values. My hypothesis is that since there were only 2 unique values of that factor in the data itself, the scale function "stopped reading in values" after two and never saw the third element "A" that should have been matched for blue rectangles.

I wondered if this pattern would extend, so I created a gist to test my hypothesis. Every result in my gist is consistent with my hypothesis. For any n unique values of a factor in the data, I can get all the fill colors to appear as long as they're listed within the first n elements of values. I also show (I think) how one can selectively cause certain values not to appear by making sure they're ordered after the nth element of values.

I really, really want to solve this problem because it's affecting my package too. Please let me know if you think my explanation matches the behavior.

[wch]

That sounds like a reasonable explanation. It has something to do with those lines changed in the commit, so it shouldn't be too hard to find and fix.

[wch]

OK, I have a fix. It also required modifying the palette functions in the scales package. One drawback to modularization of packages is that it makes it complicated to synchronize and reference fixes.

The problem was that scale$palette(n) can only take a number, when it sometimes needs a vector of names instead. For example, if the specified colors are: c(B="red", C="blue", A="green"), but the data only contains c("B", "A", "A"), then scale$palette(n) returns only the first two palette items, for B and C. This is why A ends up blank.

I modified the palette functions to optionally take a limits parameter, which is a vector of names. The limits of the data in this case are c("A", "B"), and the function returns the appropriate named palette items.

You can see the commit in scales here: wch/scales@5c74062

To try it out you can either fetch my branches of scales and ggplot2, or you can install directly. Bear in mind that I changed only two types of palettes; I think the others will also need to be changed.

# To install
install_github('wch', 'scales', 'palette-limits')
install_github('wch', 'ggplot2', 'fix-manual-palette')

@briandk: With these fixes, it passes your new tests in #264.

[hadley]

Simplified and expanded test cases:

dat <- data.frame(g = c("B","A","A"))
p <- ggplot(dat, aes(g, fill = g)) + geom_bar()
col <- c("A" = "red", "B" = "green", "C" = "blue")

cols <- function(x) ggplot_build(x)$data[[1]][, "fill"]
fill_scale <- function(order) scale_fill_manual(values = col[order], 
  na.value = "black")

# OK: Scale with three values, order ABC
p + fill_scale(1:3)
p + fill_scale(1:2)
p + fill_scale(2:1)

# NOT OK: missing colour
p + fill_scale(c(3, 2, 1))
p + fill_scale(c(3, 1, 2))
p + fill_scale(c(1, 3, 2))

# NOT OK: doesn't use na.value
p + fill_scale(c(3, 1))
p + fill_scale(c(1, 3))