bokeh server: color palette rgb is not recognized

[mdk73]

The following code plots a few lines with changing color (rgb):

from pandas import DataFrame
from bokeh.io import show, output_file, output_server
from bokeh.charts import Line, color
import subprocess
import time

if __name__ == '__main__':   
    output_file('rgb-platte.html')

    df = DataFrame({'1': [0,1],
                    '2': [1,2],
                    '3': [2,3],
                    '4': [3,4],
                    '5': [4,5]})

    # set color palette rgb
    palette_r = [50, 100, 150, 200, 250]     # changing the values for 'red'.
    palette_g = 5 * [0]    # keeping values for 'green' constant. 
    palette_b = 5 * [0]    # keeping values for 'blue' constant. 
    palette = list(zip(palette_r, palette_g, palette_b))

    # define the plot
    p = Line(df, title="palette rgb", plot_width = 300, plot_height = 200, color=palette)

    show(p)

Result:

Now I replace output_file('rgb-platte.html') with:

    # start a bokeh server
    args = ['python', '-m', 'bokeh', 'serve']
    server = subprocess.Popen(args) 
    time.sleep(1) # wait for the server to initialize
    output_server('test')

to use the bokeh server.

The resulting image has no colors:

[bryevdv]

This is clearly a bug. @mdk73 is there any JS console output or server output?

[mdk73]

This is the outout I get on the console:

DEBUG:bokeh.server.tornado:Allowed Host headers: ['localhost:5006']
DEBUG:bokeh.server.tornado:These host origins can connect to the websocket: ['localhost:5006']
DEBUG:bokeh.server.tornado:Patterns are: [('/?', <class 'bokeh.server.views.doc_handler.DocHandler'>, {'bokeh_
websocket_path': '/ws', 'application_context': <bokeh.server.application_context.ApplicationContext object at 
0x0000000006E62908>}), ('/ws', <class 'bokeh.server.views.ws.WSHandler'>, {'bokeh_websocket_path': '/ws', 'app
lication_context': <bokeh.server.application_context.ApplicationContext object at 0x0000000006E62908>}), ('/au
toload.js', <class 'bokeh.server.views.autoload_js_handler.AutoloadJsHandler'>, {'bokeh_websocket_path': '/ws'
, 'application_context': <bokeh.server.application_context.ApplicationContext object at 0x0000000006E62908>}),
 ('/static/(.*)', <class 'bokeh.server.views.static_handler.StaticHandler'>)]
INFO:bokeh.command.subcommands.serve:Starting Bokeh server on port 5006 with applications at paths ['/']
INFO:bokeh.server.views.ws:WebSocket connection opened
DEBUG:bokeh.server.views.ws:Receiver created for Protocol('1.0')
DEBUG:bokeh.server.views.ws:ServerHandler created for Protocol('1.0')
INFO:bokeh.server.views.ws:ServerConnection created
DEBUG:bokeh.server.session:pushing doc to session 'test'
INFO:bokeh.server.views.ws:WebSocket connection closed: code=1000, reason='closed'
INFO:bokeh.client.connection:Connection closed by server
INFO:tornado.access:200 GET /?bokeh-session-id=test (::1) 12.52ms
INFO:bokeh.server.views.ws:WebSocket connection opened
DEBUG:bokeh.server.views.ws:Receiver created for Protocol('1.0')
DEBUG:bokeh.server.views.ws:ServerHandler created for Protocol('1.0')
INFO:bokeh.server.views.ws:ServerConnection created
DEBUG:bokeh.server.session:Sending pull-doc-reply from session 'test'
DEBUG:bokeh.server.tornado:[pid 8704] 1 clients connected
DEBUG:bokeh.server.tornado:[pid 8704]   / has 1 sessions with 0 unused

[bryevdv]

What about the browser JavaScript console?

[mdk73]

I am not familiar with the browser development tools, I hope I found the
correct console:

bokeh.min.js?v=50ffae6…:1 Bokeh: setting prefix to /
bokeh.min.js?v=50ffae6…:2 Bokeh: setting log level to: 'info'
bokeh.min.js?v=50ffae6…:62 Bokeh: Websocket connection 0 is now open
bokeh.min.js?v=50ffae6…:2 Bokeh items were rendered successfully

[bryevdv]

That is the correct console, if that's all the output there is, this will require more investigation. The first step would be to see whether RGB tuples work in other contexts (i.e. with bokeh.plotting or bokeh.models code) That might inform as to whether this is an issue just in bokeh.charts somehow, or whether it is more general. If you have the time to run some experiments, that is certainly appreciated, but understand completely if you don't

[bryevdv]

Actually, quicker/easier thing. Make a server app (i.e. don't usurp bokeh serve foo.py directly, don't use output_server from a separate python process) and run bokeh json on it, that will spit out the serialized JSON which could be inspected for problems.

[bryevdv]

OK, this version, run as a server app, works on current master:

# saved as foo.py 

from pandas import DataFrame
from bokeh.charts import Line
from bokeh.io import curdoc

df = DataFrame({'1': [0,1],
                '2': [1,2],
                '3': [2,3],
                '4': [3,4],
                '5': [4,5]})

# set color palette rgb
palette_r = [50, 100, 150, 200, 250]     # changing the values for 'red'.
palette_g = 5 * [0]    # keeping values for 'green' constant.
palette_b = 5 * [0]    # keeping values for 'blue' constant.
palette = list(zip(palette_r, palette_g, palette_b))

# define the plot
plot = Line(df, title="palette rgb", plot_width = 300, plot_height = 200, color=palette)

curdoc().add_root(plot)

Can you try running bokeh serve on that script to see if works with the version you have installed?

---

[bryevdv]

When I run this version of the output_server version:

from pandas import DataFrame
from bokeh.io import show, output_server
from bokeh.charts import Line

if __name__ == '__main__':

    df = DataFrame({'1': [0,1],
                    '2': [1,2],
                    '3': [2,3],
                    '4': [3,4],
                    '5': [4,5]})

    # set color palette rgb
    palette_r = [50, 100, 150, 200, 250]     # changing the values for 'red'.
    palette_g = 5 * [0]    # keeping values for 'green' constant.
    palette_b = 5 * [0]    # keeping values for 'blue' constant.
    palette = list(zip(palette_r, palette_g, palette_b))

    # define the plot
    p = Line(df, title="palette rgb", plot_width = 300, plot_height = 200, color=palette)

    output_server('test')

    show(p)

I get this console output:

So my best guess is there is something weird going on with output_server and RGB tuples.

My best suggestion for a workaround, and actually my general advice in any case is to use actual bokeh serve style Bokeh apps. The output_server mode has extremely limited capability or usefulness with the new server.

[bryevdv]

The problem is definitely that the string "rgb(50, 0, 0)" is being encoded as a field for a column name, rather than a value. This seems to be happening on the python side. But I'm not sure why it would work in a server app, but not output_server context, that's actually troublesome. My best current guess is that serializing the tuple from python works (why a server app works) but that deserializing it in python (ie.e. when the server receives it from a separate script) does not.

[bryevdv]

I should add, another workaround if you don't to make a server app is to construct RGBA hex values, e.g. "#aabbcc", I just ran an output_server script that uses this format and it worked as expected.

[bryevdv]

OK, I definitely think my above hypothesis was correct. Look at:

https://github.com/bokeh/bokeh/blob/master/bokeh/core/properties.py#L1813-L1837

Color tuples get serialized to strings like "rgb(50, 0, 0)", and if these are sent directly from a server app to the browser this is fine, because the browser knows that to do with that. But in output_server cases there is an extra hop: from the python script, then to the server, then to the browser. Well the first hop turns the tuple into a string, but look at this specifically:

https://github.com/bokeh/bokeh/blob/master/bokeh/core/properties.py#L1832-L1834

In the server it is now a string, which encodes as a field.

There are a few options, any of which you could try to make a PR for, which would be a huge help.

• update isconst https://github.com/bokeh/bokeh/blob/master/bokeh/core/properties.py#L1801-L1807 to return true for strings of the form rgb(...) and rgba(...) so that they are caught before the "string" type check above
• update format_tuple https://github.com/bokeh/bokeh/blob/master/bokeh/core/properties.py#L1813-L1818 to return RGBA hex strings like "#aabb09aa", instead of the odd rgb(...) strings.

I think the second is safer and easier. Would you be interested in working up a small PR @mdk73?

[mdk73]

I tried to run the script with 'foo.py' with bokeh serve, and I ran once more in the problems I had when reporting #4092 , I updated this issue with a more complete description of steps to reproduce it.

[mdk73]

I tried to replace the rgb values in the original code with hex values, but I get an error log.

The following lines were added to use palette with hex-values (after defining the rgb-palette, before defining the plot:

    # converting into hex colors
    _palette = []
    for item in palette:
        red = item[0]
        green = item[1]
        blue = item[2]
        alpha = item[3]

        hr = format(red, '02x')
        hg = format(green, '02x')
        hb = format(blue, '02x')
        ha = format(int(alpha *255), '02x')
        _palette_ = '#{}{}{}{}'.format(hr, hg, hb, ha) # rgba
        _palette.append(_palette_)
    palette = _palette  
    print(palette) 

And the error log:

['#320000ff', '#640000ff', '#960000ff', '#c80000ff', '#fa0000ff']
Traceback (most recent call last):
  File "C:\Users\mdk\workspace\DemAn\src\qt\bokeh_palette.py", line 48, in <module>
    color=palette)
  File "C:\Python34\lib\site-packages\bokeh\charts\builders\line_builder.py", line 88, in Line
    return create_and_build(LineBuilder, data, **kws)
  File "C:\Python34\lib\site-packages\bokeh\charts\builder.py", line 67, in create_and_build
    chart.add_builder(builder)
  File "C:\Python34\lib\site-packages\bokeh\charts\chart.py", line 149, in add_builder
    builder.create(self)
  File "C:\Python34\lib\site-packages\bokeh\charts\builder.py", line 518, in create
    chart.add_renderers(self, renderers)
  File "C:\Python34\lib\site-packages\bokeh\charts\chart.py", line 144, in add_renderers
    self.renderers += renderers
  File "C:\Python34\lib\site-packages\bokeh\core\property_containers.py", line 18, in wrapper
    result = func(*args, **kwargs)
  File "C:\Python34\lib\site-packages\bokeh\core\property_containers.py", line 77, in __iadd__
    return super(PropertyValueList, self).__iadd__(y)
  File "C:\Python34\lib\site-packages\bokeh\charts\builders\line_builder.py", line 232, in yield_renderers
    **group_kwargs)
  File "C:\Python34\lib\site-packages\bokeh\charts\glyphs.py", line 169, in __init__
    super(LineGlyph, self).__init__(**kwargs)
  File "C:\Python34\lib\site-packages\bokeh\charts\models.py", line 80, in __init__
    super(CompositeGlyph, self).__init__(**properties)
  File "C:\Python34\lib\site-packages\bokeh\core\properties.py", line 701, in __init__
    setattr(self, name, value)
  File "C:\Python34\lib\site-packages\bokeh\core\properties.py", line 714, in __setattr__
    super(HasProps, self).__setattr__(name, value)
  File "C:\Python34\lib\site-packages\bokeh\core\properties.py", line 453, in __set__
    value = self.descriptor.prepare_value(obj.__class__, self.name, value)
  File "C:\Python34\lib\site-packages\bokeh\core\properties.py", line 272, in prepare_value
    raise e
  File "C:\Python34\lib\site-packages\bokeh\core\properties.py", line 265, in prepare_value
    self.validate(value)
  File "C:\Python34\lib\site-packages\bokeh\core\properties.py", line 1298, in validate
    raise ValueError("expected an element of either %s, got %r" % (nice_join(self.type_params), value))
ValueError: expected an element of either Enum('aliceblue', 'antiquewhite', 'aqua', 'aquamarine', 'azure', 'be
ige', 'bisque', 'black', 'blanchedalmond', 'blue', 'blueviolet', 'brown', 'burlywood', 'cadetblue', 'chartreus
e', 'chocolate', 'coral', 'cornflowerblue', 'cornsilk', 'crimson', 'cyan', 'darkblue', 'darkcyan', 'darkgolden
rod', 'darkgray', 'darkgreen', 'darkgrey', 'darkkhaki', 'darkmagenta', 'darkolivegreen', 'darkorange', 'darkor
chid', 'darkred', 'darksalmon', 'darkseagreen', 'darkslateblue', 'darkslategray', 'darkslategrey', 'darkturquo
ise', 'darkviolet', 'deeppink', 'deepskyblue', 'dimgray', 'dimgrey', 'dodgerblue', 'firebrick', 'floralwhite',
 'forestgreen', 'fuchsia', 'gainsboro', 'ghostwhite', 'gold', 'goldenrod', 'gray', 'green', 'greenyellow', 'gr
ey', 'honeydew', 'hotpink', 'indianred', 'indigo', 'ivory', 'khaki', 'lavender', 'lavenderblush', 'lawngreen',
 'lemonchiffon', 'lightblue', 'lightcoral', 'lightcyan', 'lightgoldenrodyellow', 'lightgray', 'lightgreen', 'l
ightgrey', 'lightpink', 'lightsalmon', 'lightseagreen', 'lightskyblue', 'lightslategray', 'lightslategrey', 'l
ightsteelblue', 'lightyellow', 'lime', 'limegreen', 'linen', 'magenta', 'maroon', 'mediumaquamarine', 'mediumb
lue', 'mediumorchid', 'mediumpurple', 'mediumseagreen', 'mediumslateblue', 'mediumspringgreen', 'mediumturquoi
se', 'mediumvioletred', 'midnightblue', 'mintcream', 'mistyrose', 'moccasin', 'navajowhite', 'navy', 'oldlace'
, 'olive', 'olivedrab', 'orange', 'orangered', 'orchid', 'palegoldenrod', 'palegreen', 'paleturquoise', 'palev
ioletred', 'papayawhip', 'peachpuff', 'peru', 'pink', 'plum', 'powderblue', 'purple', 'red', 'rosybrown', 'roy
alblue', 'saddlebrown', 'salmon', 'sandybrown', 'seagreen', 'seashell', 'sienna', 'silver', 'skyblue', 'slateb
lue', 'slategray', 'slategrey', 'snow', 'springgreen', 'steelblue', 'tan', 'teal', 'thistle', 'tomato', 'turqu
oise', 'violet', 'wheat', 'white', 'whitesmoke', 'yellow', 'yellowgreen'), Regex('^#[0-9a-fA-F]{6}$'), Tuple(B
yte(Int, 0, 255), Byte(Int, 0, 255), Byte(Int, 0, 255)) or Tuple(Byte(Int, 0, 255), Byte(Int, 0, 255), Byte(In
t, 0, 255), Percent), got '#320000ff'
DEBUG:bokeh.server.tornado:Allowed Host headers: ['localhost:5006']
DEBUG:bokeh.server.tornado:These host origins can connect to the websocket: ['localhost:5006']
DEBUG:bokeh.server.tornado:Patterns are: [('/?', <class 'bokeh.server.views.doc_handler.DocHandler'>, {'bokeh_
websocket_path': '/ws', 'application_context': <bokeh.server.application_context.ApplicationContext object at 
0x0000000006E60C88>}), ('/ws', <class 'bokeh.server.views.ws.WSHandler'>, {'bokeh_websocket_path': '/ws', 'app
lication_context': <bokeh.server.application_context.ApplicationContext object at 0x0000000006E60C88>}), ('/au
toload.js', <class 'bokeh.server.views.autoload_js_handler.AutoloadJsHandler'>, {'bokeh_websocket_path': '/ws'
, 'application_context': <bokeh.server.application_context.ApplicationContext object at 0x0000000006E60C88>}),
 ('/static/(.*)', <class 'bokeh.server.views.static_handler.StaticHandler'>)]
INFO:bokeh.command.subcommands.serve:Starting Bokeh server on port 5006 with applications at paths ['/']
DEBUG:bokeh.server.tornado:[pid 10380] 0 clients connected
DEBUG:bokeh.server.tornado:[pid 10380]   / has 0 sessions with 0 unused

[mattpap]

to return RGBA hex strings like "#aabb09aa"
I tried to replace the rgb values in the original code with hex values

Hex color syntax doesn't support alpha component (we follow CSS3 here).

[canavandl]

We've added a warning to the docs about this issue at #3904 and proposed a few work-arounds:

.. warning::
    Supplying lists of RGB or RGBA color tuples as color arguments (either
    directly or as a DataSource column reference) doesn't work. You may read a
    discussion of the issue on our project `GitHub`_ page. Suggested
    work-arounds include using lists of:

    * RGB hexadecimal values
    * `bokeh.colors.RGB` objects (i.e. ``[RGB(255, 0, 0), RGB(0, 255, 0)"]``)
    * CSS-format RGB/RGBA strings (i.e. ``["rgb(255, 0, 0)", "rgb(0, 255, 0)"]``)

.. _GitHub: https://github.com/bokeh/bokeh/issues/2622

[bryevdv]

I am like 99% sure the RGBA hex values did used to work... I thought there was actually a drawn-out issue and PR to make them work.

[bryevdv]

Also, I specifically think strings starting with "rgb" are not recognized as values which is the crux of the problem.

[canavandl]

Also, I specifically think strings starting with "rgb" are not recognized as values which is the crux of the problem.

I think it does if you enclose it in a list (similar to how we handle text values). Passing an "rgb..." string directly won't work.

I also removed mention of RGBA hex support in #3904, because it isn't supported in CSS3.

[philippjfr]

@bryevdv A number of users have been reporting this because we set the default NaN color to (0, 0, 0, 0) which becomes 'rgb(0, 0, 0, 0)'. I'd be happy to work on a fix here, but I can't quite tell what conclusions to draw from the discussion above. Would it be sufficient to allow rgb strings on the property or is that likely to cause issues elsewhere?

[bryevdv]

@philippjfr There's several intersecting things going on:

• Narrowly, this issue concerns a problem with bokeh.client and as such, from that perspective, I don't really care about it/it's not important, however

• incidentally, the problem manifested because strings like "rgb(10, 20, 30, 40)" are not recognized as color values. Offhand, I think making those strings work as color values seems reasonable and useful, and I would support a PR (with tests) that does that

• doing that should also fix the original issue here, but again, don't really care about that specifically since using bokeh.client like this here is something I'd like to discourage in any case.

• tangentially, I think there are problems (again?) with handling actual tuples as colors, and it would be nice to fix that too

TLDR; A PR to make CSS "rgb(...)" strings work is welcome. Looking into/fixing literal tuple handling is also welcome.