Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Displaying custom data on hover for RGBA image #8668

Closed
paulaceccon opened this issue Feb 18, 2019 · 13 comments · Fixed by #8773
Closed

Displaying custom data on hover for RGBA image #8668

paulaceccon opened this issue Feb 18, 2019 · 13 comments · Fixed by #8773

Comments

@paulaceccon
Copy link

It would be nice to be possible to show some sort of information on RGBA images using Bokeh (https://stackoverflow.com/questions/54753032/hover-image-rgba-bokeh/54753614#54753614).
For instance, I'm working on a project in which we provide an RGBA image where the alpha represents some sort of confidence. We would like, when hovering an image pixel, to be able to show this confidence (in this case, the rgba[:,:,3] value).

@bryevdv
Copy link
Member

bryevdv commented Feb 18, 2019

cc @jlstevens I don't recall what the obstacles to this were, offhand.

@jbednar
Copy link
Contributor

jbednar commented Feb 18, 2019

For my purposes, I'd like to be able to see the full RGBA tuple, or any particular channel, or (ideally) any other column of data sharing the same shape. E.g. if I had data that was turned into RGBA by datashader, it would be nice to hover over and see the original data value. Currently I can do that by overlaying an invisible (alpha=0) copy of the Image, but that's awkward.

@bryevdv
Copy link
Member

bryevdv commented Feb 18, 2019

Some random thought:

  • Probably good to add some new special fields: $red, $green, $blue $alpha Obviously these would only be useful or apply to hovers of ImageRGBA (or possibly also Image)

  • I think linking to data shader or other data is possible in the same way as with Image? I.e. if you have another column foo with some other data (per-pixel) then the hover tool would drill down and display @foo gives you the same per-pixel hover that Image does. Or are you asking for something else?

  • Another possible usage: coupled with CustomJSHover a per-pixel extra data column that had some external id, etc. could use that to query some other API to get lots of additional information per-pixel without having to dump everything in the CDS (useful obviously only if there is an API available)

@jbednar
Copy link
Contributor

jbednar commented Feb 18, 2019

  • The new special fields sound reasonable to me. Seems like @Alpha could apply to any glyph?

  • The datashader support is maybe something we could do in HoloViews; we transform the dataset (which is not in the browser) into an array and then to an RGBA (which is always passed to the browser); we could maybe make the array available (optionally) as a column so that it would be available for hover. That's related to things @jlstevens is already working on, so I'll leave that up to him.

  • Yes, being easily able to query (after a delay, presumably) to find out what to display in the hover would be very useful.

@bryevdv
Copy link
Member

bryevdv commented Mar 12, 2019

@jlstevens any interest/availability to make a quick stab at this?

@jlstevens
Copy link
Contributor

I'm currently working on more advanced colormapping models but I am happy to come back to this when I get a chance...

@maximlt
Copy link
Contributor

maximlt commented Oct 25, 2022

As can be seen in the Bokeh 2.4.3 docs hovering over an RGBA image shows the 32-bit RGBA int value.
image

It seems like a better default would be to display the RGBA tuple, as asked on Discourse (we also get this request on hvPlot/HoloViews).

I'm writing first in this old issue to make sure that it should not be re-opened, I'm not entirely sure the PR that closed it effectively fulfilled the original request. Happy to open a new feature request otherwise, if it's a good idea!

@mattpap
Copy link
Contributor

mattpap commented Oct 25, 2022

I would say this is quite correct behavior, because that's your data i.e. you have a 2d array of uint32. Your interpretation is different, but you don't tell bokehjs that. A better way would be to use:

img = np.empty((N, N, 4), dtype=np.uint8)

and skip the view altogether, but that currently fails in bokehjs with expected a 2D array in ImageRGBAView, but this limitation should be easy to lift. From memory usage and encoding efficiency in the protocol perspetive, both arrays are equivalent, so I would just use the one that is more semantically aligned with the use case. Still, I'm not sure how tooltip would interpret the new data. Most likely it would require improvements as well.

Alternatively or perhaps orthogonally to that, we could provide tooltip value formatters for interpreting color-like values.

@bryevdv
Copy link
Member

bryevdv commented Oct 25, 2022

@maximlt the earlier PR closed this issue because at the time it was not possible to have any hover tool for image at all for the values of RGB images. Any new refinements on top of that should definitely get their own new issues at at this point, which I stated in the PR:

We talked about maybe adding some new auxiliary fields like $red to extract the individiual RGBA channels. That seems like a great idea but I will leave it up to others to do in other issues/PRs

I do agree with @mattpap that, for the sake of consistency, @value without any other context should just display that int, because the value is. The hover tool has no way to know more unless it is told somehow.

Alternatively or perhaps orthogonally to that, we could provide tooltip value formatters for interpreting color-like values.

Don't we already? Does$color[hex]:Value" (ref) not work for images? A 1-2 line CustomJSHover is certainly also an option here. I'm definitely not opposed to adding built-in formatters either, though.

@jbednar
Copy link
Contributor

jbednar commented Oct 25, 2022

I'm confused by the claims "that's your data i.e. you have a 2d array of uint32. Your interpretation is different, but you don't tell bokehjs that" and "@value without any other context should just display that int, because the value is. The hover tool has no way to know more unless it is told somehow."

Surely somewhere in BokehJS, it is known that this uint32 is in fact being interpreted as an RGBA. Sure, the hover tool won't necessarily know that, but I think that is the essence of this request: to make the hover tool aware of and respect information that is already available to BokehJS and respected elsewhere. That way the hover information will display what other parts of BokehJS already know is true, i.e. that this is in fact an RGBA value and not simply some generic unsigned integer.

@bryevdv
Copy link
Member

bryevdv commented Oct 25, 2022

Of course, nothing is ever impossible, but there's always considerations to weigh. Here is the scenario I explicitly imagined thinking about my reply: A CDS with two int columns: one for RGBA data, and another for detector counts. You want a hover over an image that shows both. So you want the counts be shown as ints of course, right? So it would not be sufficient simply have the hover tool look at the glyph type ImageRGBA and decide "Oh I should use a color format". No, the hover tool, which is already one of the most complciated and under-tested and hard to test corners of Bokeh will have to be deeply coupled to how glyphs and data sources are set up, and see is this field used by the relevant dataspec for the the glyph (a fact that might change over time). And of course, the format itself will add a fair amount of complexity: do you want RGBA tuples? 8bit ints, or floats? Ot RGB ints and float alpha? Or hex RGBA code? And if we auto-magic this with special code paths we also have to put in an option to turn it off, in case there's a time when it is not wanted.

I suppose I could have been more clear: there is no way I personally would sign up to do any of that. I think the alternative, i.e. the user has to be a tiny bit explicit with some kind of "I want to display this as a color" hint is not unreasonable, while at the same time, far more preferable, from a testing and maintenance point of view. I guess if someone else wants to try to add auto-magic here, we can consider it, but it would have to come with lots of tests.

Edit: and what about int32 columns of ints that you want to display as colors, that are not used by an ImageRGBA glyphs? After all that, you still would need an "explicit" formatting specifier to handle that scenario, no matter what. So let's add new/better formatting options for sure (but in a new issue)

@jbednar
Copy link
Contributor

jbednar commented Oct 25, 2022

I hear your tiredness, and respect it. :-) Still, I think the implication that "auto-magic" is required is an overstatement.

My argument here is that the current implementation is inappropriately leaking the implementation detail that colors are encoded as uint32s, making something visible to the user via the hover tool that should be hidden as an internal representation. There may be plenty of good reasons why the architecture is what it is and why this leakage occurs, but it is still a leakage, i.e. a failure in the abstraction that this value is a color.

Given the clear statement of intent from the user that these items are colors, when first defined, it's a matter of plumbing, not magic, to propagate that intent all the way down to the hover tool. Putting in that plumbing may be painful, and maybe some magic approach would be more practical, but magic surely isn't required here.

I think you are talking about that plumbing when you say "deep coupling", but to me, the whole point of the hover tool is to reveal the contents of what you are hovering, similar to a conversation we had about colorbars. In both cases I'm arguing for a strong coupling so that what the user put in, what the user sees, what the hover shows, and what the colorbar shows, are all the same thing. If the user gives us an array of colors, the hover should show the color. If the plot uses a certain colormapper, the colorbar should reflect that colormapper automatically. In all cases the user should have to work hard if they want to explicitly decouple the displayed values, the hovered values, and the associated legends and colorbars. That's a battle where the software should fight the user so that only the most dedicated user could decouple those values, because the entire point of all of these ancillary mechanisms (hover, colorbars, legends, etc.) is to reveal the data (which for an RGBA image are RGBA colors, not uints).

@bryevdv
Copy link
Member

bryevdv commented Oct 25, 2022

My original (offhand) proposal was to expose new special fields $red, $green, $blue and $alpha that users could use to extract color sub-components from any value. This approach:

  • would work for any color column, whether it is used for ImagRGBA or not
  • affords a simple mechanism for users to arrange the components exactly how they want (e.g in a tuple in one line? Or separate rows for every channel) without inventing some baroque color-formatting DSL
  • Is immediately understandable by anyone who already knows what RGBA channels are (i.e. everyone who would want to be able to)
  • works well with existing formatters: want hex values? No problem, used "printf" format with $red, done!

I still (personally) think this is the bst option that trades a minuscule amount of verbosity for 1) much more flexibility for users, 2) much less maintenance burden for developers. We can cerainly continue to debate/discuss that further but at this point I really must insist on a fresh issue.

@bokeh bokeh locked as resolved and limited conversation to collaborators Oct 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants