EyeDropper API

[ipopescu93]

HIQaH! QaH! TAG!

I'm requesting a TAG review of EyeDropper API.

The EyeDropper API enables developers to use a browser-supplied eyedropper in the construction of custom color pickers.

• Explainer: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/EyeDropper/explainer.md
• Security and Privacy self-review²: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/EyeDropper/security-privacy.md
• GitHub repo (if you prefer feedback filed there): https://github.com/MicrosoftEdge/MSEdgeExplainers/
• Primary contacts (and their relationship to the specification):
  • Ionel Popescu (@ipopescu93), Microsoft,
  • Bo Cupp (@BoCupp-Microsoft), Microsoft
• Organization/project driving the design: Microsoft
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status): https://chromestatus.com/features/6304275594477568

Further details:

• I have reviewed the TAG's API Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
• The group where standardization of this work is intended to be done ("unknown" if not known): Unknown
• Existing major pieces of multi-stakeholder review or discussion of this design: Unknown
• Major unresolved issues with or opposition to this design: None at the moment
• This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option):
🐛 open issues in our GitHub repo for each point of feedback

[LeaVerou]

First, there is a need for this in the developer community and I'm happy to see this work. However, I see a number of issues with the API in its current form.

Hex colors are in sRGB, which cannot represent all on-screen colors

The most serious issue I see with this proposal is that the API returns hex colors, which on the Web platform are in sRGB. What this means is that the CSS color #ff0000 is not the brightest red the screen can produce, it's sRGB red, which is typically far less saturated than device RGB red for most modern screens (e.g. for a P3 screen, like most modern Apple devices, it would be equivalent to device 91.8% red, 20% green, 13.9% blue). This proposal seems to suggest returning a hex encoding of device RGB coordinates. We are trying to avoid exposing device RGB on the Web platform these days, as there is no way to ensure consistent results. This means that, as currently defined, this API will return colors that are not actually the color picked in any Web platform technology, defeating the purpose of an eyedropper. Furthermore, there is not even a way to convert them to the color the user actually selected.

I'm not sure what is the best way to deal with this, since none of the currently widely supported color formats can express all on-screen colors. Ideally, the API should return one of the CSS supported device independent color formats, such as lab() or lch() but support is currently limited. One may suggest returning hex when the color is within sRGB and another format when it is not, akin to computed values. However, not only is this inconsistent and would cause developer pain when parsing the value, but the cases where on-screen colors fall outside sRGB are not exactly rare: in modern P3 screens, 33% of colors are outside sRGB and thus, cannot be specified with hex colors.

Eyedropper starting point

Given the way the API is currently defined, there is no way to specify the starting point for the eyedropper, which would be jarring for end users. Typically, in most UIs of that sort, the eye dropper starts from the location of the user interaction (where applicable), e.g. see on OSX:

picker-location.mp4

Event Design

It is unclear to me why eyeDropper.open() is asynchronous with the current design, since color selection happens via the colorselect event, and not via a promise. It does not appear to tick any of the criteria for an asynchronous operation (see TAG Design Principle 5.7).

However, stepping back from the details for a moment, it appears that a lot of the complexity here has to do with using an event-based model, which has been chosen to enable selection of multiple colors. However, I am missing a list of use cases that require such multiple color selection. In most cases of eyedropper UI in the wild that I have come across, the eyedropper typically closes after color selection, which also serves as useful feedback to the end user that their selection has been accepted. If the eyedropper did not close, it is difficult to communicate to the user that their selection has been recorded, or display what their selection was, not to mention that it would be externally inconsistent with most other eyedropper interfaces. If such use cases exist for multiple selection, one could always re-open the picker after selection of the first color, since there is arbitrary open/close functionality. The spec could ensure that calling open() right after the eyedropper closes is seamless. Prioritizing the single selection common case also means that developers don't need to explicitly call close() to close the picker every time (which I can see causing many confusing bugs, e.g. when an error is thrown before the call to close(), so the picker is left open).

Playing around with these ideas, the code sample listed in the explainer could look like this with a promise-based design and ability to specify start coordinates:

// Create an EyeDropper object
let eyeDropper = new EyeDropper();

// Enter eyedropper mode
let icon = document.getElementbyId("eyeDropperIcon")
icon.addEventListener('click', async e => {
    let selection = await eyeDropper.open({left: e.clientX, top: e.clientY});
    // if we are here, the picker has closed, either by selection, or via the user choosing to exit
    // we can call eyeDropper.open() to re-open it, or do work with selection

    if (selection.color !== null) { // a color was selected
        console.log(selection.color); // logs selected color

        if (selection.position !== null) { 
            // The selected pixel came from our own document (or a document of same origin)
            // The selected pixel is located at:
            console.log(`${selection.position.left}, ${selection.position.top}`)
        }
    }
});

Please note that this is just a quick sketch to make the feedback above more concrete, and not necessarily the way you should go!

Containing color selection

This proposal does not currently define a mechanism to allow developers to hide or show the eyedropper's pixel selection UI while remaining in eyedropoper mode, but a future version may allow that, for example, to facilitate clicking on application UI elements instead of selecting a color value.

Note that to replicate the UIs that do support multiple color selection with an eyedropper, merely preventing the eyedropper from closing or re-opening it does not suffice. These UIs typically exclude an area of the screen from selection, and this area contains some of the color-related UI and provides feedback for the color selection. Take a look at this capture from Adobe Photoshop:

picker-photoshop.mp4

Note that the actual color picker dialog does not participate in the eyedropping.

Furthermore, authors may want to contain the selection within their application, or even a part of their application (e.g. the Photoshop picker does not select colors outside of the Photoshop window).

If you want to replicate these interactions on the Web, it may be worth exploring ways to specify elements to exclude, and/or contain the selection within the viewport or specific elements.

---

Btw

This proposal does not currently define an object model for a color, though it seems like something that would be a good addition to the web platform.

There is some work on that.

[tomayac]

It is unclear to me why eyeDropper.open() is asynchronous with the current design, since color selection happens via the colorselect event, and not via a promise. It does not appear to tick any of the criteria for an asynchronous operation (see TAG Design Principle 5.7).

The Explainer mentions that UAs might implement a permission prompt.

[LeaVerou]

The Explainer mentions that UAs might implement a permission prompt.

Ah, I missed that. I'm not sure why a permissions prompt would be needed however. The user is in control the entire time, and the only color actually exposed to script is the selected color, if the user does not abort. This seems analogous to a file input to me (or the font picker we were discussing in breakout B this week), both of which do not require a permission prompt.

[dbaron]

Eyedropper starting point

Is it desirable for the API to specify the starting point, or should the starting point be chosen by the implementation?

It seems to me that for mouse-based interfaces, the starting point is probably the mouse pointer location since the eyedropper essentially is the mouse cursor -- though I wonder if this is still true if the eyedropper mode is initiated via a keyboard interaction.

That said, thinking about this made me curious how this works for non-mouse-based interactions, whether keyboard navigation on a desktop or whether a touchscreen.

[BoCupp-Microsoft]

Thanks for the great comments, @LeaVerou.

Regarding this point you made:

Hex colors are in sRGB, which cannot represent all on-screen colors

We looked for a better representation of color while writing the explainer and found some of the GitHub issues you mentioned. It seems like it might take a while to close on a proper definition of a color type that can serve all the web platform's needs. Our thinking was that we could decouple from that work by returning the same color value strings that the input[type=color] element returns today from its value property. When a better color type is available we can then add a new property to the ColorSelectEvent - similar to what I'm sure we'll do for input[type=color].

Let me know if you agree with that thinking.

@dbaron, regarding the EyeDropper starting point, as you suggest, it should begin at the cursor location. For keyboard accessibility the same should be true and the the arrow keys should move the mouse cursor. Actually at any time the user should be able to move the cursor location with the mouse or with the arrow keys and go back and forth. You can see this pattern in the Snip and Sketch tool for Windows and with the new Smart Copy feature recently released for the Edge browser.

@LeaVerou, regarding this comment on selecting multiple colors:

However, I am missing a list of use cases that require such multiple color selection

You can find an example in Photoshop. The eyedropper is just another tool in the tool pallete and it remains in eyedropper mode until you switch to a different tool. I find the multiple selection model useful when the app shows a preview of what you selected as you select it. When selecting from an anti-aliased region or wherever some color dithering has happened, the user may not prefer the first color they select and will select another nearby color until they get the desired result.

@LeaVerou, regarding your comments on containing color selection:

Note that to replicate the UIs that do support multiple color selection with an eyedropper, merely preventing the eyedropper from closing or re-opening it does not suffice. These UIs typically exclude an area of the screen from selection, and this area contains some of the color-related UI and provides feedback for the color selection.

You are absolutely right. We currently have this written in the explainer:

This proposal does not currently define a mechanism to allow developers to hide or show the eyedropper's pixel selection UI while remaining in eyedropper mode, but a future version may allow that, for example, to facilitate clicking on application UI elements instead of selecting a color value.

I have it in the non-goals section, but agree the multiple color selection experience would be lacking without some mechanism to tag portions of the UI as not color selectable. Since we don't have the mechanism for doing that sorted out yet we may split the proposal into two-levels - the first being a single select model that we can deliver while we figure out the API surface for excluding portions of the document to support the multi-select model.

Note I also have an issue already opened on this topic here with one thinly defined proposal. If you have any thoughts on how you think it should work feel free to jump in.

Thanks!

[LeaVerou]

@dbaron

It seems to me that for mouse-based interfaces, the starting point is probably the mouse pointer location since the eyedropper essentially is the mouse cursor though I wonder if this is still true if the eyedropper mode is initiated via a keyboard interaction.

Ah yeah, that makes perfect sense. Even if a position is still needed for accessibility, it would make an excellent default.

@BoCupp-Microsoft

We looked for a better representation of color while writing the explainer and found some of the GitHub issues you mentioned. It seems like it might take a while to close on a proper definition of a color type that can serve all the web platform's needs. Our thinking was that we could decouple from that work by returning the same color value strings that the input[type=color] element returns today from its value property. When a better color type is available we can then add a new property to the ColorSelectEvent - similar to what I'm sure we'll do for input[type=color].

Let me know if you agree with that thinking.

<input type=color> does not have to match any specific on-screen color, it can just display an sRGB picker and suffer from no such inconsistency. With an eyedropper API, it's quite important that the color the user picked from the screen is the actual color displayed in the web application. If you generate a hex color based on device RGB, which is then interpreted as sRGB on the Web platform, it's not only the colors outside the sRGB gamut that will be different, every color that the user picks will be different. The difference is already fairly noticeable on today's P3 screens (up to 7.1 ΔΕ2000 units) and will increase further as screen gamuts expand beyond P3 in the near future.

The discussion about color types is orthogonal, as you are currently returning a string anyway, and there are many other CSS color formats beyond hex.

As an aside, even for <input type=color> there is discussion about hex values being insufficient. Regardless, as I explain above, it's not as much of a problem there because the picker UI is constrained, and not the entire screen gamut. If you are able to pick any displayable on-screen color, you need to be able to represent any displayable on-screen color.

[BoCupp-Microsoft]

@LeaVerou regarding this comment:

<input type=color> does not have to match any specific on-screen color, it can just display an sRGB picker and suffer from no such inconsistency.

Depending on the implementation, <input type=color> does have the ability to use an eyedropper to pick the color that will be turned into an RGB hex string. Here's a screencast showing that in Chromium. Isn't that the same issue?

[LeaVerou]

@LeaVerou regarding this comment:

<input type=color> does not have to match any specific on-screen color, it can just display an sRGB picker and suffer from no such inconsistency.

Depending on the implementation, <input type=color> does have the ability to use an eyedropper to pick the color that will be turned into an RGB hex string. Here's a screencast showing that in Chromium. Isn't that the same issue?

The UI for <input type=color> is largely left up to the implementation. Since no spec defines this UI, the fact that an eyedropper appears in (an unreleased version of) one browser should not set a precedent about what makes it into the Web platform.

To answer your question, yes, this suffers from the same issues. I've tried to illustrate with a video below.

chrome-picker.mp4

The gradient on the right is a gradient from P3 red to gray (you only see part of it). At first, I'm using OSX's native color meter to show you device RGB coordinates. As you can see, these are all different colors and the coordinates smoothly change as I move my pointer downwards. Then, I use the Chrome picker to consecutively pick colors lower and lower down. As you can see, they all yield #ff0000 and I need to select several until I start getting a different hex value, because apparently Chrome seems to convert the P3 coordinates to sRGB and then just clips them to the [0, 255] range. In practice, these means that the most saturated 33% of on-screen colors would be squashed onto the same hex value.

[inexorabletash]

Does the TAG have concrete suggestions for how the input element could be extended to support device independent colors in a backwards compatible way? Maybe that could be used as model here.

E.g. the <input type=color>'s value property would need to continue to be sRGB hex to avoid breaking sites, but maybe there could be a color property added which is only not undefined for color inputs, and is a CSSOM type (?) that provides forward-compatibility to access/translate colors between color spaces in a defined way. I don't think such a thing exists yet, though.

In that case, this API could follow that model - value on the event continues to be an sRGB hex, but once the device-independent, forward-compatible color type is added to the platform it could be added to the event as color.

This is a half-baked idea; I'm hoping the TAG can suggest something more practicable.

[BoCupp-Microsoft]

@LeaVerou re: this comment:

the fact that an eyedropper appears in (an unreleased version of) one browser should not set a precedent about what makes it into the Web platform.

I wasn't trying to be deceptive... just showing some new UX one of my devs recently created. You can see another screencast here I took showing Firefox, an older Chrome, and Safari all using the Mac's color sample to take a color from the screen and then expose it through the value property of input[type=color].

[LeaVerou]

We discussed this today in our VF2F and have the following feedback.

a) The biggest issue is color output. There are four ways to address this, none ideal:

1. The Web Platform is currently in a transitional state wrt color. Possibly the best solution is to wait until there are more implementations of non gamut restricted color formats and/or a Color object.
2. You could clip or gamut map the selected color to sRGB and return an sRGB hex value (or other sRGB color format). The problem is that color selection will not roundtrip for at least 1/3 of onscreen colors in today's displays (and as display technology improves, this will increase). Given that many eyedropper use cases need precision, we do not think this is a good solution. For example, a user could have a Photoshop window side-by-side with a web application and use the eyedropper to select a color from the mockup to apply to their document in the web application. The color should be identical to serve this use case. We also think returning strings is suboptimal.
3. You could return an lab() or lch() string, which can specify any visible color. This does roundtrip, though is also a string. Another problem is that implementations of these formats are currently limited.
4. You could build a special-purpose color API, with srgb and hex properties. We do not think this is ideal, as it duplicates the Color API that the Web Platform will eventually get, and it's still unclear what the primary returned value should be.

b) As long as the eyedropper cannot be used to scrub the screen, and only communicates the user's final selection back to the web application, this does not need a permissions prompt, just like the file input does not need a selection prompt, so open() does not need to be asynchronous (unless you move to a promise-based design, see below).

c) The API is designed around using a PointerEvent but it is unclear what the use cases are for reading coordinates, especially since their presence cannot be guaranteed and selection cannot be constrained to only areas where coordinates can be returned.

d) The API is designed around multiple color selection as the primary use case, requiring use of events and explicit close(). However, as pointed out by the authors, multiple color selection is primarily intended for fine tuning, but the UA can take care of the fine tuning by showing appropriate UI and only communicating the final color selection back to the app, which is also more privacy preserving. This would allow a promise based design and prevent bugs where the eyedropper mode is left open because something (e.g. an exception) intercepted the call to .close().

[plinss]

I'm becoming more convinced that AbortError in the event of the user canceling the operation is the wrong response (and the File System Access API is wrong too).

Even given the case if it should be agreed that user cancellation is an exceptional situation (which I do not agree with), a different error should be thrown. I believe AbortError should be exclusively used in response to an AbortSignal.

I'm also beginning to strongly feel that the open() method should accept an AbortSignal via an options dictionary, e.g. open({signal: signal}). This allows not only a clear mechanism for the caller to abort the operation, but for surrounding code to pass in an AbortSignal of its own.

[domenic]

a different error should be thrown. I believe AbortError should be exclusively used in response to an AbortSignal.

I strongly disagree with that. We chose AbortError to be used with AbortSignal because of its long precedent of being used for any cancelation.

[plinss]

We chose AbortError to be used with AbortSignal because of its long precedent of being used for any cancelation.

Fair enough, I wasn't considering other (systemic) reasons a process may be aborted.

My bottom line is that I'm looking for a general principle to inform these decisions, and I think there's some subtlety that should be captured here between an operation getting aborted for 'reasons' vs the user explicitly opting out of a choice.

Aside from the File Access API, do you have other examples where AbortError is thrown in response to user cancellation of an operation?

[domenic]

Sure:

• When the user presses the browser stop button to stop loading, media elements reject pending play() promises with AbortError
• When the user cancels a share operation with web share, the promise rejects with an AbortError.
• When the user aborts a payment request dialog, the promise rejects with an AbortError.

This was the result of a quick Google search, but I suspect there are more.

[ipopescu93]

We discussed this again in our plenary yesterday and while we thought this was an improvement over the previous iteration, we did have a few thoughts and questions.

Thanks for your feedback!

While it's good that there is some guidance for UAs on when to exit eyedropper mode without a selection (e.g. pressing ESC), it might be good to also provide a way for the application to explicitly close the eyedropper, e.g. a close() method or an AbortSignal parameter. Since no UI events are dispatched this would not be useful for closing the eyedropper through user interaction (e.g. via additional keyboard shortcuts), but it could be useful for closing the eyedropper (with no selection) in case another high-priority event occurs in the application, e.g. a modal dialog.

We haven't seen evidence that this is a common scenario, but I don't see any harm in providing an optional AbortSignal parameter to facilitate such a scenario.

Some of us were surprised by the promise rejection when no selection is made (vs resolving with null), since this is ordinary use and not an unusual bad state. Do you consider exiting without a color selection an error condition? Did you choose this design to be consistent with another API?

Since selecting a color is a user initiated action, I think exiting without a color selection should be treated as an exceptional case.
This is also consistent with how other APIs behave when the user aborts the operation by dismissing the UI.

While we think the promise-based design is an improvement, we did notice that it does not address your original use case of picking colors from a drawing application. Only the final composited color will be picked with no means of tracing it back to the top-most color that the user may have intended to pick since no UI events are fired throughout the selection. What are your thoughts on this? Is there another way to accomplish this with the new design or did you decide that this is a non-goal for now?

We would have to include the position of the selected color to facilitate this scenario and we have decided that this is a non-goal for now.

Relevant, note that there is now an effort to standardize a Color API for the Web Platform by a subset of the CSS WG per recent CSS WG resolution: https://github.com/WICG/color-api and serving this API is listed as an explicit goal.

I think that a Color API would be a great addition to the Web Platform and we expect that a color object will be the primary mechanism by which authors will access the selected color in the future.

[LeaVerou]

Hi @ipopescu93!

@cynthia and I looked at this during a breakout in our Gethen VF2F today. We are happy with the direction it's going and we are going to go ahead and close it.

Thanks for flying TAG!

[LeaVerou]

Hi @ipopescu93,

I noticed that the EyeDropper API was shipped in Chrome and decided to take it for a spin. I see that it is returning a hex with device RGB coordinates, despite it being labelled sRGBHex. For example, in the linked codepen, even though I'm sampling sRGB #00ff00, the color I get back is #02fe00, which is the pure sRGB green, converted to the RGB space of my (wide gamut) screen. This means that no picked color can roundtrip correctly. Is this intentional or a bug I should report?

[ipopescu93]

Hi @LeaVerou,

This looks like a bug. Could you please file a bug on http://crbug.com/ and assign it to me (iopopesc@microsoft.com)?

[LeaVerou]

Hi @ipopescu93, just filed https://bugs.chromium.org/p/chromium/issues/detail?id=1286862 but I cannot assign it to you.

[svgeesus]

I came across a Chrome extension using the EyeDropper API by @captainbrosset and tried it out. I immediately noticed that on my Wide Color Gamut screen, the results were being measured in device coordinates, but reported back as sRGB hex thus giving the wrong color.

For example, displaying a patch of #009900 the eyedropper reported #56981C which is clearly incorrect. However, using a browser that does not do color management on CSS (Firefox, which simply throws whatever RGB data at the screen) displaying #56981C gives the exact same color, visually, as displaying #009900 does in Chrome, Edge etc which do handle sRGB values in CSS correctly.

[svgeesus]

So basically the EyeDropper API is pulling raw values out of the framebuffer and then pretending those are already in sRGB