What do you want in a better clipboard API for Windows?

[metathinker]

One part of Windows we at Microsoft (myself, @ShawnSteele, and some others) have been thinking of evolving as part of Project Reunion is the API for the Windows clipboard.

We've already seen a great idea for a clipboard improvement in issue #62 - allowing apps that were previously prohibited from accessing the clipboard in the background to do so when responding to a toast notification. As we think about whether and how to decouple the clipboard API and implementation into the eventual Project Reunion codebase, we'd like to hear some more ideas.

So, if you have pain points or improvement ideas for the clipboard that aren't quite fleshed out enough for an issue of their own, feel free to post them here! We'll try to keep up and listen, and may spin off issues for ideas as they get refined.

Related Links

#62 #67

[hansmbakker]

Synchronization of the clipboard between phone and windows devices would be nice!

[metathinker]

Synchronization of the clipboard between phone and windows devices would be nice!

We actually have that now! It only works with Samsung Android phones for now, though.

If this feature were available to more phones and/or more Windows devices, how might you imagine programming it?

[JanRajnoha]

Available for all Android phones via SwiftKey keyboard, but only in text mode

[metathinker]

Clipboard APIs today (wall of text follows)

For your reference, here's the current state of the Windows clipboard API from a very high level. As we look to craft a potential Project Reunion API for clipboard, we'd want to figure out both how to help you avoid traps that the current API presents and how to give you new features you could use to make your users happier.

There are 3 basic API surfaces for interacting with the Windows clipboard:

• The classic clipboard API, including GetClipboardData() and friends in windows.h.

  • This API thinks in terms of buffers of serialized data.
  • It allows source apps for clipboard data to defer rendering (serializing) data for certain data formats they claim to have copied.
  • Before clipboard source apps cleanly exit, Windows will force them to render delay-rendered data formats. (Specifically, this happens before the window is destroyed.)

• The OLE data transfer API, including OleGetClipboard() and friends in ole2.h.

  • This API thinks in terms of "IDataObject" objects. The source app for clipboard data must provide both the code of an IDataObject implementation and the actual data accessed through that object.
  • Source apps using this API always defer rendering data formats by default.
  • Windows doesn't force source apps to render all data formats before they exit. If the source app doesn't force itself to render all formats by "flushing" the clipboard, then unrendered data formats could be lost when it exits.

• Windows.ApplicationModel.DataTransfer.Clipboard and friends.

  • This API thinks in terms of "DataPackage" objects. Windows provides the DataPackage implementation; the source app only needs to bring the data.
  • Source apps can choose to defer rendering data on a per-format basis, as with the classic clipboard API.
  • "Flushing" the clipboard is a separate concept from deferred rendering, unlike with OLE data transfer. If the source app doesn't "flush" the clipboard, then after it exits, data for both rendered and unrendered formats could be lost.

All 3 APIs are equivalent in basic clipboard functions, but Windows.ApplicationModel.DataTransfer also recently added programmatic access to the clipboard history and cloud-roamed clipboard data, as part of the addition of those features in the November 2018 update to Windows (Windows 10 version 1809).

All 3 APIs have their own advantages and pitfalls - as alluded to above, both the OLE data transfer API and Windows.ApplicationModel.DataTransfer potentially make it too easy for copied data to disappear from the clipboard when source apps are closed, frustrating users when they try to paste text from their web browser or email program and getting nothing instead, or a previously copied item they no longer want.

[metathinker]

As a general rule, APIs in the Windows App SDK are named starting with Microsoft.<whatever> as opposed to Windows.<whatever>, which is specific to the Universal Windows Platform API surface.

However, as far as I know, using WinAppSDK to build your Windows desktop app does not stop you using any part of the UWP API, including UWP clipboard (W.AM.DT.CB). Any problem caused by a "this UWP API is only for UWP apps" restriction is not caused by the use of WinAppSDK, but by the fact that you're building a desktop app and not a UWP app. (Where this claim is most likely to be wrong is around the MSIX packaging and dynamic dependencies sections of the WinAppSDK.)

[learntocode66]

Any problem caused by a "this UWP API is only for UWP apps" restriction is not caused by the use of WinAppSDK, but by the fact that you're building a desktop app and not a UWP app.

If in a UWP API's doc does not have the statement "this UWP API is only for UWP apps", then the API can be used for building non-UWP desktop app. Is this correct?

[DarranRowe]

In general, Windows Runtime APIs not supported in desktop applications is the go to documentation here.
This does give information on what runtime classes are just unavailable, runtime classes that needs some form of interop and any other requirements. The way to infer what it means is if it isn't listed then it is supported in a desktop application.
This documentation also gives a list of runtime classes that requires a package identity. But this is something that is much less accurate, the fact that it lists EventHandler<T> shows how problematic this list is.
Another example of how problematic this list of runtime classes that require a package identity is comes from the inclusion of StorageFile in this list. The reason why this is problematic is that the pickers are not part of this list. If you use the FileOpenPicker to pick a file, then it gives you a StorageFile. This means that if you read this documentation as is, the pickers aren't usable in unpackaged applications due to them returning objects that are only supported in packaged applications. Which is daft.
This requirement comes from some members do need the package identity, like StorageFile.GetFromApplicationUriAsync. This pretty much means that you have to be careful using this documentation.

[metathinker]

(Also see the previous post. That poster and I crossed paths.)

Not always. Sometimes the docs on learn.microsoft.com forget to mark a truly UWP-app-specific API as such. (Or they forget to say that a UWP API class needs to be QueryInterfaced to IInitializeWithWindow, I<ClassName>Interop, or similar to be used with desktop apps.)

UWP Clipboard is not UWP-app-specific, however.

[learntocode66]

@DarranRowe @metathinker Thank you both for your very clear explanation!

[memexy]

This discussion was posted on hacker news. I posted a comment there but I'm posting it here as well to make sure you folks see it.

Automatic saving to a specified location and OCR. The current clipboard doesn't support automatic saving so I have to use another program to do that.

[mdtauk]

I just want my copy and paste actions to work every time, without fail, and without complication

[metathinker]

I just want my copy and paste actions to work every time, without fail, and without complication

In our observation at Microsoft, a lot of problems with clipboard reliability are caused by apps that use the clipboard APIs in the "wrong" way. For example: - The classic clipboard API allows any app that is reading or writing clipboard data to accidentally lock out all other apps from accessing the clipboard - to do that, you can just forget to call CloseClipboard() when you're done. The most obvious symptom of this is a Paste operation pasting an older image or piece of text, as the most recent cut or copy attempt had silently failed due to the clipboard lockout. - As I noted above, the OLE data transfer API and Windows.ApplicationModel.DataTransfer require an explicit "flush clipboard" step. If a source app for copied clipboard data forgets to flush its written data values, they could be lost when the app exits, leading to no data being pasted on paste attempts. To get to 100% clipboard reliability, we'll need to figure out how to fix our mistakes that have led people down the wrong paths. That's part of the reason I started this discussion - we'd like to learn how to make it easy for people to do the right thing and how to make it hard or impossible for them to do the wrong thing.

[caroysMSFT]

I just came across this thread. I've been wondering how I can fix this problem for myself, because it jams me up almost daily where my clipboard does not update, and I paste the wrong stuff. I have to slow down in situations where I am trying to move as fast as possible.

I have seen many situations where time is wasted when the wrong information was pasted, and it sends people down the wrong rabbit hole. I see it all the time.

This idea of cloud clipboard makes my hair stand on end. A thing which could the clipboard more unreliable is inserting calls out to the internet in between lock/unlock. Turn off anything by default which carries a risk of compromising reliability. I would make the case that what windows users need is 100% reliability above all else. Suggestions, clipboard history and cross-device syncing is nice, but I'm not really aware of many people using these features at any scale or frequency. Just about everyone I talk to has stories of their clipboard not updating, and being annoyed by it on at least a weekly basis (sometimes daily).

It's worth noting X11 on Linux and OSX have their own clipboard challenges (even some anecdotal evidence of a similar reliability issue due to locking on OSX).

[hansmbakker]

I don't know about the feasibility / effort required of different options, but would it be possible to have a Roslyn Analyzer for that to point at mistakes in the use of these APIs (as a short term helper)?

I can imagine that changing the APIs themselves could be a longer-term project.

[metathinker]

would it be possible to have a Roslyn Analyzer for that to point at mistakes in the use of these APIs (as a short term helper)?

That's an interesting idea. From my understanding, Roslyn analyzers in particular only work for C# and VB, but other .NET managed code languages, as well as native code, could benefit from similar static analysis. I don't think we at Microsoft have really used static code analysis in Windows native code development for cases other than "this could be a security vulnerability", but perhaps we should consider broadening our view...

[sylveon]

I really want an API that allows classic & UWP app to set data it sends to the clipboard as "private." This will be useful for password manager apps, where I copy the password in the clipboard and then paste it into a web browser or other app. Currently, this goes right into the clipboard history and is then promptly synced on other devices.

I do not want my passwords in the clipboard history for the simple reason that somebody with access to the computer could press Win-V and view passwords I used in the past.
I also do not want my passwords to be synced to any remote server, (Microsoft or not) for obvious privacy and data security principles.

[sylveon]

Not sure how I managed to do that just by posting a comment, but ok

[metathinker]

I really want an API that allows classic & UWP app to set data it sends to the clipboard as "private." This will be useful for password manager apps, where I copy the password in the clipboard and then paste it into a web browser or other app. Currently, this goes right into the clipboard history and is then promptly synced on other devices.

I do not want my passwords in the clipboard history for the simple reason that somebody with access to the computer could press Win-V and view passwords I used in the past.
I also do not want my passwords to be synced to any remote server, (Microsoft or not) for obvious privacy and data security principles.

There are already 2 ways to prevent newly set clipboard data values from being synchronized by cloud clipboard or entered into the local clipboard history:

1. If you use Windows.ApplicationModel.DataTransfer, call Clipboard.SetContentWithOptions() and pass a ClipboardContentOptions record with IsRoamable and IsAllowedInHistory both set to false.
2. If you use the classic clipboard API or OLE data transfer, set the custom clipboard format ExcludeClipboardContentFromMonitorProcessing to any data value.

Are neither of those options working for you?

[sylveon]

I wasn't aware of those APIs, I just based off my assumption on the behavior of my current password manager. I'll go bother them with this then 😅

[MuleaneEve]

For privacy reasons, I think that every (approved) access to the clipboard should trigger the appearance of an icon in the taskbar (just like with geolocation access).

An extra step that I would also like to see for all these sensitive services is that there should be a way to see a history of these accesses after the fact. That way, we can audit apps to confirm that they did not abusing these services while we were distracted.

[Poopooracoocoo]

Synchronization of the clipboard between phone and windows devices would be nice!

We actually have that now! It only works with Samsung Android phones for now, though.

If this feature were available to more phones and/or more Windows devices, how might you imagine programming it?

There's also Pushbullet (off topic but it's worth mentioning alternatives to Your Phone, which is quite limited in the devices it supports)

[ElanHasson]

Are there docs for the public api?

see sample use-case: https://twitter.com/ElanHasson/status/1445784713851899912

[metathinker]

As far as I know, there is no public Windows API or documented web service API endpoint to place data items on a user's Microsoft Account or Azure AD-synced cloud clipboard. (I don't work on the clipboard anymore, so I'm just writing as an interested citizen, so to speak.)

[nickrandolph]

Has there been any further design behind what the WindowsAppSdk clipboard api would look like? Is there a draft spec we can review?

[metathinker]

I wrote a spec for a proposed v1 of the Windows App SDK clipboard API here: #119
However, there was no action taken on it before I stopped working on the clipboard about a year ago.

If you feel a better clipboard API would meet a crucial need, be sure to politely let @ShawnSteele, @andrewleader et al. know!

[dotMorten]

One of the worst things about the clipboard in UWP apps that I really hope won't come into WinUI was if you for instance had a TextBox, copying that text was only persisted while the app was running.
So often you'd launch a UWP app, find something you needed, copy the text, shut the app down and go back to where you need to paste that text, only to realize that there was nothing to paste.

There was code you had to explicitly write for every single textbox to force it to persist beyond the app session. The result was that since most developer had no idea they had to do this (because why should they have to do that for a basic feature), that copy/paste in most UWP apps seemed "broken" and unstable.

[metathinker]

I discussed this problem earlier in this thread (#374 (comment)), and fixing it was an explicit goal of my proposal for the Project Reunion SDK clipboard API: https://github.com/microsoft/WindowsAppSDK/pull/119/files#diff-dea4800272460d5fb2f1a3f21485e87774f43b694db1644af2bcfaba437346ceR53-R59

Note that if we did go this route, apps that delay rendering (serializing) some or all formats of the clipboard data item will use more time when suspending or exiting because they now have to serialize those delayed formats.

[DineshSolanki]

writing multiple items to clipboard history, currently, I have to use a delay so as to appear on history.

foreach(var item in splitOptions)
                {
                    var dataPackage = new DataPackage();
                    dataPackage.SetText(item);
                    Clipboard.SetContent(dataPackage);
                    await Task.Delay(250);
                }

[guyyst]

Yeah this is getting on my nerves as well at the moment.

A way either provide a list of data packages that will all be added to the history, or some kind of callback for when the history is ready for the next item would be nice.

[mentalisttraceur]

First, and most importantly, you should enshrine as standard terminology in docs and API that a clipboard entry is called a "pasticle". This will make everything better and will be widely considered as a great move.

Second, as a user who wants to copy and paste things, I am deeply disturbed by the idea that apps are allowed - at all - to delay clipboard rendering or flushing. This allows apps to write IOUs that they can't guarantee they can cash, or to mutate data between when I've "copied" it and when I paste it, which means it allows apps to lie to me when I copy. Over the years I've built up a habit of hitting Ctrl+C multiple times in a row whenever I need to copy something - I don't know if it makes things more reliable, but it was a response to utterly opaque unreliability. If a pasticle is a deferred pasticle, I at a minimum want some UI indicator when my pasticle is deferred/unflushed, and a way to force a real copy now.

(As a developer, I recognize that there are performance benefits to skipping a copy into the OS's clipboard if it's just going to be copied from clipboard into another user space app, but I would welcome a solution based on a copy-on-write memory mapping: instead of having deferred/unflushed copies, let me tell the OS clipboard an address and size, and then have the OS map the relevant pages with copy-on-write into the clipboard implementation - that way even if the app crashes or mutates the same data after a copy, the OS clipboard has a real copy. I think the rare cases where it is good and user-acceptable for clips to mutate after being clipped are not worth the complication and error potential it would add to the API.)

Third, as a user and developer mindful of security, I want clipboard domains/tiers: I want to be able to say "this app is not allowed to see pasticles which were copied from that app", "no app is allowed to see pasticles from this app unless I explicitly give permission", or "this app is not allowed to see any pasticles from any app".

[sylveon]

First, and most importantly, you should enshrine as standard terminology in docs and API what a clipboard entry is called: pick either "clip" or "pasticle". This will make everything better and will be widely considered as a great move.

There is no such thing as a clipboard entry in the current documentation and API term. The clipboard has content (what's currently in the clipboard) and history items (what used to be in the clipboard).

Clip is not a suitable word because the existing definition of this word implies something was removed (clipped), but copying to the clipboard is not destructive.

[mentalisttraceur]

The clipboard has content (what's currently in the clipboard) and history items (what used to be in the clipboard).

Pasticles.

Clip is not a suitable word because the existing definition of this word implies something was removed (clipped), but copying to the clipboard is not destructive.

Yeah I added "clip" as an alternative in a last second edit to my message, and I don't stand by it.