Display an input callout when tapping buttons

[wearhere]

Where I mean that, when the user taps a key, they see a "key cap" view appear above their finger:

I found a project that mimics the look and behavior of Apple's keyboard quite well https://github.com/illyabusigin/CYRKeyboardButton, the above gif is from there. It's not built for use in keyboard extensions per se but rather input accessory views (hence the buttons' ability to draw above the keyboard), but still might be adapted for use in this project, I think. The key cap view (CYRKeyboardButtonView) is even separate from the button so that might be perfect since this project already has button views.

[wearhere]

I removed the bit about drawing above the keyboard using a transparent top row since revisiting Stack Overflow didn't surface any ways to make the keyboard background transparent. Our "keycap views" will probably have to draw sideways or downwards if there isn't space above the keyboard.

(That said, a lot of keyboards will probably have some sort of toolbar on top and that would provide space for top-row keycap views to draw upward.)

[danielsaidi]

Hi! I'm finally done with the major time consuming stuff I had until the end of last week 🎉

This looks great! However, before we can implement this, we need to consider how to represent this for each action, and how to bind it to the buttons. For instance, these could perhaps be considered to be "context actions" or "callout actions", while we must also find a way to represent "secondary action"s like the ones found on iPad, when swiping down.

[wearhere]

Hey Daniel! Hope your talk went well.

I have a prototype of this feature that ought to work automatically for KeyboardButtonViews whose actions are .character or .emoji, I think (since all of those should show these sorts of keycap views, with the same character as is shown on the regular button, and with the same handling once the key is released). Will try to put it up tomorrow!

I also have work in-progress on #93. I think the new action types you’re mentioning could help there for sure.

[danielsaidi]

Cool, sounds great! I'll try to come up with a nice adaptive way to model secondary and call out actions for each action type.

[wearhere]

Some notes / thoughts on secondary and callout actions:

• I’ve come up with a way to dynamically generate all skin tone permutations of any given emoji. So their secondary actions can be automatically derived by the framework.
• Apple has three presentations of secondary actions. The first two I’ll call “no primary action”

and “primary action” (the leftmost emoji stands apart from the other secondary actions)

The framework can automatically derive the primary action too (reverse of the approach for generating the permutations), even if the button’s “regular action” is a skin tone-modified emoji (note how Apple changes the default action shown in their keyboard after you pick a skin tone the first time).

The third presentation of secondary actions is a real doozy:

That may be better than showing 5^2 images, but what a pain of custom UI and also it’s not going to fit in a keyboard extension. Thinking of ditching the keycap presentation for that one and bringing up some short of modal sheet.

My emoji-permutation technique can automatically split out the halves of the emoji, at least.

• I haven’t given much thought to callout actions (as might be used for a settings / next-keyboard menu). Maybe they could be an extension of your other project for creating menus? 😁 That has a nice model.

[wearhere]

Update: I'm waiting to PR my popups until I've completed all of the presentation styles I mention above, to make sure the architecture will handle everything.

As an example, I decided that it was necessary to ensure that the popups would be able to display upward, so that the user would not cover the popups with their finger, so I have devised a mechanism to dynamically add and remove extra vertical space:

[danielsaidi]

OMG that looks amazing @wearhere !

[wearhere]

@danielsaidi Aw thanks! The styling‘s not even complete in that shot 😄

[danielsaidi]

I think we should aim at implementing this in 3.1 or 3.2.

[danielsaidi]

@wearhere I'm working on 3.1 and to add autocomplete support to SwiftUI. After that, I'd love to take a look at your code for displaying that callout.

[reeichert]

Hi guys, any update on this feature?

[danielsaidi]

Hi! I haven't looked at it anymore, but I will be needing it for a personal project as well. That will be in SwiftUI, though.

[wearhere]

Hey @danielsaidi, it might make sense to pursue a new implementation of this. I'm still happy to share the code I have written for my callouts, but it's pretty intricate and (more importantly) not using SwiftUI, so I'm not sure it'd make sense to try to merge into the project per se.

I'm still hoping to migrate my own keyboard to SwiftUI in which case I'd need (and might be interested in contributing to) a SwiftUI implementation, but can't embark on that until there's a way to resize the keyboard—as you can see in the GIF earlier in this thread, I built a system to resize the keyboard as necessary to accommodate the callouts when they would otherwise go off the top of the keyboard—but I see a note here

IMPORTANT BUG INFORMATION Note that SwiftUI-based keyboard extensions currently doesn't resize properly when the provided view changes its height after being added to the controller.

[reeichert]

It's ok if you want make SwifUI-only, but since in my project I need to support iOS 11+, UIKit is the core for us.

Do you mind to share your implementation @wearhere ?

[danielsaidi]

I'd love to take a new look at your implementation @wearhere and at least see how you created that nice bubble effect! However, I do agree with you that it's probably based on app-specific tweaks that make it hard to generalize for a library?

I'd have to sit down and take a look at the SwiftUI approach's inability to resize. I actually do resize in an app of mine, but it's a pretty app-specific solution.

[Jinil-JP]

Hello @danielsaidi Do you find a way to add this bubble effect for accent in this project? If you found a way then it will be appreciable if not then we will found a way for that I want to implement this feature for my application also.

[danielsaidi]

Hi @Jinil-JP - I'm actually sitting with it right now. I have a working approach in a demo application and will begin adding it to the main project now. However, there is still some work left to make it fit into the rest of the design, so it may take a while before it's available. I have a model that works for both UIKit and SwiftUI as well as a bubble that works in SwiftUI.

[danielsaidi]

I have just implemented a callout bubble for secondary inputs, so I will reuse that logic to implement the typing callout as well.

[danielsaidi]

I have just released KeyboardKit 3.4, which has SwiftUI support for this.

I would love some extra eyes in the result and would love to improve the overall look and feel over time.