Implement gesture/glide/swipe typing

[williamtheaker]

I didn't see an issue for this on the 0.4.0 milestone board and want a way to follow progress on this feature since the current free/open source implementations are limited in number and accuracy. This pull request on the AnySoftKeyboard repo looks to improve gesture typing with machine learning and might be useful for reference.

[patrickgold]

I hope to be able to properly implement glide typing for v0.4.0, how exactly the implementation works I don't know yet. Thanks for referencing the PR on AnySoftKeyboard, I will look into it.

[r01k]

Just realized the [NYI] in the app stands for Not Yet Implemented. Glide typing is one of the options labeled NYI. This is the feature that I'm waiting for to switch. Other than that, the app looks and behaves great.

[akshat-ja]

Really looking forward to the glide support. I think a lot of people will switch over to FlorisBoard once this is implemented.

[raphaelh]

Waiting for this too 😄 Thanks!

[X-yl]

Hi @patrickgold, have you started working on this yet? If you don't mind, I'd like to tackle this. Although it is a fairly key feature, so I understand if you might want to do it yourself.

[patrickgold]

@X-yl if you want to, you can! This would definitely speed up the process of completing the v0.4.0 milestone, as I am currently focused on the suggestions and dictionary stuff + general bug fixing etc.

There are a few thoughts I already had on glide typing that I want to share with you:

• My idea was that the glide typing line doesn't use an overlay UI element (like the popups or the clipboard manager do), but that the line of which the pointer traveled on the keyboard layout is drawn directly on top of the text input manager layout. This would require to make a custom view out of the LinearLayout root in text_input_layout.xml and implementing the touch handling and drawing in there.
• I also planned to correct the estimated word with the help of a dictionary (if the detected word does exist in the dictionary then use it, else try to find the best match for the current locale). The problem is that the dictionary component is currently missing (there's the Flictionary but this API will completely change soon because of the suggestions backend rework). So it would be best to build the input logic so that it can correct detected gesture words with an dictionary, but for now it just skips this step.
• All gesture related preferences should go to Settings > Gestures > Glide typing group. There's already a NYI enabled stub preference for glide typing, which you can use. Glide typing preferences reside under the glide namespace.

I very much appreciate your contribution! If you have any questions during development, feel free to ask, I am always here to help!

[X-yl]

My idea was that the glide typing line doesn't use an overlay UI element (like the popups or the clipboard manager do), but that the line of which the pointer traveled on the keyboard layout is drawn directly on top of the text input manager layout. This would require to make a custom view out of the LinearLayout root in text_input_layout.xml and implementing the touch handling and drawing in there.

I had the same thought, however I was planning to put it inside text.keyboard.KeyboardView, I think that would make more sense since if I were to subclass the LinearLayout root intext_input_layout.xml, glide typing would work in the smartbar, clipboard history, emoji selector, etc which doesn't really make any sense.

I also planned to correct the estimated word with the help of a dictionary (if the detected word does exist in the dictionary then use it, else try to find the best match for the current locale). The problem is that the dictionary component is currently missing (there's the Flictionary but this API will completely change soon because of the suggestions backend rework). So it would be best to build the input logic so that it can correct detected gesture words with an dictionary, but for now it just skips this step.

From what I understand, supporting dictionaries should be as simple as adding the words from the dictionary into the internal list used for gesture matching, so it should be no trouble supporting it. I think also it's a good idea to have an abstraction around the gesture classifier, in case someone comes along with a better algorithm in the future. (For now, I'll just use the same thing mentioned in that AnySoftKeyboard PR)

All gesture related preferences should go to Settings > Gestures > Glide typing group. There's already a NYI enabled stub preference for glide typing, which you can use. Glide typing preferences reside under the glide namespace.

That's great. Just wanted to check though, is 'glide' definitely the right word? Because I'm not sure if it's the actual generic term or just Google's word for it (given it starts with g). Definitely don't want to get into trouble because google trademarked 'glide typing' or something.

Not a question, but I happened across this function:

florisboard/app/src/main/java/dev/patrickgold/florisboard/ime/text/gestures/SwipeGesture.kt

Lines 190 to 201 in 670e6ca

	private fun angle(diffX: Double, diffY: Double): Double {
	val tmpAngle = abs(360 * atan(diffY / diffX) / (2 * PI))
	return if (diffX < 0 && diffY >= 0) {
	180.0f - tmpAngle
	} else if (diffX < 0 && diffY < 0) {
	180.0f + tmpAngle
	} else if (diffX >= 0 && diffY < 0) {
	360.0f - tmpAngle
	} else {
	tmpAngle
	}
	}

And I'm forced to conclude that you have not heard of Math.atan2, which you have basically reimplemented here lol.

Anyway, I'm getting started, I'll let you know if I have any questions.

[rugk]

glide typing'

I only know that as "gesture typing" or "swiping"… but I dunno.

[patrickgold]

I had the same thought, however I was planning to put it inside text.keyboard.KeyboardView, I think that would make more sense since if I were to subclass the LinearLayout root intext_input_layout.xml, glide typing would work in the smartbar, clipboard history, emoji selector, etc which doesn't really make any sense.

The reason why I wanted to place the glide logic into the text input manager layout is, that in Gboard for instance you can glide over the characters and if you see a generated suggestion which fits you you glide to the smartbar and then release the pointer ontop of the generated suggestion, which will then be inserted. Of course you are right that glide typing shouldn't be available in the clip or media layout, but it shouldn't be too hard to check in the touch logic if the current context is text and the current keyboard mode is characters.

From what I understand, supporting dictionaries should be as simple as adding the words from the dictionary into the internal list used for gesture matching, so it should be no trouble supporting it.

Great!

That's great. Just wanted to check though, is 'glide' definitely the right word?

Well I dunno what's the right word for this. These are terms what I've found for this feature:

• gesture typing
• swipe typing
• glide typing
• slide typing
• flow typing

I don't know exactly if the term "glide typing" is a Gboard only thing. If it is, my personal favorite for an alternative would be "swipe typing".

And I'm forced to conclude that you have not heard of Math.atan2, which you have basically reimplemented here lol.

Well... 😄
At least my angle function does take the parameters in the x,y form and not y,x which looks so confusing when all other methods in the Android ecosystem also use x,y...

[X-yl]

The reason why I wanted to place the glide logic into the text input manager layout is, that in Gboard for instance you can glide over the characters and if you see a generated suggestion which fits you you glide to the smartbar and then release the pointer ontop of the generated suggestion, which will then be inserted. Of course you are right that glide typing shouldn't be available in the clip or media layout, but it shouldn't be too hard to check in the touch logic if the current context is text and the current keyboard mode is characters.

Err I don't understand. As far as as I can see gboard doesn't have suggestions while gesture typing, it only has one entry in the smart bar bit, and that shows what will he entered once the finger is released. (and releasing the pointer over the suggestion doesn't do anything special). It would be pretty hard to implement this feature, since it's not really possible to guess when the gesture ends and when the moving to the smart bar begins.

Anyway, in other news, I have a basic proof of concept working! It's mostly just a port of the code in that PR right now.

Not done yet though. Need to fix a few bugs, do the preferences, clean up the code and polish it up. Although this is currently how the AnySoftKeyboard version works, I'd like to see if I can work on the performance and stuff and get it to show a preview of the word while drawing (like gboard). I have a couple ideas about how to tweak the algorithm as well, which I will look into.

Finally, I just wanted to ask you about the dictionary API. Basically in order to classify the gestures, I need to have a word-frequency list. For now, what I'm going to do is just add in a stub method that just returns the data for the English subtype, so you can go ahead and fix it up to work properly later, alright?

[patrickgold]

Err I don't understand. As far as as I can see gboard doesn't have suggestions while gesture typing, it only has one entry in the smart bar bit, and that shows what will he entered once the finger is released. (and releasing the pointer over the suggestion doesn't do anything special). It would be pretty hard to implement this feature, since it's not really possible to guess when the gesture ends and when the moving to the smart bar begins.

I agree, for now its best to just leave this out and focus on the keyboardview and its gesture implementation.

Anyway, in other news, I have a basic proof of concept working! It's mostly just a port of the code in that PR right now.

Looks promising!

Finally, I just wanted to ask you about the dictionary API. Basically in order to classify the gestures, I need to have a word-frequency list. For now, what I'm going to do is just add in a stub method that just returns the data for the English subtype, so you can go ahead and fix it up to work properly later, alright?

As long as the dictionary calling is bundled within a small amount of methods and not spread all over the place I am totally ok with you adding it for English for now and me fixing it up later :)

[bugnano]

Finally, I just wanted to ask you about the dictionary API. Basically in order to classify the gestures, I need to have a word-frequency list. For now, what I'm going to do is just add in a stub method that just returns the data for the English subtype, so you can go ahead and fix it up to work properly later, alright?

The word frequency list could be implemented in such a way that it automatically updates based on how much you type on the keyboard.
My reasoning here is that each one of us has a different style of writing, and tends to use about the same words. But these same words are not the same for everyone, so a dictionary that dynamically updates its frequency count based on how much that word that particular person uses, I think that would immensely increase the accuracy of the keyboard.

[X-yl]

The word frequency list could be implemented in such a way that it automatically updates based on how much you type on the keyboard.
My reasoning here is that each one of us has a different style of writing, and tends to use about the same words. But these same words are not the same for everyone, so a dictionary that dynamically updates its frequency count based on how much that word that particular person uses, I think that would immensely increase the accuracy of the keyboard.

I believe that eventually, yes, that is how it will work.

Anyway, I have just opened a pull request, check it out if you want to try out the gesture typing, notice any bugs, or want to have a look through the code.