New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ace: add external keyboard support on iOS (except history) #3172

Closed
wants to merge 5 commits into
base: master
from

Conversation

Projects
None yet
8 participants
@etamponi
Contributor

etamponi commented Dec 20, 2016

This adds some basic external keyboard handling and fixes part of #37 .

What is supported:

  • Arrow keys
  • Selection and navigation using the keyboard
  • Copy and paste using keyboard

What is not supported:

  • History management using Cmd+Z and Shift+Cmd+Z
  • Touch (except for moving the caret to a certain position)

However, this provides a pretty usable experience.

How does it work?

The only event that iOS sends to textareas and other editable elements when any "special" character is pressed is selectionchange. If we pick the text in the textarea in the correct way, it is possible to use this event to detect what special character has been pressed.

It is possible to modify the code to also support history and touch, but I'll leave it for another PR :)

Show outdated Hide outdated lib/ace/css/editor.css Outdated
var iOS = /iPad|iPhone|iPod/.test(navigator.userAgent) && !window.MSStream;
if (iOS) {
this.textInput = new TextInput_iOS(renderer.getTextAreaContainer(), this);

This comment has been minimized.

@pketh

pketh Dec 20, 2016

@etamponi re comment above, could you get away with appending a class to the editor here? and using the css to target based on that classname rather than viewport size?

@pketh

pketh Dec 20, 2016

@etamponi re comment above, could you get away with appending a class to the editor here? and using the css to target based on that classname rather than viewport size?

@pketh

This comment has been minimized.

Show comment
Hide comment
@pketh

pketh Dec 20, 2016

this looks awesome now ❤️

pketh commented on e8e8616 Dec 20, 2016

this looks awesome now ❤️

@nightwing

This comment has been minimized.

Show comment
Hide comment
@nightwing

nightwing Dec 20, 2016

Member

This looks awesome!
Though i don't quite understand why this works 🤔. I have seen this approach tried a couple of times and usually ios stops sending selectionchange events once cursor gets to the end of text in the textarea and resetting selection after that doesn't help.

I only found a couple of small issues:

  • why is selectionchange listener attached to the document instead of textarea? would be good to have a comment with explaination
  • when typing fast cursor often moves down a line
Member

nightwing commented Dec 20, 2016

This looks awesome!
Though i don't quite understand why this works 🤔. I have seen this approach tried a couple of times and usually ios stops sending selectionchange events once cursor gets to the end of text in the textarea and resetting selection after that doesn't help.

I only found a couple of small issues:

  • why is selectionchange listener attached to the document instead of textarea? would be good to have a comment with explaination
  • when typing fast cursor often moves down a line
@etamponi

This comment has been minimized.

Show comment
Hide comment
@etamponi

etamponi Dec 21, 2016

Contributor

Thanks @nightwing :)

Ok, so this works by: 1) using a specific placeholder to detect special keys and 2) resetting the selection after every selectionchange event that has been handled. I don't know why this approach didn't work before though... 🤔 It's my first attempt, and it worked :D

Anyway, for the sake of explanation, let's make an example:

the hidden textarea contains: "\n aaaa a\n", and the initial selection is (4, 5), that is, the third a is selected. What happens when the user hits ArrowRight? that the selection is now empty, and the cursor is at position 5 (right after the third a). We have successfully detected ArrowRight! Let's reset the selection to (4, 5). Now the user hits Alt+ArrowRight. What happens? that now the cursor is at position 7, that is, right before the last a... we detected Alt+ArrowRight! And so on and so on...

Needing to attach it to the document was a surprise for me too! It's part of iOS quirks... selectionchange is only fired on document... I couldn't make it work on any other element. That's why I need to check activeElement. I'll add a comment.

Thanks for discovering the fast typing bug :) I think the reason is that I reset typing too fast. Better add a timeout... I'll let you know if this works!

Contributor

etamponi commented Dec 21, 2016

Thanks @nightwing :)

Ok, so this works by: 1) using a specific placeholder to detect special keys and 2) resetting the selection after every selectionchange event that has been handled. I don't know why this approach didn't work before though... 🤔 It's my first attempt, and it worked :D

Anyway, for the sake of explanation, let's make an example:

the hidden textarea contains: "\n aaaa a\n", and the initial selection is (4, 5), that is, the third a is selected. What happens when the user hits ArrowRight? that the selection is now empty, and the cursor is at position 5 (right after the third a). We have successfully detected ArrowRight! Let's reset the selection to (4, 5). Now the user hits Alt+ArrowRight. What happens? that now the cursor is at position 7, that is, right before the last a... we detected Alt+ArrowRight! And so on and so on...

Needing to attach it to the document was a surprise for me too! It's part of iOS quirks... selectionchange is only fired on document... I couldn't make it work on any other element. That's why I need to check activeElement. I'll add a comment.

Thanks for discovering the fast typing bug :) I think the reason is that I reset typing too fast. Better add a timeout... I'll let you know if this works!

@etamponi

This comment has been minimized.

Show comment
Hide comment
@etamponi

etamponi Dec 21, 2016

Contributor

Ok, as I suspected, adding a timeout fixes it.

Contributor

etamponi commented Dec 21, 2016

Ok, as I suspected, adding a timeout fixes it.

@@ -74,7 +75,13 @@ var Editor = function(renderer, session) {
this.renderer = renderer;
this.commands = new CommandManager(useragent.isMac ? "mac" : "win", defaultCommands);
this.textInput = new TextInput(renderer.getTextAreaContainer(), this);
var iOS = /iPad|iPhone|iPod/.test(navigator.userAgent) && !window.MSStream;

This comment has been minimized.

@nightwing

nightwing Jan 9, 2017

Member

why does this check for MSStream?

@nightwing

nightwing Jan 9, 2017

Member

why does this check for MSStream?

This comment has been minimized.

@tqc

tqc Apr 13, 2017

Contributor

That is because MS tried to break browser detection - see http://stackoverflow.com/a/9039885/101970

@tqc

tqc Apr 13, 2017

Contributor

That is because MS tried to break browser detection - see http://stackoverflow.com/a/9039885/101970

this.textInput = new TextInput(renderer.getTextAreaContainer(), this);
var iOS = /iPad|iPhone|iPod/.test(navigator.userAgent) && !window.MSStream;
if (iOS) {

This comment has been minimized.

@nightwing

nightwing Jan 9, 2017

Member

would be better to move navigator.userAgent test to https://github.com/ajaxorg/ace/blob/master/lib/ace/lib/useragent.js#L104 and use useragent.isIOS here.

@nightwing

nightwing Jan 9, 2017

Member

would be better to move navigator.userAgent test to https://github.com/ajaxorg/ace/blob/master/lib/ace/lib/useragent.js#L104 and use useragent.isIOS here.

var text = dom.createElement("textarea");
text.className = "ace_text-input ace_text-input-ios";
if (useragent.isTouchPad)

This comment has been minimized.

@nightwing

nightwing Jan 9, 2017

Member

this can be removed

@nightwing

nightwing Jan 9, 2017

Member

this can be removed

.ace_text-input-ios {
position: absolute !important;
top: -100000px !important;
left: -100000px !important;

This comment has been minimized.

@nightwing

nightwing Jan 9, 2017

Member

why is this needed? this will break composition handling.

@nightwing

nightwing Jan 9, 2017

Member

why is this needed? this will break composition handling.

This comment has been minimized.

@tqc

tqc Apr 13, 2017

Contributor

Moving the text box off screen prevents iOS rendering selection handles, which are not affected by opacity settings.

@tqc

tqc Apr 13, 2017

Contributor

Moving the text box off screen prevents iOS rendering selection handles, which are not affected by opacity settings.

return text.selectionStart === 0 && text.selectionEnd === text.value.length;
};
// IE8 does not support setSelectionRange
if (!text.setSelectionRange && text.createTextRange) {

This comment has been minimized.

@nightwing

nightwing Jan 9, 2017

Member

could you remove ie8 code from here?

@nightwing

nightwing Jan 9, 2017

Member

could you remove ie8 code from here?

return text.selectionStart === 0 && text.selectionEnd === text.value.length;
};
// IE8 does not support setSelectionRange
if (!text.setSelectionRange && text.createTextRange) {

This comment has been minimized.

@nightwing

nightwing Jan 9, 2017

Member

could you remove ie8 code from here?

@nightwing

nightwing Jan 9, 2017

Member

could you remove ie8 code from here?

break;
}
}
text.style.height = "100px";

This comment has been minimized.

@nightwing

nightwing Jan 9, 2017

Member

why do we need to set height? this may be better to do somewhere else

@nightwing

nightwing Jan 9, 2017

Member

why do we need to set height? this may be better to do somewhere else

@tqc

This comment has been minimized.

Show comment
Hide comment
@tqc

tqc Apr 13, 2017

Contributor

Not sure what best to do with it since this isn't my PR, but I have a cleaned up version of this at
https://github.com/tqc/ace/tree/ipadkb

Contributor

tqc commented Apr 13, 2017

Not sure what best to do with it since this isn't my PR, but I have a cleaned up version of this at
https://github.com/tqc/ace/tree/ipadkb

@nightwing

This comment has been minimized.

Show comment
Hide comment
@nightwing

nightwing Apr 21, 2017

Member

@tqc your branch looks good, could you confirm in a comment here that the changes you've made are under the terms of the project's BSD license. I can merge after that, thanks!

Member

nightwing commented Apr 21, 2017

@tqc your branch looks good, could you confirm in a comment here that the changes you've made are under the terms of the project's BSD license. I can merge after that, thanks!

@tqc

This comment has been minimized.

Show comment
Hide comment
@tqc

tqc Apr 21, 2017

Contributor

@nightwing Confirmed.

Contributor

tqc commented Apr 21, 2017

@nightwing Confirmed.

@etamponi

This comment has been minimized.

Show comment
Hide comment
@etamponi

etamponi Apr 24, 2017

Contributor

Unfortunately I don't have time to keep working on this, sorry for the inconvenience. It was meant as a simple proof of concept and as someone stated in a different bug report, it might cause problems with other plugins (like vim), but I am happy that @tqc cleaned it up :)

Contributor

etamponi commented Apr 24, 2017

Unfortunately I don't have time to keep working on this, sorry for the inconvenience. It was meant as a simple proof of concept and as someone stated in a different bug report, it might cause problems with other plugins (like vim), but I am happy that @tqc cleaned it up :)

@tqc

This comment has been minimized.

Show comment
Hide comment
@tqc

tqc Apr 24, 2017

Contributor

@etamponi Do you have a link to the specific bug report? If you mean #3157, I don't think there is a problem - this update doesn't fully support vim on iOS, but it doesn't make anything worse either - vim needs a lot more special keys than can be detected with selectionChange.

Contributor

tqc commented Apr 24, 2017

@etamponi Do you have a link to the specific bug report? If you mean #3157, I don't think there is a problem - this update doesn't fully support vim on iOS, but it doesn't make anything worse either - vim needs a lot more special keys than can be detected with selectionChange.

@etamponi

This comment has been minimized.

Show comment
Hide comment
@etamponi

etamponi Apr 24, 2017

Contributor

@tqc perhaps I was dreaming... :D I was pretty sure there was a bug somewhere related to my changes that asked about support for plugins like the vim one. Since I can't find it anymore, I guess it is fine 👍

Contributor

etamponi commented Apr 24, 2017

@tqc perhaps I was dreaming... :D I was pretty sure there was a bug somewhere related to my changes that asked about support for plugins like the vim one. Since I can't find it anymore, I guess it is fine 👍

@aaronvegh

This comment has been minimized.

Show comment
Hide comment
@aaronvegh

aaronvegh May 21, 2017

I, for one, would love to see this get finished and merged... :-)

aaronvegh commented May 21, 2017

I, for one, would love to see this get finished and merged... :-)

@nightwing nightwing referenced this pull request Jun 5, 2017

Merged

Ios keyboard #3310

@idchlife

This comment has been minimized.

Show comment
Hide comment
@idchlife

idchlife Jun 7, 2017

Hey guys, didn't they fix this in new iOS 11? I didn't test it yet, but maybe someone knows. Arrow keys.

idchlife commented Jun 7, 2017

Hey guys, didn't they fix this in new iOS 11? I didn't test it yet, but maybe someone knows. Arrow keys.

@aaronvegh

This comment has been minimized.

Show comment
Hide comment
@aaronvegh

aaronvegh Jun 7, 2017

I've just checked on an iPad running iOS 11. The Kitchen Sink demo https://ace.c9.io/build/kitchen-sink.html does not work with the arrow keys. This merge is still very much needed. Thanks!

aaronvegh commented Jun 7, 2017

I've just checked on an iPad running iOS 11. The Kitchen Sink demo https://ace.c9.io/build/kitchen-sink.html does not work with the arrow keys. This merge is still very much needed. Thanks!

@idchlife

This comment has been minimized.

Show comment
Hide comment
@idchlife

idchlife Jun 7, 2017

@aaronvegh thanks for checking! I didn't yet buy iPad because of this. Also, some offtopic, did you by any chance use your ipad with keyboard for coding in terminal vim as ide or something? Very interested in experience. If you even have video of this process somewhere public, would be very interested to see.

idchlife commented Jun 7, 2017

@aaronvegh thanks for checking! I didn't yet buy iPad because of this. Also, some offtopic, did you by any chance use your ipad with keyboard for coding in terminal vim as ide or something? Very interested in experience. If you even have video of this process somewhere public, would be very interested to see.

@aaronvegh

This comment has been minimized.

Show comment
Hide comment
@aaronvegh

aaronvegh Jun 7, 2017

Here's a video you can look at: I recorded this using iOS 11's screen recorder. It's Panic's Coda running in the terminal with a keyboard. You'll see nano, and using the arrow keys to navigate history.

http://d.pr/v/lVwUlA

aaronvegh commented Jun 7, 2017

Here's a video you can look at: I recorded this using iOS 11's screen recorder. It's Panic's Coda running in the terminal with a keyboard. You'll see nano, and using the arrow keys to navigate history.

http://d.pr/v/lVwUlA

@idchlife

This comment has been minimized.

Show comment
Hide comment
@idchlife

idchlife Jun 7, 2017

@aaronvegh thank you very much. If you have reddit account, I can gild it 🙇

idchlife commented Jun 7, 2017

@aaronvegh thank you very much. If you have reddit account, I can gild it 🙇

@nightwing

This comment has been minimized.

Show comment
Hide comment
@nightwing

nightwing Jun 16, 2017

Member

merged as #3310

Member

nightwing commented Jun 16, 2017

merged as #3310

@nightwing nightwing closed this Jun 16, 2017

@nightwing

This comment has been minimized.

Show comment
Hide comment
@nightwing

nightwing Jun 17, 2017

Member

It is possible to modify the code to also support history and touch, but I'll leave it for another PR :)

@etamponi what approach did you have in mind for these? Do not leave us in mystery like Fermat did:)

Member

nightwing commented Jun 17, 2017

It is possible to modify the code to also support history and touch, but I'll leave it for another PR :)

@etamponi what approach did you have in mind for these? Do not leave us in mystery like Fermat did:)

@etamponi

This comment has been minimized.

Show comment
Hide comment
@etamponi

etamponi Jun 17, 2017

Contributor

@tqc I found the mention about vim support, it is on #37 on which I cited this PR. I thought the comment meant that my PR broke vim support on iOS, but apparently that wasn't the case :)

@nightwing: you now put me in a hard situation... Should I do like Fermat, so that I have at least something in common with him, or should I answer your question and lose this great chance? :)

I'm writing from my phone right now, so I can't elaborate a lot, I'll be more precise tomorrow. To make history work, it should be enough to track when the content of the hidden textbox changes without any other event has been detected (keypress, copy/paste etc). This should be pretty easy to do for Cmd+Z (undo), less so for Cmd+Shift+Z (redo).

I'll try to recollect my original thoughs and elaborate more on this tomorrow :) right now I don't remember anything on how to make touch selection work too.

Contributor

etamponi commented Jun 17, 2017

@tqc I found the mention about vim support, it is on #37 on which I cited this PR. I thought the comment meant that my PR broke vim support on iOS, but apparently that wasn't the case :)

@nightwing: you now put me in a hard situation... Should I do like Fermat, so that I have at least something in common with him, or should I answer your question and lose this great chance? :)

I'm writing from my phone right now, so I can't elaborate a lot, I'll be more precise tomorrow. To make history work, it should be enough to track when the content of the hidden textbox changes without any other event has been detected (keypress, copy/paste etc). This should be pretty easy to do for Cmd+Z (undo), less so for Cmd+Shift+Z (redo).

I'll try to recollect my original thoughs and elaborate more on this tomorrow :) right now I don't remember anything on how to make touch selection work too.

@etamponi

This comment has been minimized.

Show comment
Hide comment
@etamponi

etamponi Jun 19, 2017

Contributor

More details on history support:

  1. the hidden textarea starts with "\n aaaa a\n"; the user types a letter, say b, and the text becomes \n aaba a\n; a keypress event is triggered, so we know from where the b comes. We handle the event and reset the textarea using KeyboardEvents in order to preserve history (if this is still feasible. I attempted at using keyboard events to write to a textarea in Chrome and I couldn't figure out how to do it). For example, we can make sure that the history is:
\n aaua a\n   - for undo
\n aaaa a\n   - current value
\n aara a\n   - for redo
  1. The user hits Cmd+Z or Cmd+Shift+Z: the text in the textarea changes, selectionchange is triggered, but no keypress event. It can't be an arrow key event since the content of the textarea has changed, so it has to be an history event. Is it Cmd+Z or Cmd+Shift+Z? It's enough to check if the value is \n aaua a\n or \n aara a\n.

This would be my plan, but again, it depends on the possibility of making programmatic changes to a textarea preserving/changing its history.

Contributor

etamponi commented Jun 19, 2017

More details on history support:

  1. the hidden textarea starts with "\n aaaa a\n"; the user types a letter, say b, and the text becomes \n aaba a\n; a keypress event is triggered, so we know from where the b comes. We handle the event and reset the textarea using KeyboardEvents in order to preserve history (if this is still feasible. I attempted at using keyboard events to write to a textarea in Chrome and I couldn't figure out how to do it). For example, we can make sure that the history is:
\n aaua a\n   - for undo
\n aaaa a\n   - current value
\n aara a\n   - for redo
  1. The user hits Cmd+Z or Cmd+Shift+Z: the text in the textarea changes, selectionchange is triggered, but no keypress event. It can't be an arrow key event since the content of the textarea has changed, so it has to be an history event. Is it Cmd+Z or Cmd+Shift+Z? It's enough to check if the value is \n aaua a\n or \n aara a\n.

This would be my plan, but again, it depends on the possibility of making programmatic changes to a textarea preserving/changing its history.

@ildodo12

This comment has been minimized.

Show comment
Hide comment
@ildodo12

ildodo12 Sep 24, 2017

guys in my experience the arrow keys on an external keyboard still break typing in sharelatex on ios...

ildodo12 commented Sep 24, 2017

guys in my experience the arrow keys on an external keyboard still break typing in sharelatex on ios...

@nightwing

This comment has been minimized.

Show comment
Hide comment
@nightwing

nightwing Sep 24, 2017

Member

@ildodo12 sharelatex still uses 1.2.5 (check by typing ace.version in the browser console), ask them to update to 1.2.8

Member

nightwing commented Sep 24, 2017

@ildodo12 sharelatex still uses 1.2.5 (check by typing ace.version in the browser console), ask them to update to 1.2.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment