Add On Screen buttons

[S0und]

My phone has capacitive buttons, and because of that i don't have on-screen buttons. A command line argument for showing the buttons would be nice.

[rom1v]

Yes, we discussed it internally, but it's not trivial in SDL since there are no widgets.

But there are shortcuts:

• right-click or Alt+b (or Alt+Backspace) for BACK,
• middle-click or Alt+h for HOME,
• Alt+s for the third button, APP_SWITCH.

EDIT: now shortcuts use Alt by default.
https://github.com/Genymobile/scrcpy#shortcuts

[davcec]

+1 for that! I hope you will find time to investigate that...

[Kyle-Falconer]

If this isn't easily done onscreen, then perhaps a sidebar would be doable, such as how the Android Emulator or Vysor handles this.

In addition to the navigation buttons, I'd love to have buttons for screenshots and video capture as well, but perhaps that's out of scope for this issue specifically and should have its own issue.

Edit: I see that issue #21 has been opened to request a keyboard shortcut to start screen record/capture.

[rom1v]

For information, on dev, I just implemented new shortcuts:

• Right-click presses BACK (when screen is on) (675704c)
• Middle-click presses HOME (c87d94e)

I think it may help.

[snowdream]

For Macbook， “Esc” is back, And "Fn + Left" is Home.

[AndroidDeveloperLB]

Can you please use the same keys as used in the emulator, and even allow to customize?
Here are those that are used there, for MAC :
https://i.stack.imgur.com/eNB8x.png
There are others for Windows/Linux.

For me, the most commonly used are Home (which is the home button on the keyboard) and Back (which is Escape button on the keyboard).

[rom1v]

For me, the most commonly used are Home (which is the home button on the keyboard)

It also works.

and Back (which is Escape button on the keyboard).

Escape sends ESCAPE to the device, which behaves similar to BACK (but not totally). I don't understand the logic though. Maybe it's the same on the emulator…

[AndroidDeveloperLB]

@rom1v Home button seems to work.
But Escape button doesn't do anything.

[arnoldlaishram]

@Kyle-Falconer Hey, how did you get this working on emulator?

Works fine for devices but I get this error when i run for emulator

➜  ~ scrcpy
/usr/local/share/scrcpy/scrcpy-server.jar: ...le pushed. 1.3 MB/s (19038 bytes in 0.014s)
2018-03-13 12:20:57.823 scrcpy[6894:419807] INFO: Initial texture: 540x960
ERROR: Exception on thread Thread[main,5,main]
java.lang.IllegalStateException
	at android.media.MediaCodec.native_stop(Native Method)
	at android.media.MediaCodec.stop(MediaCodec.java:1901)
	at com.genymobile.scrcpy.ScreenEncoder.streamScreen(ScreenEncoder.java:69)
	at com.genymobile.scrcpy.Server.scrcpy(Server.java:21)
	at com.genymobile.scrcpy.Server.main(Server.java:70)
	at com.android.internal.os.RuntimeInit.nativeFinishInit(Native Method)
	at com.android.internal.os.RuntimeInit.main(RuntimeInit.java:251)
2018-03-13 12:20:58.428 scrcpy[6894:419909] ERROR: Could not open video stream

[arnoldlaishram]

I tested on emulator with Oreo. Works for only Oreo. Doesn't seem to work for Nougat, Marshmellow and below.

[AndroidDeveloperLB]

What does the mouse-right-click do?
To me it seems to only be able to wake the device.

[rom1v]

What does the mouse-right-click do?

If the device screen is off, it turns it on. Otherwise, it presses BACK. But currently, it's only on the dev branch (675704c).

[AndroidDeveloperLB]

@rom1v It doesn't do BACK for me. Only wakes the device.

[rom1v]

@AndroidDeveloperLB Did you compile from the dev branch (server included)?

[AndroidDeveloperLB]

@rom1v I only launched the binaries for Windows OS. Did you mean the BACK will work on the dev branch?

[rom1v]

Did you mean the BACK will work on the dev branch?

Yes. It has been implemented after the release.

[AndroidDeveloperLB]

@rom1v OK sorry.

[mmm4m5m]

@rom1v :
Please, bring "APP_SWITCH" to the mouse! This is the only missing thing! (Maybe using mouse wheel up/down).

"Home" button is NOT working here. Other keys - Ctrl+H and Middle click - both works fine.

"Long press" is not supported.
"Long press Ctrl+H/Home" is NOT working like "long press HOME" on device.
(My setup is "show notifications').
"Long press Ctrl+B" or "Right click and hold" is NOT working like "long press BACK" on device.
(My setup is "lock device").

"Double tap" is not supported.
"Double press Ctrl+S" is NOT like "double press RECENTS" on device.
(Default action is "switch to last used application". Like press once Alt+Tab on PC).

Fast press Ctrl+P and Ctrl+Down (or Ctrl+Down and Ctrl+P) is NOT creating screenshot.

(And last - for screen resize, there could be "resize screen to actual device size").

Version 1.11, device OnePlus 3T. I am using Debian buster but installed the packages from Debian sid/testing repository (dpkg with "--force-depends" because libavcodec58, libavformat58, libsdl2-2.0-0): scrcpy_1.11+ds-1_i386.deb, scrcpy_1.11+ds-1_i386.deb

Very useful application.
Thanks!

[rom1v]

Please, bring "APP_SWITCH" to the mouse! This is the only missing thing!

That's true, it's annoying to use the keyboard for that. But there are only 3 buttons on many mice (click, home, back).

(Maybe using mouse wheel up/down).

Mouse wheel up/down are used for scrolling. I would like to keep Ctrl+wheel for zoom or something if we need to. Maybe Cltr+click_wheel?

"Home" button is NOT working here. Other keys - Ctrl+H and Middle click - both works fine.

That's on purpose: 5d473ef.

"Long press" is not supported.

Hmm, it is. If I long press with left-click, it acts as if I long pressed with the finger.

"Long press Ctrl+H/Home" is NOT working like "long press HOME" on device.
(My setup is "show notifications').

I don't know, it correctly sends AKEYCODE_HOME with ACTION_DOWN when you press, and AKEYCODE_HOME with ACTION_UP when you release.

"Long press Ctrl+B" or "Right click and hold" is NOT working like "long press BACK" on device.
(My setup is "lock device").

Same with AKEYCODE_BACK.

"Double tap" is not supported.

If I double click on a text, it selects it, while if I single-click, it does not. What do you mean by "double tap" is not supported?

"Double press Ctrl+S" is NOT like "double press RECENTS" on device.
(Default action is "switch to last used application". Like press once Alt+Tab on PC).

On my device (OnePlus 7 Pro), it behaves that way.

Fast press Ctrl+P and Ctrl+Down (or Ctrl+Down and Ctrl+P) is NOT creating screenshot.

On my device, it does. (I didn't know that, but I just tested)

(And last - for screen resize, there could be "resize screen to actual device size").

Ctrl+g

[mmm4m5m]

Mouse wheel up/down are used for scrolling. I would like to keep Ctrl+wheel for zoom or something if we need to. Maybe Cltr+click_wheel?

No, "Ctrl" is again a keyboard key.

"Long press" is not supported.
"Double tap" is not supported.

Both are about HOME, BACK, APP_SWITCH (Recents). Android 9 Pie can assign actions (in Setting/Buttons/Buttons).

"Double press Ctrl+S" is NOT like "double press RECENTS" on device.
(Default action is "switch to last used application". Like press once Alt+Tab on PC).

On my device (OnePlus 7 Pro), it behaves that way.

I did mistake! Maybe my double press was very slow.
It works for HOME and APP_SWITCH.
It does NOT work for BACK - Ctrl+b, Ctrl+Backspace, Right-click.

Fast press Ctrl+P and Ctrl+Down (or Ctrl+Down and Ctrl+P) is NOT creating screenshot.

On my device, it does. (I didn't know that, but I just tested)

I did mistake! Somehow my screenshot action (application?) is completely disabled.

(And last - for screen resize, there could be "resize screen to actual device size").

Ctrl+g

Ctrl+G is 1:1 pixels, I mean 1:1 physical size/inches (approximately).

Sorry for my mistakes! :-/

[rom1v]

I don't know, it correctly sends AKEYCODE_HOME with ACTION_DOWN when you press, and AKEYCODE_HOME with ACTION_UP when you release.

Oh, actually, it only does that for keyboard shortcuts.

With the mouse, it sends both down/up immediately. It will be a bit more complex for BACK, because the right click is used both for "power on" and "back", and the choice is done by the server (on the device).

Could you please open a separate issue for correctly handling long press of HOME and BACK using the mouse?

[mmm4m5m]

Created issue 1062 . (Mouse and keys also, because I do not understand fully your answer)
Thanks! :)

[AndroidDeveloperLB]

What was done? Did you add long pressing on navigation buttons (both mouse and keyboard) ?

[mmm4m5m]

@AndroidDeveloperLB - No. Sorry for confusion. Just created new issue for long press. Edited my previous comment.
Also, based on above talk, I added one feature request - 1063 - "APP_SWITCH with middle-click".

[AndroidDeveloperLB]

@mmm4m5m
Wait, so how do you use HOME using the mouse?
I thought that maybe the middle click would be for home

[mmm4m5m]

@AndroidDeveloperLB - Yes, middle-click is currently for "HOME". I think, it is better if both are switched - keyboard for HOME, mouse for APP_SWITCH. Command line parameter to change this behavior could help.
Anyway, the described workaround can be used in both cases! And with "long press", (when it is fixed/implemented), we will have even more options.

[AndroidDeveloperLB]

So what would long pressing the middle mouse button do? Will it go to recent tasks?

I also wonder if there is a function to switch to the recent-tasks (double click on recent-tasks button), or even switch left/right (using the new navigation gestures). I ask this because my mouse actually have 2 more buttons. One for back-navigation and one for forward . I use them mostly on the web browser

[mmm4m5m]

@AndroidDeveloperLB - It will do the action you assign in your phone settings.
Android 9 Pie, in settings/buttons/buttons, you can assign additional actions for "double tap" and "long press". I have android 9, but this feature is maybe available for older android versions also.
The only missing part, the only needed thing is - "long press" does not work with scrcpy program.

Switching to last task works fine. If you press "double Ctrl+S" will send "double APP_SWITCH" (double RECENTS) to device.

If your mouse is smart enough, or maybe using some software, then you can assign "Ctrl+S" to your mouse extra buttons.

Gestures, I assume, are more complicated to assign to a key/mouse button. But you can do gestures using the mouse pointer.

[AndroidDeveloperLB]

Maybe you have some non-vanilla rom, because I have Pixel 4 and I have no setting of what to do upon long/double pressing the home button.
The only thing there is, is the assistant settings.

[rom1v]

And with "long press", (when it is fixed/implemented), we will have even more options.

The fix for #1062 is to correctly send DOWN and UP actions separately for mouse shortcuts (like keyboard shortcuts). The "long press" is still (implicitly) managed by the device.

[AndroidDeveloperLB]

Maybe it's best to just make a nice UI for the app?
One that allows us to set the various settings of it?
For example, I could set what the side buttons of the mouse would do... Maybe one of them should show the recent tasks, for example...

[rom1v]

Maybe it's best to just make a nice UI for the app?

"just"

[AndroidDeveloperLB]

@rom1v Sorry. Isn't Google in some good relationship with Genymobile? Maybe they could help?
This is such a great tool. Can't they buy it or something?

[rom1v]

Isn't Google in some good relationship with Genymobile?

I don't know. I'm not a Genymobile employee anymore.

Since the initial release of the project has been published, I develop/maintain scrcpy on my free time (which is scarce, I have a full time job on other projects).

Maybe they could help?

Anyone could help. Several people have already contributed.

[AndroidDeveloperLB]

Sorry for that. Seeing "Genymobile" in the repository, I assumed this repository belongs to them.

[Peter-maximus]

Please add the toolbar (out of phone screen) with virtual buttons (back, home, appswitch) and there can be also save screenshot, capture video buttons.
All other android remote viewers have it (vysor, teamviewer).

There are still plenty of phones with hardware buttons, so no virtual buttons on screen possible in that case.

And it is also not really convenient to remember mouse / keyboard shortcuts... (especially if do not use them on daily basis).

[lostest4352]

+1. I hope you add an optional feature for the phones with hardware buttons. Alt+S is not very intuitive compared to buttons.