Documentation for control protocol

[jze]

I would like to use the scrcpy server in connection with machine learning. With some adjustments relatd to the new version 1.9 of the server I have been able to use the py-scrcpy Python code https://github.com/jze/py-scrcpy receive the video stream. Is is correct to simple receive the video stream via the first socket. When looking at the original data stream of scrcpy using Wireshark there are some additional characters like OKAY.

Now I am looking for a way to send commands back to the Android device. I think https://github.com/Genymobile/scrcpy/blob/master/server/src/main/java/com/genymobile/scrcpy/ControlMessageReader.java contains the necessary information to understand the control protocol. However, it would be much easier if there was some written documentation or simple demo code.

[rom1v]

Hi,

The protocol between the client and the server is internal and private, and may (and will) change at any time.

Recently, as you noticed, the communication has changed to use 2 sockets in order to support copy-paste (ec71a3f).

In the near future, to reduce latency (see #646), each packet in the video stream will be prefixed with headers (even when recording is disabled). (But I will probably keep the option on the server-side to not send these headers, even if scrcpy itself will never use it, not to break unnecessarily usages of scrcpy-server.jar).

Also, the mouse events format will evolve to support multitouch.

Without this flexibility, I won't be able to make it evolve as necessary. So it's not a public interface, that's why there is no documentation.

The "real" documentation for this protocol is complete unit tests set on both sides:

• ControlMessage (from client to device) deserialization | serialization
• DeviceMessage (from device to client) serialization | deserialization

In the long term, to support your use case, I have 2 options I would like to work on:

• make a library (libscrcpy) used by the command line/SDL app;
• expose a "remote interface" to receive the video stream and send commands.

[razumeiko]

Hey @rom1v i am trying to send commands from python and i can already mimic this protocol but i have two problems that i can't fix, i don't understand what data format is it. In tests you have this:

0x00, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, 0xc8, // 100 200

Means first 4 bytes is 100 and second 4 bytes is 200. From python i can easily unpack it:

>>> struct.unpack('>I', b'\x00\x00\x00\x64')
(100,)

But when i want to pack it back and send:

# Now try to pack 100
>>> struct.pack('>I', 100)
b'\x00\x00\x00d'

I am receiving different format. Same for screen res

0x04, 0x38, 0x07, 0x80, // 1080 1920
>>> struct.unpack('>h', b'\x04\x38')
(1080,)
>>> struct.pack('>h', 1080)
b'\x048'

It looks like it's hex format but split for 2 bytes. Can you clarify please what format is it and how to build it?

[rom1v]

>>> struct.unpack('>I', b'\x00\x00\x00\x64')
(100,)

But when i want to pack it back and send:

# Now try to pack 100
>>> struct.pack('>I', 100)
b'\x00\x00\x00d'

It's the same, \x64 is d is ASCII.

0x04, 0x38, 0x07, 0x80, // 1080 1920
>>> struct.unpack('>h', b'\x04\x38')
(1080,)
>>> struct.pack('>h', 1080)
b'\x048'

Same here, \x38 is 8 in ASCII.

[razumeiko]

I create python class that you can use to send swipe event, this can be extended if needed:
https://github.com/razumeiko/py-android-viewer