New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remote desktop support #467

Open
beidl opened this Issue Jul 17, 2018 · 28 comments

Comments

Projects
None yet
7 participants
@beidl

beidl commented Jul 17, 2018

I'd be willing to implement support for remote desktop functionality into Mir.
Are there any current plans, expectations and/or code snippets available?
If not, what would be in the expected feature set (application-only remoting, protocol support, etc)?

@AlanGriffiths

This comment has been minimized.

Show comment
Hide comment
@AlanGriffiths

AlanGriffiths Jul 19, 2018

Member

Hello @beidl,

thanks for taking an interest. This is certainly something we'd welcome.

There are no plans currently. We had some internal discussions a few years ago, but the context has changed a lot since then.

As your post implies, the requirements are the first thing that needs capturing.

The most recent relevant discussion was with the UBports group: https://forums.ubports.com/topic/1389/reverse-convergence-view-control-your-phone-from-computer-like-vnc-rdp/

Clearly, that's not the only usecase to consider as we'd want to support desktop sharing and connecting to other types of remote devices/computer.

The approach I outlined in the above UBports discussion (putting the support into the "android" platform) is not the the most flexible, it would better to integrate support into libmirserver.

That raises a host of concerns about security and choice of protocols that we would need to consider. I've not researched this (yet) do you have some ideas?

Also, any sort of remote desktop implementation would need to concern itself with efficiency. That would probably entail getting into the damage tracking in the compositor logic. But, being a long term proponent of "making working code fast" over "making fast code work", I think that's probably best left until something is working.

As for prior art:

You can find the code for "screencast" support starting from: src/include/server/mir/frontend/screencast.h

We did have some support for "display casting": https://wiki.ubuntu.com/Touch/DisplayCasting (I'd have to check the extent to which that was tied into the android/libhybris code - @albaguirre were you involved?).

Both the "graphics platform" and "input platform" are dynamically loaded, this might also provide a route into the code.

Mentioning @gerboland @RAOF and @wmww as they may have further thoughts.

Member

AlanGriffiths commented Jul 19, 2018

Hello @beidl,

thanks for taking an interest. This is certainly something we'd welcome.

There are no plans currently. We had some internal discussions a few years ago, but the context has changed a lot since then.

As your post implies, the requirements are the first thing that needs capturing.

The most recent relevant discussion was with the UBports group: https://forums.ubports.com/topic/1389/reverse-convergence-view-control-your-phone-from-computer-like-vnc-rdp/

Clearly, that's not the only usecase to consider as we'd want to support desktop sharing and connecting to other types of remote devices/computer.

The approach I outlined in the above UBports discussion (putting the support into the "android" platform) is not the the most flexible, it would better to integrate support into libmirserver.

That raises a host of concerns about security and choice of protocols that we would need to consider. I've not researched this (yet) do you have some ideas?

Also, any sort of remote desktop implementation would need to concern itself with efficiency. That would probably entail getting into the damage tracking in the compositor logic. But, being a long term proponent of "making working code fast" over "making fast code work", I think that's probably best left until something is working.

As for prior art:

You can find the code for "screencast" support starting from: src/include/server/mir/frontend/screencast.h

We did have some support for "display casting": https://wiki.ubuntu.com/Touch/DisplayCasting (I'd have to check the extent to which that was tied into the android/libhybris code - @albaguirre were you involved?).

Both the "graphics platform" and "input platform" are dynamically loaded, this might also provide a route into the code.

Mentioning @gerboland @RAOF and @wmww as they may have further thoughts.

@AlanGriffiths

This comment has been minimized.

Show comment
Hide comment
@AlanGriffiths

AlanGriffiths Jul 19, 2018

Member

A couple of prior art links (courtesy of "GizmoChicken" on the community forums):

Member

AlanGriffiths commented Jul 19, 2018

A couple of prior art links (courtesy of "GizmoChicken" on the community forums):

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 19, 2018

AFAICT the aethercast support follows a "push" approach (well, Miracast) where as remote desktop protocols involve a pull approach, but there could be some useful pieces in there.

Implementing support into libmirserver would imply making the display server itself aware of remote connections. Is this intended and what are the pros and cons?

Also, the Mir screencast utility seems to use Mir's internal protocol. Are input events also exposed through the Mir socket? Then remote screencasting could be implemented as a separate application/service proxying events to the compositor, allowing a single solution for all graphics platforms. PipeWire could also be implemented on this proxy service side, keeping the display server/compositor process clean from networking code.

On the other hand:

  • How's the long-term plan with keeping support for Mir's own protocol (is the internal protocol even affected by the move to the Wayland client protocol?)
  • What kinds of input events are expected to be passed through? (IMO preferably all, including touch)

beidl commented Jul 19, 2018

AFAICT the aethercast support follows a "push" approach (well, Miracast) where as remote desktop protocols involve a pull approach, but there could be some useful pieces in there.

Implementing support into libmirserver would imply making the display server itself aware of remote connections. Is this intended and what are the pros and cons?

Also, the Mir screencast utility seems to use Mir's internal protocol. Are input events also exposed through the Mir socket? Then remote screencasting could be implemented as a separate application/service proxying events to the compositor, allowing a single solution for all graphics platforms. PipeWire could also be implemented on this proxy service side, keeping the display server/compositor process clean from networking code.

On the other hand:

  • How's the long-term plan with keeping support for Mir's own protocol (is the internal protocol even affected by the move to the Wayland client protocol?)
  • What kinds of input events are expected to be passed through? (IMO preferably all, including touch)
@AlanGriffiths

This comment has been minimized.

Show comment
Hide comment
@AlanGriffiths

AlanGriffiths Jul 19, 2018

Member

Yes, the references were intended as a way into the code rather than something that's directly useful.

As I said before, we've no current plans to the extent of not having gathered requirements. My instinct would be to have a local process to connect to the mir server and act as an intermediary but I've not thought through the implications. Can you provide any guidance?

Mir's internal protocol and the libmirclient API are "legacy". We're not using or developing them for any new features. There are projects using it (and our test suite) but they are on notice that support will be withdrawn at some (distant) point.

It would likely be useful to find a forum with more bandwidth to discuss this. Can you use google hangouts? What timezone are you in? When is a good time?

Member

AlanGriffiths commented Jul 19, 2018

Yes, the references were intended as a way into the code rather than something that's directly useful.

As I said before, we've no current plans to the extent of not having gathered requirements. My instinct would be to have a local process to connect to the mir server and act as an intermediary but I've not thought through the implications. Can you provide any guidance?

Mir's internal protocol and the libmirclient API are "legacy". We're not using or developing them for any new features. There are projects using it (and our test suite) but they are on notice that support will be withdrawn at some (distant) point.

It would likely be useful to find a forum with more bandwidth to discuss this. Can you use google hangouts? What timezone are you in? When is a good time?

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 19, 2018

Yes, the intermediary process would be what I described as proxy service.

I'm Central Eastern, though I'm on a capped connection right now so Hangouts wouldn't work out quite well, I don't know when this particular situation is going to change.

beidl commented Jul 19, 2018

Yes, the intermediary process would be what I described as proxy service.

I'm Central Eastern, though I'm on a capped connection right now so Hangouts wouldn't work out quite well, I don't know when this particular situation is going to change.

@RAOF

This comment has been minimized.

Show comment
Hide comment
@RAOF

RAOF Jul 19, 2018

Member
Member

RAOF commented Jul 19, 2018

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 19, 2018

@RAOF having multiple graphics/input platforms handle this kind of use case implies treating every platform the same, where as a Screenbuffer-Sharing platform (to me at least) seems to have different intrinsic behaviour (take something someone else rendered and present it VS render and present by itself).
This also brings networking into play where a full networking stack (specifically, having remote socket connections to open) would have to be loaded into the compositor process, which is something I'm not really fond of for a core component like a system's compositor.

beidl commented Jul 19, 2018

@RAOF having multiple graphics/input platforms handle this kind of use case implies treating every platform the same, where as a Screenbuffer-Sharing platform (to me at least) seems to have different intrinsic behaviour (take something someone else rendered and present it VS render and present by itself).
This also brings networking into play where a full networking stack (specifically, having remote socket connections to open) would have to be loaded into the compositor process, which is something I'm not really fond of for a core component like a system's compositor.

@AlanGriffiths

This comment has been minimized.

Show comment
Hide comment
@AlanGriffiths

AlanGriffiths Jul 19, 2018

Member

I too like the combined input/graphics platform approach. One might side-step the multiple platform issue by implementing as an "adaptor" that can load another platform. Alternatively, it might be better to solve that part of the problem in libmirserver.

Member

AlanGriffiths commented Jul 19, 2018

I too like the combined input/graphics platform approach. One might side-step the multiple platform issue by implementing as an "adaptor" that can load another platform. Alternatively, it might be better to solve that part of the problem in libmirserver.

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 21, 2018

Implementing parts ("most things") in platform code makes sense indeed, though I would really like to have networking separated into a different process instead of dragging protocol-specific issues into the main compositor process, which means some kind of fd passing, protocol, what have you, between processes. This also raises the question of making the server itself aware of remoting.

Is the "adaptor" idea already at some stage or should we discuss this first before proceeding?

beidl commented Jul 21, 2018

Implementing parts ("most things") in platform code makes sense indeed, though I would really like to have networking separated into a different process instead of dragging protocol-specific issues into the main compositor process, which means some kind of fd passing, protocol, what have you, between processes. This also raises the question of making the server itself aware of remoting.

Is the "adaptor" idea already at some stage or should we discuss this first before proceeding?

@wmww

This comment has been minimized.

Show comment
Hide comment
Contributor

wmww commented Jul 22, 2018

@GizmoChicken

This comment has been minimized.

Show comment
Hide comment
@GizmoChicken

GizmoChicken Jul 22, 2018

Reportedly, Fedora plans to rely on PipeWire and GNOME Remote Desktop (in conjunction with ScreenCast and RemoteDesktop APIs provided by xdg-desktop-portal) to bring Wayland remote desktop support to Fedora 29. More information is available here, here, and here.

Would it be feasible to implement APIs that would allow using PipeWire and GNOME Remote Desktop with Mir?

P.S. I don't mean to assert that the GNOME/Fedora approach is better (or worse) than another other approach. (I'm not qualified to make such an assertion.) But the GNOME/Fedora approach will be widely adopted, for what that's worth.

P.P.S. This discussion is way over my head, so unless prompted, this will likely be my only comment in this thread.

GizmoChicken commented Jul 22, 2018

Reportedly, Fedora plans to rely on PipeWire and GNOME Remote Desktop (in conjunction with ScreenCast and RemoteDesktop APIs provided by xdg-desktop-portal) to bring Wayland remote desktop support to Fedora 29. More information is available here, here, and here.

Would it be feasible to implement APIs that would allow using PipeWire and GNOME Remote Desktop with Mir?

P.S. I don't mean to assert that the GNOME/Fedora approach is better (or worse) than another other approach. (I'm not qualified to make such an assertion.) But the GNOME/Fedora approach will be widely adopted, for what that's worth.

P.P.S. This discussion is way over my head, so unless prompted, this will likely be my only comment in this thread.

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 22, 2018

I've had experience with a similar approach, pushing input events through DBus in a compositor-type scenario.
Eavesdropping on the bus doesn't seem to be of any concern as they seemingly aim to solve sandboxing issues through DBus APIs.

This would mean:

  • implementing an input injection DBus listener object on the input platform side (as some kind of "adaptor" as Alan called it)
  • passing a PipeWire stream from Mir through another DBus listener object when requested, replacing the internal Mir protocol in use by mirscreencast in the long run
  • have an application outside of Mir's scope handle transport and encryption shenanigans

My current concerns here revolve around DBus eavesdropping and dbus-daemon CPU usage when spamming the host with input events (I'm running dbus-broker on Arch, so my mileage regarding DBus performance varies), as well as replacing all this fine C++ code. :)

EDIT: additionally, this answers the remote-session awareness question for the display server, there'd be need for some accounting regarding RemoteDesktop and ScreenCast session lifecycles.

beidl commented Jul 22, 2018

I've had experience with a similar approach, pushing input events through DBus in a compositor-type scenario.
Eavesdropping on the bus doesn't seem to be of any concern as they seemingly aim to solve sandboxing issues through DBus APIs.

This would mean:

  • implementing an input injection DBus listener object on the input platform side (as some kind of "adaptor" as Alan called it)
  • passing a PipeWire stream from Mir through another DBus listener object when requested, replacing the internal Mir protocol in use by mirscreencast in the long run
  • have an application outside of Mir's scope handle transport and encryption shenanigans

My current concerns here revolve around DBus eavesdropping and dbus-daemon CPU usage when spamming the host with input events (I'm running dbus-broker on Arch, so my mileage regarding DBus performance varies), as well as replacing all this fine C++ code. :)

EDIT: additionally, this answers the remote-session awareness question for the display server, there'd be need for some accounting regarding RemoteDesktop and ScreenCast session lifecycles.

@wmww

This comment has been minimized.

Show comment
Hide comment
@wmww

wmww Jul 23, 2018

Contributor

It seems to me it would be more appropriate to write or pull in a Wayland protocol for input injection instead of using D-Bus. Something along the lines of this: https://github.com/KDE/kwayland/blob/master/src/client/protocols/fake-input.xml (but with auth moved to an external protocol, and some other changes needed)

Contributor

wmww commented Jul 23, 2018

It seems to me it would be more appropriate to write or pull in a Wayland protocol for input injection instead of using D-Bus. Something along the lines of this: https://github.com/KDE/kwayland/blob/master/src/client/protocols/fake-input.xml (but with auth moved to an external protocol, and some other changes needed)

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 23, 2018

So, what we are currently discussing seems to be which input protocol to use, which could be changed over time (if and/or when needed) through the input platform facilities in Mir.

I'd agree that adding DBus in the mix would cause more complexity in terms of security concerns, having a private IPC channel between display server and remote desktop server ala KWayland makes sense here, keeping display-server <-> wl-client and display-server <-> remote-desktop-server communications distinctly separate.

Does adding a platform-type abstraction layer for display buffer sharing make sense? I'm not sure if PipeWire is exactly there yet.

beidl commented Jul 23, 2018

So, what we are currently discussing seems to be which input protocol to use, which could be changed over time (if and/or when needed) through the input platform facilities in Mir.

I'd agree that adding DBus in the mix would cause more complexity in terms of security concerns, having a private IPC channel between display server and remote desktop server ala KWayland makes sense here, keeping display-server <-> wl-client and display-server <-> remote-desktop-server communications distinctly separate.

Does adding a platform-type abstraction layer for display buffer sharing make sense? I'm not sure if PipeWire is exactly there yet.

@AlanGriffiths

This comment has been minimized.

Show comment
Hide comment
@AlanGriffiths

AlanGriffiths Jul 23, 2018

Member

I concur with "have an application outside of Mir's scope handle transport and encryption shenanigans" I'll call it "proxy-mir" for the sake of having a name.

A platform-type abstraction makes sense for both display and input remoting. I'm open to argument whether this belongs outside or inside libmirserver. Without a clear argument either way I'd suggest trying "outside" first (as I think it would likely be easier to migrate out to in than in to out when we gain more knowledge).

There's a bunch of stuff to address that might be beyond the scope of a "platform-type abstraction": is the connection an additional display device (cf the UBports discussion)? or a view onto existing display devices? Controlling that certainly feels more like a separate "proxy-mir" app with a UI than part of the server.

Protocols: Between Mir and "proxy-mir" I don't know of good candidates (that's ignorance, not an assertion they don't exist).

Between "proxy-mir" and a remote desktop I guess VNC has wide support.

Member

AlanGriffiths commented Jul 23, 2018

I concur with "have an application outside of Mir's scope handle transport and encryption shenanigans" I'll call it "proxy-mir" for the sake of having a name.

A platform-type abstraction makes sense for both display and input remoting. I'm open to argument whether this belongs outside or inside libmirserver. Without a clear argument either way I'd suggest trying "outside" first (as I think it would likely be easier to migrate out to in than in to out when we gain more knowledge).

There's a bunch of stuff to address that might be beyond the scope of a "platform-type abstraction": is the connection an additional display device (cf the UBports discussion)? or a view onto existing display devices? Controlling that certainly feels more like a separate "proxy-mir" app with a UI than part of the server.

Protocols: Between Mir and "proxy-mir" I don't know of good candidates (that's ignorance, not an assertion they don't exist).

Between "proxy-mir" and a remote desktop I guess VNC has wide support.

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 23, 2018

Remote desktop functionality to my understanding inherently means showing content of an existing & connected display, but thinking of Windows Server RDP sessions I can see an argument for headless systems as well. Ideally we would solve more than just Remote Desktop-related problems. What this basically comes down to is transfering prerendered display frames to the RDP/VNC-enabled proxy (though I wouldn't put this functionality in scope of the Mir project yet, except for a reference implementation), which could ease work for implementing convergence over similar technologies (having the PC/device render display content offscreen onto a virtual display), basically passing the "hot potato" around. Where the hot potato comes from shouldn't matter, we'd just pass it to the proxy (over PipeWire, shared memory, memfd or whatever).

Additionally, passing damage information about which parts of the screen were re-rendered to the proxy would be useful (where was the potato pierced?). Can this be accounted for along the way to mirscreencast with existing code? I'd suggest adding the abstraction near where ever this information is stored.

beidl commented Jul 23, 2018

Remote desktop functionality to my understanding inherently means showing content of an existing & connected display, but thinking of Windows Server RDP sessions I can see an argument for headless systems as well. Ideally we would solve more than just Remote Desktop-related problems. What this basically comes down to is transfering prerendered display frames to the RDP/VNC-enabled proxy (though I wouldn't put this functionality in scope of the Mir project yet, except for a reference implementation), which could ease work for implementing convergence over similar technologies (having the PC/device render display content offscreen onto a virtual display), basically passing the "hot potato" around. Where the hot potato comes from shouldn't matter, we'd just pass it to the proxy (over PipeWire, shared memory, memfd or whatever).

Additionally, passing damage information about which parts of the screen were re-rendered to the proxy would be useful (where was the potato pierced?). Can this be accounted for along the way to mirscreencast with existing code? I'd suggest adding the abstraction near where ever this information is stored.

@RAOF

This comment has been minimized.

Show comment
Hide comment
@RAOF

RAOF Jul 24, 2018

Member

This is (one of) the advantage(s) of having the Mir end of this use the existing graphics platform API - there's still reasonably rich scene information available at the DisplayBuffer level. A good Renderer implementation will pass through windows individually, making all sorts of optimisations possible or easier. If the remoting protocol allows it, you could even do the final compositing step on the client, which would make things like moving windows around basically free. We don't actually track damage information currently, but once we do we would provide it to the DisplayBuffer (things like DisplayLink USB hardware can optimise based on framebuffer damage, too).

This would also fit in nicely with the long-term hybrid-graphics plans. Our hybrid support is currently the simplest thing that could work - a single mesa-kms instance drives all GPUs. This has drawbacks, the most obvious of which is that it only works if all the GPUs are supported by Mesa. The long-term plan is to make each GPU be driven by a separate platform instance, with render-GPU communicating with display-GPU via an explicit dma-buf interface.

Implementing this as a graphics platform makes no demands on how you get the “hot potato” to the remote desktop process - the graphics platform can register whatever Wayland extensions it desires (indeed, it has to register at least one in order for clients to submit GPU buffers), the graphics platform can expose a DBus interface, or whatever other IPC is necessary between Mir and the remote desktop server process. (A similar hook for input platforms to register Wayland extensions would be trivial).

Member

RAOF commented Jul 24, 2018

This is (one of) the advantage(s) of having the Mir end of this use the existing graphics platform API - there's still reasonably rich scene information available at the DisplayBuffer level. A good Renderer implementation will pass through windows individually, making all sorts of optimisations possible or easier. If the remoting protocol allows it, you could even do the final compositing step on the client, which would make things like moving windows around basically free. We don't actually track damage information currently, but once we do we would provide it to the DisplayBuffer (things like DisplayLink USB hardware can optimise based on framebuffer damage, too).

This would also fit in nicely with the long-term hybrid-graphics plans. Our hybrid support is currently the simplest thing that could work - a single mesa-kms instance drives all GPUs. This has drawbacks, the most obvious of which is that it only works if all the GPUs are supported by Mesa. The long-term plan is to make each GPU be driven by a separate platform instance, with render-GPU communicating with display-GPU via an explicit dma-buf interface.

Implementing this as a graphics platform makes no demands on how you get the “hot potato” to the remote desktop process - the graphics platform can register whatever Wayland extensions it desires (indeed, it has to register at least one in order for clients to submit GPU buffers), the graphics platform can expose a DBus interface, or whatever other IPC is necessary between Mir and the remote desktop server process. (A similar hook for input platforms to register Wayland extensions would be trivial).

@RAOF

This comment has been minimized.

Show comment
Hide comment
@RAOF

RAOF Jul 24, 2018

Member

For requirements gathering, I think we have three use-cases:

  1. Desktop sharing - remote access to an existing desktop session, where both the existing and new user want to interact with the desktop.
  2. Remote login - remote access to a new session on a server; each connection is basically a new seat, with a separate set of input/output devices
  3. a. (ish) Desktop extension - the UBports use case: add an extra display to an existing desktop which just happens to be driven by a GPU not physically attached to the system.
    b. (ish) Weird Input - related UBports use case: use an app on your phone to turn your touchscreen into a touchpad for the desktop.

I am undecided whether all 3ish of these should use the same infrastructure. 2. is easily and obviously implementable as a combination input/graphics platform exactly like the X11 platform, right now. 3a looks like a graphics platform, and 3b looks like an input platform, but implementing them as such requires core changes to allow multiple platform implementations to be loaded.

1 looks the least like a graphics platform - rather than adding a new output, this wants to clone (some subset of?) the existing outputs. This can expressed by a graphics platform - simply set each relevant output to a clone - but this is not necessarily what is meant to be expressed by a DisplayConfiguration? Maybe we'd need an extra is_virtual bit to help things which inspect display configuration?

Member

RAOF commented Jul 24, 2018

For requirements gathering, I think we have three use-cases:

  1. Desktop sharing - remote access to an existing desktop session, where both the existing and new user want to interact with the desktop.
  2. Remote login - remote access to a new session on a server; each connection is basically a new seat, with a separate set of input/output devices
  3. a. (ish) Desktop extension - the UBports use case: add an extra display to an existing desktop which just happens to be driven by a GPU not physically attached to the system.
    b. (ish) Weird Input - related UBports use case: use an app on your phone to turn your touchscreen into a touchpad for the desktop.

I am undecided whether all 3ish of these should use the same infrastructure. 2. is easily and obviously implementable as a combination input/graphics platform exactly like the X11 platform, right now. 3a looks like a graphics platform, and 3b looks like an input platform, but implementing them as such requires core changes to allow multiple platform implementations to be loaded.

1 looks the least like a graphics platform - rather than adding a new output, this wants to clone (some subset of?) the existing outputs. This can expressed by a graphics platform - simply set each relevant output to a clone - but this is not necessarily what is meant to be expressed by a DisplayConfiguration? Maybe we'd need an extra is_virtual bit to help things which inspect display configuration?

@RAOF

This comment has been minimized.

Show comment
Hide comment
@RAOF

RAOF Jul 25, 2018

Member

We discussed this in the regular Mir team meeting last night, and the consensus was that making a “remoting-proxy” input/graphics platform would be the sensible way to begin this. That would let you satisfy use-case (2) with the least amount of work (particularly, with no core Mir code changes), and can be incrementally extended to cover the other use-cases.

Suggested implementation plan

Because of the current state of the graphics platform API you'll need to implement both a RenderingPlatform and a DisplayPlatform. The simplest thing to do here is to copy src/platforms/mesa/server/kms and the associated helpers into a new platform.

You should be able to leave most of the code as-is; the Display implementation is the relevant point of difference (for configuration and cursor management), and particularly the DisplayBuffer implementation which is what handles the actual rendered output. (This needs to implement renderer::gl::RenderTarget). GBMOutputSurface, and particularly ::lock_front(), give you a gbm_bo* you can slurp the dma-buf from.

For input I'd suggest checking out src/platforms/mesa/server/x11/input; that should be a good example of creating input devices to proxy from another source.

That will give you all the display output and allow you to inject input events, so all the necessary Mir-side interfaces.

We don't particularly mind which project you choose to do the remote-desktop-protocol bit, nor do we mind how you choose to communicate between the “remoting-proxy” platform and that process. Using whatever GNOME is planning to use is a reasonable choice, but there may well be other reasonable choices. As a loadable platform module it's quite easy to have multiple implementations, and select the one you want to use at runtime.

This should give a viable implementation for remoting into a headless server. Next steps would be to add dynamic platform loading and better separate display from rendering; then the platform can be used for the other use-cases.

Member

RAOF commented Jul 25, 2018

We discussed this in the regular Mir team meeting last night, and the consensus was that making a “remoting-proxy” input/graphics platform would be the sensible way to begin this. That would let you satisfy use-case (2) with the least amount of work (particularly, with no core Mir code changes), and can be incrementally extended to cover the other use-cases.

Suggested implementation plan

Because of the current state of the graphics platform API you'll need to implement both a RenderingPlatform and a DisplayPlatform. The simplest thing to do here is to copy src/platforms/mesa/server/kms and the associated helpers into a new platform.

You should be able to leave most of the code as-is; the Display implementation is the relevant point of difference (for configuration and cursor management), and particularly the DisplayBuffer implementation which is what handles the actual rendered output. (This needs to implement renderer::gl::RenderTarget). GBMOutputSurface, and particularly ::lock_front(), give you a gbm_bo* you can slurp the dma-buf from.

For input I'd suggest checking out src/platforms/mesa/server/x11/input; that should be a good example of creating input devices to proxy from another source.

That will give you all the display output and allow you to inject input events, so all the necessary Mir-side interfaces.

We don't particularly mind which project you choose to do the remote-desktop-protocol bit, nor do we mind how you choose to communicate between the “remoting-proxy” platform and that process. Using whatever GNOME is planning to use is a reasonable choice, but there may well be other reasonable choices. As a loadable platform module it's quite easy to have multiple implementations, and select the one you want to use at runtime.

This should give a viable implementation for remoting into a headless server. Next steps would be to add dynamic platform loading and better separate display from rendering; then the platform can be used for the other use-cases.

@Saviq

This comment has been minimized.

Show comment
Hide comment
@Saviq

Saviq Jul 25, 2018

Member

Would the same approach allow for zero-copy in-GPU encoding? Most GPUs have built-in efficient h264(5?) encoders which can cater for screencasting, as we implemented for Ubuntu Touch and its Miracast support.

Member

Saviq commented Jul 25, 2018

Would the same approach allow for zero-copy in-GPU encoding? Most GPUs have built-in efficient h264(5?) encoders which can cater for screencasting, as we implemented for Ubuntu Touch and its Miracast support.

@RAOF

This comment has been minimized.

Show comment
Hide comment
@RAOF

RAOF Jul 25, 2018

Member
Member

RAOF commented Jul 25, 2018

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 25, 2018

Getting headless remote desktop support into Mir could require awareness about sessions, creating a new session/seat for a new user.
This would additionally require configurability through the shell in the long run, allowing shell implementations to select behaviour through libmirserver.
Should an incoming remote connection create a new session/seat when connecting or are those bound to a VT at the moment? May I join the next Mir meeting to discuss details?

beidl commented Jul 25, 2018

Getting headless remote desktop support into Mir could require awareness about sessions, creating a new session/seat for a new user.
This would additionally require configurability through the shell in the long run, allowing shell implementations to select behaviour through libmirserver.
Should an incoming remote connection create a new session/seat when connecting or are those bound to a VT at the moment? May I join the next Mir meeting to discuss details?

@RAOF

This comment has been minimized.

Show comment
Hide comment
@RAOF

RAOF Jul 27, 2018

Member

Mir won't require awareness about sessions and such; that's a job for some other subsystem - for example, you can get LightDM to handle remote connections by starting up a remote display server. I believe GDM has some similar sort of support.

We suggest the implementation plan above because we think it's the path of least work required to get something working that can then be incrementally expanded to cover all the identified use-cases.

Particularly, it doesn't require any core Mir infrastructure before it can work, but can take advantage of (and drive) core Mir changes once they happen.

This is only a suggestion; I certainly wouldn't reject a merge proposal that covered desktop sharing but not headless remoting out of hand, but I think it would likely be a large PR that would need to be broken up into multiple stages, each one of which wouldn't make complete sense without the whole series.

A remote-proxy platform would be self-contained, an easy PR to review and accept. Further PRs to implement the core Mir changes would then have an existing user, and be easier to do incrementally while validating that they actually do what we want.

Member

RAOF commented Jul 27, 2018

Mir won't require awareness about sessions and such; that's a job for some other subsystem - for example, you can get LightDM to handle remote connections by starting up a remote display server. I believe GDM has some similar sort of support.

We suggest the implementation plan above because we think it's the path of least work required to get something working that can then be incrementally expanded to cover all the identified use-cases.

Particularly, it doesn't require any core Mir infrastructure before it can work, but can take advantage of (and drive) core Mir changes once they happen.

This is only a suggestion; I certainly wouldn't reject a merge proposal that covered desktop sharing but not headless remoting out of hand, but I think it would likely be a large PR that would need to be broken up into multiple stages, each one of which wouldn't make complete sense without the whole series.

A remote-proxy platform would be self-contained, an easy PR to review and accept. Further PRs to implement the core Mir changes would then have an existing user, and be easier to do incrementally while validating that they actually do what we want.

@AlanGriffiths

This comment has been minimized.

Show comment
Hide comment
@AlanGriffiths

AlanGriffiths Jul 27, 2018

Member

@beidl I imagine the next Mir meeting is at a bad time for you (Tuesday, 10:30 UTC).

I hope you already have enough answers but if not, and you're online at the same time as me (Western European Time) or @RAOF (Australian Eastern Time) then either of us can help.

Member

AlanGriffiths commented Jul 27, 2018

@beidl I imagine the next Mir meeting is at a bad time for you (Tuesday, 10:30 UTC).

I hope you already have enough answers but if not, and you're online at the same time as me (Western European Time) or @RAOF (Australian Eastern Time) then either of us can help.

@beidl

This comment has been minimized.

Show comment
Hide comment
@beidl

beidl Jul 27, 2018

Yup, should be enough. Agreed, core Mir infrastructure should be changed when needed. I can't give you an estimate when I have an initial implementation + PR available, maybe within a month.

beidl commented Jul 27, 2018

Yup, should be enough. Agreed, core Mir infrastructure should be changed when needed. I can't give you an estimate when I have an initial implementation + PR available, maybe within a month.

@Flohack74

This comment has been minimized.

Show comment
Hide comment
@Flohack74

Flohack74 Aug 9, 2018

Thanks @AlanGriffiths for pointing me on this one.

As I am working in both "worlds" of Windows and Linux, I was always missing the quick & decent way of getting a remote graphical desktop for a Linux machine. First, on servers this was never installed. Okay, you might argue that a server does not need a GUI - but, in todays graphic world, this is not so true anymore. Imagine downloading a needed software pack with Lynx. Getting a spreadsheet report of data transfer per day. Quickly dumping a few files in a diff viewer - all that is troublesome on a text terminal.

X11 is too complicated and has no easy usable software for Windows. VNC is not the same, first of all you must be logged in to the machine, and then the protocol has issues with screen scaling and color tearing etc.

Plus with VNC only one person can work on a machine. In some scenarios, I can imagine having Windows clients or Thin clients deployed to access Linux machines.

I strongly would vote for implementing RDP right away. It has performance, authentication and encryption built-in, can convey Video, Mouse, Keyboard, Sound, Printers, Local drives, etc. I am just not sure all of it is open sourced :)

Flohack74 commented Aug 9, 2018

Thanks @AlanGriffiths for pointing me on this one.

As I am working in both "worlds" of Windows and Linux, I was always missing the quick & decent way of getting a remote graphical desktop for a Linux machine. First, on servers this was never installed. Okay, you might argue that a server does not need a GUI - but, in todays graphic world, this is not so true anymore. Imagine downloading a needed software pack with Lynx. Getting a spreadsheet report of data transfer per day. Quickly dumping a few files in a diff viewer - all that is troublesome on a text terminal.

X11 is too complicated and has no easy usable software for Windows. VNC is not the same, first of all you must be logged in to the machine, and then the protocol has issues with screen scaling and color tearing etc.

Plus with VNC only one person can work on a machine. In some scenarios, I can imagine having Windows clients or Thin clients deployed to access Linux machines.

I strongly would vote for implementing RDP right away. It has performance, authentication and encryption built-in, can convey Video, Mouse, Keyboard, Sound, Printers, Local drives, etc. I am just not sure all of it is open sourced :)

@Flohack74

This comment has been minimized.

Show comment
Hide comment
@Flohack74

Flohack74 Aug 9, 2018

Regarding sessions, RDP has the notion of the "Admin" console vs. new logins. Connecting to the Admin console will simply connect the user with the session associated with the local hardware. Interestingly in such a case the local console will log off and transfer the session to RDP. Other people cannot spy on you what you are doing.

Also, if the logged on user is a different identity he will be informed that someone wants to connect to his session. Around that principle Microsoft put the whole blingbling of Remote Support technology where in the end it IS possible to share the screen and the mouse inputs ;)

Flohack74 commented Aug 9, 2018

Regarding sessions, RDP has the notion of the "Admin" console vs. new logins. Connecting to the Admin console will simply connect the user with the session associated with the local hardware. Interestingly in such a case the local console will log off and transfer the session to RDP. Other people cannot spy on you what you are doing.

Also, if the logged on user is a different identity he will be informed that someone wants to connect to his session. Around that principle Microsoft put the whole blingbling of Remote Support technology where in the end it IS possible to share the screen and the mouse inputs ;)

@GizmoChicken

This comment has been minimized.

Show comment
Hide comment
@GizmoChicken

GizmoChicken Sep 23, 2018

From an article published on Phoronix:

The KDE Plasma/KWin developers have been pursuing Wayland remote desktop support along a similar route to the GNOME Shell camp by making use of PipeWire and the XDG-Desktop-Portal. Bits are already in place for KDE Plasma 5.13 and the upcoming 5.14 release, but for the 5.15 release is now where it sounds like the support may be in good shape for end-users.

GizmoChicken commented Sep 23, 2018

From an article published on Phoronix:

The KDE Plasma/KWin developers have been pursuing Wayland remote desktop support along a similar route to the GNOME Shell camp by making use of PipeWire and the XDG-Desktop-Portal. Bits are already in place for KDE Plasma 5.13 and the upcoming 5.14 release, but for the 5.15 release is now where it sounds like the support may be in good shape for end-users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment