New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CNTK (Transfer learning) not working with remote desktop #1841

Closed
Bishwa-Subedi opened this Issue May 4, 2017 · 8 comments

Comments

Projects
None yet
4 participants
@Bishwa-Subedi

Bishwa-Subedi commented May 4, 2017

I am trying to implement transfer learning concept (https://github.com/Microsoft/CNTK/wiki/Build-your-own-image-classifier-using-Transfer-Learning).
We have set up an Remote GPU machine and I want to access it using MS remote desktop from my laptop. While running TransferLearning.py it says "Specified device id (0) is invalid".
While trying to run it on CPU instead of GPU it says "Batch Normalization training on CPU is not yet implemented"

I found that GPU features are disabled when using MS remote desktop. Is above error "Specified device id (0) is invalid" is due to the fact that I am using remote desktop? Is there any workaround for it?

Thanks a lot in advance.
Bishwa

@Sadler2

This comment has been minimized.

Show comment
Hide comment
@Sadler2

Sadler2 May 4, 2017

Yes, this is the property of RDP:

On the server, RDP uses its own video driver to render display output by constructing the rendering information into network packets by using RDP protocol and sending them over the network to the client.
https://msdn.microsoft.com/en-us/library/aa383015.aspx

The only workaround I may suggest as an ordinary user of CNTK is to close your RDP session during the whole learning proccess or at least during device initialization phase.

Sadler2 commented May 4, 2017

Yes, this is the property of RDP:

On the server, RDP uses its own video driver to render display output by constructing the rendering information into network packets by using RDP protocol and sending them over the network to the client.
https://msdn.microsoft.com/en-us/library/aa383015.aspx

The only workaround I may suggest as an ordinary user of CNTK is to close your RDP session during the whole learning proccess or at least during device initialization phase.

@Bishwa-Subedi

This comment has been minimized.

Show comment
Hide comment
@Bishwa-Subedi

Bishwa-Subedi May 4, 2017

Hi Sadler2,

Thanks for your reply.
I could not understand your workaround. My GPU machine is few hundreds kilometer away from me and I am accessing GPU machine through RDP. In order to run TransferLearning.py in GPU machine. I should create a RDP session between my laptop and remote machine.

How can I run any script on remote by closing an RDP session? Am I missing any fundamental point about RDP here?

Thanks a lot in advance.
Bishwa

Bishwa-Subedi commented May 4, 2017

Hi Sadler2,

Thanks for your reply.
I could not understand your workaround. My GPU machine is few hundreds kilometer away from me and I am accessing GPU machine through RDP. In order to run TransferLearning.py in GPU machine. I should create a RDP session between my laptop and remote machine.

How can I run any script on remote by closing an RDP session? Am I missing any fundamental point about RDP here?

Thanks a lot in advance.
Bishwa

@Sadler2

This comment has been minimized.

Show comment
Hide comment
@Sadler2

Sadler2 May 4, 2017

Well, it's as easy as ping 127.0.0.1 -n 10 && your_command, you will have around 10 seconds to close your remote session Not too elegant, but I've used it so many times with BOINC client, should work here too. Also you might think about replacing RDP with another remote desktop solution (e.g. VNC, TeamViewer).

Sadler2 commented May 4, 2017

Well, it's as easy as ping 127.0.0.1 -n 10 && your_command, you will have around 10 seconds to close your remote session Not too elegant, but I've used it so many times with BOINC client, should work here too. Also you might think about replacing RDP with another remote desktop solution (e.g. VNC, TeamViewer).

@Bishwa-Subedi

This comment has been minimized.

Show comment
Hide comment
@Bishwa-Subedi

Bishwa-Subedi May 5, 2017

Hi Sadler2,

Thanks for the solution, but still I am facing "Specified device id (0) is invalid".

Thanks,
Bishwa

Bishwa-Subedi commented May 5, 2017

Hi Sadler2,

Thanks for the solution, but still I am facing "Specified device id (0) is invalid".

Thanks,
Bishwa

@cha-zhang

This comment has been minimized.

Show comment
Hide comment
@cha-zhang

cha-zhang May 5, 2017

Member

One workaround is to use VNC, although I seem to be able to use RDP for GPU tasks without any problem. Maybe it's a driver issue?

Member

cha-zhang commented May 5, 2017

One workaround is to use VNC, although I seem to be able to use RDP for GPU tasks without any problem. Maybe it's a driver issue?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 11, 2017

@Bishwa-Subedi if you use RDP version 8.0+ on the host and client, GPU should be accessible without any issues via Remote Desktop.

  • For GTX cards, the reference working setup would be Windows 2012 R2 hosts and NVIDIA driver version 21.21.13.7290 (3.7230) or higher. You would need to configure each host's windows registry to eliminate TDR timeout for the GTX card.
  • Windows 7 (with updated RDP client), Windows 10 and Windows 2012 clients have access to the host and its GPU via Remote Desktop feature.

If you want to access the host through Remote Desktop Services/RDS - you would need Tesla card (TCC mode) as the GPU needs to be virtualized properly via RemoteFX/vGPU.

As other have mentioned, you can also try using VNC - which is similar to Remote Desktop (as opposed to Remote Desktop Services).

ghost commented May 11, 2017

@Bishwa-Subedi if you use RDP version 8.0+ on the host and client, GPU should be accessible without any issues via Remote Desktop.

  • For GTX cards, the reference working setup would be Windows 2012 R2 hosts and NVIDIA driver version 21.21.13.7290 (3.7230) or higher. You would need to configure each host's windows registry to eliminate TDR timeout for the GTX card.
  • Windows 7 (with updated RDP client), Windows 10 and Windows 2012 clients have access to the host and its GPU via Remote Desktop feature.

If you want to access the host through Remote Desktop Services/RDS - you would need Tesla card (TCC mode) as the GPU needs to be virtualized properly via RemoteFX/vGPU.

As other have mentioned, you can also try using VNC - which is similar to Remote Desktop (as opposed to Remote Desktop Services).

@Bishwa-Subedi

This comment has been minimized.

Show comment
Hide comment
@Bishwa-Subedi

Bishwa-Subedi May 11, 2017

Hi cha-zhang, avader906

Thanks a lot for your suggestion. I will try your solutions next week and let you know the result. I am not allowed to have any third party remote access tool except MS RDP :)

Once again, Thanks a lot for your time.

Bishwa

Bishwa-Subedi commented May 11, 2017

Hi cha-zhang, avader906

Thanks a lot for your suggestion. I will try your solutions next week and let you know the result. I am not allowed to have any third party remote access tool except MS RDP :)

Once again, Thanks a lot for your time.

Bishwa

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 11, 2017

@Bishwa-Subedi the easiest would be uninstall Remote Desktop Services Role/Feature from the server (you need exclusive access to GPU), check NVIDIA driver version and enable the simple Remote Desktop access. Then you would need to disable TDR timeout (documentation here) on the server via regedit.exe and modifying TdrLevel key to 0 (please read the documentation and disclaimer about editing registry settings). If you have the opportunity to spec - the manufacturer supported setup would be Tesla hardware (and not GTX cards).

ghost commented May 11, 2017

@Bishwa-Subedi the easiest would be uninstall Remote Desktop Services Role/Feature from the server (you need exclusive access to GPU), check NVIDIA driver version and enable the simple Remote Desktop access. Then you would need to disable TDR timeout (documentation here) on the server via regedit.exe and modifying TdrLevel key to 0 (please read the documentation and disclaimer about editing registry settings). If you have the opportunity to spec - the manufacturer supported setup would be Tesla hardware (and not GTX cards).

@wolfma61 wolfma61 closed this Jul 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment