Connecting issue with AirSim

[edimoon777]

I trained the model successfully and tried to run TestModel.ipynb.
but connection with Airsim is not works.

1. I changed MODEL_PATH with trained model.
   MODEL_PATH = 'model/models/model_model.26-0.0002362.h5'
   -> Runs well.

2. Run AD_Cookbook_Start_AirSim.ps1 using PowerShell.
   -> Runs well.

3. No further process during connecting to AirSim.
   Waiting for connection:
   -> Not works.

I've already tested examples of CNTK-AirSim (Drone, Car) and it works well in my laptop.
Test Environments
OS : Windows 10 64bit.
GPU : GTX1060 6GB
Anaconda3

[adshar]

Did you make sure AirSim was running in a separate window before trying to connect to it?

[edimoon777]

What does a separate window means? I’ve run Airsim and Jupyter notebook in a computer. Do you mean that it requires two computers to run Test code through network? Jungho Moon

…

On 8 Mar 2018, at 7:10 AM, Aditya Sharma ***@***.***> wrote: Did you run AirSim in a separate window before trying to connect to it? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[mitchellspryn]

1. Are you passing the 'landscape' argument to the ps script? It should be started with

.\AD_Cookbook_Start_AirSim.ps1 landscape -windowed

2. What is the output of
   netstat -a
   You should see a TCP connection on port 42451. If it's not there, then AirSim didn't start correctly. If, for some reason AirSim is starting on a different port, then you'll need to modify AirsimClient.py to point to the correct port (inside the constructor for CarClient).

[edimoon777]

I tried to run AirSim seperately with powershell and checked netstat.
but it does not connect to AirSim.

1. Run AirSim in a seperated windows -> success.
2. Checked TCP port -> normal
3. Waiting for connection: -> no further progress

-----------Powershell------
PS C:\AD_Cookbook_AirSim\AD_Cookbook_AirSim> .\AD_Cookbook_Start_AirSim.ps1 landscape -windowed
Configuring AirSim for scenario landscape...
Creating configuration JSON for scenario landscape...
Attempting to write configuration json to C:\Users\Jungho\Documents\AirSim\settings.json...
Configuration json successfully written.
Starting AirSim for scenario landscape...
Executing C:\AD_Cookbook_AirSim\AD_Cookbook_AirSim\Landscape_Neighborhood\JustAssets.exe landscape -WINDOWED...
PS C:\AD_Cookbook_AirSim\AD_Cookbook_AirSim>

----------- netstat -a -----------
...
TCP 127.0.0.1:42451 DESKTOP-ONVOC78:51845 ESTABLISHED
...
TCP 127.0.0.1:51845 DESKTOP-ONVOC78:42451 ESTABLISHED

[mitchellspryn]

Dumb question, but did you try running the rest of the cells? The cell should finish without any errors if the connection is successful. Usually, when a connection failure happens, a stack trace is printed. Is the cell freezing with the [*] forever, or does it get a number?

[edimoon777]

The cell gets a number. but It prints "Waiting for connection:" and remained.
I'll check next steps ignoring this cell.

[mitchellspryn]

If the cell gets a number (and doesn't print a big angry stack trace), then that means that it finished running and connected as expected. You can verify this by adding a print statement at the bottom of the cell.

[edimoon777]

At the last cell, errors came as below.
It seems to be related to CUDA and cuDNN.

---

2018-03-12 23:02:16.754965: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_blas.cc:443] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018-03-12 23:02:16.767295: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_blas.cc:443] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018-03-12 23:02:16.777790: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_blas.cc:443] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018-03-12 23:02:18.146084: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2018-03-12 23:02:18.156164: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018-03-12 23:02:18.166822: F C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

[shrimalmadhur]

@edimoon777 Looks like a similar issue which has been discussed here. There are some solutions mentioned there. Can you try those?

[mitchellspryn]

Is this issue solved?

[edimoon777]

It is not solved yet. But I think you can close this issue related with airsim connection. It is another problem related with CuDNN. I’ll test it using various versions of cuDNN. Can you tell me versions of cuDNN and CUDA which are used for testing? Jungho Moon

…

On 19 Mar 2018, at 7:48 AM, Mitchell Spryn ***@***.***> wrote: Is this issue solved? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mitchellspryn]

Ok.

I have CUDA 9.1 and CuDNN 7.1.1.

[edimoon777]

Thank you everybody. I solved this issue.
"waiting for connection" message remains even when the connection is normal.
so I suggest you to add a message such as 'Connection successful' if it's connected. (It is confusing)

I wrote a new issue that I fixed about GPU memory errors.

[NotAnyMike]

Based on this, the problem of failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED and Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED is because tf is using all GPU memory, you can avoid it by allocating memory dynamically, the following code will make tf use only the memory it needs

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True  # dynamically grow the memory used on the GPU

sess = tf.Session(config=config)
set_session(sess)  # set this TensorFlow session as the default session for Keras

That solved the problem for me

[yosra5]

whene i running tensorflow-gpu with airsim i haved this error :
ValueError: cannot reshape array of size 1 into shape (0,0)
how can i solve it ?