Unity Connection time out!

[simonjeger]

I'm using flightmare on branch "dev/version_22" (2.5.2022) on Ubuntu 18.04 with Unity 2020.3.36f1.

I can train a policy with "python3 run_control_demo.py --train 1" without any problems.
Now I want to test and visualize it. I press play on unity, type "python3 run_control_demo.py --train 0 --render 1" and the following things happen:

• The python script tries to connect to unity but doesn't succeed, eventually saying "Unity Connection time out!".
• On Unity, the splash screen gets deleted, but other than that everything stays exactly the same (Camera moving around as before). It also throws an error "NullReferenceException: Object reference not set to an instance of an object" (presumably because the splash screen is not there anymore).

I managed to run all of this successfully your master branch, so I think I'm using the right commands.

[simonjeger]

In the "dev/version_22" branch different type of objects (dynamic and static objects) can be passed to flightmare_unity (see flightmare/flightlib/src/envs/vision_env/vision_env.cpp, line 67-82) and that's not yet implemented on the flightmare_unity "master" branch.

So I propose to use flightmare_unity "dev/agile_flight" branch instead. Now if you don't have any static objects in a .csv file, you'll still get an error. I therefore adapted the following lines to catch that case (not sure if that's a good fix but it worked for me):

In flightmare_unity/Asset/Flightmare/Script/CameraController.cs replace the following (line 706 - 722)

void instantiateObjects()
    {
      config_object.ReadCSVFile(internal_state, settings.object_csv, ListToVector3(settings.render_offset));
      // 
      // Initialize additional objects
      foreach (var obj_state in settings.static_objects)
      {
        GameObject prefab = Resources.Load(obj_state.prefabID) as GameObject;
        // Debug.Log("obj_state id : " + obj_state.ID);
        // GameObject obj = internal_state.getGameobject(obj_state.ID, gate_template); 
        GameObject obj = internal_state.getGameobject(obj_state.ID, prefab);
        obj.transform.localScale = ListToVector3(obj_state.size);
        Vector3 obj_position = ListToVector3(obj_state.position) + ListToVector3(settings.render_offset);
        obj.transform.SetPositionAndRotation(obj_position, ListToQuaternion(obj_state.rotation));
        obj.transform.localScale = ListToVector3(obj_state.size);
        // obj.layer = 9;
      }

with

void instantiateObjects()
    {
      try
      {
        config_object.ReadCSVFile(internal_state, settings.object_csv, ListToVector3(settings.render_offset));
        // 
        // Initialize additional objects
        foreach (var obj_state in settings.static_objects)
        {
          GameObject prefab = Resources.Load(obj_state.prefabID) as GameObject;
          // Debug.Log("obj_state id : " + obj_state.ID);
          // GameObject obj = internal_state.getGameobject(obj_state.ID, gate_template); 
          GameObject obj = internal_state.getGameobject(obj_state.ID, prefab);
          obj.transform.localScale = ListToVector3(obj_state.size);
          Vector3 obj_position = ListToVector3(obj_state.position) + ListToVector3(settings.render_offset);
          obj.transform.SetPositionAndRotation(obj_position, ListToQuaternion(obj_state.rotation));
          obj.transform.localScale = ListToVector3(obj_state.size);
          // obj.layer = 9;
        }
      }
      catch (Exception)
      {
        print("No static objects found");
      }
    }

[RibhavOjha]

Hi
Which directory did you put the "flightmare_unity" repo?

[simonjeger]

I don't think it matters where you put the flightmare_unity folder. You just need to click play when you run flightmare.

[RibhavOjha]

I am also in the dev/version_22 branch to run the RL example.
I have trained the RL model, and wanted to visualise it user --render 1.

Flightmare has this in their documentation:
"
To use unity rendering, you need first download the binary from Releases and extract it into the flightrender folder. To enable unity for visualization, double click the extracted executable file RPG_Flightmare.x84-64 and then test a pre-trained controller
"

I double click the RPG_Flightmare.x84-64 in the directory: flightmare/flightrender/RPG_Flightmare/RPG_Flightmare.x84-64. I get this:

After that, i go to flightpy/flightrl and run python3 run_control_demo.py --train 0 --render 1

It just says unity connection timed out.
Could you please tell me what might be wrong here? You pointed out that it is something to do with flightmare_unity, but where are we even using it?

Thank you

[RibhavOjha]

Hey, I cloned flightmare_unity, built everything from scratch from https://github.com/uzh-rpg/flightmare/blob/master/docs/source/building_flightmare_binary/standalone.rst
And then followed steps as outlined by you. I can see the drone on unity now.

Thanks!