PyCall does not close window correctly

[fplk]

I need to use Python bindings for a Unity 3D environment which provides explicit methods to start and end scenes. However, while the end_scene() method will close the opened window with the 3D environment under Python, when executed via PyCall from Julia the environment will just stay open. This is a major issue, since terminating the entire process and starting a new one would entail prohibitive overhead for thousands of scenes.

---

I have built a minimal example. Unfortunately, it requires three files:

• https://github.com/NextCenturyCorporation/MCS/releases/download/0.0.10/MCS-AI2-THOR-Unity-App-v0.0.10.x86_64
• https://github.com/NextCenturyCorporation/MCS/releases/download/0.0.10/MCS-AI2-THOR-Unity-App-v0.0.10_Data.tar.gz
• https://evaluation2-training-scenes.s3.amazonaws.com/interaction-scenes.zip

Note that these are from a well-known DARPA-affiliated company and the details are explained here, so there should be no security concerns about running it. [It is also fully public and I have explicit clearance to post the code below.] You need to make the first file executable (Linux binary!), extract the second file right next to it and finally extract the third archive somewhere. Afterwards please just adapt scene_path below to point to the file with the same name in extracted folder from the third file as well as mcs_executable_path to point to the executable from file 1.

Optional explanation: You do not need to point to the second file, since it is right next to the executable. If the executable MCS-AI2-THOR-Unity-App-v0.0.10.x86_64 is in a folder, the folder MCS-AI2-THOR-Unity-App-v0.0.10_Data extracted from file 2 needs to be right next to it in the same folder. That's all. [Also note that the solution is built in a convoluted way since that's what I had at hand - of course, you would do this in a single vcat() command in any production code.]

using PyCall
@pyimport machine_common_sense as mcs
MCS=mcs.MCS
MCS_Step_Output=mcs.MCS_Step_Output
MCS_Object=mcs.MCS_Object

pathlib = pyimport("pathlib")

solution = push!([push!([push!([push!([push!([push!([
      "RotateLook, rotation=-40"], "MoveAhead, amount=1.0",
      "MoveAhead, amount=0.5", "RotateLook, rotation=-90");
      repeat(["MoveAhead, amount=1.0"], 3)], "MoveAhead, amount=0.5");
      repeat(["RotateLook, rotation=-40"], 9);
      repeat(["MoveAhead, amount=1.0"], 4)], "RotateLook, rotation=-45");
      repeat(["MoveAhead, amount=1.0"], 8)], "RotateLook, rotation=37");
      repeat(["MoveAhead, amount=1.0"], 2)],
      "PickupObject, objectId=96c608b1-8b50-4dea-a12e-541a3204912a",
      "RotateLook, rotation=185"); repeat(["MoveAhead, amount=1.0"], 6)],
      "RotateLook, horizon=45", "DropObject, objectId=96c608b1-8b50-4dea-a12e-541a3204912a")

scene_path="/home/falk/mitibm/AI2Thor_MCS/dataset_eval2/interaction_scenes/transferral_goal-0772.json"
mcs_executable_path="/home/falk/mitibm/AI2Thor_MCS/MCS-AI2-THOR-Unity-App-v0.0.10.x86_64"
config_data, status = MCS.load_config_json_file(scene_path)
controller = MCS.create_controller(mcs_executable_path)
output = controller.start_scene(config_data)
for next_action in solution
      output = controller.step(next_action)
end
classification = "classification"
confidence = 1.0
controller.end_scene(classification, confidence)

As aforementioned, the last line should close the window and free the resources for that particular scene so I can start a second scene via controller.start_scene(config_data), but instead the window remains open and occupies system resources thus preventing me from executing the thousands of scenes I need to run.

Any ideas what could cause this or how I could solve this would be highly appreciated. Thank you in advance.

PS:
This is on Ubuntu 20.04.1 LTS (64 bit, 5.4.0-42-generic) with PyCall 1.91.4 and Julia 1.3.1.

[bzinberg]

I don't even see how calling end_scene directly from Python is closing the window. AFAICS, end_scene is implemented here, and the call to super().end_scene does nothing. I'm not seeing any foreign bindings being called. @fplk, do you know how it's happening?

[fplk]

The Python equivalent of the Julia snippet above would be

from machine_common_sense.mcs import MCS

scene_path="/home/falk/mitibm/AI2Thor_MCS/dataset_eval2/interaction_scenes/transferral_goal-0772.json"
mcs_executable_path="/home/falk/mitibm/AI2Thor_MCS/MCS-AI2-THOR-Unity-App-v0.0.10.x86_64"

solution = []
solution += ['RotateLook, rotation=-40'] + ['MoveAhead, amount=1.0'] + \
           ['MoveAhead, amount=0.5'] + ['RotateLook, rotation=-90'] + \
           ['MoveAhead, amount=1.0'] * 3 + ['MoveAhead, amount=0.5'] + \
           ['RotateLook, rotation=-40'] * 9 + ['MoveAhead, amount=1.0'] * 4 + \
           ['RotateLook, rotation=-45'] + ['MoveAhead, amount=1.0'] * 8 + \
           ['RotateLook, rotation=37'] + ['MoveAhead, amount=1.0'] * 2 + \
           ['PickupObject, objectId=96c608b1-8b50-4dea-a12e-541a3204912a'] + \
           ['RotateLook, rotation=185'] + ['MoveAhead, amount=1.0'] * 6 + \
           ['RotateLook, horizon=45'] + \
           ['DropObject, objectId=96c608b1-8b50-4dea-a12e-541a3204912a']

config_data, status = MCS.load_config_json_file(scene_path)
controller = MCS.create_controller(mcs_executable_path)
output = controller.start_scene(config_data)
for next_action in solution:
      output = controller.step(next_action)
classification = "classification"
confidence = 1.0
controller.end_scene(classification, confidence)

and that properly terminates the window.

However, I think @bzinberg's comment is very interesting - from reading the external code and stepping through it with a debugger it indeed seems like the method currently does nothing and just allows begin_scene to overwrite the scene which is in direct conflict to what I've been told (but I should have known better and checked for myself). I will for now close this issue, since it is i) not clear there is a bug in PyCall and ii) since this might provide a good solution to the problem. Thank you, Ben - very helpful. And apologies for filing a most likely unnecessary bug report.