AirSim causes Heap Corruption exceptions with Unreal in HITL mode

[MSBGit]

When I try to run an Unreal environment with AirSim in HITL mode, the VS2019 debugger throws two exceptions: Source Not Available followed by Heap Corruption. Sometimes I don't get the Source Not Available exception and I see the actual source of where the exception occurred. I have included screenshots for both cases here. Caveat: I am not a Windows developer, I am an embedded and Linux developer, so VS2019 is like black magic to me and my expertise with it is relegated to pushing buttons following online build instructions.

Environment:

• AirSim 1.3.1 (previous comment said: 1.2.0 Beta which was based on incorrect version info in the Unreal plugin file) in HITL mode with a Pixhawk 4 running PX4 1.10.1
• Unreal 4.24
• Windows 10
• Blocks environment (from AirSim repo)

Steps to reproduce:

1. I plug my Pixhawk into my computer via USB.
2. I open the sln file in VS2019, select the environment as the startup project, select DebugGameEditor and Win64 mode.
3. Press F5 to start the build.
4. The Unreal Editor comes up with the Blocks environment loaded.
5. I press Play and VS2019 will shortly breakpoint and I will either see Source Not Found followed by Unhandled Exception: Heap Corruption after I press Continue or I will see Exception Thrown: Breakpoint followed by Unhandled Exception: Heap Corruption. I suspect these two are identical, one is just that VS2019 can't find the source for some reason. It's troubling to me that my toolchain is producing inconsistent results on something as simple as finding the source code when debugging.

Pressing Continue allows the simulation to continue and then it seems to run well. However, this bug causes me to be unable to run the simulation outside of the VS2019 debugger.

If I do not plug in the Pixhawk and start the simulation, I do not get these exceptions. If I then plug in the Pixhawk while the simulation is running, I will see a message that AirSim has connected to the Pixhawk but HITL doesn't work since the Pixhawk doesn't get GPS data from AirSim (that's OK since I don't expect that AirSim would support hot plugging a flight controller while the simulation is running). If I then stop the simulation and restart it, I will get the same set of exceptions as above.

[madratman]

1.2.0 is way too old. Current airsim master and release (1.3.1) works with px4 1.10.1

[MSBGit]

How do I tell if the version that Unreal Editor plugin manager is telling me I have installed is actually correct? I cloned the AirSim repo earlier today and built master then dropped the built AirSim plugin bits into my own custom environment, and in both my environment and in Blocks I see 1.2.0 Beta. So I wonder if the plugin info for Unreal hasn't been updated (but the actual plugin bits are current) or if I'm doing something wrong when I copy the plugin into my environment.

[MSBGit]

Looking at the AirSim.uplugin file in master, I see Version 1.2.0:

{
  "FileVersion" : 3,
  "Version" : "1.2.0",
  "VersionName": "1.2.0",
...

and cloning the repo again and doing all the build steps from scratch as described in the AirSim docs, I still see Version 1.2.0 Beta in Unreal Editor in the Blocks environment. The repo itself has very recent commits so I'm assuming it's the correct repo to use.

So now I'm confused. Is it just the case that the version info in the AirSim.uplugin file needs to be updated?

[saihv]

If you've cloned master, you should have the latest bits - it does look like the .uplugin needs to be updated.

Regarding your actual issue, could you perhaps post the output log from VS? Bottom right in the debugger, Output tab. Upon first looks, it seems like the exception is happening in the joystick part of code. Also, I find this interesting:

Pressing Continue allows the simulation to continue and then it seems to run well. However, this bug causes me to be unable to run the simulation outside of the VS2019 debugger.

So the code throws an unhandled exception and keeps going? And what exactly is the runtime error you get when you run this outside of the debugger?

[MSBGit]

@saihv I attached a zipfile with logs from both the standard Blocks environment and my custom Oceans environment. Both environments behave the same way with respect to the crash at simulation startup.

I wrote:

Pressing Continue allows the simulation to continue and then it seems to run well. However, this bug causes me to be unable to run the simulation outside of the VS2019 debugger.

So the code throws an unhandled exception and keeps going?

Yes when I press CONTINUE in VS2019 the second time, the simulation seems to run fine from that pint on. I have run it for a few hours without any issue that I could see,

And what exactly is the runtime error you get when you run this outside of the debugger?

Double-clicking on Blocks Unreal Engine project file:

I get a popup that says:

I press Yes, Unreal builds a bunch of stuff and then Unreal Editor 4.24 starts up with Blocks loaded. I press PLAY, I see the Windows blue circle spinning for a while and then the whole thing dies without any additional error messages (so perhaps it's hitting the unhanded exception that the debugger catches).

Selecting Launch game on Blocks Unreal Engine project file:

I get a popup that says:

I press Yes, then after building, I get the Blocks banner and then the simulation starts. What happens at this point is what I think is a very important clue:

If I have just plugged in the Pixhawk prior to doing the Launch game, I'll get a black screen for about 5 seconds and then the simulation will die.

If I have used the Pixhawk with the simulator (even in the case above where the simulator dies in the previous attempt), the simulation will frequently run, sometimes correctly, sometimes with the drone just spinning end over end for a bit before I use my R/C controller to try to get it to fly correctly.

To be clear about this - in all three cases (VS2019 DebugGame mode, UE Editor mode, Launch game mode), the behaviour of the simulator is pretty uch the same with respect to how it behaves if the Pixhwak has just been plugged in or if it has been used with the simulator in a previous session.

So I am suspecting that perhaps the HITL code isn't properly initializing the Pixhawk in some way or that on the first communication with a just-booted Pixhawk there might be some MAVLINK messages that are causing the AirSim code to corrupt the heap. The InitDirectInput() references in the crash might be a reg herring where the heap has been corrupted some time before that call.

Contents of my Blocks Unreal Engine project file:

{
	"FileVersion": 3,
	"EngineAssociation": "4.24",
	"Category": "",
	"Description": "",
	"Modules": [
		{
			"Name": "Blocks",
			"Type": "Runtime",
			"LoadingPhase": "Default",
			"AdditionalDependencies": [
				"AirSim"
			]
		}
	],
	"Plugins": [
		{
			"Name": "AirSim",
			"Enabled": true
		}
	]
}

AirSim Heap Corruption #2583.zip

Blocks UE Project.zip

[MSBGit]

I went back to an older version of Blocks and Unreal 4.22 and confirmed that the crash does not happen in that configuration (still HITL with Pixhawk 4 and PX4 1.10.1). I hope this is helpful and I'll also compare the AirSim sources in what might be likely areas to see what's changed between then and now.

AirSim>git show
commit 2450acfea564da1e426a5a5152ce4881ae76ccbb (HEAD -> master, origin/master, origin/HEAD)
Merge: 9625639f 613385b4
Author: msb336 <32399587+msb336@users.noreply.github.com>
Date:   Wed Jun 26 09:12:21 2019 -0700
    Merge pull request #2015 from xmyqsh/patch-1
    fix/add_missing_return

[lovettchris]

Can you try this PR and see if that fixes the heap corruption?
#2634

[MSBGit]

Can you try this PR and see if that fixes the heap corruption?
#2634

I'll try this on Friday. Thanks!

[MSBGit]

Can you try this PR and see if that fixes the heap corruption?
#2634

My initial testing (Pixhawk.PX4 HITL, AirSim latest master version, Unreal 4.24.3, VS2019 used for everything) shows that the heap corruption issue isn't occurring anymore - thanks! I'll be doing more testing over the weekend and update this issue on Monday.

[lovettchris]

Great, thanks for trying it.