MeshAgent keeps crashing

[PathfinderNetworks]

Something is going on with the current MeshAgent continually crashing in Windows- at least on Windows 10 (including Server2016/2019). I haven't tested others OS's yet.

I noticed this happening yesterday but thought it was just the Internet connection on the remote end so didn't dig in to it. This evening I'm working on a Windows 10 desktop that has a fast fiber internet connection. What I'm seeing is the desktop image will freeze (on the browser side of thing) for a few seconds. Then it will start working again. So I looked in the Event Logs on the remote device and can see that the MeshAgent is crashing every few minutes while I'm remotely connected. This is what is causing the apparent freeze I'm seeing.

There are two Application log entries every time it crashes. The first is below:
Faulting application name: MeshAgent.exe, version: 0.2.1.3, time stamp: 0x5d85151d
Faulting module name: MeshAgent.exe, version: 0.2.1.3, time stamp: 0x5d85151d
Exception code: 0xc0000005
Fault offset: 0x000000000000a90f
Faulting process id: 0x1e1c
Faulting application start time: 0x01d57e35bd2c7c84
Faulting application path: C:\Program Files\Mesh Agent\MeshAgent.exe
Faulting module path: C:\Program Files\Mesh Agent\MeshAgent.exe
Report Id: fc7db4d9-cd3b-4328-ada2-9eaa5fc2cf20
Faulting package full name:
Faulting package-relative application ID:

Then there is Information entry at the same time:
Fault bucket 1760899226197263994, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: MeshAgent.exe
P2: 0.2.1.3
P3: 5d85151d
P4: MeshAgent.exe
P5: 0.2.1.3
P6: 5d85151d
P7: c0000005
P8: 000000000000a90f
P9:
P10:

Attached files:
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB3C8.tmp.dmp
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB437.tmp.WERInternalMetadata.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB457.tmp.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB455.tmp.csv
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERB485.tmp.txt

These files may be available here:
\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_MeshAgent.exe_deef26c3db96e166f2427be6c56a9dab1e8ba_065bce66_647cf10d-3765-4f9b-a42c-2dbfb775bd10

Analysis symbol:
Rechecking for solution: 0
Report Id: fc7db4d9-cd3b-4328-ada2-9eaa5fc2cf20
Report Status: 268435456
Hashed bucket: 0e79a4ca93909def686ff8843ebf867a
Cab Guid: 0

Do you want/need me to grab those dump files?

[krayon007]

The dump file might be large, but it will definitely help with trying to debug the crash. You may want to upload it somewhere and email me the link directly instead of posting the dump here, because the dump contains a memory dump of the process.

[PathfinderNetworks]

I just sent you a zip file via email. There is no .dmp file (for some reason) in any of crash reports in the WER folder. So I sent you what was there- which is just a couple small text based files.

[krayon007]

Is your remote desktop connection freezing and unfreezing? That might explain why the dump file isn't there, if it's actually the child KVM process that is crashing, and not the main agent process. Only the parent process creates a dump file....

Currently if the child KVM process crashes or exits, the parent just spawns a new child process (and switches permissions each time), and reattaches the pipes. It does this, because currently, this behavior is expected when you login and logoff, because the method call to scrape the screen will produce an access violation due to permissions, because the agent process doesn't have UI rights, and the user process doesn't have rights to scrape the logon screen.

I'll see if I can tweek a few things so I can generate a dump from the child KVM process if it unexpectedly crashes.

[PathfinderNetworks]

Yes, this is exactly what is happening. The connection freezes (but doesn't appear to disconnect on my browser end) then starts working again a few seconds later.

[PathfinderNetworks]

Hi Bryan and Ylian,
Has the new agent to address this been released? I hadn't seen any notice if it had been- other than it being delayed last week. I know Ylian is going to be traveling the next few weeks so just wanted to know when it was going to be available (if it hasn't already been released).

Thanks!

[MailYouLater]

I believe the delay is because Ylianst isn't comfortable with releasing a new (relatively untested) agent right before he leaves for a while, as then if there were issues with it, he wouldn't be around to fix them, therefore I wouldn't expect the new release until after he's returned in a few weeks. Do note however, that krayon007 will still be around to work on the agent, and he mentioned that the new version is available in the MeshAgent GitHub repo. I don't see any binaries released over there, and I'm not sure how exactly to go about building the new version myself, but I'm sure if you wanted to help test the new version before it gets released, Ylianst and krayon007 wouldn't mind getting any feedback you have about it here (regarding if it fixes this issue for you) or at the MeshAgent repo (if you find other issues with it).

@krayon007 / @Ylianst, any chance you could give a basic rundown of how someone could build the new version to help test it before release?

[krayon007]

If you guys are interested in helping us test the agent, I'd be happy to compile everything and post a link or something. I spent a few days and tracked down some issues that caused Windows Agent crashes, when an object was garbage collected. I'll compile various binaries, and post a link somewhere with instructions of how they can be used for testing. I'm not sure if you guys are planning on testing individual client machines, or if you guys have a test server setup, where you could just drop in the binary, and have it auto update the connected agents. I suppose the easiest way to distribute the test binaries would be to just checkin the binaries into the repo, into a test folder or something.

[MailYouLater]

If you just want a place to store the files to link them in a comment on a GitHub issue, you can actually just put them in a .zip file and drag & drop them into your comment. (It won't accept a .exe unless you put it in a .zip.) Seems to me like that would probably be the easiest, and won't also pollute your git repository with binaries (which git isn't really very good at handling).

[PathfinderNetworks]

Any updates on this? The agent is still crashing (and driving me pretty much absolutely insane the last few days). Like right now, I'm trying to remotely work on a device and the meshagent keeps crashing roughly every 2 or 3 minutes. The desktop connection will go black. Maybe half the time it will automatically reconnect and show the desktop. The other times I have to disconnect then reconnect to see the desktop again. It seems to be happening mostly when I'm moving the mouse cursor around (but not certain- in this case I was trying to move a bunch of files from one folder to another as a drag and drop procedure).
It's getting to the point where it's becoming unacceptable as far as my work is concerned.

[krayon007]

Is your agent configured to generate a dump file? The agent update after this thread was created includes a fix, so that the child kvm process can also generate a dump file. I think in your case, it is the child process that is crashing, not the parent process, which is why your original attempt to generate a dump file failed.

[MailYouLater]

Good question. For reference, to enable generating a dump file, you add a line that says coreDumpEnabled=1 to the .msh file and restart the agent.

[PathfinderNetworks]

Ok, I was able to get a device to generate a dmp file by adding the coreDumpEanbled=1 command to the meshagent.msh file. It didn't take long (maybe a few minutes) for it to crash and generate the MeshAgent_kvm.dmp file. Hopefully this will help to get this resolved. I'm seeing this constant crashing on every device I work on. Bryan, I am going to email you a zip that contains the crash dump as well as the .wer file Windows generated. Let me know if you need more examples.

[krayon007]

@PathfinderNetworks , do you happen to have a little context on what was going on when it crashes? I ask, becuase I found where it crashed based on your dumps. They are both in the same exact place... It's because in the method where it was scraping the screen, I found a bug where some edge case error conditions were accidentally returning success instead of failure, which caused a NULL reference exception, which is why it crashed...

I fixed it, so it will correctly return an error, however this won't really fix your stuttering problem, because all this will do is cause the child process to exit gracefully, at which point the parent will try to respawn the process... I need to figure out why win32 is failing to initialize the graphics context after 2-3 minutes, as you say... So far, the only conditions I've seen where this would happen, is if you logout/logon or if the system goes to sleep...

[PathfinderNetworks]

It seems to happen as I'm moving the mouse around. I don't seem to notice it if I'm just entering text via the keyboard or just watching the remote screen. So maybe something to do with how it locates/reports the mouse location?

[krayon007]

Interesting. I'll take a look at the code for detecting the mouse cursor. Unlike Linux where I get an event on cursor change, on windows I have to poll for it, so I simply fetch it whenever new mouse coordinates are injected.

[anishsalpy]

Hi , Mesh Agent crashes in windows 7 32 bit and 64 bit systems every 4-5 minutes intervel. I am using Symantec endpoint protection 14.x version. After disable the AV the crashes stops and it's working fine. I have updated the exception in AV but the issue still persist. Symantec team also checking the issue more than a month.
So please help me to get the solution.

[anishsalpy]

Any update Mr.developer.

[btcsupport]

Hi,
I’m using latest version of Meshcentral (branch 0_9_16).
I found the similar error in the event log.
What method should I use to solve this issue?

Faulting application name: MeshAgent.exe, version: 0.2.1.3, time stamp: 0x612d4568 Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000 Exception code: 0xc0000005 Fault offset: 0x00dbf918 Faulting process id: 0x169c Faulting application start time: 0x01d7bf63fedf90d6 Faulting application path: C:\Program Files (x86)\Mesh Agent\MeshAgent.exe Faulting module path: unknown Report Id: 50055802-02a1-4c5a-ae46-d0c182936b26 Faulting package full name: Faulting package-relative application ID:

[dinger1986]

seems fine now @si458

[si458]

closing as stale