Server Crash on "quit"

[MAGNAT2645]

This extensions crashes my Linux server (local on Arch) when i run quit (via LinuxGSM's tf2server stop) to stop the server.
There are no any errors or dumps when this happens. Accelerator doesn't seem to notice this kind of crash because it happens on SDK unload, after these lines:

SDK unload
metamod SDK unload

Then all i see in logs is the message about server restarting in 10 seconds.

And server crashes even if i don't use extension features in any plugin, the extension just need to be loaded to crash.
But it doesn't crash if i unload it via sm exts unload though.

[MAGNAT2645]

So it actually seems that this crash could happen due to these:
JetPack Fix (but last time I checked with it there wasn't such crash and actually idk if I still should use this ext)
source-vehicles
Slender Fortress Modified Versions
One or two of my plugins which utilize CBaseNPC's CEntityFactory to create new entities.

Without these the server stops fine after almost 2 seconds (via LinuxGSM).

[Kenzzer]

JetPack_Fix has been merged into live tf2, it's safe to remove.

Source-vehicles is unlikely the source of your crash. Try to isolate the root cause, by running either Slender or Source-Vehicles only.

[MAGNAT2645]

They are not running together at the same time. My GameMode Manager disables plugins of the current gamemode and enables plugins of the next gamemode in OnMapEnd (before switching to the next map).
So if I'm switching from, for example, a trade map to slender_ map, it will unload and move to disabled all listed plugins in my config (including vehicles) and will load all disabled plugins needed for SF2.
Crash happens if either Vehicles are running or SF2.
But I will try running these without any other plugins (excluding standard SM ones and including my own plugins) and probably extensions. It will require a lot of server restarts though 😢

[Kenzzer]

Those could be still different crash then. It'd be tremendously useful if you could somehow generate a minidump.

[MAGNAT2645]

Here's what I found out:
If I run these standard plugins and extensions with vehicles.smx included

It will throw these errors

L 06/26/2024 - 12:21:48: [SM] Unable to load extension "madt.ext": /home/tf2server/serverfiles/tf/addons/sourcemod/extensions/madt.ext.so: cannot open shared object file: No such file or directory
L 06/26/2024 - 12:21:48: [SM] Unable to load extension "SteamWorks.ext": /home/tf2server/serverfiles/tf/addons/sourcemod/extensions/SteamWorks.ext.so: cannot open shared object file: No such file or directory
L 06/26/2024 - 12:21:48: [SM] Unable to load plugin "vehicles.smx": Required extension "More ADTs" file("madt.ext") not running
L 06/26/2024 - 12:21:48: [SM] Unable to load plugin "updater.smx": Required extension "SteamWorks" file("SteamWorks.ext") not running

indicating that vehicles could not load but if I try to stop the server it still crashes.
When I moved vehicles.smx to disabled then started and stopped the server again, it stopped fine without crashing.
But one thing I noticed is that I use modified version of vehicles plugin and these are all of the changes I made to it:

#undef REQUIRE_EXTENSIONS
#tryinclude <loadsoundscript>
+#tryinclude <cbasenpc>
#define REQUIRE_EXTENSIONS

+#if defined _CBASENPC_EXTENSION_INC_
+	#undef REQUIRE_PLUGIN
+	#define DONT_DEFINE_ENTITY_EFFECTS
+	#tryinclude <aon/store-inventory>
+	#define REQUIRE_PLUGIN
+#endif

...

public void OnLibraryAdded(const char[] name)
{
	if (StrEqual(name, "LoadSoundscript"))
	{
		g_LoadSoundscript = true;
+		return;
	}
+#if defined _aon_store_inventory_included
+	if ( StrEqual( name, "aon_store_inventory" ) )
+	{
+		Store_SetEntClassNameAnnotationTitleHandler( VEHICLE_CLASSNAME, VehicleAnnotationHandler );
+		ItemAnnotationAction action = new ItemAnnotationAction( "Enter/Exit", VehicleAnnotationHandler );
+		Store_AddAnnotationActionToEntClassName( VEHICLE_CLASSNAME, action );
+	}
+#endif
}

public void OnLibraryRemoved(const char[] name)
{
	if (StrEqual(name, "LoadSoundscript"))
	{
		g_LoadSoundscript = false;
	}
}

+#if defined _aon_store_inventory_included
+Action VehicleAnnotationHandler(
+    StorePlayer player,
+    CBaseEntity entity,
+    ItemAnnotationAction action,
+    ItemAnnotationActionCBType nCallbackType,
+    char[] szBuffer,
+    int iMaxLen)
+{
+	switch ( nCallbackType )
+	{
+		case ItemAnnotationAction_DisplayTitle:
+		{
+			int iEnt = entity.index;
+			VehicleConfig pConfig;
+			if ( GetConfigByVehicleEnt( iEnt, pConfig ) )
+			{
+				if ( TranslationPhraseExists( pConfig.name ) )
+				{
+					FormatEx( szBuffer, iMaxLen, "%T", pConfig.name, player );
+				}
+				else
+				{
+					strcopy( szBuffer, iMaxLen, pConfig.id );
+				}
+			}
+			else
+			{
+				strcopy( szBuffer, iMaxLen, VEHICLE_CLASSNAME );
+			}
+
+			int iDriver = entity.GetPropEnt( Prop_Data, "m_hPlayer" );
+			if ( iDriver != -1 )
+			{
+				Format( szBuffer, iMaxLen, "%s\n%T", szBuffer, "#AnnotationAction_DriverName", player, iDriver );
+			}
+		}
+		case ItemAnnotationAction_Draw:
+		{
+			int iClient = player.Index;
+			if ( !IsInAVehicle( iClient ) && !CanEnterVehicle( iClient, entity.index ) )
+			{
+				return Plugin_Handled;
+			}
+		}
+		case ItemAnnotationAction_Display:
+		{
+			int iClient = player.Index;
+			FormatEx( szBuffer, iMaxLen, "%T", IsInAVehicle( iClient )
+				? "#AnnotationAction_Exit" : "#AnnotationAction_Enter", iClient );
+			return Plugin_Changed;
+		}
+	}
+	return Plugin_Continue;
+}
+#endif

Since aon_store_inventory never gets loaded as it's disabled, these changes should not affect the plugin at all.
If I use the last version of vehicles from GitHub it does not crash.
The SF2M (modified versions are using CBaseNPC if I remember correctly) also crashes the server without any other plugins being loaded.
So it seems that this crash happens to every plugin that has included CBaseNPC.

[MAGNAT2645]

That's so weird

[Kenzzer]

That makes your issue more clear. Now we can clearly establish a link to CBaseNPC, however without any minidump I'm afraid this issue will remain in limbo for another good while.

[MAGNAT2645]

But I really can't get any minidump when the server is stopping it does not seem to track any crash to generate the dump for.

[Kenzzer]

You'll need to attach a debugger to the srcds process, so that it can outlive metamod and sourcemod. And then generate a minidump from that. This will require familiarity with debuggers however.

Alternatively, if you can create the absolute minimal reproduction case, no extra extensions, no extra plugins, and transmit it as a zip file. I could investigate this on my own.

[MAGNAT2645]

So the gdb thrown this:

Thread 1 "srcds_linux" received signal SIGSEGV, Segmentation fault.
0xe1b2be39 in CBaseNPCExt::SDK_OnUnload (this=0xe1c12e14 <g_CBaseNPCExt>) at /home/runner/work/CBaseNPC/CBaseNPC/CBaseNPC/extension/extension.cpp:293
293	/home/runner/work/CBaseNPC/CBaseNPC/CBaseNPC/extension/extension.cpp: No such file or directory.
(gdb) continue
Continuing.

Thread 1 "srcds_linux" received signal SIGSEGV, Segmentation fault.

Idk if this is the line it's talking about:

	if (g_pSDKHooks) {
		g_pSDKHooks->RemoveEntityListener(this);
	}

[Kenzzer]

Definitively dangling pointers

CBaseNPC/extension/extension.cpp

Lines 260 to 274 in b5dd18a

	bool CBaseNPCExt::QueryInterfaceDrop(SMInterface* interface) {
	if (interface == g_pBinTools) {
	return false;
	}
	if (interface == g_pSDKHooks) {
	return false;
	}
	if (interface == g_pSDKTools) {
	return false;
	}
	return IExtensionInterface::QueryInterfaceDrop(interface);
	}

They should set back to nullptr here. I'm surprised the dependency system doesn't catch this

CBaseNPC/extension/extension.cpp

Lines 107 to 109 in b5dd18a

	sharesys->AddDependency(myself, "bintools.ext", true, true);
	sharesys->AddDependency(myself, "sdktools.ext", true, true);
	sharesys->AddDependency(myself, "sdkhooks.ext", true, true);

In any case, if you feel up to it. You can compile the extension yourself with the fix I mentioned, otherwise I'll open a pull request later today.

[MAGNAT2645]

Did you mean directly setting the pointer to null like this?

 	if (interface == g_pSDKHooks) { 
 	        g_pSDKHooks = nullptr;
 		return false; 
 	} 

And should I do the same with other interface pointers?

[MAGNAT2645]

Yeah, it seems that this fixed the crash. My problem was that I first built the ext on my Arch (btw) and as it turned out I have a little higher GLIBC version (2.38) than the one on my Ubuntu server (2.35). So I had to clone all of the stuff (SDK, SM, MM:S, AMBuild) and install some new stuff (Clang is one of them) on my server. Then I forgor to apply the fix and thought it didn't work 😆

[MAGNAT2645]

I will enable all of the disabled plugins to see if it's not crashing anymore and if it's fine I'll close this issue.

[MAGNAT2645]

Yeah, I have enabled all of the problematic plugins which use CBaseNPC. Now my server stops correctly.

[Kenzzer]

Thank you for all your troubleshooting, and field testing the fix. Feel free to PR in the fix !