Game crash (on Linux) TM:PE V11 STABLE

[darthroe]

Temporary solution

• Use latest development build or TEST workshop version

Describe the problem

I run Cities:Skylines on LInux (Manjaro),
Hardware if it matters: AMD FX 8370, Nvidia GeForce 750 Ti (Diver 440.64)
When I start a saved city everything is fine until I un-pause it, then it will either freeze or crash. When I unsubscribe to this mod the city will load and run.

Steps to reproduce

1. Load saved city.
2. un-pause game

Log files

https://ufile.io/9vld1vi2

Savegame?

https://ufile.io/fyy8h170

Screenshots?

None

Notes or questions?

[AllegedlyLuca]

You haven't provided the TMPE.log, which might contain more information we need. Could you upload that somewhere?

This were two stacktraces I was able to fine. The first is near identical to a different error someone else reported on the Discord server back on Tuesday. The second I haven't seen before but seems unrelated to TMPE.

AchievemenStacktrace:


Native stacktrace:

	/home/darthroe/.local/share/Steam/steamapps/common/Cities_Skylines/Cities_Data/Mono/x86_64/libmono.so(+0x91f55) [0x7f458c5e1f55]
	/usr/lib/libpthread.so.0(+0x147ff) [0x7f4592ee47ff]
	/usr/lib/libc.so.6(gsignal+0x145) [0x7f45929dbce5]
	/usr/lib/libc.so.6(abort+0x12a) [0x7f45929c5856]
	./Cities.x64() [0x9f864b]
	/home/darthroe/.local/share/Steam/steamapps/common/Cities_Skylines/Cities_Data/Mono/x86_64/libmono.so(+0xd1086) [0x7f458c621086]
	/home/darthroe/.local/share/Steam/steamapps/common/Cities_Skylines/Cities_Data/Mono/x86_64/libmono.so(+0x34c3d) [0x7f458c584c3d]
	/usr/lib/libpthread.so.0(+0x147ff) [0x7f4592ee47ff]
	[0x410a1f49]

Debug info from gdb:

ERROR: ld.so: object '/home/darthroe/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
I refuse to debug myself!
No threads.

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

The LD_PRELOAD error is reportedly quite common with Steam games running on Linux and is usually safe to ignore. Still don't know anything about the stacktrace or the reason for SIGABRT yet though.

System.SystemException: Error running gmcs: Cannot find the specified file
  at Mono.CSharp.CSharpCodeCompiler.CompileFromFileBatch (System.CodeDom.Compiler.CompilerParameters options, System.String[] fileNames) [0x00000] in <filename unknown>:0 
  at Mono.CSharp.CSharpCodeCompiler.CompileAssemblyFromFileBatch (System.CodeDom.Compiler.CompilerParameters options, System.String[] fileNames) [0x00000] in <filename unknown>:0 
  at Mono.CSharp.CSharpCodeCompiler.CompileAssemblyFromFile (System.CodeDom.Compiler.CompilerParameters options, System.String fileName) [0x00000] in <filename unknown>:0 
  at System.Xml.Serialization.XmlSerializer.GenerateSerializers (System.Xml.Serialization.GenerationBatch batch, System.CodeDom.Compiler.CompilerParameters cp) [0x00000] in <filename unknown>:0 
  at System.Xml.Serialization.XmlSerializer.RunSerializerGeneration (System.Object obj) [0x00000] in <filename unknown>:0 
Unknown prefab: Cosmopolitan Portland  [Serialization]

This second error seems related to a custom asset.

Could you verify the platform you're using to play the game and the game version you're using? I figure you're on Steam, but I cannot find references to the game version in your log.

[krzychu124]

Hi, I suggest disabling Steam game overlay, to get rid of errors. Maybe that the cause??

[originalfoo]

System.SystemException: Error running gmcs: Cannot find the specified file
  at Mono.CSharp.CSharpCodeCompiler.CompileFromFileBatch ...

That might be caused by some mod that bundles it source code. Look in your workshop folders (on Windows they're in Steam\steamapps\workshop\content\255710 so probably something similar on Linux) and see if there are any Source folders buried in there and unsubscribe associated items.

[darthroe]

Okay, so I disabled Steam game overlay, but that doesn't seem to be the cause. In fact I'm not entirely sure TM:PE is the cause. I say this because I went through my log and removed some assets that were causing errors. I also cleaned up my RICO file and files that were responsible for a lot of log entries. I also didn't notice anything in the logs that would suggest TM:PE is the cause of the freeze or crash.

I then proceed to load that save a few times, and sometimes it ran without problem and sometimes it would freeze. So I then made a new city and it ran, but trying to load it again it would sometimes run and sometimes freeze or crash.
Here are the logs from the test city that crashed.

Player.log: https://ufile.io/mhznwnx9
TMPE log: https://ufile.io/8hku7vfk

[originalfoo]

Try unsubscribing Surface Painter and see if that makes a difference.

Also, unsub this asset (despite what author says, it's horrendously broken).

Other than that I'm out of ideas. Does the problem remain if you unsub TMPE?

[neuryt]

I am also using Manjaro, here are my PC specs. I am also affected by this bug but I noticed it is not pausing itself when I try to load game with loading screen mod "remove all citizens, pedestrians and recover from simulations errors" on save load. Maybe it can be temporary workaround also for original issue poster? It is beaming fine for me for few loads after checking those settings – however after some time problem is returning...

On next time when I reproduce this issue I will upload required logs and my save game here – I sadly came into this issue after workarounding it...

System:    Host: klocek Kernel: 5.6.0-1-MANJARO x86_64 bits: 64 Desktop: KDE Plasma 5.18.3 Distro: Manjaro Linux 
CPU:       6-Core: AMD Ryzen 5 3600 type: MCP speed: 4069 MHz 
Graphics:  OpenGL: renderer: Radeon RX 580 Series (POLARIS10 DRM 3.36.0 5.6.0-1-MANJARO LLVM 9.0.1) v: 4.5 Mesa 19.3.5 
Drives:    Local Storage: total: 462.05 GiB used: 85.48 GiB (18.5%) 
Info:      Processes: 258 Uptime: 1d 13h 37m Memory: 31.38 GiB used: 21.57 GiB (68.7%) Shell: bash inxi: 3.0.37 

[krzychu124]

I found some info that mix Linux - AMD - Steam may cause similar issues.
I'm not sure how good you are at linux os, but I think you could try this: https://inxile.zendesk.com/hc/en-us/articles/115004658528-File-handles-bug-causing-crashes-Linux-
It shouldn't break anything, but since Cities Skylines tends to open and read a lot of files I think it's worth trying.

[darthroe]

To narrow things down, I uninstalled every asset and mod. I then reset CS configuration, I also cleaned up residual xml files and such left over from uninstalled mods.

I then re-subscribed to TM:PE and was able to play. I then re-subscribed to Loading Screen Mod. This is when I started to get freezing and crashes.
Basically with TM:PE alone, I seem to have no problem, with Loading Screen Mod alone I also seem to have no problem. Having both seems to cause random freezing/crashes.

Here's the latest log file from the city that crashed: https://ufile.io/ndrk6tta
TM:PE log file: https://ufile.io/ap4pelqv

[krzychu124]

I've just found that guide, I hope that there is an answer for this issue: https://steamcommunity.com/sharedfiles/filedetails/?id=467062052
Let us know if something from there help you

edit: I've just noticed that @aubergine10 is the author 😄

[CandyAngel]

I seem to also have this issue where both Stable and Labs causes a crash on, or shortly after, unpausing the game. (Arch Linux, nVidia GPU)

This happens even when TMPE is the only mod I am subscribed to. I load up the "Clean up Crew" scenario and the game will crash as soon as I unpause or very shortly after[1]. Unsubscribing from TMPE and the game runs without issue. I don't think the logs are very helpful though. Please let me know if I can do anything to help!

Player.log: http://dpaste.com/0DJAZEX
TMPE.log: http://dpaste.com/3PPJQ8D

EDIT #1: This is from a crash with TMPE Stable.

[1] For reference, I moved the water pump while it is paused, then unpause, then connect the water and then the electricity to the moved pump, but these aren't steps required to trigger the crash. Just noting!

[neuryt]

About this guide – nothing new, I have kernel 5.6.0, use LSM, it also happened once when I was playing offline due to my ISP outage so my steam was in offline mode I guess.

Oh. And my game is also playable when paused alter load and freeze after unpausing.

[chris-simpson]

TL;DR: Possible workaround: load C:S, start a New Game, press "Play" (this seems to be important), then hit Esc and Load your "broken" game. If assets seem to be missing, Exit to Main Menu, and then Load your broken game again.

I have similar behaviour to that described here on a linux machine. I've got Industries/Parklife/Matchday DLCs and the only mods I have now are TMPE, Loading Screen, Metro Overhaul Mod, Network Extensions 2, Move It!, and their required dependencies. I had more mods when I started a city a couple of days ago and grew it to 8000 population before saving, but then I experienced the freezing/crashing described here. By unsubscribing from all mods and re-subscribing to the 5 mods listed (which are kind of essential for good gameplay), I was able to load that game (I'll call it Game 0) and proceed. But when I made a few modest developments to the city and saved it, I was unable to run this new save. So I've tried to troubleshoot methodically.

Issues arise when I add a new intersection (which makes me think it might have something to do with TM:PE). I can make minimal changes to Game 0 (just let it run for a minute, or just extend a road slightly) and save it (call it Game 1), then completely exit Steam and load Game 1 without problems. However, if I add an intersection (on an urban road or a highway) things start to go wrong. If I save this game (Game 2), I can exit to the main menu and continue, I can exit to the main menu and then continue by selecting "load game" and Game 2, but I cannot exit Cities:Skylines and load Game 2 from there.

What I can do is load Game 0 or Game 1 (which almost always works) and then hit Esc and choose "Load Game" and load Game 2, and it seems to be fine. In fact, I was able to load Game 1, then choose to load a game I'd abandoned at a population of 20k because it had been developed while mods were being updated to be compatible with Sunset Harbor and I had read that some of these had not been completely fixed. So this leads to the possible workaround at the start of this post.

This made me think it was something to do with the Loading Screen Mod, so I unsubscribed from that, but now even Game 1 crashed, so I resubscribed. What I've also noticed is that if the game freezes so badly I have to kill -9 it but leave Steam running, then all games will crash -- I have to exit Steam completely.

The last few lines in Player.log are always something like this:

Receiving unhandled NULL exception
#0  0x007ff5213ad240 in funlockfile
#1  0x007ff5213ad7e8 in (Unknown)
#2  0x007ff5213ad7f8 in (Unknown)

So this may or may not have anything to do with TM:PE, but it seems to have something to do with the way Steam/C:S caches which I know nothing about. I'm posting here mainly in the hope that other people who have this problem can try my workaround and maybe continue to enjoy the game.

[originalfoo]

Can you try this mod - it exposes more details about errors, including some errors that are silently hidden by the vanilla game: https://steamcommunity.com/sharedfiles/filedetails/?id=2055465280

[CandyAngel]

Can you try this mod

Here is the Player.log from a crash after installing that mod: http://dpaste.com/1EGCWEG

[krzychu124]

I installed Manjaro Linux and can confirm issue with simulation freezing / crashing. I have no idea what is causing it and no option to test it easily because debugger cannot be attached under Linux :/ I suspect an issue with Harmony patching or sim thread lock/hang. In fact no errors were reported and simulation worked only for fraction of a second after starting.
No issues when TMPE disabled.

BTW, it looks like game runs way faster than on Windows (exactly the same settings) 🤔

[neuryt]

What debugger are you using for cities? Maybe mono debugger from VS code will be helpful? (I just read than it can be used fro cities here: https://skylines.paradoxwikis.com/Debugging_Mods and Linux version of vs code also have mono debugger so this is just idea, I hope it is at least a bit helpful.)

And about this BTW – why do you think I am using Linux? It is free and it is working better. (I have to say it now – wolne oprogramowanie means free software, not slow software xD)
My spec have on average 33% more FPS on Linux.

[krzychu124]

I use dnSpy and debug version of Mono.dll. I can't find debug binaries for linux so I can't use debugger (no way to attach it). Not to mention that laucher prevents from running game directly (debugger detaches itself) :/

[neuryt]

Oh. BTW, my TMPE log here, it finally crashed – I did my workaround (delete all vehicle agents by loading screen mod) and now it is again working again.
http://paste.asie.pl/vw4o <= log after freeze

I will try to build debug mono and attach it to cities myself maybe. If I succeed I will post my results here, I want to help.

[krzychu124]

Thanks, log is "empty" (nothing new there). I would check output_log(windows)/player.log(linux/macos), because if we(TMPE) fail to catch an error it should be logged there anyways.
I'll build debug mono too, but I have to figure out how to run game in debug mode, because mono has to start with these params to be visible for any debugger: --debugger-agent=transport=dt_socket,server=y,address=127.0.0.1:55555,defer=y,no-hide-debugger

[neuryt]

https://files.catbox.moe/86ilfl.log –> Player log, however I see this funlockfile null also here. :< Again delete all vehicle agents helped, this time i had to also restart steam.

[csupi]

I had the same issue on Ubuntu 19.10 and as aubergine10 suggested I found a directory named 'Source' under .steam/steam/steamapps/workshop/content/255710. It was in the directory of the mod 'Building Theme: European Suburbia Decorated'. I unsubscribed to this and it solved this issue for me.
thx.

[darthroe]

I tried the workaround that neuryt suggested (delete all vehicle agents by loading screen mod) and it seems to have worked. I was able to play without the city freezing. I'll try some more to ensure that it works each time.

[darthroe]

Unfortunately the workaround I tried does not always work. I tried a few more times and the freezing then crash continue to happen.

[ScottNAtlanta]

I have this problem, but to get my game to play, I have to unsubscribe all my mods, exit, go back and subscribe again, exit...then I can play, but after saving game and exit I have to repeat it all to play. This is at the end of the saved game log which leads me here

NullReferenceException: Object reference not set to an instance of an object
at TrafficManager.LoadingExtension.OnReleased () [0x00000] in :0
at LoadingWrapper.OnLoadingExtensionsReleased () [0x00000] in :0
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
LoadingWrapper:OnLoadingExtensionsReleased()
LoadingWrapper:Release()
LoadingManager:ReleaseRelay()
LoadingManager:OnDestroy()

Here is log after I've saved it
https://ufile.io/9hea9uvc
Here is the log when I try again after exiting and it crashes
https://ufile.io/9wjt4b34

I have the Mod Compatibility Checker which shows no conflicts
Ubuntu 19.10 Linux
steam v 020

[atomt]

I think I'm seeing the same bug. Linux, Ubuntu 20.04, TMPE stable 10.3.2, Sunset Harbor.
I've noticed that it randomly works sometimes. Say, after loading the same save about 5-6 times, it suddenly just works fine, and remains stable for as long as I keep the game running.

Unsubscribing from TMPE makes it load correctly 100% of the time.

Not sure when it started happening, had a hiatus from january to Sunset Harbor.

Some timing related problem during startup?

[jnnks]

Same issues here.

Solus with 5.5.11 and STABLE 11.3.2

Player.log
TMPE.log

[gmjck]

Unsubscribed for all Workshop items, unchecked Compatibility settings for CS in the Steam Library, then uninstalled CS. Removed ~/.paradoxlauncer, ~/.config/unity3d/Colossal Order, ~/.local/share/Colossal Order, and steamapps/common/Cities_Skylines to purge any remaining CS files.

Reinstalled CS, subscribed to Harmony 2.0.4-2 on Workshop, copied 1060 branch TMPE to local Mods directory. Started new game with stock Black Woods map and with TMPE on. Saved game before unpausing. Unpaused, immediate SIGSEGV. TMPE off, no SIGSEGV. Happens every time.

Zip file with TMPE.log, Player.log, Steam system_info, and save game:

https://ufile.io/1verse3o

[krzychu124]

Thank you I'll test it in about an hour, but I don't see any crash stacktrace or SIGSEGV in the log 🤔

[gmjck]

Yeah, it's very odd. I removed the 1060 branch TMPE and tried both the Stable and Labs from the workshop. I can load and unpause that NewBlackWoods save just fine with those versions turned on (the crashes I was experience before were always after I had built up more networks, never with a brand new game). Unsubscribed, reinstalled 1060 locally, back to crashing immediately upon unpause.

Trying to figure out how to get more debug info out of mono, since the CS logs aren't capturing any errors and the coredumps are useless.

[krzychu124]

Yeah, it's very odd. I removed the 1060 branch TMPE and tried both the Stable and Labs from the workshop. I can load and unpause that NewBlackWoods save just fine with those versions turned on (the crashes I was experience before were always after I had built up more networks, never with a brand new game). Unsubscribed, reinstalled 1060 locally, back to crashing immediately upon unpause.

Trying to figure out how to get more debug info out of mono, since the CS logs aren't capturing any errors and the coredumps are useless.

I can confirm that game crashes on unpause (Ubuntu and Manjaro too). I tried to catch that signal and at least figure out which method it is but GDB stops automatically when I call mono_backtrace to describe methods from the stack 😕 It's something on simulation thread...
I tried version from previous commit(Citizen AI patches not migrated) TMPE.zip and it works without any issue... I don't get it.

[EDIT]
I found the problem in patch causing StackOverflowException. I'll push a fix, so you can try again.

[krzychu124]

@gmjck you can download latest version (use link from #1060 ).
Everything should work. If you spot any bug or weird vehicle/citizen behavior, let me know😉

[gmjck]

Excellent! I've restored all my mods, assets, and saves, and everything that used to crash after unpause is now working perfectly. Thanks for this critically important mod, and now that I have a dev environment set up I guess I'd better figure out how to contribute.

[mmatis]

I can confirm the #1060 build as of this morning does not crash after unpausing both when I run C:S using the Steam Linux Runtime AND when I load it with my system's native libs (Mint 20.1 Cinnamon). Thank you so much!!!!!

[tonatiuhmira]

I can also confirm that the last build (#1060) solves unpause crash/freeze in (Manjaro) Linux. Thank you for this great mod!

[darthroe]

I too have been testing out the latest build from #1060 and so far (haven't played too much) it seems to have fixed the problem. Thanks for all the hard work!

[studio752]

Running Steam on Debian, with the stable release I almost always got either a crash to desktop on load or a freeze at simulation start. I've ran the game a dozen times so far using the #1060 build and it's working fine.

[flarn2006]

#1060 fixed it for me as well. I'm using Arch Linux, which I think is what Manjaro is based on.

[darthroe]

I wonder does anyone still have crashes when using #1060, I know I no longer have any crashes, should we mark this as fixed and close it?

[flarn2006]

Not sure if this is related to the actual issue or just a result of another asset/ mod/ the testing state of the Harmony migration, but the following massage appears loading my city with the testing version (copy the raw message did not work).

The game itself runs fine so far.

I wish I'd get a massage when an error occurs...

[Superkikim]

Hello. When clicking "Clear traffic", immediately after, I have Array index out of range errors in series.... Is it realated to TPME or something else ?

Simulation error: Array index is out of range. at (wrapper dynamic-method) TramBaseAI.TramBaseAI.SimulationStep_Patch0 (TramBaseAI,uint16,Vehicle&,UnityEngine.Vector3) <0x00226> at VehicleManager.SimulationStepImpl (int) <0x004d8> at SimulationManagerBase2.SimulationStep (int) <0x00040>
at VehicleManager.ISimulationManager.SimulationStep (int) <0x0001e>
at SimulationManager.SimulationStep () <0x0064c>
at SimulationManager.SimulationThread () <0x0015e>
[Core]

(Filename: /home/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
`

[krzychu124]

Which version of TM:PE do you use? I suggest switch to master since #1060 has been merged and I also fixed compatibility issue with More Vehicles mod

[Superkikim]

TM:PE enabled. Version 11.5.3, Build 11.5.3.3965 STABLE for game version 1.13.0-f8

Just downloaded the latest master. I'll try it.

[Superkikim]

Info 0.0114718: TM:PE enabled. Version 11.5.4, Build 11.5.4.21361 STABLE for game version 1.13.0-f8

Works fine now :) Thank you

[Superkikim]

Haha... Funny.... With the new version of TM:PE, there's a impact on Improved Public Transport....

The vehicule title is wrong, the description is correct.

At least I've never seen that before.

[samtoxie]

Hi I just wanted to say thanks for your hard work on this ticket! @krzychu124 :)

[toddmallen]

I had always missed this mod since the issues started. Every so often, I'd start to see traffic build up, and think for a second "Oh I'll just...ARGGHHHH can't do it."

So, I thought I'd check in if the issue got fixed, and find out that the dev not only ran across what no one wants to ("Oh it's a race condition...IN SOMEONE ELSE'S CODE"), but actually spent the better part of a year troubleshooting and fixing it. And with the latest master branch, I'm back to using one of my favorite mods.

Impressive work indeed. Thank you for it.

[krzychu124]

@toddmallen you can use "live" TEST version, already available to the workshop, albeit still unlisted(not available in the workshop search results) - waiting for last feature, new speed limits tool😉

[Raspikabek]

So far so good in the last 12 hours of playtime with #1060 !

Is there any ETA to merge this into the Stable version ?