`arkmanager installmod [modid]` fails to install

[josephshanak]

I recently had issues where all of my mods were wiped out and I was no longer able to start any of the servers. It appears to be related to mishandling of the output of steamcmd in arkmanager. I'm posting here first, since I'm not sure if there's something specific to the image or not.

It began on 2024-04-08 and it seems to coincide with a recent steamcmd update (and docker image update):

The version of steamcmd in my container is 1743712175

$ steamcmd
Redirecting stderr to '/home/steam/.steam/logs/stderr.txt'
Logging directory: '/home/steam/.steam/logs'
Looks like steam didn't shutdown cleanly, scheduling immediate update check
[  0%] Checking for available updates...
[----] Verifying installation...
UpdateUI: skip show logo
Steam Console Client (c) Valve Corporation - version 1743712175
-- type 'quit' to exit --
Loading Steam API...IPC function call IClientUtils::GetSteamRealm took too long: 49 msec
OK

I believe it's caused by a slight change in output of steamcmd ... +workshop_download_item

The first indication that something was wrong were these logs:

2025-04-09 07:00:59.212	Performing ARK update ...  -/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-/|\-/ Update to 13834083 complete
2025-04-09 07:00:59.270	Running command 'start' for instance 'main'
2025-04-09 07:00:59.303	Checking for updates before starting
2025-04-09 07:00:59.311	Checking for update; PID: 190
2025-04-09 07:01:00.511	Downloading mod 1404697612Executing /arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd +@NoPromptForPassword 1 +login anonymous +workshop_download_item 346110 1404697612 +quit
2025-04-09 07:01:00.514	/usr/local/bin/arkmanager: line 728: /arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd: No such file or directory
2025-04-09 07:01:00.515	2025-04-09 07:01:00.519	
tail: cannot open '/home/steam/.steam/logs/workshop_log.txt'$'\n''/home/steam/.steam/steam/logs/workshop_log.txt' for reading: No such file or directory
2025-04-09 07:01:00.521	Mod 1404697612 download failed
2025-04-09 07:01:00.521	Mod 1404697612 was not successfully downloaded
2025-04-09 07:01:01.067	Downloading mod 1609138312Executing /arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd +@NoPromptForPassword 1 +login anonymous +workshop_download_item 346110 1609138312 +quit
2025-04-09 07:01:01.071	/usr/local/bin/arkmanager: line 728: /arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd: No such file or directory
2025-04-09 07:01:01.076	tail: cannot open '/home/steam/.steam/logs/workshop_log.txt'$'\n''/home/steam/.steam/steam/logs/workshop_log.txt' for reading: No such file or directory

These logs come from here:
https://github.com/arkmanager/ark-server-tools/blob/a05d763b8e2f49a7ea82759728bf682d24fa254d/tools/arkmanager#L2696

And then later after some troubleshooting (sorry, I forgot exactly what I did):

Logging directory: '/home/steam/.steam/logs'
[  0%] Checking for available updates...
[----] Verifying installation...
UpdateUI: skip show logo
Steam Console Client (c) Valve Corporation - version 1743712175
-- type 'quit' to exit --
Loading Steam API...IPC function call IClientUtils::GetSteamRealm took too long: 49 msec
OK
@NoPromptForPassword 1 
"@NoPromptForPassword" = "1"


Connecting anonymously to Steam Public...OK 2025-04-13 11:01:50.539	
Waiting for client config...OK 2025-04-13 11:01:50.541	
Waiting for user info...OK 2025-04-13 11:01:50.566	
workshop_download_item 346110 2307661303 2025-04-13 11:01:56.877	
Downloading item 2307661303 ...  2025-04-13 11:01:56.910	
Success. Downloaded item 2307661303 to "/home/steam/.steam/SteamApps/workshop/content/346110/2307661303" (24589452 bytes) quit 
Mod 2307661303 was not successfully downloaded

These logs come from here: https://github.com/arkmanager/ark-server-tools/blob/a05d763b8e2f49a7ea82759728bf682d24fa254d/tools/arkmanager#L2689

I eventually worked around the issue by changing this line: https://github.com/arkmanager/ark-server-tools/blob/a05d763b8e2f49a7ea82759728bf682d24fa254d/tools/arkmanager#L2689

to this (to try to remove any ansi escape sequences):

modsrcdir="$(echo "$output" | sed -n 's@Success. Downloaded item [0-9][0-9]* to "\([^"]*\)" .*@\1@p' | sed -r "s/\x1B\[(([0-9]{1,2})?(;)?([0-9]{1,2})?)?[m,K,H,f,J]//g")"

And reinstalling the mods with arkmanager installmod ..., before restarting. To

It appears that steamcmd now includes ANSIs escape sequence

You can reproduce this by doing this in a container (I'm basically replicating what arkmanager does):

$ source $(which arkmanager)
$ output=$(runSteamCMDspinnerSubst 2 +workshop_download_item "346110" "1404697612")
$ modsrcdir="$(echo "$output" | sed -n 's@^Success. Downloaded item [0-9][0-9]* to "\([^"]*\)" .*@\1@p')"
$ echo "$modsrcdir"

$ modsrcdir="$(echo "$output" | sed -n 's@Success. Downloaded item [0-9][0-9]* to "\([^"]*\)" .*@\1@p')"
$ echo "$modsrcdir"
/home/steam/.steam/SteamApps/workshop/content/346110/1404697612
steam@ark-ark-cluster-island-7787888f5f-l84q4:~$ cat $modsrcdir/mod.info
cat: ''$'\033''[0m/home/steam/.steam/SteamApps/workshop/content/346110/1404697612/mod.info': No such file or directory
$ cat /home/steam/.steam/SteamApps/workshop/content/346110/1404697612/mod.info
AwesomeSpyglassAwesomeSpyglass>$
$ sudo apt update
$ sudo apt install -y less
$ echo "$modsrcdir" | less
$ echo "$output" | less

ESC[0m/home/steam/.steam/SteamApps/workshop/content/346110/1404697612

Redirecting stderr to '/home/steam/.steam/logs/stderr.txt'
Logging directory: '/home/steam/.steam/logs'
[  0%] Checking for available updates...
[----] Verifying installation...
UpdateUI: skip show logo
Steam Console Client (c) Valve Corporation - version 1743712175
-- type 'quit' to exit --
Loading Steam API...ESC[0mIPC function call IClientUtils::GetSteamRealm took too long: 51 msec
OK
ESC[0mESC[1m@NoPromptForPassword 1
ESC[0m"@NoPromptForPassword" = "1"
ESC[0m
Connecting anonymously to Steam Public...ESC[0mOK
ESC[0mWaiting for client config...ESC[0mOK
ESC[0mWaiting for user info...ESC[0mOK
ESC[0mESC[1mworkshop_download_item 346110 1404697612
ESC[0mDownloading item 1404697612 ...ESC[0m
ESC[0mSuccess. Downloaded item 1404697612 to "/home/steam/.steam/SteamApps/workshop/content/346110/1404697612" (3418407 bytes) ESC[0mESC[1mquit
ESC[0m

[DrPsychick]

Well, steamcmd is a bit of a pain indeed and handling it with shell (arkmanager) not the simplest task on earth to be honest. These things have caused issues in the past and mods only complicate the issue.

What I found when looking at this is that arkmanager overwrites the steamcmd root when "automanagedmods" is active (see https://github.com/arkmanager/ark-server-tools/blob/a05d763b8e2f49a7ea82759728bf682d24fa254d/tools/arkmanager#L2667).
This would match the error in your first output and explain why downloading mods fails.

2025-04-09 07:01:00.514	/usr/local/bin/arkmanager: line 728: /arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd: No such file or directory

What we could do in this image is cover the case when a) automanagedmods is active and b) /arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd is missing: simply stop and display an error.
Since I don't use this image actively anymore, @josephshanak could you please check if /arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd exists or when it appears? I would assume it's part of the game files, but maybe I'm wrong.

PS: by default arkflag_automanagedmods is not set.
https://github.com/arkmanager/ark-server-tools/blob/master/tools/arkmanager.cfg.shellcheck#L282

[josephshanak]

Looking at my commit history, it looks like I did enable arkflag_automanagedmods at one point.

/arkserver/Engine/Binaries/ThirdParty/SteamCMD/Linux/steamcmd exists currently, but I believe that's because I added it while trying to troubleshoot this.

Stepping back for a minute, let me give you the full context.

Full context

I've been troubleshooting something similar to SickHub/ark-server-charts#31, which has been happening regularly, but I haven't been able to gather an enough info find the cause. I've been trying to understand the update mechanism (hence the "issue" I reported here SickHub/ark-server-charts#28), and I believe it has something to with how the mods are extracted. I've observed that when issue occurs for me, the ".mod" file seems to be missing (https://github.com/arkmanager/ark-server-tools/blob/a05d763b8e2f49a7ea82759728bf682d24fa254d/tools/arkmanager#L2953).

One part that has been particularly frustrating is that when that issue occurs, if the server somehow manages to start without the mods, any structures or items from that are erased. Luckily, I've been able to restore from backups.

It's possible I enabled arkflag_automanagedmods while trying to troubleshoot that issue (i.e. "maybe if steam manages the updates, it won't start without the mods).

However, I think the issue with arkmanager installmod still exists (and perhaps I need to report it upstream): It doesn't seem to properly parse the steamcmd output, and as a result it does not copy the downloaded mod into the mods folder. Thus if for some reason an update needs to occur it will fail.

For now I've tried to disable any kind of updates occurring by disabling the default updates from the crontab (Due to #17 I did this by overwriting the run.sh)

an aside about SickHub/ark-server-charts/issues/31

I wonder if part of my issue with mods disappearing is the container OOMing, or getting killed due to a liveness check during an arkmanager update from the crontab?

As I understand it https://github.com/SickHub/ark-server-charts does not use a staging directory (https://github.com/SickHub/ark-server-charts/blob/c8dd3cb581d0a91f7cff74f8bfc4ed587e25337a/charts/ark-cluster/templates/deployment.yaml#L126C21-L127), so it ends up deleting the mod entirely before extracting it into the /Mods directory.

Currently I have 4Gi request and an 8Gi limit for the Island server, which is the one that performs the updates.

Next Steps

0. I think we can ignore the specifics of arkflag_automanagedmods=true for this issue, as I no longer have it enabled.
1. Should I report the steamcmd parsing problem in arkmanager upstream?
2. I'm not sure I understand why am_arkStagingDir is incompatible with ARKSERVER_SHARED, and I think extracting to a separate directory and then copying each file (like AM does when there is a staging directory https://github.com/arkmanager/ark-server-tools/blob/a05d763b8e2f49a7ea82759728bf682d24fa254d/tools/arkmanager#L2992-L3021) might be beneficial to avoid mod corruption in the presence of crashes.
3. Given that updates (or at least mod updates) have been the root cause of most of my issues, it would be nice to be able to have more control over them. In this image there doesn't appear a way to disable updates at startup (

   arkserver/run.sh

   Line 207 in 88f7704

   arkmanager update --force --no-autostart

   ) and I've had issues changing the crontab in the helm charts (Make crontab work #17 fix(deployment.yaml): add an emptyDir volume for /ark/config ark-server-charts#51).

[DrPsychick]

I think you should report to upstream, yes (1).

If I remember correctly, my main reason for not using the staging directory back then was saving disk space, but I'm not 100% sure on that. Feel free to give it a try and create a PR if it works well.

As for force-updating the server and usage of mods: Usually you cannot avoid updating the Client and thus the server needs to be updated as well, so you can actually connect. That, IIRC, is the main reason why the server updates automatically on start.

The mods have cost me several hours back then as well. I remember I was once building my own download and unpack logic, before switching to use arkmanager. To avoid loss of structures or the like, it would definitely be good to have the startup abort when a mod cannot be downloaded or extracted.

I'm not running any ARK servers anymore and I will not spend time on these issues myself, but I will gladly review and accept any suggestions that make the image better.

[josephshanak]

Sounds good.

I filed arkmanager/ark-server-tools#1281 for the issue with arkmanager.

I'll play around and see if I can make the mod update more atomic.

[DrPsychick]

They already released a fix and I updated the "latest" images as well.

[DrPsychick]

I believe this is solved then. If not, let me know or open a new issue. Thanks for reaching out!