New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Steam Cloud path conflicts breaks game-save syncing #60

Closed
da2x opened this Issue Jan 28, 2018 · 44 comments

Comments

9 participants
@da2x
Copy link

da2x commented Jan 28, 2018

TL;DR: Paths are broken because of XDG overwrites. Skip to the code example.

Update: Read this analysis of this bug which includes two suggestions for possible fixes. The below ended up serving as drafts for that analysis and is less less accurate.


For most games, Steam Cloud syncs saves and configurations to and from /home/user/.local/share/studio/gamename. This is the LinuxXdgDataHome data root path in Steam Cloud, which documentation suggests is corresponding to $XDG_DATA_HOME. It appears that game developers use $XDG_DATA_HOME (default in e.g. Unity3D) but Steam itself doesn’t actually read the $XDG_DATA_HOME variable but just hardcodes $HOME/.local/share/ or $HOME/.config/.


Many games, like RimWorld , won’t read from /home/user/.local/share/studio/gamename where saves are synced by Steam but instead only read from and write to /home/user/.var/app/com.valvesoftware.Steam/data/studio/gamename. These won’t be synced and players won’t have access to existing saves.

Some games, like Cities Skylines, are synced to /home/user/.local/share/studio/gamename and will read from both locations, but only write changes and new saves to /home/user/.var/app/com.valvesoftware.Steam/data/studio/gamename. Players have access to existing syncs, but can’t update existing saves or add new ones.

A few games, like FTL, are synced to /home/user/.var/app/com.valvesoftware.Steam/data/studio/gamename and everything appears to work out of the box. These games seem to be the exception.


I tested with restoring the sandboxed $HOME prefix stuff in the wrapper committed by @alexlarsson and reverted by @kujeger. This seems to fix cloud syncing with many games, but broke some games like Cities Skylines.

Combining this special sandboxed $HOME variable with having $XDG_DATA_HOME point to /home/user/.var/app/com.valvesoftware.Steam/home/.local/share/ and $XDG_CONFIG_HOME point to /home/user/.var/app/com.valvesoftware.Steam/home/.config/ should fix pretty much every game as Steam and games then should agree on where saves are kept. Should be a workable solution that works for a majority of games (at least Unity3D titles, and I suggest most others too) — unless anyone has contacts at Valve and can make them fix this upstream?

Here is a suggested patch for `steam-wrapper:

export HOME="$HOME/.var/app/com.valvesoftware.Steam/data/home"
export XDG_CONFIG_HOME="$HOME/.config"
export XDG_DATA_HOME="$HOME/.local/share"

OLD_DATA_DIR=$HOME/.var/app/com.valvesoftware.Steam/data/Steam
if [ -d $OLD_DATA_DIR ]; then
    mkdir -p $HOME
    mkdir -p $XDG_CONFIG_HOME
    mkdir -p $XDG_DATA_HOME
    rm -rf $OLD_DATA_DIR
fi

The good:

  • All games† now only have one data directory and everyone agrees on where that directory is located.
  • All games† now sync to and from Steam Cloud, meaning players get back years worth of game progression.

The bad:

  • The old Steam library and install is no good. It can’t be migrated as moving it breaks some games that use absolute install paths and causes crashes with the Steam overlay.
  • Local saves inside and outside the Flatpak container is lost. These are recoverable but we can’t migrate the data for the user without causing Steam Cloud conflicts. We also don’t know what the actual paths that need to be migrated are. Some are inside the Flatpak app directory, some are in the home directory, some are partially in one place and partially in the other.

† Works with at least RimWorld, City Skylines, Borderlands 2, Project Highrise, Sim Airport (game doesn’t work, but sync does), Transport Fever (game doesn’t work, but sync does), Mini Metro, Dream Daddy, and Space Run (game doesn’t work, but sync does).

The following titles have broken cloud sync implementations and can’t be fixed without creative game-specific patching that has nothing to do with Flatpak:

  • FTL (casesensitive path mixup)
  • Bridge Constructor Portal (cloud profile doesn’t match actual storage location)
@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jan 31, 2018

It sounds to me like this thing is so overly messed up there should be a counterpart ticket on Steam for Linux project

@da2x

This comment has been minimized.

Copy link

da2x commented Feb 5, 2018

It sounds to me like this thing is so overly messed up there should be a counterpart ticket on Steam for Linux project

Valve’s only fault here is that they make a poor recommendation in their documentation. I’ve contacted them about that specifically.

@da2x

This comment has been minimized.

Copy link

da2x commented Feb 5, 2018

I wrote up a more thorough analysis of this bug on my blog. It’s way less rambly than the above.

@kujeger

This comment has been minimized.

Copy link
Contributor

kujeger commented Feb 5, 2018

Just to clarify: #22 does not give steam access to the actual host $HOME -- it simply persists everything written to $HOME inside the steam flatpak.

(the option to give access to host $HOME is --filesystem=home)

Great job describing this mess, I never discovered these issues when testing.

It seems that simply setting $XDG_DATA_HOME to $HOME/.local/share (along with the other XDG*s) would fix the issue, but would also lose the progress of any applications actually honoring the XDG paths, unless manually moved, as you point out.

@da2x

This comment has been minimized.

Copy link

da2x commented Feb 5, 2018

[…] Just to clarify: #22 does not give steam access to the actual host $HOME -- it simply persists everything written to $HOME inside the steam flatpak.

I don’t think this can be solved in any other way than providing documentation. Maybe xdg-open "https://github.com/flathub/com.valvesoftware.Steam/wiki" on upgrade or steam-wrapper first-run to a page describing where to find game saves and where to move them to recover data.

@kujeger

This comment has been minimized.

Copy link
Contributor

kujeger commented Feb 6, 2018

https://support.steampowered.com/kb_article.php?ref=6736-QEIG-8941#conflicts

It looks like steam will detect and require manual intervention if there are sync conflicts, so it's possible that we can just copy everything from the old XDG_DATA_HOME to a new one.

If so the wrapper could just more or less be

cp -a $XDG_DATA_HOME/* $HOME/.local/share/
export XDG_DATA_HOME=$HOME/.local/share/

and similar for cache and config, with something to make sure that it only runs once.
There's no need to change $HOME.

@swick

This comment has been minimized.

Copy link

swick commented Feb 6, 2018

Steam installs itself into $XDG_DATA_HOME so the script would also copy steam and all games.

@kujeger

This comment has been minimized.

Copy link
Contributor

kujeger commented Feb 6, 2018

Then perhaps moving $XDG_DATA_HOME/Steam, and copying everything else.

@da2x

This comment has been minimized.

Copy link

da2x commented Feb 6, 2018

You can’t move $XDG_DATA_HOME/Steam. Some paths in that directory itself are hardcoded.

@alexlarsson

This comment has been minimized.

Copy link
Member

alexlarsson commented Feb 8, 2018

You can move it and leave around a relative symlink to the new place.

@kujeger

This comment has been minimized.

Copy link
Contributor

kujeger commented Feb 8, 2018

That's a good thought; instead of changing the env variables we could

  • move/copy $HOME/.local/share/* to $XDG_DATA_HOME/
  • (relatively) symlink $HOME/.local/share to $XDG_DATA_HOME
@alexlarsson

This comment has been minimized.

Copy link
Member

alexlarsson commented Feb 8, 2018

Same with .config

@alexlarsson

This comment has been minimized.

Copy link
Member

alexlarsson commented Feb 8, 2018

The blog about this mentioned that some apps did not handle symlinks well (even crashing). Should get some data about that first though.

@da2x

This comment has been minimized.

Copy link

da2x commented Feb 20, 2018

The problem with moving or symlinking anything is that there are already two directories per game. One created by the game executable ($XDG_CONFIG_HOME/game), and one by Steam Auto-Cloud ($HOME/.config/game). Which of the two directories do the user want to keep? I’d personally want my cloud saves back; there are a lot of hours in there — but others may want to keep their more recent game saves instead.

These games crash with symlinked data directories: Oil Rush and Colony Survival.

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Feb 20, 2018

Any reasonable way to query the choice from end-user?

@TingPing

This comment has been minimized.

Copy link
Member

TingPing commented Feb 20, 2018

If you symlink $HOME/.config to $XDG_CONFIG_HOME the broken games would be fine and steam would use the right directory?

@da2x

This comment has been minimized.

Copy link

da2x commented Mar 12, 2018

We need to make a decision on what to break to proceed with this issue. Local saves or cloud saves.

I’ve experimented a bit with merging local folders on top of steam cloud folders. A dialog in Steam will ask players to choose which they want to keep. However, we shouldn’t overwrite either the local copy of the Steam Cloud or the local game progress to make it possible to recover either two.

I believe this should fix the issues and preserve both sets of data:

  1. Create ~/.var/app/com.valvesoftware.Steam/home/ and set it as $HOME in the launcher.
  2. Create ~/.var/app/com.valvesoftware.Steam/home/.cache/ and set it as $XDG_CACHE_HOME in the launcher.
  3. Create ~/.var/app/com.valvesoftware.Steam/home/.config/ and set it as $XDG_CONFIG_HOME in the launcher.
  4. Create ~/.var/app/com.valvesoftware.Steam/home/.local/share/ and set it as $XDG_DATA_HOME in the launcher.
  5. Copy ~/.var/app/com.valvesoftware.Steam/config/* to ~/.var/app/com.valvesoftware.Steam/home/.config/
  6. Move ~/.var/app/com.valvesoftware.Steam/data/Steam/ to ~/.var/app/com.valvesoftware.Steam/home/.local/share/Steam/
  7. Copy ~/.var/app/com.valvesoftware.Steam/data/* to ~/.var/app/com.valvesoftware.Steam/home/.local/share/

The Steam library folder shouldn’t be copied as it’s where all games is installed. It can be enormous. The cache folder doesn’t need to be copied.

@TingPing

This comment has been minimized.

Copy link
Member

TingPing commented Mar 15, 2018

@da2x If that works it sounds fine.

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Mar 29, 2018

I'd kinda lean towards backupping local saves, replacing with cloud saves where available and alert user they need to manually override cloud data with local data if they didn't like the choice. It's harder to recover from the point if you break the user's cloud data

TingPing added a commit that referenced this issue May 7, 2018

Workaround sync issues documented in #60
This seems to work but the old config dir cannot be symlinked
which is likely to cause a problem.
@AsciiWolf

This comment has been minimized.

Copy link
Contributor

AsciiWolf commented Jul 15, 2018

Any way to properly fix this?

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jul 24, 2018

@AsciiWolf possibly although complex to fix. @TingPing has a branch with WiP patch in case you want to contribute

@gasinvein

This comment has been minimized.

Copy link
Contributor

gasinvein commented Jul 24, 2018

@da2x why symlink $XDG_DATA_HOME to hardcoded default ~/.local/share/, and not the opposite?

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jul 25, 2018

Just a note here: symlinks in the direction as in the patches (for at least XDG_CONFIG_HOME) break the Flatpak's ability to start even bash. It's basically a no-go

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jul 25, 2018

I managed to now get the migration logic started by @TingPing working as part of #110
I'm leaving the PR open for a bit in case you guys want to discuss but it should be mostly good to merge at this point

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jul 25, 2018

I recommend building from wip branch yourself if you want to try it out
http://docs.flatpak.org/en/latest/first-build.html is sufficient in helping you to do a local build for testing
Note that the migration will permanently change through symlinks (non-overridden Flatpak values)
$XDG_CONFIG_HOME -> ~/.var/app/com.valvesoftware.Steam/.config
$XDG_DATA_HOME -> ~/.var/app/com.valvesoftware.Steam/.local/share
$XDG_CACHE_HOME -> ~/.var/app/com.valvesoftware.Steam/.cache
ALERT: the migration process merges directories so obtaining the original directory structure through even manual rollbacking is most likely really hard.
You can take backups before migration if you like. Note that $XDG_DATA_HOME/Steam is massive, keeping a copy of it might not be practical
This is mostly about trying to find games that crash with the new structure.

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jul 26, 2018

There's a backup plan of trying bindfs if symlinks done this way still causes crashing but if there's no issues with this approach, we can adopt this. Imo if further changes are done, the bootstrapper should be turned into a Python script that execs Steam since the migrating logic is getting quite involved. We may also potentially need to start versioning migrations in case we need to migrate again in the future

@TingPing

This comment has been minimized.

Copy link
Member

TingPing commented Jul 27, 2018

So I did very basic testing of this:

  • Migrated existing data
  • Made a cloud save in Civ V
  • Restart app and load cloud save

Not sure if that is good enough but it worked at least.

@da2x

This comment has been minimized.

Copy link

da2x commented Jul 27, 2018

Its import to test what happens with games that have a local game save before the migration, and also have an existing cloud save that will start syncing after the migration. Steam should ask you to resolve sync conflicts. You should not loose either the local or cloud saves without Steam t least asking you to choose. (I don't have any time to look into this for at least a few weeks.)

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jul 27, 2018

That's probably fine. It's not as if this issue is likely to go anywhere in the meantime. It'd be really nice to get also some more information on those crashes to understand better why they happened.

@michel-slm

This comment has been minimized.

Copy link

michel-slm commented Jul 31, 2018

@nanonyme @TingPing @da2x - are there test scenarios you'd like help with? (If insufficient testing is what's preventing the test flatpak from being committed).

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Jul 31, 2018

@michel-slm we basically just need that migration scenario described by @da2x tested with as many games as possible. Rollback is basically impossible if we get crashes

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 1, 2018

@da2x do you happen to by the way know any sample games that currently save to "wrong" place? I'm not totally sure if I have any in my catalog.

@da2x

This comment has been minimized.

Copy link

da2x commented Aug 1, 2018

@nanonyme, there is a short list of popular titles with the issue confirmed near the end of the initial post.

Statistically, almost every game built on Unity 3D engine version 4 or newer will use $XDG_CONFIG_HOME in-game and then use $HOME/.config in their Steam Cloud configuration. Look at 2013 or newer here in this list:
https://en.wikipedia.org/wiki/List_of_Unity_games

@da2x

This comment has been minimized.

Copy link

da2x commented Aug 1, 2018

Thanks to GDPR you can inspect your Steam Cloud including seeing the time stamp for when files were last modified or uploaded via the web now:
https://store.steampowered.com/account/remotestorage

Should help with testing and verifying that things are uploaded. Thanks European Parliament and the Council of the European Union!

nanonyme added a commit that referenced this issue Aug 1, 2018

Workaround sync issues documented in #60
This seems to work but the old config dir cannot be symlinked
which is likely to cause a problem.
@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 1, 2018

Looks like I have exactly one game on the list and I don't have enough disk space to install it currently. I just built a new bundle to same location since wip branch was rebased to have the new Steam version too

nanonyme added a commit that referenced this issue Aug 13, 2018

Workaround sync issues documented in #60
This seems to work but the old config dir cannot be symlinked
which is likely to cause a problem.
@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 13, 2018

Tested with Wasteland 2 that this results in Steam to upload data as exacted. Haven't tested a conflict scenario yet

  1. Install old app
  2. Run game
  3. Do a save
  4. Update to new app

Errors when launching Wasteland 2 disappeared as a result

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 14, 2018

Test round 2 with Wasteland 2 with existing cloud data was a success

  1. Wiped entire .var/app/com.valvesoftware.Steam
  2. Installed new app
  3. Ran game again
  4. Continuing the previous game that was previously uploaded to cloud
@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 14, 2018

Test round 3 with both local and cloud data failed

Conclusion is that the migration script wipes local data on name conflicts so on eg Wasteland 2 you lose all local progress without any promptms. No go, we need to improve the config directory migration

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 15, 2018

Still haven't figured out a way to get Steam to trigger cloud sync conflict page so user could choose which stuff to use

@da2x

This comment has been minimized.

Copy link

da2x commented Aug 15, 2018

If I remember correctly, then conflicts dialog is triggered when file on disk is newer than cloud copy, at which point Steam compares checksums, and asks users whether to override the local file or not.

This is documented in detail somewhere in the documentation for Steam Cloud on Steamworks.

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 15, 2018

Yeah, I haven't managed to trigger it here yet. Also I have a suspicion currently that preserving your Steam data dir breaks the test setup since cloud sync seemed to keep working even without migration

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 17, 2018

Tested latest revision of the migrate script
Local data was preserved and uploaded into cloud. However I'm a bit concerned I still haven't seen the conflict dialog ever during my tests @da2x

@nanonyme

This comment has been minimized.

Copy link
Collaborator

nanonyme commented Aug 21, 2018

Closing as fixed, sync seems to now work

@michel-slm

This comment has been minimized.

Copy link

michel-slm commented Aug 21, 2018

Glad the issue is solved, but for reference the game I tested with (that works using the test Steam flatpak but broke with the earlier published flatpak) is Invisible Inc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment