Reorganize userdata folder

[Pentarctagon]

Steam's cloud setup (without using the SDK) only allows us to sync a single folder across devices which right now is just the saves folder, but there are multiple things that would be useful to keep synced - preferences (achievements, campaign completion, general settings) and map editor maps/scenarios come to mind in particular. It would be useful to have these all located under a single folder so that they can all be synced.

/[version]/userdata/data/add-ons/
/[version]/userdata/logs/
/[version]/userdata/screenshots/
/[version]/userdata/save_index
/[version]/userdata/credentials
/[version]/userdata/lan_config.cfg


/[version]/userdata/sync/editor/
/[version]/userdata/sync/persist/
/[version]/userdata/sync/saves/
/[version]/userdata/sync/lua_command_history
/[version]/userdata/sync/preferences

[stevecotton]

The settings would need to be split in two, for example display settings should be per-device.

[Pentarctagon]

Loading two files, combining the contents, and then continuing as normal wouldn't be much of an issue.

[CelticMinstrel]

So I guess the question would be, what should be synched and what shouldn't?

• cache - seems like it probably shouldn't be, but not sure?
• data - this is actually add-ons; I'm not sure if it should be synched
• editor - somewhat legacy, but a place to store custom maps for synching isn't a bad idea
• logs - why would you want to sync that?
• history files - sure, why not (I only have a Lua console history file, but I suspect there could be more?)
• persist - is related to saved games so should definitely be synched
• preferences - mixed bag, as mentioned it should probably be split in two
• save_index - related to saved games, but I'm not sure… it's generated automatically, after all, but it's also tiny, so…
• saves - obviously
• screenshots - dunno, maybe
• credentials - definitely not, as they're encrypted with a machine-dependent key

Is there anything else I missed?

I guess the idea is to move all the stuff that should be synched into a common folder; I'm not sure what to call it though. If we can't think of anything better, I suppose "sync" or "cloud" would do.

Loading two files, combining the contents, and then continuing as normal wouldn't be much of an issue.

Apparently it already does exactly this if the build environment defines a certain constant (though for some reason it doesn't clear the prefs before loading in that case, which seems wrong). See load_base_prefs() in preferences/general.cpp.

For the preferences, couldn't we just run the preprocessor on it? The existing "preferences" file would then be a fixed file that never changes (with a comment at the top noting as such), and it just references all the necessary files containing various bits of relevant information that are stored wherever they need to be. If there are builds that use a "default preferences file", that could probably be merged into the core preferences file.

Obviously, writing the preferences would become completely different, as it would write some things to one file and some things to another file. There'd need to be a method of deciding what goes to which file somehow. Perhaps the preferences themselves would become more structured, with fewer toplevel keys; the contents of each tag might go in a different file.

[Pentarctagon]

I'd say:

• cache - no
• data - no (could end up too big)
• editor - yes
• logs - no
• history files - yes
• persist - yes
• preferences - as much as makes sense
• save_index - no
• saves - yes (is what's currently synced)
• screenshots - no
• credentials - no

I don't think the preprocessor could handle this though? Given on Linux for example you have the preferences and lua_command_history in ~/.config/wesnoth and the userdata in ~/.local/share/wesnoth/1.17.

[Wedge009]

I guess the idea is to move all the stuff that should be synched into a common folder; I'm not sure what to call it though. If we can't think of anything better, I suppose "sync" or "cloud" would do.

If all this is specific to Steam, should we call it steam? If a theoretical second cloud-based platform were to be supported in the future, would it use the same directory structure? If we assume it could, then maybe better to use a generic term like sync.

[CelticMinstrel]

I don't think we should call it "steam", as it's very reasonable to think there might be another cloud-based platform supported in the future.

[Pentarctagon]

Probably going to work on this after #8791, so the new folder structure per version would be:

/[version]/userdata/data/add-ons/
/[version]/userdata/logs/
/[version]/userdata/screenshots/
/[version]/userdata/save_index
/[version]/userdata/credentials
/[version]/userdata/lan_config.cfg

/[version]/userdata/sync/editor/
/[version]/userdata/sync/persist/
/[version]/userdata/sync/saves/
/[version]/userdata/sync/lua_command_history
/[version]/userdata/sync/preferences

This would also then resolve #4163 as well.

[CelticMinstrel]

Isn't it [version]/userdata/whatever? Or I guess it might depend on the platform.

Before working on it, I think we need to discuss what preferences should be synched and not synched. I'm pretty sure there's at least some things in there that shouldn't be synched.

[Pentarctagon]

I don't really care whether the version number comes before or after userdata in the path, but the goal would of course be to have all platforms use the same structure.

The preferences are a bit of an unknown at the moment just since I don't think it's known what the full list of preferences that currently exist are.

[CelticMinstrel]

Right, and we need to work that out in order to decide which ones need to be synched.

[stevecotton]

/userdata/[version]/cache/

Cache should be kept separately, as it's normal to generate it locally rather than transmit it over a network. Some OSes have specific directories for caches, so that backup programs or roaming profiles can automatically ignore it.

[soliton-]

The stuff in the sync dir is what gets synced.

[stevecotton]

There's more than just Steam's synchronisation to consider, which is why I gave the examples of backups and roaming profiles. Wesnoth already supports XDG_CACHE_HOME on Linux for treating cache specially.

[Pentarctagon]

Wesnoth supports the XDG paths for more than just the cache, which actually just annoys me most of the time. I don't like needing to look in five different folders depending on what data I want.

[soliton-]

Oh, is this a suggestion to disregard the XDG specification? I don't think that's useful.

[CelticMinstrel]

I don't think this is a suggestion to do that, unless XDG specifies that the 4 things noted as belonging in the sync folder should be in different places.

…so maybe it's a suggestion to ignore it only for preferences?

[Pentarctagon]

I suppose I don't really care where the cache goes. I don't know why it makes sense to store the preferences, credentials, and lua command history separately from everything else though.

[Pentarctagon]

Something that occurred to me - does anyone know if the Steam Autocloud functionality supports syncing of subfolders, or does it only sync the files immediately under the folder it's setup to sync?

[Pentarctagon]

Tested - Steam's Autocloud functionality doesn't sync subfolders, so this wouldn't actually accomplish anything.

[gfgtdf]

Maybe this can be configuured? https://partner.steamgames.com/doc/features/cloud#steam_auto-cloud explicitly talk about some "Recursive" option

[Pentarctagon]

Oh, uh... yeah, that's a thing that exists actually. I've updated that setting and can confirm that it does look in subfolders now for storing files in the steam cloud.

[CelticMinstrel]

Probably going to work on this after #8791, so the new folder structure per version would be:

/userdata/[version]/cache/
/userdata/[version]/data/add-ons/
/userdata/[version]/logs/
/userdata/[version]/screenshots/
/userdata/[version]/save_index
/userdata/[version]/credentials
/userdata/[version]/sync/editor/
/userdata/[version]/sync/saves/
/userdata/[version]/sync/history
/userdata/[version]/sync/preferences

This would also then resolve #4163 as well.

Just realized that this is missing the persist directory, which should probably go at /userdata/[version]/sync/persist/.

[Pentarctagon]

True, added.