Autogems UI save issue

[wiktor-obrebski]

Hi.
Thanks for you work on DFHack, amazing tool.
I still learning new plugins and today I tried to use autogems. So, I trying to define autogems rules in the UI: o -> W -> G
I make some changes and use "Esc" to save.
In the moment the plugin seems to have a crash. Game works correctly, but I get following error in the DFHack console.

./hack/lua/json.lua:25: Could not write to /opt/dwarffortress/data/save/DeepDeep/autogems.json
stack traceback:
	[C]: in function 'error'
	./hack/lua/json.lua:25: in function 'json.encode_file'
	/opt/dwarffortress/hack/scripts/gui/autogems.lua:105: in global 'save'
	/opt/dwarffortress/hack/scripts/gui/autogems.lua:237: in function </opt/dwarffortress/hack/scripts/gui/autogems.lua:236>
	[C]: in ?
	[C]: in field 'dismiss'
	./hack/lua/gui.lua:556: in method 'dismiss'
	/opt/dwarffortress/hack/scripts/gui/autogems.lua:229: in function </opt/dwarffortress/hack/scripts/gui/autogems.lua:222>
	[C]: in ?
./hack/lua/json.lua:25: Could not write to /opt/dwarffortress/data/save/DeepDeep/autogems.json
stack traceback:
	[C]: in function 'error'
	./hack/lua/json.lua:25: in function 'json.encode_file'
	/opt/dwarffortress/hack/scripts/gui/autogems.lua:105: in global 'save'
	/opt/dwarffortress/hack/scripts/gui/autogems.lua:237: in function </opt/dwarffortress/hack/scripts/gui/autogems.lua:236>
	[C]: in ?
	[C]: in field 'dismiss'
	./hack/lua/gui.lua:556: in method 'dismiss'
	/opt/dwarffortress/hack/scripts/gui/autogems.lua:229: in function </opt/dwarffortress/hack/scripts/gui/autogems.lua:222>
	[C]: in ?

The changes in gems are not saved after all.
I am playing on Linux.
Version: DFHack version 0.47.04-r2 (release) on x86_64

[lethosor]

Does the /opt/dwarffortress/data/save/DeepDeep folder exist (if not, which of its parents exist, if any)? Do you have permission to write to it (or any of its parents)? Did you install DF/DFHack through a package manager?

[wiktor-obrebski]

no. I have dwarffortress install at ~/.dwarffortress directory. I have full permissions to folders inside.
my save is in ~/.dwarffortress/data/save/DeepDeep.
I used package manager to install dfhack.
I missed the broken path before. I will check what's going on, maybe I have double installation or something. Willl let you know here.

[wiktor-obrebski]

ok. I am on linux arch OS and I used yay package manager (it's overlay app on classic arch pacman package manager).
I see now that dfhack and Dwarf_Fortress is installed in /opt/dwarffortress/... directory.
but dwarffortress saves are localed in user directory ~/.dwarffortress.

In the same folder there are symlinks to all needed files:

# ls -l ~/.dwarffortress
dfhack -> /opt/dwarffortress/dfhack
dfhack-run -> /opt/dwarffortress/dfhack-run
hack -> /opt/dwarffortress/hack
libs -> /opt/dwarffortress/libs
raw -> /opt/dwarffortress/raw
stonesense -> /opt/dwarffortress/stonesense
# ... and more

but data directory (where saves are stored) are only in ~/.dwarffortress/data directory. it is not located in /opt/dwarffortress.

I just fixed the problem by creating a symlink:
sudo ln -s ~/.dwarffortress/data/save /opt/dwarffortress/data/save
and it helped, but it's quite nasty in my opinion.

I am not sure if other linux users have same problem, or it just specific issues with arch package manager installation.

[lethosor]

Interesting. Just to make sure there aren't any other symlinks in play, does realpath /opt/dwarffortress/libs/Dwarf_Fortress print /opt/dwarffortress/libs/Dwarf_Fortress or something else?

To my knowledge, DF always looks for saves in the data/save folder relative to itself (on Linux, relative to the libs folder). I am genuinely not sure how this could work without the symlink you created. Lots of things in DFHack calculate the save path in the same way, so I would try reporting this to the Arch package maintainer - I suspect the symlink you made (or a similar one for the top-level data folder) is necessary.

[wiktor-obrebski]

realpath /opt/dwarffortress/libs/Dwarf_Fortress prints /opt/dwarffortress/libs/Dwarf_Fortress.

I guess the maintainer goal was to be possible to play dwarffortress by few users with my game installation, but with separated worlds.

To my knowledge, DF always looks for saves in the data/save folder relative to itself (on Linux, relative to the libs folder). I am genuinely not sure how this could work without the symlink you created.

Since now I had no problems with raw game, dfhack and soundsense.

[lethosor]

On second thought, it's possible that 56e43a0 broke it (this change was first in 0.47.04-r2, which could make sense). Basically, getHackPath() and functions it depends on, like getSavePath(), used to return the current working directory, but now return the directory containing libs. This is consistent with other platforms now (and the previous behavior was a bug that interfered with DFHack/scripts#152) but if the Arch package sets the working directory to ~/.dwarffortress to make DF save there, that's no longer enough for DFHack tools to find the right save path.

@Ziusudra it looks like you're a maintainer of the Arch package, right? I'm not sure what the best way to fix this is, but I'd be happy to make any necessary core changes.

[Ziusudra]

Yes, I'm a co-maintainer. This also causes exportlegends to fail unless an absolute path is provided. (see comments on the package page.)

Yeah, so, the DF package makes DF multiuser by having the common game files in system folders and the user specific files in ~/.dwarffortress/ along with links to the system files. (That is an official package maintained by someone else.) The dfhack packages attempt to work with that.

The only things I can think of is an option that the launcher script could invoke to tell dfhack to use the CWD. Or for dfhack to check if the CWD contains the libs link to what it found for exe_path and if it does use CWD for cached_path.

[lethosor]

Just to clarify: the DF package works by setting the working directory to ~/.dwarffortress on startup, which causes DF to look there for all of its folders, and the non-writable folders are symlinks to corresponding folders in /opt?

Does the DFHack package install any actual files in ~/.dwarffortress, besides symlinks to /opt? dfhack-config is about the only user-writable folder I can think of, but I'm not sure how best to check the behavior of the package from a non-Arch system.

I think the best thing to do here is to look for user-writable locations (data/save and dfhack-config) relative to the DF root (/opt/dwarffortress in this case), then in the current directory (~/.dwarffortress) if they don't exist in the DF root. This would minimize the risk of breaking existing installations where everything is in just one folder. Would this work, or do the Arch packages also set up corresponding folders in /opt? It sounds like /opt/dwarffortress/data/save does not exist, which is good, but I'm not sure about /opt/dwarffortress/dfhack-config or others.

Sorry about the number of questions - I think I have a good idea of how to do this if my assumptions are correct. There are a relatively small number of places in DFHack that deal with user-writable folders currently, but none of them are quite the same (Core.cpp alone has 3 different ways of finding a save folder!) so there is some risk here.

[Ziusudra]

Just to clarify: the DF package works by setting the working directory to ~/.dwarffortress on startup, which causes DF to look there for all of its folders, and the non-writable folders are symlinks to corresponding folders in /opt?

Mostly, the data/ folder is slightly complicated. The DF package installs all of DF's files to /opt/dwarffortress/. It includes a launch script which is here. When run by a user that checks if ~/.dwarffortress exists and if not it creates it, links the raw/ and libs/ folders, and copies data/init/. The script then always copies other files from data/ to make sure they're updated (and should be using the --update argument for cp, but that's not relevant). It finallly cds to ~/.dwarffortress and executes ./libs/Dwarf_Fortress "$@"

Does the DFHack package install any actual files in ~/.dwarffortress, besides symlinks to /opt? dfhack-config is about the only user-writable folder I can think of, but I'm not sure how best to check the behavior of the package from a non-Arch system.

The DFHack package does much the same and it does copy some files rather than just linking. Its scripts are here. The PKGBUILD is the script that builds the package and the *.sh scripts are what run DFHack executables. The launch scripts (link) do everything that the DF one does and also check to make sure the dfhack folders have been linked. However, those launch scripts copy the /opt/dwarffortress/dfhack-config/ folder (and dfhack.init-example as dfhack.init) so that it can be unique to each user.

I think the best thing to do here is to look for user-writable locations (data/save and dfhack-config) relative to the DF root (/opt/dwarffortress in this case), then in the current directory (~/.dwarffortress) if they don't exist in the DF root. This would minimize the risk of breaking existing installations where everything is in just one folder. Would this work, or do the Arch packages also set up corresponding folders in /opt? It sounds like /opt/dwarffortress/data/save does not exist, which is good, but I'm not sure about /opt/dwarffortress/dfhack-config or others.

So, all of DF's and DFHack's folders do exist in /opt/dwarffortress/ but none of them are user writable. The dfhack-config/ folder will exist in ~/.dwarffortress/ - as a copy not a link - and be user writable. The data/save/ folder will not exist anywhere until DF creates it in ~/.dwarffortress/ and I'm not sure when that happens - might not happen until the first time DF tries to save. Even if it happens the first time DF launches will it happen before DFHack goes looking for it? Either way, a writable ~/.dwarffortress/dfhack-config/ will exist (unless the user does something bad like run out of storage space).

Sorry about the number of questions - I think I have a good idea of how to do this if my assumptions are correct. There are a relatively small number of places in DFHack that deal with user-writable folders currently, but none of them are quite the same (Core.cpp alone has 3 different ways of finding a save folder!) so there is some risk here.

Not a problem, glad to do what I can to help.

[lethosor]

So, all of DF's and DFHack's folders do exist in /opt/dwarffortress/ but none of them are user writable. The dfhack-config/ folder will exist in ~/.dwarffortress/ - as a copy not a link - and be user writable.

Got it, so I should do the opposite: check in the working directory first, then in the DF root. The working directory essentially never changes on other platforms, so that should be fine.

The data/save/ folder will not exist anywhere until DF creates it in ~/.dwarffortress/ and I'm not sure when that happens - might not happen until the first time DF tries to save. Even if it happens the first time DF launches will it happen before DFHack goes looking for it? Either way, a writable ~/.dwarffortress/dfhack-config/ will exist (unless the user does something bad like run out of storage space).

It's not possible to load a save if the data/save folder doesn't exist, so we hopefully won't run into any issues there. The only possible one I can think of is that during world generation, the name of the region folder is determined by DF and stored in its usual place in memory, but not created until the world is saved. This is already an edge case that isn't addressed cleanly - the only change on Arch is that data/save wouldn't exist yet either, but I'm pretty sure that was also the case before 0.47.04-r2 (and it might actually get created earlier, to allow temporary data to be stored in data/save/current).

Glad to see that the package already handles dfhack-config! I hadn't thought of data/init, but I don't think DFHack does anything with that.

[lethosor]

I've changed my mind somewhat - I now think that the most straightforward approach would be to imitate the previous behavior and define the DF path as the initial working directory. It should effectively behave the same way as before 0.47.04-r2, and would hopefully fix all package-related issues like this one.

The issue with exportlegends that led to 56e43a0 was that getDFPath() would always return the working directory on Linux, so it couldn't be used to change back to the DF path after calling chdir(). #1672 has added some support for tracking the initial working directory, so I think changing getDFPath() to return that (on all platforms, or at least Linux/macOS) would resolve the issue.

[lethosor]

This should be fixed as of #1672 (so it'll be in r4). You can also grab an unstable build from https://dfhack.org/builds if you want to test it out, although installing it over a package manager-provided DFHack installation could be risky.

[Ziusudra]

I uninstalled dfhack, made a test package using DFHack version 0.47.04-r3 (development build 0.47.04-r3-115-gfce10e59) on x86_64 [build ID: 201112001], and installed that.

It was able to successfully do exportlegends all. It created /home/ziusudra/.dwarffortress/legends-region1-00125-01-01, populated it with the expected files, and did not report any errors.

I was also able to go into a fortress and change the autogems options without any error.

Anyone too impatient to wait for the release can do a manual DF+DFHack install : https://dwarffortresswiki.org/index.php/DF2014:Installation#Manual_or_multiple_installations

[lethosor]

Awesome, thanks for checking!