Import saves, settings, and content from a previous pack

[Pidgeot]

I've been using a pretty simple batch script for a while, but it would be really easy to do better.

• if no saves are found when PyLNP runs, offer a popup "No saves detected. Import from a previous pack?" (yes/no/don't ask again)
• GUI to navigate to the old DF directory. <handle versions, errors, etc> then copy saves over.
• additional files to copy can be glob-specified by a pack maintainer for each pack, and the union of the sets is copied to the new pack (after user confirmation).

[Issue created by PeridexisErrant: 2015-11-27]
[Last updated on bitbucket: 2016-08-07]

[Comment created by PeridexisErrant: 2016-08-07]
Closed by pull request #65.

Future work may add further import strategies, as noted in core/importer.py, but is not a high priority.

[Comment created by PeridexisErrant: 2016-07-22]
'Old DF folder' can be anywhere; and I think I only removed my todo notes on sanity-checking to avoid diff-clutter. I'll add a decent amount of logging, since that's enough of an interface to hook it all up to, and better messages / return options is part of that.

Future upgrades... I have lots of ideas, but the big one is ship something. Everything else can wait, and honestly I don't see it as a priority.

[Comment created by Pidgeot: 2016-07-21]
This should work for a basic UI, though you may want to tweak some text or something:

#!patch
diff -r f88d2c9baaa4 tkgui/tkgui.py
--- a/tkgui/tkgui.py	Mon Jul 11 03:15:32 2016 +0200
+++ b/tkgui/tkgui.py	Fri Jul 22 00:43:00 2016 +0200
@@ -16,6 +16,7 @@
     from tkinter.ttk import *
     import tkinter.messagebox as messagebox
     import tkinter.simpledialog as simpledialog
+    import tkinter.filedialog as filedialog
     import tkinter.font as tkFont
     #pylint:disable=redefined-builtin
     basestring = str
@@ -26,6 +27,7 @@
     from ttk import *
     import tkMessageBox as messagebox
     import tkSimpleDialog as simpledialog
+    import tkFileDialog as filedialog
     import tkFont
 # pylint:enable=wrong-import-order
 
@@ -61,7 +63,7 @@
 from core.helpers import get_resource
 from core.lnp import lnp, VERSION
 from core import df, launcher, log, paths, update, mods, download, baselines
-from core import terminal
+from core import terminal, importer
 
 has_PNG = has_PIL or (TkVersion >= 8.6)  # Tk 8.6 supports PNG natively
 
@@ -380,6 +382,10 @@
             menu_file.add_command(
                 label="Reload/Choose DF folder", command=self.reload_program)
 
+        menu_file.add_command(
+            label='Import from previous install...',
+            command=self.migrate_settings)
+
         if sys.platform != 'darwin':
             menu_file.add_command(
                 label='Exit', command=self.exit_program, accelerator='Alt+F4')
@@ -701,4 +707,17 @@
                 self.root.title(),
                 'All settings reset to defaults!')
 
+    def migrate_settings(self):
+        """Migrates settings from a previous DF install."""
+        old_df = filedialog.askdirectory(
+            title='Locate previous DF...', mustexist=True)
+        if old_df:
+            if importer.do_imports(df):
+                messagebox.showinfo(
+                    self.root.title(), 'Settings have been imported.')
+            else:
+                messagebox.showinfo(
+                    self.root.title(),
+                    'Import failed, please check the log for details.')
+
 # vim:expandtab

A couple of notes regarding this:

• It would be nice if the import method would return an error code or text of some sort, so the UI could display something more sensible on failure
• You're probably going to want to add a sanity check that the selected folder is even a DF folder to begin with - that's probably best suited for the importer itself, IMO.
• Eventually we might want to have a fuller UI to only import certain things? Not sure about that, and not something that's necessary for now, at least

[Comment created by PeridexisErrant: 2016-07-21]
https://bitbucket.org/PeridexisErrant/python-lnp/branch/import-user-content#diff

A simple backend, which should easily support more sophisticated import strategies in future - currently either copies files or dirs (non-recursive, no overwriting) or prepends logfile content.

@Pidgeot - if this looks workable to you, do you mind adding the interface for it? All it needs is a menu entry to call it, and a 'choose old DF folder' dialogue when called.

[Comment created by Pidgeot: 2016-07-21]
Will take a look when I can, should be able to do that this week.

UI shouldn't be an issue - but just to clarify, is that "old DF folder" in the same base directory, or an arbitrary place on the harddrive?

[Comment created by PeridexisErrant: 2016-07-12]
Oh, of course - the whole point of using PyLNP instead of the old simple shell script is to do it 'properly'.

[Comment created by Pidgeot: 2016-07-12]
Regarding d_init.txt and init.txt, do note that they state the following:

WARNING: Do NOT copy over the (d_)init.txt from an earlier version of DF.
Always read the file carefully, including the comments.

We can probably do some kind of merging with PyLNP, but simple copying should probably be avoided. (This sort of thing might also apply to other files in that directory.)

[Comment created by PeridexisErrant: 2016-07-12]
Discussion on Reddit about what to import to a new pack, including a nice list of files: https://www.reddit.com/comments/4s630y//d584mhh

[Comment created by kazimuth: 2016-01-21]
Why not store user-generated files versioned?

So you have $userprofile/34.11, $userprofile/42.05, etc.

With non-breaking upgrades - insofar as those exist - it's a straight upgrade path, you just copy the folder. You could also allow users to unpack a downloaded pack over an existing one, so they don't have to mess around with copying files.

With breaking upgrades, you display a warning and leave files in place, and create a new folder.

There are issues with finding $userprofile, and having multiple $userprofiles on a system - like if you're using both a USB drive and a permanent installation. I think those can be worked out though.

(You could even allow users to make their own custom versions for mods - $userprofile/42.05-masterwork, etc.)

[Comment created by fricy: 2016-01-16]
USB drives would benefit from this, as I have stated I don't want the paths to be hardcoded, and it'd be very easy to add a simple browse dialog panel to the launcher so the user can tell Pylnp to look for the files on usb. Or wherever. Or just add a fallback check, so if there's a User Content folder next to the launcher use that, if there isn't, look in Documents. If nothing else: ask the user if it's a first start, and create from scratch, or import from somewhere. etc.

And please give me an example where versioning would be a problem, I have a hard time imaging a valid scenario where things would be worse than now.

[Comment created by PeridexisErrant: 2016-01-16]
For much the same reasons given in #18, I'm not keen on this - for additional cons, consider the problems of versioning, portable versions (eg on a USB drive), and multiple incompatible versions in parallel.

However, this has encouraged me to think of backup-old-pack and import-from-backup functions. The obvious intermediate format is a zip archive with subfolders. When creating a backup, subdirs in the archive are created based on a PyLNP.json setting - some (saves, macros, etc) would be standard and others added by pack authors. When importing, known subdirs would be unpacked automatically and the user asked about any unknown ones.

The copy-from-old-pack config should probably be similar and reusable, though I think it should still be a separate feature.

[Comment created by fricy: 2016-01-15]
I'd like to propose a different approach:
Instead of copying settings/files from previous version, my preferred method would be to store all User generated content under username/documents/folder (largely as proposed in my previous suggestion), and when a new version is installed/detected let's just create a backup of the save files for safety reasons, and create symlinks in DF that point to the correct user folders. If the User generated content folder doesn't exists the launcher should create it from scratch, if it's there, then just link the content to the appropriate folders in df.

These paths should be stored in Pylnp.json like:

"Userdata": [
[
"user save",
"%userprofile%/Documents/Dwarf Fortress Content/Saves"
],
[
"target save",
"/data/save"
],
[
"user macro",
"%userprofile%/Documents/Dwarf Fortress Content/Macros"
],
[
"target macro",
"/data/init/macros"
],
[
"user soundsense",
"%userprofile%/Documents/Dwarf Fortress Content/Soundsense content/packs"
],
[
"target soundsense",
"LNP/Utilities/soundsense/packs"
],
etc.

To make the system as flexible as possible we should be able to add new path to the config with the user/target designation to mark which paths should be symlinked by the launcher.
Perhaps the save folder should even be "%userprofile%/Documents/Dwarf Fortress Content/Saves/packversion, that way we can let multiple packs use the same folder at the same time, each seeing its own saves, but all the files without interfering with the other.

Pros:
All user data is at the same place, so backing up (on the user's end) is easier
User macros, stockpile settings, soundsense files can be preserved without hassle
Less files to copy, faster upgrades (only the save folder needs to be copied to backup, everything else can left alone)
OSX pack would become self-contained, and easier to upgrade
DF takes less space: eg: soundsense tracks need only be downloaded and stored once, but can be used by multiple versions at the same time. I'm not sure how much zwei uses for soundsense distribution, but the reduction from this could be substantial for him.

Cons:
Harder to implement than a copy script.
Needs a backup handling function in the launcher to clean up uneeded saves/restore older saves.
Not sure how well symlinks work on windows XP. It's a breeze on *nix, and everything post Vista should be OK.