Migrate persistent data from old versions

[ThePotatoGuy]

Is there any reason to not use the existing persistent variables structure Renpy provides instead of creating a separate file handler?

I was able to save dicts and lists using those persistent variables, and I think those structures are sufficient for saving already said text / playernames / dates / whatever.

I'd be happy to implement this kind of saving / loading if you want.

[therationalpi]

I didn't describe the problem well.

The current version uses persistent variables that are stored outside of the game's directory. Those persist between version changes and updates just fine.

The problem is that there's no handler right now for when those data structures need to be changed because of API changes, and some of the logic is messed up as far as updating persistent variables when new content is added. A tangentially related issue is that there's not a good way to pull in persistent variables from the player's DDLC game, which breaks some of the immersion for Monika's fourth wall-breaking.

But here's an example of what I mean...

Between version 0.2.2 and 0.3.0, I changed the persistent variable for tracking which topics Monika had already used. Originally, it was a list of numbers, because all of Monika's topics in DDLC were numbered sequentially. In 0.3.0, this was switched over to each topic having a unique topic_id, and the persistent variable for tracking unheard topics was changed to a list of strings for the topic id's. There wasn't a handler for converting between those, so if you heard topic number 1 in version 0.2.2, then switched to version 0.3.0 where topic 1 was renamed "monika_god" you might end up hearing the same topic twice.

Similarly, whenever version 0.3.4 gets pushed out, there will be new topic_ids that need to be added to the persistent random topic list on the first load after the update. There needs to be some logic added to check if there's a new version since the last load (there's a persistent version number already, so that check is easy), and then make any appropriate changes to bring the data structures up to date with the new version.

[ThePotatoGuy]

Ah I see what you mean now. In that case I have an idea for handling this then.

The following structures would be needed:

• list that contains seen topic ids (if there isnt one already)
• dict where the key is a version number, value is a dict consisting of:
  • key: old topic id
  • value: new topic id
    • this would be for adjusting topics that change ids
    • this also would grow substantially over time, but there isn't much that can be done here besides maybe reading from xml/json or something (if renpy allows that)
• list of all topics (including new ones) (basically just regenerating the list upon a new version)
  • this would also be temporary

The following logic would take place upon a version change:

1. If a fresh game:
   a. save the list of all topics as the unseen list
2. otherwise:
   a. The version number dict would be iterated through first, adjusting both and seen and unseen topics accordingly
   b. the list of all topics would be iterated through:
   1. if a topic id is in the seen or unseen list, move on
   2. otherwise add that topic id to the unseen list

If this seems like a good design to you, I can start working on it tomorrow after work

[therationalpi]

Honestly, I think taking a more straightforward approach would be better, here. No need to try generalizing a specific problem.

The best way I can see to do this is iterative. You check what version the current save has, then you just update that to the next version. Keep updating the file, version by version, until you get to the current release. That way, when we come out with a new release, we just have to include the specific changes needed to update from the last release.

The end result is that you have a series of update scripts.

update_ddlc_0.2.2
update_0.2.2_0.3.0
update_0.3.0_0.3.1
update_0.3.1_0.3.2
update_0.3.2_0.3.3

Each feeds into the next in the chain, and does whatever specific changes to the persistent structures are needed.

The nice thing about doing it this way is that it's flexible for idiosyncratic changes between versions.

[ThePotatoGuy]

Would these update scripts be separate files or ran as python sections in a, lets say, update.rpy file of some sort?

Also would the replacement logic be similar to what I've described in terms of updating id names and adding new topics to the unseen list?

[therationalpi]

I think so.

Here's the instantiation for the random topic list in version 0.2.2. You can see it's just a range of numbers from 1 to 60, with two scripts removed.

To compare, here's an entry in the analogous structure in version 0.3.0.

So, to go from 0.2.2 to 0.3.0, you would loop through persistent.monikatopics, and for each number use your dictionary to pull the corresponding id and add it to a new persistent.monika_random_topics structure, also, you'd need to set persistent.monika_random_built to 1. So:

1 -> 'monika_god'
2 -> 'monika_death'
3 -> 'monika_bad_day'
...

You would also want to append every new id that was added to the random list, that wasn't part of the original 59 scripts, which would include everything after "monika_route" that includes a line like

if not (persistent.monika_random_built) : persistent.monika_random_topics.append('monika_programming')

Just to save you the trouble, I used grep to pull out every key for version 0.3.0

'monika_god'
'monika_death'
'monika_bad_day'
'monika_sleep'
'monika_sayori'
'monika_japan'
'monika_high_school'
'monika_nihilism'
'monika_piano'
'monika_twitter'
'monika_portraitof'
'monika_veggies'
'monika_saved'
'monika_color'
'monika_music'
'monika_listener'
'monika_spicy'
'monika_why'
'monika_okayeveryone'
'monika_ghosts'
'monika_archetype'
'monika_tea'
'monika_favoritegame'
'monika_smash'
'monika_lastpoem'
'monika_anxious'
'monika_friends'
'monika_college'
'monika_middleschool'
'monika_outfit'
'monika_horror'
'monika_rap'
'monika_wine'
'monika_date'
'monika_kiss'
'monika_yuri'
'monika_writingtip'
'monika_habits'
'monika_creative'
'monika_deleted'
'monika_keitai'
'monika_simulated'
'monika_rain'
'monika_closeness'
'monika_confidence'
'monika_carryme'
'monika_debate'
'monika_internet'
'monika_lazy'
'monika_mentalillness'
'monika_read'
'monika_festival'
'monika_tsundere'
'monika_introduce'
'monika_cold'
'monika_housewife'
'monika_route'
'monika_programming'
'monika_vn'
'monika_credits_song'
'monika_poetry'
'monika_vidya'
'monika_books'
'monika_natsuki'
'monika_hedgehog'
'monika_freewill'
'monika_technique'
'monika_contribute'
'monika_drawing'
'monika_mc'
'monika_waifus'

You can use the same approach to find all the new keys in 0.3.1 and 0.3.2 (they should be the last ones in the list, or you can use a diff command to strip out all the old ones)

Just a heads up on this, the 0.3.0 and 0.3.1 releases were on my fork before I was added as a contributor on this repo. If you need to look at those for reference.

[therationalpi]

Like I said, each version change is idiosyncratic, and I'll take better care to consider backwards compatibility going forward.

Best way to figure out what to compare each major release to the previous, and find every place with a new "persistant." variable. Compare view is really good for this.

[ThePotatoGuy]

Sounds good, I'll get started on this.

[therationalpi]

Handled excellently! Thanks so much!

#30