Original Arena saved game decoding and import

[Allofich]

This may already be in your longer-term plans, but I think it would be very useful for testing and reverse-engineering the original game data to be able to import native save game files from original Arena, as Daggerfall Unity can do with saved games from original Daggerfall.

There are some saved game editors on UESP for Arena, and these are useful for finding out how data is stored in the save files.

[Allofich]

Here are basics on how Arena handles saved game files, based on testing with a newly created character on a fresh Arena installation:

Files modified when saving, where ## is the save number (00, 01, etc.):
AUTOMAP.64 (Identical to AUTOMAP.##?)
AUTOMAP.##
CITYDATA.## (Has city names)
LOG.## (empty on new game)
NAMES.DAT
SAVEENGN.## (contains character stats data)
SAVEGAME.##
SPELLS.## (empty on new game, even with a spellcaster. Maybe records custom-created spells?)
SPELLSG.## (standard spells for sale?)
STATES.## (empty on new game)
WILD001.## (wilderness data probably)
WILD002.##
WILD003.##
WILD004.##

Files modified when program started:
CITYDATA.64
LOG.64
SPELLS.64
SPELLSG.64
STATES.64

Files modified when loading game:
AUTOMAP.64
CITYDATA.64
LOG.64
SAVEENGN.64
SAVEGAME.64
SPELLS.64
SPELLSG.64
STATES.64
WILD001.64
WILD002.64
WILD003.64
WILD004.64

Temporary files that can be created. Deleted on program close:
CHARSHET.$$$
SAVEFACE.$$$
SCREENSV.$$2
TMP2.$$$

**.64 files remain even after closing the program, but are probably active data being used in the current session.

[Allofich]

NAMES.DAT

Saved game names that appear on the load screen. No actual link to the rest of the save data.
Each name can have up to 47 characters. This leaves space for at least 1 00 to terminate the string.
Any leftover character spaces are filled out with 00.

[Allofich]

LOG.**

Journal entries.
Format (all numbers stylized as "code" are in hexadecimal):
26 (&) - Skips down 1 line, hides preceding 2 characters if they would render and sets following text to be rendered in red color, but only if there are 2 or fewer characters between the start of the file and 26, or if 26 was preceded by 2A (perhaps other conditions as well). Otherwise all that happens is "&" is rendered. Used before date entries. Red color lasts until the next "new line" (0A or 2A).
0D - appears at end of date entry. Doesn't render anything, purpose unknown.
0A - new line
20 - appears before 2A at end of text entries. Doesn't render anything, purpose unknown.
2A (*) - Used at end of text entry. Causes previous byte to not be rendered if it would have rendered something, starts a new line.

Basic usage for a log entry seems to be:
26 Date entry 0D 0A
Text entry (using 0A for new lines as needed) 20 2A

26 (&) behavior seems a bit weird, but for example:
If the file starts with:
&Text, a&Text or aa&Text then a line is skipped and "Text" is rendered in red, with the preceding "a"s not shown.
If the file starts with aaa&Text or with more before the &, it will just be rendered as it is, in the default color and without skipping a line.
If you just want to read in the automatically-generated journal entries, though, I don't think it would be necessary to replicate the behavior this exactly.

Up to 13 rows of text can appear. More than that and you get a "More..." button to turn the page.

[Allofich]

SAVEENGN.**

(all numbers stylized as "code" are in hexadecimal)
Offset:
04, byte, Race ID
Values:
60 Redguard
61 Breton
62 Dark Elf
63 Nord
64 Wood Elf
65 High Elf
66 Argonian
67 Khajiit

Setting to an invalid value will make the race name blank, and the character invisible on the character sheet and the character face in the menu bar, with a black background on the character sheet.

[Allofich]

I'll stop here and wait for feedback. I'm trying to keep this organized but I have a feeling it could easily become a mess if we aren't careful.

It would probably be good to separate out each type of data to be imported into its own issue. Anyway, I'll wait for your opinion.

[Carmina16]

I will describe the save file format in the wiki.

[afritz1]

Yeah, I was about to suggest that. Having the save file format in the wiki would be preferable for long-term records. We can keep this issue here as a reminder.

[Allofich]

I didn't even notice the wiki until now. Good to see Carmina16 has already made a lot of progress.

It might be further useful to have individual wiki pages on data elements when there is more information. For example, the valid race ID values. Or just a comment may be enough.

The text formatting and journal information (such as the maximum number of lines per journal page) also seem like they could go in the wiki.

[Allofich]

@Carmina16, I'm happy to see so much progress on the wiki, including not just saved game structure but things like travel code!

[afritz1]

I am as well. I'm not sure what the extent of their knowledge is, but they seem to know a lot about Arena's inner-workings. Although I might not have the chance to implement all of those things in the wiki right away, I plan on looking into each one over time. I've still got a lot of renderer stuff and level data I need to work on first.

[CyberistX]

I've gathered a bit more info about the spell format (which by the way is also used in the file SPELLS.LST). Referring to the structure in the wiki:

WORD Param0[3];
WORD Param1[3];
WORD Param2[3];
WORD Param3[3];
WORD Param4[3];
WORD Param5[3];
BYTE TargetType;
BYTE unk;
BYTE Element;
WORD Flags;
BYTE Effects[3]; // e.g. 'Fortify'; 0xFF = no effect
BYTE SubEffects[3]; // e.g. 'Attribute:'
BYTE AffectedAttr[3]; // e.g. 'Strength'
WORD Cost;
char Name[33];

The parameters param0-5 contain three entries, each of which is relative to one of the three possible effects a spell can have (specified in Effects[3]).

Valid values for TargetType are:

        0x00 self
	0x01 touch
	0x02 target

Values for Element are:

        0x00 Fire
	0x01 Ice
	0x02 Poison
	0x03 Shock
	0x04 Magic
	0x05 None

Possible values for Effects:

	0x00    Cause 
	0x01    Continuous Damage
	0x02    Create 
	0x03    Cure 
	0x04    Damage Health
	0x05    Designate as non target
	0x06    Destroy wall
	0x07    Disintagrate
	0x08    Dispel
	0x09    Drain Attribute
	0x0A    Elemental resistance
	0x0B    Fortify attribute
	0x0C    Heal Attribute
	0x0D    Transfer Attribute
	0x0E    Imprison
	0x0F    Invisibility
	0x10    Levitate
	0x11    Light
	0x12    Lock	
	0x13    Open
	0x14    Regenerate
	0x15    Silence
	0x16    Spell Absorption
	0x17    Spell Reflection
	0x18    Spell Resistence

Different spell effects use different parameters:

        Cause
		param0 chance to inflict {disease/poison/curse/paralysis}, deterioration is
		param3 pts of ?? every round + param1% every param4 levels. duration is
		param5 rounds per level
	
	Continuous damage {health/fatigue/spell points}
		param0 to param1 pts of damage every param4 rnds. duration is param5 rnds.
		damage is param2 to param3 pts every param4 levels

	Create Shield
		param0 hit pts shield created, +param1 pts every param4 levels

	Create Wall/Floor
		param0 walls/floors created

	Cure 
		param0% chance to cure {disease/poison/para/curse} of equal level to the
		caster. + param1% every	param4 levels

	Damage {health/fatigue/spell points}	
		param0 to param1 pts damage +param2 to param3 pts per param4 levels
	
	Designate non target
		param0% chance caster is ignored + param1% per param4 levels
		param5 rounds per param2 levels. (you may cast other spells: check flags)s

	Destroy wall/floor
		param0 walls permanently negated

	Disintagrate
		param0% chance to disintegrate +param1% every param4 levels	

	Dispel
		param0% chance + param1% every param4 level. Area is param5 meter radius

	Drain Attribute
		param0 pts of attribute drained for param5 rounds per level. Target recover
		attribute at param1 pts per param4 rounds

	Elemental resistance
		resistance to element +param0% for param5 rounds per param2 levels of caster
		+param1% per param4 levels
	
	Fortify attribute
		+param0 pts of attribute for param5 rnds per level. attribute loss of param1
		pts per param4 rounds.

	Heal attribute
		param0 to param1 pts to attribute +param2 to param3 per param4 levels

	Transfer attribute
		Tranfers param0 pts of attribute from target to caster

	Imprison
		param0 hit pts cage created, + param1 pts per param4 levels

	Invisibility
		param0 rounds + param1 rounds per param4 levels

	Levitate
		param0 rounds + param1 rounds per param4 levels

	Light
		param5 rounds per level

	Lock
		param0 chance of locking. duration is param5 rounds per param2 levels
		lock strength + param1 % per param4 levels

	Open
		param0 chance of unlocking, + param1% per param4 levels

	Regenerate
		param0 health every param1 rounds for param4 rounds per level

	Silence
		param0 chance of silencing for param5 rounds per param2 levels, + param1 % 
		per param4 levels

	Spell Absorption
		param0 chance to absorb + param1 % per param4 levels. duration is param5 
		rounds per param2 levels

	Spell reflection
		param0 chance to reflect + param1 % per param4 levels. duration is param5 
		rounds per param2 levels

	Spell resistence
		param0 chance to resist + param1 % per param4 levels. duration is param5 
		rounds per param2 levels

What needs further investigation is the meaning and possible values of Flags/SubEffects/AffectedAttr for the different Spell Effects. Hope it's helpful.

[Allofich]

Looks like player coordinates in a level are at offset 0x105C8 in SAVEGAME.xx. First a DWORD that is the X coordinate, then a DWORD that is the Y coordinate. This could be useful for cheating one's way to a particular square in an Arena save.

[Allofich]

Originally in #100

F8 - Toggle compass

Toggling compass should be a XOR of 0x2000 or 0x0020, possibly on the GameOptions WORD in the > SAVEENGN file, as described in the Wiki. For a new character I created the changing value was at offset > 0x425 in the SAVEENGN file.

[satribe]

This seemed the appropriate place, so I hope this isn't necro posting. I'd just like to leave a caution note about importing original save games into OpenTESArena. The original game has many ways that you can corrupt the save game files which end up causing glitches later on. Simply opening the "SAVE/LOAD" menu before creating a character will introduce glitches into your game. This example will cause the starting date to be the 1st of Morningstar instead of the 1st of Hearthfire as intended. This can cause many glitches and even the 1.06 patch notes recommends against using this option which is part of the NEW GAME option.
"Can't get an Artifact Quest: This seems to be related to picking the Start New Game option while in the middle of another game. We recommend that you do not choose this option."
There appear to be several ways to mess up the saved games, but the glitches show up later and we usually blame the game rather that the save file. Which is my point. I'd hate to see people loading their corrupted save files and blaming problems on OpenTESArena. Control of input is crucial. It's still a great Idea, but I'd recommend heavy-handed control over the process. (So grateful for this project that has allowed me to peek under the hood - great work!)

[afritz1]

Thanks for the info. It's sometimes reasonable to do sanitizing of data read from the original game if it means more quality of life for the modern engine. From your example though, I can't see how that would break a save if the month was different. Seems like some global variable bug in the original executable that presumably gets saved to the save file. I don't know how artifact quests were implemented originally but that bug would probably not carry over to here.

[Allofich]

If there are known problems with saves from the original game, those could be perhaps be (optionally?) patched if importing them into OpenTESArena.

I made created this issue after helping work on Daggerfall Unity. Saves from the original game were useful to have when researching and implementing various game mechanics, so I figured they would also be useful here.

Another big reason, though, is because something that happened in both Daggerfall Unity and OpenMW that would be nice to prevent as much as possible: having to worry patch or worry about conflicts with old versions of OpenTESArena-format saves. As in, say, you release OpenTESArena with save/load support (in a unique OpenTESArena format), then later change the way some kind of data is stored in order to more accurately recreate something from the original game, etc. You then have the problem of supporting, or not supporting, the old version's saves. In OpenMW, for example, I've seen "that would change the save format" often come up as an obstacle to why something can't be easily changed.

I thought that perhaps, if importing saves from the original game is implemented well, maybe for some time those could be used, perhaps exclusively, for testing/development before releasing OpenTESArena save/load capability "out into the wild", which might minimize the problem of changing save game formats, since it would be somewhat mature/complete already from staying close to the original data format. @afritz1, it's all your call of course, but I'm just letting you know why I kind of wanted to push the idea of importing original saves and opened this issue.

[Carmina16]

I don't think it's going to be a problem. Most of the save file is just a dump of internal variables not needed for this engine, so we can easily and safely cherrypick what we actually need.

[satribe]

@afritz1
The month change was just the obvious notice that your game was glitched. It was the other glitches that came along with it that were the problem. I still don't know how these cause such problems from the save data. If you press ESC when asked if you want to be Male or Female, it will default to Female, but create some weird offsets. When you try to change weapons, it will instead change your character's face. (the weapon works, but sprites in the menu are screwy.)

@Allofich
I "do" think it's a good idea and would have many benefits, I just wanted to caution to ensure it didn't carry bugs over which leads me to my next comment.

@Carmina16
That's good. I don't know how this engine works (nor the original for that matter), I just remember how frustrating it was working on a project where I had control over the program, but no control over the input files. They were all over the place. I think they thought the program was just supposed to magically know. "I only moved one thing. It's obvious." But yes, that's what I was hoping. Cherry picking which allows you to test if they are valid variables. Thanks.

[Thunderforge]

Regarding the calendar, this page says that the reference sheet included with the game shows a Tamriel-ified Gregorian calendar (which is pretty much what Morrowind and later use), but in the actual game, every month has 30 days and each month begins on Tirdas (Tuesday) the 1st, despite the each month ending on Middas (Wednesday) the 30th.

The actual implementation details are probably better suited for the Time & Calendar wiki, but I'll add my two cents for game importing. My understanding is that the game stores time as minutes since the start of the game (12:00 noon, Tirdas, 1st of Hearthfire in the year 3E 389). If we have OpenTESArena use a "fixed" calendar that matches the Arena reference sheet and later games (and I think we should instead of using the broken existing calendar) I would be okay with having the internal minutes since game start stay the same and the cosmetic display change. For instance, I would be okay with 90 days after game start being:

• Original Arena: Middas, 30th of Sun's Dusk
• OpenTESArena: Sundas, 29th of Sun's Dusk

So long as our save import documentation explains that this is happening and that this is cosmetic; no quest deadlines or whatever are being modified because internally they are still being calculated as minutes from game start.