Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Starfield #1882

Closed
pStyl3 opened this issue Jun 25, 2023 · 44 comments
Closed

Add support for Starfield #1882

pStyl3 opened this issue Jun 25, 2023 · 44 comments
Assignees
Labels
Milestone

Comments

@pStyl3
Copy link
Member

pStyl3 commented Jun 25, 2023

After Starfield was first announced during E3 2018, it is now scheduled to be released on the 6th of September 2023.

Platforms:

There might be GOG/EGS releases planned at some point, but as far as I know this hasn't been officially talked about yet.

Minimum System Requirements:

  • OS: Windows 10 version 22H2 (10.0.19045)
  • Processor: AMD Ryzen 5 2600X, Intel Core i7-6800K
  • Memory: 16 GB RAM
  • Graphics: AMD Radeon RX 5700, NVIDIA GeForce 1070 Ti
  • DirectX: Version 12
  • Storage: 125 GB available space
  • Additional Notes: SSD Required

It probably might be a good idea to calculate with more than 125 GB storage space, official DLCs and mods will come on top of that.

Videos:

Creation Engine 2

https://twitter.com/BethesdaStudios/status/1404124165658009601

Starfield will be our first game to use Creation Engine 2. We've spent years developing it to power the next generation of immersion and exploration.

Starfield is build using the CE 2, which is said to be vastly improved compared to it's predecessors. There are some snippets of information floating around, but not much is known until now. We should very carefully observe everything around the CE 2, since it might mean that there are substantial differences in regards to load order as well.

Of course, besides updating the LOOT application itself, a new masterlist will need to be created, too.

Closing Thoughts:
I bought the Digital Premium Edition from Steam, as such I will have early access to it 5 days prior to the official release, and I will also get the first planned story expansion, aka DLC "Shattered Space". For testing purposes, it might be good to have a few people owning the game through the Microsoft Store, any help on this front will be welcome!

Let's see how the game will perform and get received, I'm looking forward to it.

@pStyl3 pStyl3 added the backend label Jun 25, 2023
@loot loot deleted a comment Jul 6, 2023
@pStyl3
Copy link
Member Author

pStyl3 commented Aug 17, 2023

Starfield has gone gold

Starfield has gone gold! Preloads begin tomorrow for @Xbox
X|S and Windows PC and August 30 for Steam.

People have started preloading the game now, and while the files are still encrypted, the files within Starfield's game folder (within this "preloaded state") are:

\Config
\D3D12
\Data
\media
appxmanifest.xml
bink2w64.dll
CreationKitTooltips.ini
DefaultGlobalGraphicsSettings.json
gamelaunchhelper.exe
High.ini
layout_ff5f9afb-a689-44d8-85cd-06cf9139122e.xml
Low.ini
Medium.ini
MicrosoftGame.Config
resources.pri
ssce5564.dll
Starfield.exe
Starfield.ini
Starfield_it.ini
Starfield_pl.ini
Starfield_ptbr.ini
Starfield_zhhans.ini
Ultra.ini
XCurl.dll

The files within the Data folder are also known, though in order to avoid potential spoilers, I'm only listing the files that are relevant for us:

Starfield.esm
BlueprintShips-Starfield.esm
BlueprintShips-Starfield - Localization.ba2
Constellation.esm
Constellation - Localization.ba2
Constellation - Textures.ba2
OldMars.esm
OldMars - Localization.ba2
OldMars - Textures.ba2

As such the .esm plugin format as well as the .ba2 archive format will continue to be used.
It remains to be seen, if they have been updated for Starfield in any way, shape or form.

@Ortham
Copy link
Member

Ortham commented Sep 1, 2023

I've taken a look through the code to see what questions need to be answered for Starfield support:

  • What is Starfield's minimum plugin header version?
    • All 4 plugins have a HEDR subrecord with the first 4 data bytes being 8F C2 75 3F, which is ~ 0.96 as a float32, suggesting that's it.
  • What (if any) generic Registry key install path value gets created by Starfield?
    • There is no launcher, pressing Play in Steam immediately launches the game, and it doesn't appear to create a relevant Registry key.
  • Does the Steam install path contain an installscript.vdf file?
    • No, but it has a steam_api64.dll file.
  • What is Starfield's app name on the Microsoft Store?
    • BethesdaSoftworks.ProjectGold
  • What local app data folder name does the Microsoft Store version of Starfield use?
    • Starfield
  • What local app data folder name does the Steam version of Starfield use?
    • Starfield
  • Does Starfield support activating plugins using sTestFile1 through sTestFile10 ini file properties?
    • Yes, the properties are recognised in the [General] section of Starfield.ini, Starfield_<lang>.INI (e.g. Starfield_en.INI) and My Games/Starfield/StarfieldCustom.ini. The 3 files are merged in that order, so properties defined in the last override earlier inis. All 3 files are encoded using Windows-1252. The numbering of the ini properties does not necessarily reflect plugin load order.
    • How does Starfield decide which Starfield_<lang>.INI file to load?
      • Steam installs use the language configured in Steam's settings for Starfield. (Instructions here). The language is stored in Steam's appmanifest_1716740.acf file as the value of AppState.UserConfig.language, and the value there needs to be mapped to the language code used by Starfield:
        "english" => "en",
        "german" => "de",
        "french" => "fr",
        "italian" => "it",
        "spanish" | "latam" => "es",
        "schinese" => "zhhans",
        "japanese" => "ja",
        "polish" => "pl",
        "brazilian" => "ptbr",
        
      • MS Store installs apparently use the first preferred language configured in Windows' settings, but I haven't gotten Starfield to launch in any language other than English.
  • Does Starfield support light plugins?
    • Yes, but the flag is 0x100.
    • If Starfield supports light plugins, what is their valid object index range?
      • 0x0 to 0xFFF
  • Does Starfield support .esl plugins?
    • Starfield.exe contains thet strings .ESM, .ESP and .ESL next to each other. It also contains .esp in a different location.
    • Yes
    • Does having an .esl extension cause the game to set the light plugin flag on load?
      • Yes, I renamed my test plugin to test.esl, made sure it didn't have the flag set, added it to StarfieldCustom.ini as sTestFile1=test.esl under [General], and then used FindForm to check its record's FormID in game.
    • Does having an .esl extension cause the game to set the master flag on load?
      • Yes, I created two plugins that both overrode the Eon pistol's record, test.esl and test.esp. test.esl changed the Eon's name to ESL and test.esp changed the Eon's name to ESP. Neither plugin had the master flag set. I then added both to StarfieldCustom.ini as sTestFile1=test.esp and sTestFile2=test.esl. In-game, the pistol's name was ESP. I then renamed test.esl to test2.esp, and the in-game name was then ESL.
  • Does having an .esm extension cause the game to set the master flag on load?
    • Yes, I created two plugins that both overrode the Eon pistol's record, test.esm and test.esp. test.esm changed the Eon's name to ESM and test.esp changed the Eon's name to ESP. Neither plugin had the master flag set. I then added both to StarfieldCustom.ini as sTestFile1=test.esp and sTestFile2=test.esm. In-game, the pistol's name was ESP. I then renamed test.esm to test2.esp, and the in-game name was then ESM.
  • What are the rules for what plugins load what BA2 archives?
    • Looks like a plugin A.esm can load A - Localization.ba2, A - Textures.ba2 and A - Voices_en.ba2 (replacing en with the currently configured language code, i.e. the current value of sLanguage in the game ini file).
  • Is plugin load order even relevant to Starfield?
    • Looks like yes, records are still used, they've still for 4-byte FormIDs that can get overridden by later-loading plugins.
  • How is load order stored in Starfield?
    • No plugins.txt is created when Starfield launches, however Starfield.exe does contain the string %sPlugins.txt. The same is true for SkyrimSE.exe: the %s is probably there as a placeholder for the parent path.
    • If I use the sTestFile1 ini property with a handmade test.esm plugin that overrides the "Eon" pistol's record (000476C4), I can see the change in-game. However, if I create a plugins.txt containing *test.esm instead of using the ini property, I do not see the change in-game.
    • However, in Process Monitor I can see that Starfield.exe does find and open test.esm a bunch of times, and plugins.txt is also accessed. There are a few more interactions with Constellation.esm and OldMars.esm than test.esm though.
    • If there is no plugins.txt, I can see attempts to access it using the filename Plugins.txt.
    • I've found code in Starfield.exe that reads Plugins.txt, ignoring blank lines and lines starting with #, and treating lines starting with * differently (operating on the rest of the line), which suggests that it's the same system as Skyrim SE and Fallout 4, but that it's currently disabled somehow.
    • There's now an SFSE plugin available that enables plugins.txt and Starfield.ccc, or you can just rename BlueprintShips-Starfield.esm. I've verified they work the same way as in Fallout 4.
  • How does Starfield keep track of which plugins are active?
    • As above, it looks like plugins listed in Plugins.txt that are prefixed with an asterisk are treated as active.
  • Which plugins are hardcoded to load?
    • Starfield.exe contains the strings TestData.esm, BlueprintShips-Starfield.esm, Constellation.esm, OldMars.esm. Deleting any of the latter three (the first doesn't exist) causes the game to warn when loading a save, saving content is missing. Deleting Starfield.esm causes the game to crash before it reaches the title screen.
  • What order are the plugins hardcoded to load in?
    • Starfield.esm, Constellation.esm, OldMars.esm, BlueprintShips-Starfield.esm.
    • BlueprintShips-Starfield.esm is hardcoded to be active but does not have a hardcoded load order position: if other masters are listed in plugins.txt and it is not, it will load after them.
  • What is the structure of Starfield's plugin file format? (Judging by file extensions it'll be similar to the other games)
    • Are records and groups arranged like in all games after Morrowind?
      • Yes, they look the same as in SSE.
    • Is the plugin header record still TES4?
      • Yes
    • Does the plugin header record still have an SNAM subrecord that holds the plugin description?
      • Looks like it, though the SNAM subrecord is missing on BlueprintShips-Starfield.esm, and the other three official .esm files have a value of Script (their CNAM / Author field is Python).
    • Does the plugin header record still have a record and group count subdrecord field, and if so which is it?
    • There's a HEDR subrecord of the correct size, and assuming its fields are unchanged since SSE, Starfield.esm has 9F 5C 3A 00, BlueprintShips-Starfield.esm has 10 D3 16 00, Constellation.esm has 39 00 00 00 and OldMars.esm has 27 00 00 00. Their file sizes are 1.35 GB, 287 MB, 38.2 KB and 23.7 KB, and counting form IDs after parsing those files indicates that the field's value is roughly accurate (it's equal to the record count plus one for three plugins, and a few thousand off for BlueprintShips-Starfield.esm).
    • How long is the record header, and what is its structure?
      • Same length of header for records and groups as in SSE, 24 bytes, fields look like they have the same meanings (though flag values may differ, all official plugins have at least 0x80 and 0x1 set in their TES4 record, which make sense as localised and master flagged respectively).
    • How are large subrecords handled in Starfield?
      • Looks like the same XXXX subrecord method as in previous games.
  • What is the structure of Starfield's BA2 file format?
    • Looks like the header is the same as for FO4, except that the version field is now 2, the two official BA2 types are still GNRL and DX10, and file paths are still stored at the end of the file.
    • According to Elminster, Zilav shared that the version field can also be 3. Sure enough, all the Starfield - Textures*.ba2 and Starfield - LODTextures.ba2 files have a version of 3. It doesn't look like version 3 has any differences in the header's paths offset or how the file paths are stored.
  • Are there any official BA2 files that contain the same file paths and so need hash collision warnings suppressed?
    • It doesn't seem so.
  • Does esplugin need to do anything about the 0x200 flag "squashing" FormIDs into the 00 mod index?
    • The only thing it would impact is calculating if two plugins' FormID sets overlap. Usually two plugins that each add a new record can't have those records' FormIDs overlap, but if both plugins have 0x200 set then the FormIDs would both be squashes down into the 00 mod index, so if their object indexes were the same then they'd both be setting the same FormID. However, that's not really overlap, which is about when two plugins both override the same record - in this case it's not an override, the squashed records could be completely different. It sounds more like a plugin validity issue than an normal overlap issue. esplugin would need to expose the presence of the flag to detect the validity issue though.
  • What happens in the squashed FormID is the same as an existing FormID?
  • What happens if two plugins try to squash FormIDs with the same object index?

Thanks to Pickysaurus from Nexus Mods I now have a copy of Starfield on Steam and a copy from the Microsoft Store through a 3-month Game Pass subscription, so I'll be able to get answers to these questions myself. I expect that most will be fairly straightforward to answer, the plugin file format questions will probably need a little trial-and-error parsing to check assumptions. It may be tricky to work out the ins and outs of how load order works, but the basics should be obvious with a little testing.

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 1, 2023

Steam version

grafik
Version of Starfield.exe: 1.7.23.0

Does the Steam install path contain an installscript.vdf file?

It doesn't seem so.

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 1, 2023

C:\Users\USER\AppData\Local\<game> usually includes plugins.txt.

For Starfield - C:\Users\USER\AppData\Local\Starfield - contains only a single file for me:
Pipeline.cache

It's size is around 62 MB and it can't be opened with a simple text editor.

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 1, 2023

Viewing Starfield.esm with a hex editor suggests, that both TES4 as well as SNAM continue to be there.

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 1, 2023

While viewing Starfield.exe with a hex editor and searching for .esm will find these entries:

.ESM
TestData.esm
Starfield.esm
BlueprintShips-Starfield.esm
Constellation.esm
OldMars.esm

Searching for .esp will find:

.ESP
.esp

Searching for .esl will find:

.ESL

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 1, 2023

I'm playing a bit with console commands. In C:\Users\USER\Documents\My Games\Starfield there is a StarfieldConsoleHistory.log file that logs every console command input.

In FO4, you could use

ShowGlobalVars
GetGlobalValue <var>

where <var> was one of the listed variables of ShowGlobalVars and it would return the correct value.

I'm currently unable to replicate this behaviour in Starfield. While ShowGlobalVars works, using GetGlobalValue <var> afterwards results in the following message:

Item '<var>' not found in parameter Global.
Compiled script not saved!

On the other hand

player.getvalue luck
player.setvalue luck 2

worked, and using another player.getvalue luck afterwards returned 2 as the result, while before it was 1.

@Geeknasty
Copy link

In FO4, you could use

ShowGlobalVars
GetGlobalValue <var>

where <var> was one of the listed variables of ShowGlobalVars and it would return the correct value.

I'm currently unable to replicate this behaviour in Starfield. While ShowGlobalVars works, using GetGlobalValue <var> afterwards results in the following message:

Item '<var>' not found in parameter Global.
Compiled script not saved!

I was able to use GetGlobalValue <var> to work by using the FormID value of the global variables instead of the EditorID.

So instead of GetGlobalValue Locklevel_Novice I would use GetGlobalValue AEF4F

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 2, 2023

Starfield Default Values for All Known Valid INI Settings compiled by @DoubleYouC
In case we want to make use of some of these settings.

@Ortham
Copy link
Member

Ortham commented Sep 2, 2023

In Process Monitor it looks like Starfield also looks for some files (including plugins and BA2 files) under %USERPROFILE%\Documents\My Games\Starfield, e.g.

%USERPROFILE%\Documents\My Games\Starfield\Data\Starfield.esm
%USERPROFILE%\Documents\My Games\Starfield\DATA\DATA\Starfield.esm
%USERPROFILE%\Documents\My Games\Starfield\Data\Starfield.ccc
%USERPROFILE%\Documents\My Games\Starfield\Starfield.ccc

%USERPROFILE%\Documents\My Games\Starfield\DATA\Data\Constellation - Localization.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\Constellation - Localization.ba2
%USERPROFILE%\Documents\My Games\Starfield\DATA\Data\Constellation - Textures.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\Constellation - Textures.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\Constellation - Voices_en.ba2
%USERPROFILE%\Documents\My Games\Starfield\Constellation - Voices_en.ba2

%USERPROFILE%\Documents\My Games\Starfield\DATA\Data\OldMars - Localization.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\OldMars - Localization.ba2
%USERPROFILE%\Documents\My Games\Starfield\DATA\Data\OldMars - Textures.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\OldMars - Textures.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\OldMars - Voices_en.ba2
%USERPROFILE%\Documents\My Games\Starfield\OldMars - Voices_en.ba2

%USERPROFILE%\Documents\My Games\Starfield\DATA\DATA\test.esm
%USERPROFILE%\Documents\My Games\Starfield\Data\test.esm

%USERPROFILE%\Documents\My Games\Starfield\Data\BlueprintShips-Starfield - Main.ba2
%USERPROFILE%\Documents\My Games\Starfield\BlueprintShips-Starfield - Main.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\BlueprintShips-Starfield - Textures.ba2
%USERPROFILE%\Documents\My Games\Starfield\BlueprintShips-Starfield - Textures.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\BlueprintShips-Starfield - Voices_en.ba2
%USERPROFILE%\Documents\My Games\Starfield\BlueprintShips-Starfield - Voices_en.ba2
%USERPROFILE%\Documents\My Games\Starfield\Data\BlueprintShips-Starfield - Localization.ba2

It's hopefully not using %USERPROFILE% to resolve the path, IIRC there's a folder ID for getting the My Games path, but I can't remember what it's called. I imagine it's using that, so it's not necessarily at %USERPROFILE%\Documents\My Games, but I'm not going to mess with my user account to find out.

Other observations from Process Monitor:

  • It looks like the game looks for each BA2 in My Games before the install path. E.g. My Games\Starfield\DATA\Data\OldMars - Textures.ba2, then steamapps\common\Starfield\DATA\Data\OldMars - Textures.ba2, then My Games\Starfield\Data\OldMars - Textures.ba2, then steamapps\common\Starfield\Data\OldMars - Textures.ba2.
  • Starfield.ccc is searched for at My Games\Starfield\Data\Starfield.ccc, then steamapps\common\Starfield\Data\Starfield.ccc, then My Games\Starfield\Starfield.ccc, then steamapps\common\Starfield\Starfield.ccc.
  • OldMars.esm (and other plugins) is searched for at My Games\Starfield\DATA\DATA\OldMars.esm then steamapps\common\Starfield\DATA\DATA\OldMars.esm, then My Games\Starfield\Data\OldMars.esm, then steamapps\common\Starfield\Data\OldMars.esm.
  • test.esm is a plugin I've created to test loading from plugins.txt. It's in the game install's Data folder, and even without it listed in plugins.txt the game still looks for it in the same folders as the other plugins.
  • BlueprintShips-Starfield - Main.ba2 is the only Main-suffixed BA2 that's searched for. There's a block in Starfield.exe that is: Textures.ba2� - Main.ba2�.ba2�������� - Voices_�� (all the question marks are nulls). Main.ba2 also appears later in the executable in Starfield - Animations.ba2������Main.ba2�. Localization appears in the block - Localization��%s*.ba2�- Main.�- Textures.�. I don't see any other relevant strings starting with - or - .
  • Main.ba2 appears as the value of sDLCArchiveNameSuffix in StarfieldCustom.ini, according to the link that pStyl3 shared above. My guess is that's what the second appearance of Main.ba2 in Starfield.exe is for. However, setting a different value for sDLCArchiveNameSuffix doesn't cause Starfield to try opening files with that suffix, it still looks for BlueprintShips-Starfield - Main.ba2.
  • The Voices_<language> language value comes from the Starfield.ini (and language-specific ini files)'s sLanguage property. It's not set in Starfield.ini, sLanguage=en is the implied default, but e.g. in the German ini file it's sLanguage=de, and if I add it to Starfield.ini I see the in-game language changes and the relevant BA2 files are opened.

libloot already has support for loading from multiple data paths, but the CCC file being loaded from multiple paths is new to me and not currently supported. It's currently read by LOOT and libloadorder, and it's probably easier to just add special handling for it.

@Ortham
Copy link
Member

Ortham commented Sep 2, 2023

I've been able to handmake a plugin that changes the name of the "Eon" pistol: it's FormID 000476C4, and the name is the first FULL subrecord in the record. It's a localised string by default, so I set the TES4 header's flags so that 0x80 was not set and then edited the field to a 3-letter name (as its original value is a 4-byte localised string index, and I didn't want to mess with offsets/sizes).

However, I'm only able to see the change in-game when I replace one of the hardcoded plugins (e.g. OldMars.esm with my plugin. Enabling the TestData.esm plugin might also work, but I don't think that approach would really tell me anything new.

I also haven't been able to create a new record (not one that just overrides one in Starfield.esm) and have that show up in game, and none of the records added by OldMars.esm or Constellation.esm seem to be inventory items (which are easy to spawn in using player.additem), so I've so far been unable to check what the hardcoded load order is.

@Ortham
Copy link
Member

Ortham commented Sep 3, 2023

Interesting... I created a copy of the Eon pistol's record and gave it a different EDID value (ZVP - again I only had 3 bytes to play with) and a FormID of 01FFFFFF, and then used FindForm in-game, and it said the FormID was FE001FFF. The plugin's TES4 flags were 0x101. When I change the flags to 0x0 or 0x1 the in-game FormID is 01FFFFFF, and when the flag is 0x100 it's FE001FFF again. If I keep the flag at 0x100 and change the FormID to 01000000, it shows in-game as FE001000.

So it looks like:

  • the light plugin flag is 0x100 in this game, not 0x200
  • the valid object ID range for light plugin records is 0x0 to 0xFFF.

I then changed the first instance of OldMars.esm in Starfield.exe to OldMars.esl and renamed by test plugin to match, clearing its TES4 flags, and when loading my save game Starfield prompted about mods being loaded causing achievements to be disabled, but FindForm did not find my new record. I then changed its FormID back to 01FFFFFF, but it still wasn't found. I then also changed the second instance of OldMars.esm in Starfield.exe, but that made no difference.

I'm not sure what to draw from this, as despite changing both strings the game recognised that the plugins it was trying to load weren't the usual official plugin names, but then loading seems to have failed as the changes weren't visible in-game (not just my new pistol, but also the edit to the Eon didn't show up). I'm not sure this says much about plugin loading in general either, as the official plugins may well go through a different path to the rest.

I also tried changing the file extension to .esp, but got the same results as for .esl.


Unsetting the light plugin flag for copies of the three hardcoded plugins revealed the order they load in is:

  • 01 Constellation.esm
  • 02 OldMars.esm
  • 03 BlueprintShips-Starfield.esm

It may be worth noting that BlueprintShips-Starfield.esm is part of the base game (it appears in my Game Pass copy, while the others don't).

@Ortham
Copy link
Member

Ortham commented Sep 3, 2023

I renamed Starfield - Meshes01.ba2 to test out what happened when I tried loading a save with it using different filenames, and:

  • OldMars - Meshes01.ba2: crash
  • OldMarsTextures.ba2: crash
  • OldMars - Voices_fr.ba2: crash (with language set to English)
  • OldMars.ba2: crash
  • OldMars - Main.ba2: success
  • OldMars - Textures.ba2: success
  • OldMars - Localization.ba2: success
  • OldMars - Voices_en.ba2: success (with language set to English)

That's pretty much what I was expecting, except OldMars.ba2 not getting loaded.

@Infernio
Copy link
Member

Infernio commented Sep 3, 2023

aers on the RE discord confirmed that .esl still forces the master flag.

@Ortham
Copy link
Member

Ortham commented Sep 4, 2023

aers on the SkyrimSE RE discord has also found that the 0x200 TES4 flag (which was previously the light plugin flag) now suppresses the light plugin flag being set for plugins with a .esl extension (but doesn't do anything if that flag is already set in the file). Here is where most of the discussion starts.

The logic (as written by Elminster) boils down to:

if the 0x200 flag is set then begin
  if there is no master list, or the 0x100 flag is set then
    remove the 0x200 flag
end else
  if the extension is .esl then
    force 0x100 flag

I've verified this in-game: I renamed my test plugin to test.esl, made sure it didn't have the flag set, added it to StarfieldCustom.ini as sTestFile1=test.esl under [General], and then used FindForm to check its record's FormID in game. Without 0x200 set, it was FE002FFF, while with the flag it was 00FFFFFF. The latter FormID is unexpected: if I change the file extension to .esm and unset the flag, the FormID is 02FFFFFF in-game. The FormID in the plugin is 01FFFFFF. It looks like 0x200 "squashes" the records down into the same FormID range as Starfield.esm uses.

I should update esplugin to account for the suppression behaviour, so that plugins don't get treated as light when the flag is suppressed. I'm not yet sure what to do about the "squashing".

@aers
Copy link

aers commented Sep 4, 2023

It's hopefully not using %USERPROFILE% to resolve the path, IIRC there's a folder ID for getting the My Games path, but I can't remember what it's called. I imagine it's using that, so it's not necessarily at %USERPROFILE%\Documents\My Games, but I'm not going to mess with my user account to find out.

It uses ShGetFolderPathA to resolve the user's personal Documents folder, so yes it will work no matter where it is located.

@Ortham
Copy link
Member

Ortham commented Sep 4, 2023

It uses ShGetFolderPathA to resolve the user's personal Documents folder, so yes it will work no matter where it is located.

Thanks, I was thinking of SHGetKnownFolderPath, and I thought there was a FOLDERID for My Games but it looks like I was wrong, there's only FOLDERID_SavedGames.

@Ortham
Copy link
Member

Ortham commented Sep 4, 2023

@aers I ran those tests you suggested, plus a few extra:

One plugin:

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

In-game:
- the Eon (000476C4) is renamed ESM.
- FindForm ZVM returns 02FFFFFF

---

One plugin:

test.esm
Flags: 0x200
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

In-game:
- the Eon (000476C4) is renamed ESM.
- FindForm ZVM returns 00FFFFFF

---

One plugin:

test.esl
Flags: 0x300
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

In-game:
- the Eon (000476C4) is renamed ESM.
- FindForm ZVM returns FE002FFF

---

One plugin:

test.esl
Flags: 0x200
Master: None

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

In-game:
- the Eon (000476C4) is not renamed
- FindForm ZVM returns 02FFFFFF
- I ran `player.additem 020476C4 1` but that caused the game to crash

---

Two plugins:

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

test2.esm
Flags: 0x200
Master: Starfield.esm

Records:
- FormID: 000476C4, EDID: Eon, FULL: ES2
- FormID: 01FFFFFF, EDID: ZV2, FULL: ZV2

Loaded sTestFile1=test2.esm, sTestFile2=test.esm

In-game:
- the Eon (000476C4) is renamed ES2.
- FindForm ZVM returns 02FFFFFF
- FindForm ZV2 returns 00FFFFFF

---

Two plugins:

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

test_dependent.esm
Flags: 0x0
Master: test.esm

Records:
- FormID: 00FFFFFF, EDID: ZVL, FULL: ESL
- FormID: 01FFFFFF, EDID: YVL, FULL: YVL

Loaded sTestFile1=test.esm, sTestFile2=test_dependent.esm

In-game:
- the Eon (000476C4) is renamed ESM
- FindForm ZVM returns nothing
- FindForm ZVL returns 02FFFFFF
- FindForm YVL returns 03FFFFFF

---

Two plugins:

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

test_dependent.esm
Flags: 0x200
Master: test.esm

Records:
- FormID: 00FFFFFF, EDID: ZVL, FULL: ESL
- FormID: 01FFFFFF, EDID: YVL, FULL: YVL

Loaded sTestFile1=test.esm, sTestFile2=test_dependent.esm

In-game:
- the Eon (000476C4) is renamed ESM
- FindForm ZVM returns nothing
- FindForm ZVL returns nothing
- FindForm YVL returns 02FFFFFF

Most of the tests above suggest that a plugin that has the 0x200 flag applied (after the logic found by aers and described by Elminster runs) does not take up a mod index slot, which affects checking if active plugin limits have been reached and displaying plugin load order indexes, so esplugin will need to expose that info for use by libloadorder and LOOT.

However, the last test shows that there are cases in which the 0x200 flag is ignored other than those already described. I need to do more investigation to narrow down the cause for that test case. I also don't understand why the ZVM/ZVL record vanished in that test case, All I did was change the flags on test_dependent.esm.

@Ortham
Copy link
Member

Ortham commented Sep 4, 2023

Doing some more testing to understand the game's behaviour with the 0x200 flag:

Two plugins:

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

test_starfield_dependent.esm
Flags: 0x200
Masters: Starfield.esm, test.esm

Records:
- FormID: 01FFFFFF, EDID: ZVL, FULL: ESL
- FormID: 02FFFFFF, EDID: YVL, FULL: YVL

Loaded sTestFile1=test.esm, sTestFile2=test_starfield_dependent.esm

In-game:
- the Eon (000476C4) is renamed ESM
- FindForm ZVM returns nothing
- FindForm ZVL returns 02FFFFFF
- FindForm YVL returns 00FFFFFF

That makes sense to me: ZVL overrides ZVM, so doesn't use up a new mod index, while YVL is a new record that gets squashed down to Starfield.esm's mod index. Is it squashing down to Starfield.esm's mod index though, or the mod index of the first master listed?

Two plugins:

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

test_starfield_dependent.esm
Flags: 0x200
Masters: test.esm, Starfield.esm

Records:
- FormID: 00FFFFFF, EDID: ZVL, FULL: ESL
- FormID: 02FFFFFF, EDID: YVL, FULL: YVL

Loaded sTestFile1=test.esm, sTestFile2=test_starfield_dependent.esm

In-game:
- the Eon (000476C4) is renamed ESM
- FindForm ZVM returns nothing
- FindForm ZVL returns nothing
- FindForm YVL returns 02FFFFFF

This gets the same result as the last test case in my previous comment, and what's common between them is that the first master is not Starfield.esm. I was expecting ZVL to be 02

ZVM should have mod index 02, but it and ZVL have vanished and YVL has mod index 02. I think what's happened is that ZVM was overridden by ZVL, but they were both overwritten by YVL, because all three have the same FormID.

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

test_starfield_dependent.esm
Flags: 0x200
Masters: test.esm, Starfield.esm

Records:
- FormID: 02FFFFFE, EDID: ZVL, FULL: ESL
- FormID: 02FFFFFF, EDID: YVL, FULL: YVL

Loaded sTestFile1=test.esm, sTestFile2=test_starfield_dependent.esm

In-game:
- the Eon (000476C4) is renamed ESM
- FindForm ZVM returns nothing
- FindForm ZVL returns 02FEFFFF
- FindForm YVL returns 02FFFFFF

Here ZVL and YVL are both new records that have been squashed down into test.esm's mod index, and ZVM no longer exists because it's been overwritten (not overridden) by YVL.

Two plugins:

test.esm
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

test_starfield_dependent.esm
Flags: 0x200
Masters: test.esm, Starfield.esm

Records:
- FormID: 00FFFFFF, EDID: ZVL, FULL: ESL
- FormID: 02FFFFFE, EDID: YVL, FULL: YVL

Loaded sTestFile1=test.esm, sTestFile2=test_starfield_dependent.esm

In-game:
- the Eon (000476C4) is renamed ESM
- FindForm ZVM returns nothing
- FindForm ZVL returns 02FFFFFF
- FindForm YVL returns 02FFFFFE

Here YVL uses a different object index from ZVM/ZVL, so ZVL overrides ZVM as expected, and YVL squashes into test.esm as expected.

@Ortham
Copy link
Member

Ortham commented Sep 4, 2023

I think I've got everything I need to add support for Starfield to esplugin and loot-condition-interpreter, and to libloot itself. However, as I've been so far unable to figure out how to get the game to load non-hardcoded plugins without using ini settings, I can't complete support in libloadorder.

It looks like adding support for Starfield will involve a couple of breaking changes to libloot's API, so I might just assume that Starfield uses the same load order system as Skyrim SE and Fallout 4 so that I can include the breaking changes in the next libloot release - I don't have any reason to expect figuring out the load order system to involve breaking API changes in libloot or libloadorder, so it would be easier to patch in any corrections later.

Aside from basic functionality, it would be good to support:

  • Loading plugins and archives from My Games before looking in the install path (but I think it would be premature to add support for reading Starfield.ccc from there, as it may end up elsewhere).
  • Displaying an error message for invalid override plugins (i.e. plugins that set the 0x200 flag but contain new records).

@TechAngel85
Copy link

TechAngel85 commented Sep 4, 2023

I think I've got everything I need to add support for Starfield to esplugin and loot-condition-interpreter, and to libloot itself. However, as I've been so far unable to figure out how to get the game to load non-hardcoded plugins without using ini settings, I can't complete support in libloadorder.

I don't know if this will be helpful, but Elminster (xEdit dev) mentioned in the Starfield Discord that it's likely Bethesda has blocked or commented out the code for loading plugins.txt. It looks like it's still present, just block off; likely until an official update for "mod support".

@Ortham
Copy link
Member

Ortham commented Sep 4, 2023

I don't know if this will be helpful, but Elminster (xEdit dev) mentioned in the Starfield Discord that it's likely Bethesda has blocked or commented out the code for loading plugins.txt. It looks like it's still present, just block off; likely until an official update for "mod support".

I'm already aware, I've been talking to him saying the same sort of thing, though what you've said isn't quite correct: the game does read plugins.txt, but we don't yet know what it does with the contents. I've got the code open in in a disassembler, but I've been avoiding it (I don't enjoy reading assembly).

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 4, 2023

Starfield Repository
The first few commits have been made and Starfield's masterlist.yaml has been created.
@MacSplody @sibir-ine @Sabre142 I'd appreciate it, if you could take a look. 😄

@Ortham
Copy link
Member

Ortham commented Sep 5, 2023

Correction about BA2 loading: if I've got sTestFile1=test.esm in my ini file, the game will attempt to load test - Main.ba2. Using the same test setup as earlier I've verified that this does load assets successfully. I went back and tested OldMars - Main.ba2 and that actually works, so I've updated that test result above (I must have had a typo in the BA2 name or something like that).

@Ortham
Copy link
Member

Ortham commented Sep 5, 2023

Another test case checking if the 0x200 flag affects the master flag getting set for a .esm plugin:

Two plugins:

test.esp
Flags: 0x0
Master: Starfield.esm

Records
- FormID: 000476C4, EDID: Eon, FULL: ESP
- FormID: 01FFFFFF, EDID: ZVP, FULL: ZVP

test.esm
Flags: 0x200
Masters: Starfield.esm

Records:
- FormID: 000476C4, EDID: Eon, FULL: ESM
- FormID: 01FFFFFF, EDID: ZVM, FULL: ZVM

Loaded sTestFile1=test.esp, sTestFile2=test.esm

In-game:
- the Eon (000476C4) is renamed ESP
- FindForm ZVM returns 00FFFFFF
- FindForm ZVP returns 03FFFFFF

So the master flag is still applied. I renamed test.esm to test.esl and got the same results.

@Ortham
Copy link
Member

Ortham commented Sep 5, 2023

I've been slowly making my way through the disassembly for some of the Starfield.exe code that uses the %sPlugins.txt string that's hardcoded in Starfield.exe, and I'm pretty confident that it's using the same format of plugin.txt as in Skyrim SE and Fallout 4: lines beginning with # are skipped, and lines beginning with * are treated differently. I've identified one function that cuts off any leading * before doing something with the rest of the line, and another function that only does stuff for lines that begin with *. I haven't figured out what is done in either case, but it seems likely that the * is still used to indicate an active plugin. I haven't yet spotted anything that would explain why listing plugins in plugins.txt doesn't seem to have any impact in-game.

@Ortham Ortham self-assigned this Sep 5, 2023
@Ortham
Copy link
Member

Ortham commented Sep 6, 2023

Now that my Game Pass copy of Starfield has unlocked, I've started it up and gotten past character creation then made a save, and taken a look at what files were accessed where:

  • My Games/Starfield was used for loading plugins and BA2s and Starfield.ccc as in the Steam version
  • %LOCALAPPDATA%\Starfield\Plugins.txt was opened

Weirdly, although I made a save and loaded it, I can't find it anywhere, and ProcMon doesn't show any .sfs file accesses, aside from one attempt to open a save at D:\Games\XboxGames\Starfield\Content\Saves\Exitsave0_37CE184B_4A6F6C656E65_000008_20230906070421_1_0_4.sfs (D:\Games\XboxGames\Starfield is where I installed the game). LOOT doesn't use saves so it's not an issue.

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 7, 2023

BGS Discord

Cartogriffi
Hi folks, with Starfield launched we’re opening a dedicated channel for discussing Starfield Mods.
Official mod support is planned for the future, and we look forward to sharing more details. There is no ETA on when this will be, but keep an eye on this channel and #⁠modding-news!

And according to the following japanese news article - the section that includes Mod within the question:

Will we be able to place new creatures and other things on the planet with the mod?

Todd:
When we can do mods, we'll be able to do almost anything we've done in the past, and the mods will be supported next year, but we'll do it in a big way because we love it too.

(DeepL translation)

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 13, 2023

Starfield has received it's first post-launch patch - Link
The version of Starfield.exe has been bumped from 1.7.23.0 to 1.7.29.0.

@3ventic
Copy link
Member

3ventic commented Sep 17, 2023

What local app data folder name does the Microsoft Store version of Starfield use?

Microsoft Store version of the game also uses %LocalAppData%\Starfield

@Ortham
Copy link
Member

Ortham commented Sep 18, 2023

What local app data folder name does the Microsoft Store version of Starfield use?

Microsoft Store version of the game also uses %LocalAppData%\Starfield

Thanks, I'd already got an answer for this one (I've got Game Pass), but forgot to tick it off, sorry!

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 25, 2023

A new Patch is available - Notes
The new version of Starfield.exe is 1.7.33.0.

@Ortham
Copy link
Member

Ortham commented Sep 26, 2023

While Starfield.exe looks for Starfield.ccc, plugins listed in it aren't currently loaded, same as plugins.txt.

I've just finished adding support for reading sTestFile ini properties to libloadorder, which is used by other games and is also currently the only way to load plugins for Starfield, but I don't intend to support changing those ini property values (they're also only used to activate plugins, not set load order).

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 26, 2023

Considering that

  • the normal way to load a plugin in Starfield is currently "disabled"
  • we don't know how to enable or disable plugin loading via plugins.txt yet
  • it could take potentially months until Bethesda enables it

How about adding a basic ("view only") support for Starfield to LOOT? As in, you can select the game, see plugins that are found in the Data folder, you can even try to sort your load, but the sorting result get's either dismissed automatically, or the load order is written to some sort of temporary file e.g. order.txt (so that it doesn't get confused with plugins.txt)? Or disable the ability to sort the Starfield load order to begin with.

And as soon as loading plugins via plugins.txt works, we can also activate it in LOOT.

Just some ideas ..

@Ortham
Copy link
Member

Ortham commented Sep 26, 2023

How about adding a basic ("view only") support for Starfield to LOOT?

Yeah, that's the direction I'm heading in, I've got it so that libloadorder will error if you try to save changes, so I was going to disable the apply changes button in LOOT, but thinking about it again, disabling sorting entirely makes more sense since you can't do anything with the results.

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 26, 2023

This is what LOOT using loot_0.20.0-99-gb743ba8_starfield-win64.7z looks like after starting LOOT with the plugins mentioned here, so no sorting until now.

snip1

I think it's great that LOOT immediately informs the users about things like header versions being lower than the game's minimum header version.

The ability to sort currently enables me to see the CRC32 values of all the different plugins, as well as if LOOT detects plugins to be invalid:

snip2

These information are nice to have and we would miss out on them for Starfield, as long as sorting would be disabled. But I think it would be okay, given the circumstances. Starfield being recognized by LOOT and the ability to see the plugins (and potential messages for them) seems like a good first step.

@Ortham
Copy link
Member

Ortham commented Sep 26, 2023

@pStyl3 It's not obvious, but you get CRC calculation and full plugin parsing if you use the conflict filter.

@pStyl3
Copy link
Member Author

pStyl3 commented Sep 26, 2023

True. Seems like I haven't used the conflict filter for quite some time. 😄 Well, then all the more reason to disable sorting for Starfield, until loading plugins via plugins.txt is working.

@Ortham
Copy link
Member

Ortham commented Sep 26, 2023

I've disabled sorting and checking for ambiguous load orders when the current game is Starfield, as in both cases there's nothing the user can do with the result, and I've updated the warning message displayed for Starfield to reflect this.

In other news, I've hit an issue with implementing support for the sTestFile1 through sTestFile10 ini properties: Starfield supports them being defined in the Starfield_<language>.ini (e.g. Starfield_en.ini) file for the current language, but figuring out which is the current language is not straightforward:

  • For Steam installs, the language is set in Steam's game properties, and is stored in the game's appmanifest ACF file, so libloadorder needs to parse that and map it to the appropriate language code (I think the game gets it through the Steam API DLL). I've gotten that working thanks to a parser library for the file format, though it doesn't seem particularly high quality or well maintained, so I'm not sure I want to go ahead with depending on it.
  • For Microsoft Store installs, the language is taken from your Windows system configuration. Unfortunately, it's not clear what config is relevant or what OS languages/locales are supported (though there are lists in appxmanifest.xml and MicrosoftGame.Config), in part because I've found that Bethesda's instructions for setting the language don't actually work...

The Starfield_<language>.ini is one of three ini files you can use to set sTestFile<N>, and is not a likely choice (if Bethesda are going to set sTestFile<N> in a future update, it'll probably be in Starfield.ini, and a user would probably use StarfieldCustom.ini), so I might just ship without support for it and leave it as a todo for some other time.

@Ortham
Copy link
Member

Ortham commented Sep 26, 2023

Also, I've intentionally limited support in libloadorder so that it will ignore plugins.txt and Starfield.ccc since the game doesn't currently activate plugins listed in those files if they exist.

@Infernio
Copy link
Member

Nukem has posted an SFSE plugin that enables plugins.txt support: https://www.nexusmods.com/starfield/mods/4157

@Ortham
Copy link
Member

Ortham commented Sep 28, 2023

@Infernio Do you know where (if anywhere) the source code is available for that? I don't really want to use it to verify game behaviour without knowing how much it changes.

@Ortham
Copy link
Member

Ortham commented Sep 28, 2023

So in what is probably today's winner for "weird but true fact", pStyl3 shared with me a screenshot of a screenshot that Picksaurus had shared in the Starfield Modding Discord server of a comment Nukem made in Discord that deleting BlueprintShips-Starfield.esm is enough to get plugins.txt. I've verified that this is true. You can also just rename BlueprintShips-Starfield.esm and activate it in plugins.txt to avoid losing its content.

What I haven't seen mentioned elsewhere yet is that it also (like the SFSE plugin linked a few comments above) makes the game pay attention to Starfield.ccc, so that also works if you create it and list plugins in there. With testing, I've also verified that the system works like Fallout 4, in that if you have sTestFile<N> set, that causes the game to ignore plugins.txt and Starfield.ccc. Test cases follow.

Also, BlueprintShips-Starfield.esm is hardcoded to be always active, but doesn't have a fixed load order position, it's like Update.esm in Skyrim in that it'll load after masters listed in plugins.txt if it is not also listed.


Starfield with plugins.txt enabler SFSE plugin:

With plugins.txt:

*test.esp

It loaded at index 02.


With plugins.txt:

*test.esm
*test2.esm

test.esm loaded at 01, test2.esm loaded at 02


With plugins.txt:

*test2.esm
*test.esm

test.esm loaded at 02, test2.esm loaded at 01


If I edit OldMars.esm to not be a light plugin, with plugins.txt:

*test2.esm
*test.esm

OldMars.esm loads at 01, test2.esm loaded at 02, test.esm loaded at 03


If I edit Constellation.esm to not be a light plugin, with plugins.txt:

*test2.esm
*test.esm

Constellation.esm loads at 01, test2.esm loaded at 02, test.esm loaded at 03


With OldMars.esm back to normal and plugins.txt:

*test.esm
*test2.esm
*test.esp

test.esm loads at 01, test2.esm loads at 02, and test.esp loads at 04. That means BlueprintShips-Starfield.esm must load at 03, so it's implicitly active but not an early loader.


With plugins.txt:

*test2.esm
*test.esp

and sTestFile1=test.esm, only test.esm is loaded


With plugins.txt:

*test2.esm
*test.esp

and Starfield.ccc:

test.esm

test.esm loads at 01, test2.esm loads at 02, and test.esp loads at 04


With Starfield.ccc:

test2.esm

and sTestFile1=test.esm, only test.esm is loaded


With Starfield.ccc:

test2.esm
test.esp

test.esm loaded at 02, test2.esm loaded at 01


With Starfield.ccc:

test.esm
test2.esp

test.esm loaded at 01, test2.esm loaded at 02

@Ortham
Copy link
Member

Ortham commented Sep 29, 2023

Done as of 9dbc06f.

@Ortham Ortham closed this as completed Sep 29, 2023
@pStyl3 pStyl3 added this to the 0.22.0 milestone Sep 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

7 participants