FO3 & FNV Windows Store Support #617
Assignees
Labels
A-new-game
Area: New Game Implementation (new subpackage in the game package)
A-windows-store
Area: Windows Store Support (game/windows_store_game.py, game/ws_* and env)
C-enhancement
Category: Enhancement, a request to add or enhance a feature
G-fallout3
Game: Fallout 3
G-falloutnv
Game: Fallout: New Vegas
M-relnotes
Misc: Issue should be listed in the version history for its milestone
Milestone
Comments
Infernio
added
C-enhancement
Infernio
added
the
M-relnotes
Infernio
added
the
A-windows-store
Infernio
added a commit
that referenced
this issue
Jul 24, 2022
310 is the culmination of years of work, finally moving us to Python 3 (3.9 to be exact - we can't upgrade to 3.10+ yet, see #619). All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### New BP Tweaks (#602) The first major change in 310 was a contribution by @sibir-ine, who added six new settings tweaks in a423f34. @sibir-ine also contributed further patcher and tweak updates throughout 310's development, especially targeting consistency of patched records and subrecords across games. Some of the relevant commits: - 41ecb03 & 4ea307f: FO3/FNV patcher updates - 6513e3d: TES5 patcher updates - 80f2338: Improvements to tweaks - fe70f2d: New settings tweak Many thanks for the great contributions (and endless help with the Discord server)! ----------------------------------------------------------------------- ### Python 3 (#460) This is of course the highlight of the release. The merge that finally completed this goal is f0a803b, but we had to squash the (lengthy) branch that completed the upgrade down to one commit so as to not break dev. See that merge's commit message for more details. There is still some followup work to do (and a bunch that was already done throughout the rest of 310), but this is such a massive step forward it's hard to overstate. Hacks gone, bugs fixed, dependencies upgraded, new stdlib tools starting to get employed (see e.g. 43fc54c, which allowed us to get rid of our bundled lzma.exe in favor of py3's stdlib lzma module) - and of course the main benefit being that we're no longer stuck on obsolete technology. Massive thanks to everyone who has contributed to this herculean effort over the past four releases! ----------------------------------------------------------------------- ### Locale crashes (#610) One of the biggest remaining issues in WB post-wx3-upgrade were the locale crashes. This proved to be a wild goose chase. Every time we thought we'd solved it for good, some user with a different unusual setup came along and proved us wrong. The core problem is that wxWidgets, wxPython and Python itself are all trying to handle locales, and all of them end up fighting over it. We initially thought we'd finally fixed it in 2ebc752, part of the py3 merge, but that *still* didn't do the trick. 0e93496 seems to have finally fixed it, but at the cost of *completely disregarding the user's locale* and always setting the C locale. Far from an ideal solution, if you can even call it that. wxPython 4.2.0, which will be based on wxWidgets 3.2.x, should hopefully help here, when it finally (it's been *two years* since the last wxPython release!) gets a release. ----------------------------------------------------------------------- ### Skyrim AE (#616) Midway through 310's development, Skyrim Anniversary Edition was released. As this is really just an update to the executable and an optional batch purchase of Creation Club plugins, adding support was easy. 937b182 completed the task. However, an unexpected annoyance arose from this. All of the Creation Club plugins included in the AE bundle are localized, and it turns out WB's boot process did not love this. Which is why f789663 was necessary. This merge significantly improves the boot performance in AE (~50% on my machine, meaning an improvement from 2s down to 1s). ----------------------------------------------------------------------- ### Windows Store versions of FO3/FNV (#617) Quick and easy to support. Note that Microsoft has *once again* changed how Windows Store games are installed, and 310 does not support this new method yet. See #639. ----------------------------------------------------------------------- ### FName (#543) After the py3 upgrade mentioned above, the next biggest beast lurking in 310 is #543. What started as an innocent enough idea (bolt.Path is pretty heavyweight and unnecessary for use cases like masters and filenames, which never have path separators in them - would be great if we could replace all that with a more appropriate and lightweight class) quickly ballooned into a massive refactoring of much of the codebase, eventually culminating in one hell of a rabbit hole whose event horizon spanned many merges and three WB releases. See 308 and 309's release commits for notes on #543 progress back then. Reiterating all of this here would take hundreds of not thousands of words, so this section will only focus on #543 during 310's development. 15b979f reduced Path usages, replaced some LBYL patterns with EAFP ones and wrapped common patterns as preparation for the final #543 merge. Some more refactoring, especially regarding BAIN, happened shortly afterwards in 97768fd. The next few commits (specifically c55b60d and 9c25544) mostly focused on renames and some minor refactoring in preparation for further attacks on Path and its usages. In 8555285, many more Path usages were pruned. And in 9a0db27, the converters part of the codebase (the code that handles BCFs) received significant touchups. More explicit Path usages gone, more LBYL -> EAFP refactoring and improvements to this somewhat ancient and dusty API. And then it was time for the big one. b3b730c introduced bolt.FName, the ultimate result of all this effort. This gave performance and memory usage improvements, improved code style in many places, etc. But the main benefit, as mentioned all the way back in the 308 release notes, is that this paves the way for further work on #480 and related issues in 311+. ----------------------------------------------------------------------- ### Preliminary macOS support (#622) f085cf0 introduced some WIP support for macOS. Just like with Linux, this does not mean you can actually use WB for that OS yet, but it's an eventual goal. The immediate benefit of adding partial support for other operating systems is the resulting refactoring and isolation of Windows-specific code. Some more refactoring towards this goal landed in 14c7639, which refactored our archives handling (used for reading and writing archives, e.g. .zip and .7z files) to be OS-agnostic. ----------------------------------------------------------------------- ### Skyrim patchers (#151) As of 802423a, the last patcher (barring FormID Lists, which will need rewriting - see #497) has been ported to Skyrim (to be exact, it was Tweak Names). This finally brings the eight-year project of porting patchers to Skyrim to an end. ----------------------------------------------------------------------- ### FOMODs (#627, #603 and #593) Ever since 307, where FOMOD support was first added to WB in #380 (see 307 release notes for commit links), we've been steadily improving BAIN's support for FOMODs. 310 continues this theme. d636b5b implemented the last few parts of the FOMOD schema that hadn't been implemented yet (foseDependency, fommDependency, alwaysInstall and installIfUsable). Next up, 870c0ed implemented schema validation using lxml. bc6d1c0 resolved a long-standing feature request by adding a link to capture the output of running an FOMOD installer into a new project. And finally, 2c603d6 added support for generating an empty ModuleConfig when creating a new project via BAIN. ----------------------------------------------------------------------- ### Usability (#628) Continuing the fight for better usability (aka me sticking search bars everywhere), the doc browser got a bit of a makeover in c7e555c. On top of small improvements like "Forget Doc" reselecting a document after being clicked, the doc browser now also contains a dropdown menu listing all your load order's plugins and has a search bar to help you narrow those plugins down. fdb34dd cleaned up the Installers context menu even further (this started way back in 307) by moving archive-specific links into an 'Archive..' submenu that only appears for archives and similarly moving project-specific links into a 'Project..' submenu. WB has support for various keyboard shortcuts. Not that you'd ever know that unless you read the docs (wish more people did that to be honest, but oh well...). To help with this discoverabilty issue, b8d3e8d added keyboard hints next to links that perform the equivalent functionality. And while we're on the topic of keyboard shortcuts, 70ca87e and d8fdd57 added some new ones. Told you this is just me sprinkling search bars all over WB: 03122cc added search bars to every list patcher in the BP dialog. That means tweakers ("Tweak X" patchers) and importers ("Import X" patchers, including "Leveled Lists" and "FormID Lists") now have a search bar you can use to filter their sources. Especially useful for Tweak Settings, which has more than *eighty* tweaks in Oblivion! There are also, as per usual, a bunch more minor commits improving random bits of WB with regards to usability in 310 that aren't mentioned here. ----------------------------------------------------------------------- ### Patchers (#312) Yep, still going on (and will be going on for a while longer...). e3b93d1 refactored a lot of Oblivion-specific constants and code, especially stuff related to MGEF handling. This stood in the way of extending WB to handle more games in the future, since no one was sure how exactly to update these constants (or if they even needed updating for other games). ----------------------------------------------------------------------- ### Records (#480) And where #312 dwells, #480 is not far behind. One of the big goals for 311+ will be FormIDs handling, and 2e3e5ee functions as a prelude in that direction. Especially nice as it showcases how Python 3 (specifically its ordered dicts) can help in refactoring. ----------------------------------------------------------------------- ### Debug mode (or lack thereof) (#638) We were getting mighty sick of people not understanding how to properly report bugs and volunteers having to spend time over and over explaining the same thing. So in c9a5036, we changed WB's debug mode so that it's always enabled in standalone builds. That's great and all, but debug mode has always had one problem: if you enable it, you lose the "native" error reporting mechanism provided by the environment you're running WB in. Most notably the wxPython error window that would otherwise pop up if you hit a traceback. So in d66da27, we dropped debug mode entirely and always enabled debug printing. To solve the problem of the native error reporting mechanism getting disabled, we now copy the debug print output to both the native stdout and to the BashBugDump. That way reporting is as easy as possible: users get notified of a traceback and the BashBugDump already contains full debug prints. The reason not to do this in the past was the overhead of deprint, but since fab8b90 deprint is pretty much as fast as a regular print (all the overhead came from usage of the inspect APIs, which statted the source/bytecode files). ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BeermotorWB, @warmfrost85, @GandaG and many more that GitHub's contribution tracker doesn't list.
Infernio
added a commit
that referenced
this issue
Jul 24, 2022
310 is the culmination of years of work, finally moving us to Python 3 (3.9 to be exact - we can't upgrade to 3.10+ yet, see #619). All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### New BP Tweaks (#602) The first major change in 310 was a contribution by @sibir-ine, who added six new settings tweaks in a423f34. @sibir-ine also contributed further patcher and tweak updates throughout 310's development, especially targeting consistency of patched records and subrecords across games. Some of the relevant commits: - 41ecb03 & 4ea307f: FO3/FNV patcher updates - 6513e3d: TES5 patcher updates - 80f2338: Improvements to tweaks - fe70f2d: New settings tweak Many thanks for the great contributions (and endless help with the Discord server)! ----------------------------------------------------------------------- ### Python 3 (#460) This is of course the highlight of the release. The merge that finally completed this goal is f0a803b, but we had to squash the (lengthy) branch that completed the upgrade down to one commit so as to not break dev. See that merge's commit message for more details. There is still some followup work to do (and a bunch that was already done throughout the rest of 310), but this is such a massive step forward it's hard to overstate. Hacks gone, bugs fixed, dependencies upgraded, new stdlib tools starting to get employed (see e.g. 43fc54c, which allowed us to get rid of our bundled lzma.exe in favor of py3's stdlib lzma module) - and of course the main benefit being that we're no longer stuck on obsolete technology. Massive thanks to everyone who has contributed to this herculean effort over the past four releases! ----------------------------------------------------------------------- ### Locale crashes (#610) One of the biggest remaining issues in WB post-wx3-upgrade were the locale crashes. This proved to be a wild goose chase. Every time we thought we'd solved it for good, some user with a different unusual setup came along and proved us wrong. The core problem is that wxWidgets, wxPython and Python itself are all trying to handle locales, and all of them end up fighting over it. We initially thought we'd finally fixed it in 2ebc752, part of the py3 merge, but that *still* didn't do the trick. 0e93496 seems to have finally fixed it, but at the cost of *completely disregarding the user's locale* and always setting the C locale. Far from an ideal solution, if you can even call it that. wxPython 4.2.0, which will be based on wxWidgets 3.2.x, should hopefully help here, when it finally (it's been *two years* since the last wxPython release!) gets a release. ----------------------------------------------------------------------- ### Skyrim AE (#616) Midway through 310's development, Skyrim Anniversary Edition was released. As this is really just an update to the executable and an optional batch purchase of Creation Club plugins, adding support was easy. 937b182 completed the task. However, an unexpected annoyance arose from this. All of the Creation Club plugins included in the AE bundle are localized, and it turns out WB's boot process did not love this. Which is why f789663 was necessary. This merge significantly improves the boot performance in AE (~50% on my machine, meaning an improvement from 2s down to 1s). ----------------------------------------------------------------------- ### Windows Store versions of FO3/FNV (#617) Quick and easy to support. Note that Microsoft has *once again* changed how Windows Store games are installed, and 310 does not support this new method yet. See #639. ----------------------------------------------------------------------- ### FName (#543) After the py3 upgrade mentioned above, the next biggest beast lurking in 310 is #543. What started as an innocent enough idea (bolt.Path is pretty heavyweight and unnecessary for use cases like masters and filenames, which never have path separators in them - would be great if we could replace all that with a more appropriate and lightweight class) quickly ballooned into a massive refactoring of much of the codebase, eventually culminating in one hell of a rabbit hole whose event horizon spanned many merges and three WB releases. See 308 and 309's release commits for notes on #543 progress back then. Reiterating all of this here would take hundreds of not thousands of words, so this section will only focus on #543 during 310's development. 15b979f reduced Path usages, replaced some LBYL patterns with EAFP ones and wrapped common patterns as preparation for the final #543 merge. Some more refactoring, especially regarding BAIN, happened shortly afterwards in 97768fd. The next few commits (specifically c55b60d and 9c25544) mostly focused on renames and some minor refactoring in preparation for further attacks on Path and its usages. In 8555285, many more Path usages were pruned. And in 9a0db27, the converters part of the codebase (the code that handles BCFs) received significant touchups. More explicit Path usages gone, more LBYL -> EAFP refactoring and improvements to this somewhat ancient and dusty API. And then it was time for the big one. b3b730c introduced bolt.FName, the ultimate result of all this effort. This gave performance and memory usage improvements, improved code style in many places, etc. But the main benefit, as mentioned all the way back in the 308 release notes, is that this paves the way for further work on #480 and related issues in 311+. ----------------------------------------------------------------------- ### Preliminary macOS support (#622) f085cf0 introduced some WIP support for macOS. Just like with Linux, this does not mean you can actually use WB for that OS yet, but it's an eventual goal. The immediate benefit of adding partial support for other operating systems is the resulting refactoring and isolation of Windows-specific code. Some more refactoring towards this goal landed in 14c7639, which refactored our archives handling (used for reading and writing archives, e.g. .zip and .7z files) to be OS-agnostic. ----------------------------------------------------------------------- ### Skyrim patchers (#151) As of 802423a, the last patcher (barring FormID Lists, which will need rewriting - see #497) has been ported to Skyrim (to be exact, it was Tweak Names). This finally brings the eight-year project of porting patchers to Skyrim to an end. ----------------------------------------------------------------------- ### FOMODs (#627, #603 and #593) Ever since 307, where FOMOD support was first added to WB in #380 (see 307 release notes for commit links), we've been steadily improving BAIN's support for FOMODs. 310 continues this theme. d636b5b implemented the last few parts of the FOMOD schema that hadn't been implemented yet (foseDependency, fommDependency, alwaysInstall and installIfUsable). Next up, 870c0ed implemented schema validation using lxml. bc6d1c0 resolved a long-standing feature request by adding a link to capture the output of running an FOMOD installer into a new project. And finally, 2c603d6 added support for generating an empty ModuleConfig when creating a new project via BAIN. ----------------------------------------------------------------------- ### Usability (#628) Continuing the fight for better usability (aka me sticking search bars everywhere), the doc browser got a bit of a makeover in c7e555c. On top of small improvements like "Forget Doc" reselecting a document after being clicked, the doc browser now also contains a dropdown menu listing all your load order's plugins and has a search bar to help you narrow those plugins down. fdb34dd cleaned up the Installers context menu even further (this started way back in 307) by moving archive-specific links into an 'Archive..' submenu that only appears for archives and similarly moving project-specific links into a 'Project..' submenu. WB has support for various keyboard shortcuts. Not that you'd ever know that unless you read the docs (wish more people did that to be honest, but oh well...). To help with this discoverabilty issue, b8d3e8d added keyboard hints next to links that perform the equivalent functionality. And while we're on the topic of keyboard shortcuts, 70ca87e and d8fdd57 added some new ones. Told you this is just me sprinkling search bars all over WB: 03122cc added search bars to every list patcher in the BP dialog. That means tweakers ("Tweak X" patchers) and importers ("Import X" patchers, including "Leveled Lists" and "FormID Lists") now have a search bar you can use to filter their sources. Especially useful for Tweak Settings, which has more than *eighty* tweaks in Oblivion! There are also, as per usual, a bunch more minor commits improving random bits of WB with regards to usability in 310 that aren't mentioned here. ----------------------------------------------------------------------- ### Patchers (#312) Yep, still going on (and will be going on for a while longer...). e3b93d1 refactored a lot of Oblivion-specific constants and code, especially stuff related to MGEF handling. This stood in the way of extending WB to handle more games in the future, since no one was sure how exactly to update these constants (or if they even needed updating for other games). ----------------------------------------------------------------------- ### Records (#480) And where #312 dwells, #480 is not far behind. One of the big goals for 311+ will be FormIDs handling, and 2e3e5ee functions as a prelude in that direction. Especially nice as it showcases how Python 3 (specifically its ordered dicts) can help in refactoring. ----------------------------------------------------------------------- ### Debug mode (or lack thereof) (#638) We were getting mighty sick of people not understanding how to properly report bugs and volunteers having to spend time over and over explaining the same thing. So in c9a5036, we changed WB's debug mode so that it's always enabled in standalone builds. That's great and all, but debug mode has always had one problem: if you enable it, you lose the "native" error reporting mechanism provided by the environment you're running WB in. Most notably the wxPython error window that would otherwise pop up if you hit a traceback. So in d66da27, we dropped debug mode entirely and always enabled debug printing. To solve the problem of the native error reporting mechanism getting disabled, we now copy the debug print output to both the native stdout and to the BashBugDump. That way reporting is as easy as possible: users get notified of a traceback and the BashBugDump already contains full debug prints. The reason not to do this in the past was the overhead of deprint, but since fab8b90 deprint is pretty much as fast as a regular print (all the overhead came from usage of the inspect APIs, which statted the source/bytecode files). ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BeermotorWB, @warmfrost85, @GandaG and many more that GitHub's contribution tracker doesn't list.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Turns out those got released at some point. Shouldn't be hard, but I'll probably need to subscribe to GamePass again to get testing done (ugh).
The text was updated successfully, but these errors were encountered: