New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wxPython 3 Upgrade #15
Comments
I have thought of this a lot cause I tried (and managed) to collapse the installers under any particular marker in BAIN. I spent last 2012/2013 Christmas to do this and posted in the forums - one of these days I will upload the code. This made me realize that the "view" and "model" code are simply the same thing. What must be done IMO is : after the patchers are out we must define a GUI API (with classes as MainFrame, Tab, etc) and decouple the library from the main Bash code. So we will have a Tab.py and then a TabImpl.py which has the actual imports to wx and refactor basher.py to use the Tab.py API. Then we go to TabImpl and import wxWidgets 3 instead and then later on Phoenix - without changing a single line in Bash code. I hope come weekend I will be able to look into this |
Hmm, I understand that Bash creates a bunch of custom UI elements, and moving such UI code so that it isn't mixed in with 'worker' code sounds like a good idea to me, but I'm not sure about writing a full API layer between Bash's main 'worker' code and the wxPython code. That sounds like a fair bit of unnecessary work, though maybe I've misunderstood you. |
I admit I haven't still looked at how this could be done and I remember I don't really know if it's worth the trouble to be honest but I'd say yes On Tue, Feb 18, 2014 at 4:41 PM, WrinklyNinja notifications@github.comwrote:
|
I believe the switch to 3.0 shouldn't be too hard. I may give it a whack myself in a new branch, dunno. I know a few of the GUI classes we use changed slightly. I had at one point ported Bash to Python 3 syntax, but the GUI was messed up due to whatever changes wxPheonix had done at the time. Once those are figured, I'm sure it'll be easy enough to swap to 3.0 As far as refactoring the GUI code, the idea is sound for sure. There's a lot of interconnection going on there though, between all the monkey-patching that has happened over the years and between the fact that some tab's GUI use the Tank class and some don't. I honestly don't know which direction would be the best to go from there, but yes I looked at standardizing all the tabs, and gave up. It's certainly doable, but it will take a bit of work, and I had more important things to spend my time on at the time. |
I've been working with Project Phoenix for the last year or so getting most stuff up to par with PY3 and phoenix. Also, using the pure python agw.aui for the redesigned GUI is almost a must have... As to quit living in the stone age of user usability. Just please don't suck-it-up like everyone else and move the GUI to PySide/PyQt. |
Do you have any idea when that is likely to be? I was under the impression that Phoenix was still a way off. |
I run phoenix with my projects & python2 fine right now. Some of the python3 stuff is still being worked on or needs tweaked. It is getting closer everyday. I think Robin at most might do a final Classic update once after Phoenix goes live, then you might as well consider Classic DEAD. This would probably be a wx3.0.1 or wx3.1 Classic release. And since wrye bash never used custom built control widgets, this isn't a real big deal. |
It sounds like moving to wxPython 3.0 should be approached as a springboard to Phoenix then, giving us a chance to clean up the UI code and make the transition to Python 3 and Phoenix as painless as possible in advance. Thanks for the info. Though saying that, maybe Phoenix will be released by the time we get around to this. TBH, I'd consider refactoring Bash's existing code (not just patchers), a more pressing "heavy work" issue, we don't need to update the UI library, but we do need game-specific code out of the bulk of the Bash code, for example (or Skyrim's Bashed Patch will stay crippled, and Fallout 3/NV support won't happen). |
Yeah, basically. Get the GUI working with 3.0 Classic, then switching to phoenix will be simple enough. Most of the PY2/PY3 cross-compatible syntax stuff is relatively the same with some exceptions for stuff like... for key value in dict.iteritems: in PY2 six.py handles most anything you would come across, unless wrye bash is solely going PY3 only, then six wouldn't be needed. I have a small utility I've been working on that will help with the "Classic vs. Phoenix" stuff. |
Actually I could probably skim over most of it in a week or so to help get the classic and phoenix checks all rounded up, since I work with it everyday anyway... |
@Metallicow: we are at the process of refactoring so it is easy to mess up - and the messy repo reflects both this and our transition to git. Read couple last threads to get a feeling of what's going on. Mainly the master branch is an SVN clone - branch off there to give a whirl to basher - but we will be discussing how we will finally split it - most likely it will become a module - basher/. Keep in mind it's better to wait till the patchers are out cause this will have impact on basher eventually and it is happening now - there must be no conflicts. How familiar are you with git, branching, rebasing, conflict resolution etc ? |
Well, I feel comfortable using basic git to create a repo, create a branch, send commits to branch. I'm not looking at doing any of the actual refactoring code(game based stuff) ATM(as I can see that end is still a mess), just getting everything running/starting up with wx3 Classic/Phoenix. Basically fixing up the wxWrapper and the changes for widgets methods Ex: Lists and stuff have changed. Most of this stuff is easy recognisable for me with a quick search and wouldn't require everyone else to go digging though the new docs, which are still WIPz but are atleast useable now. Basically, I would just fix up the GUI to run up to date wx and then you all can look at how the canges where made in basher and use the different syntax parts for your refactor(moving/splitting/etc of the code). Most everything wx-wise is split up well enough for me to do this without jumping around a whole lot. 2.8-3.0 is 7-8 years old BTW, for those who didn't keep count on the "quote" between stable version changes. Just give me an Idea of where to start and hopefully maybe without having to install all the game stuff on this HHD, and this could be done pretty easily in my off-time in probably about a week or so. So, what would be the best plan of attack? |
@Metallicow : I'd suggest you clone the repo, branch off from master as in $git checkout -b mettallicow_wx3 and edit basher (keep your commits as clean and self contained (should compile individually and should be able to revert skip without affecting the rest of the branch) as possible - don't take this personally we're trying to establish common good practices for git) and then issue a pull request Try to keep off bosh.py - but after all I think you could edit basher at least now nobody is editing heavily. Also keep in mind you might need to rebase on master before pushing the request - comment back here when ready |
Ok, I have a few spare minutes for this tonight. might as well go straight to phoenix. Ok, Classic3.0... Anyway... Fired up fine-minus some basic gui issues. About the 2 last lines about R6025 and the - pure virtual function call Any Ideas? Is this bash spitting out these messages? or something else?:
I need to find the small cause of the crash on exit first. This is the most important first off and might be a bit tricky, but I ran into this a few times with SourceCoder and made the appropriate menuing patches/fixes. This might be tied to the custom statusbar also... EDIT:
|
this is from Bash
This is wxPython > 2.8. I've noticed this myself. Some chance in (i believe the ListCtrl?) is incompatible with what Bash is expecting. Even with wxPython 2.9 the same errors occur. |
Ok. good enough. I haven't did a lot with the ListCtrl stuff yet. Most of that stuff is littered around outside of balt. So, does that mean CBash.dll needs changes. C++ isn't exactly my cup of soup, tho I understand it a little better than I did way back when. ListCtrl and similar ctrls will need a pretty good check through as bigger wx changes happen with them. ...I will still need to pinpoint the crash on exit tho... will see if I can figure it out...hopefully the easy way, because it might not be CBash that is causing the crash, but then again it might be... Usually crash on exit has to do with the gui stuff... Does CBash try and do anything in the close function? |
Note: There is no R6025 and Pure virtual function call with phoenix at this point.... hmmm wonder what it was in classic? Something strange is probably lingering in the classic close code, but phoenix doesn't mind...??? |
Back burner - notice though that it is taken care of at #163. The de-wx'ing of the code is an essential step towards being able to swap our graphics library without rewriting the UI |
@wrye-bash/bashers Notes on my wx3-phoenix compat branch @ runtime currently:
But at least I can say now OFFICIALLY Wrye Bash works on wx3.0.2.0 Classic and Phoenix snapshots, even if poorly because of bash's refactoring at the moment. This is major Step forward if nothing else. |
? |
There are probably some other wx fixes to be found and an Crash(No trace/swallowed) on closing issue lost somewhere in the code that needs fixed. |
Most of the GUI work can be done in SourceCoder in less than an hour. Time yourself. Destruction magick imminent. |
- Destroy does not do the trick still appears behind the Refreshing Installers one... (???) Another attempt for a fixup: Mopy/bash/bosh/bain.py: @@ -1731,2 +1731,3 @@ def refresh_installer(self, package, is_project, progress, installer.refreshBasic(progress, recalculate_project_crc=_fullRefresh) + progress(1.0, _(u'Done')) if do_refresh: - progress window for project did not disappear immediately (??) This does not always do the trick either............ Mopy/bash/bolt.py: Progress(object)
@Infernio repushed the branch but messed up a bit rebase so let me rerepush - meanwhile I noticed yet another bug
and we close the window by the main X button python crahes and the windows dialog for stopping/debugging is displayed
|
Ok rerepushed - included my hacky workaround which should land on dev along with patcher fixups (maybe squashed? or non ff merge?) 8eb0e39 and BAIN ones Squash that in too b962d8d Re: load order - good job! I think that if we implement moving the active lists in BashLoadOrder.dat we can close that issue with a nice non ff merge Of course I may be missing something and maybe some of it needs more testing |
I may have missed smth (or added something stupid) but you get the picture - another annoying TODO: chardet merge should try and download chardet in the python version of the installer then we can merge that too |
Oh god, this sounds like fun. Will spend some time looking into it.
Yep, that one has been getting a lot of attention on Discord. No idea what causes it yet.
|
Thanks - we could maybe add a Link.Frame.Show() when progress is destroyed although this might cause Wrye to pop up while in the background? For another example related check ModChecker - push that button "scan for dirty edits" then cancel the progress - Bash disappears (!) |
Decided to do the records/patcher merge as |
So, the real bug here seems to be that clicking the X of the main window shouldn't even be possible, since Edit: Scratch that, it's not working with |
I think I know why it happens, but this might not be fixable on wx3. wx makes its modal dialogs work by creating a temporary event loop and pausing the application's main loop. However, these modal dialogs happen before we've created our main loop ( Edit: Looks like it would definitely be fixable on wx4, but wx3 doesn't seem to ever call |
If it's fixable on wx4 then I'd say we can live with it as no user is supposed to open Bash just to peek those warnings. Good you investigate wx boot however - fixing the weird focus issues may be also related to all this |
If you wrap every call if anything has a widget, it will help, tho testing old code and new will result in a test havok if tests are made. Ex: dont write unittests for 2.8 and expect them to work for 4.0 without full attention for example. If you can wrap all wx4 code before you get there... even better, tho all logic code is your own problem. |
Interesting factoid: the 'progress dialog causes WB to lose focus' bug doesn't seem to happen with pyinstaller 🤔 |
@Infernio Most widget focus bugs are worked out in wx4.0, tho they might still appear in 2.8 <> 4.0. |
I suspect that the loose focus bug has to do with saving window size/position - whenever I experience it on boot the warnings windows are messed up (minimum size ?) - if the bug does not kick in they display normally - so related to the code that saves position for not maximized windows Just wild guesses |
After the wx3 upgrade (see #15), the next step towards Python 3 is moving to wx4, which is the first wxPython version to support py3. Note that I dropped it from the installer - that's broken, but it's already broken on dev and nightly since it doesn't install all our other dependencies like: - chardet - lz4 - pyyaml I don't see any way to update it do to these either, we'd have to hardcode direct links to PyPI resources, and I doubt PyPA would like that :) So I propose we just drop the ability of the installer to install python versions. Seeing as not a single person has complained about this being broken since we moved to pypi chardet, I doubt anyone uses it - plus would simplify the UI a bit (no more checkmarks). Done in the next commit. wx4: Build fixups Yay, it's a mess :/ Note I dropped MPR.dll, has been in our nightlies for a long time now but clearly *shouldn't* be there, it's a standard Windows DLL for communicating with network drivers. Closes #488 Under #460 Co-authored-by: MrD <the.ubik@gmail.com> Co-authored-by: lojack5 <1458329+lojack5@users.noreply.github.com>
For reference's sake: the locale situation seems to maybe have finally gotten better in wx4.1: wxWidgets/Phoenix#1702 |
307 is a gigantic release that has been cooking for almost five years at this point. In fact, it is even bigger than 306: 305 -> 306: 48,748 additions and 44,482 deletions 306 -> 307: 172,916 additions and 111,125 deletions Where 306 was mostly about refactoring the codebase to save it from becoming inoperable, 307 is all about using the newly refactored codebase to design new features, support more games and, of course, continue the everlasting refactoring war. The following notes are an attempt to summarize every important thing that's happened in 307, grouped as topics in a semi-chronological order. Do note that this is mostly a futile effort due to the 2000+ commits and 280000+ lines worth of changes that make up 307's development (and the fact that I only joined the project two thirds of the way through 307 :P). It is also highly recommended to read this on GitHub, with gitk open in the background. That way you can click on issue numbers to open them, while pasting commit SHAs into gitk, where the individual commits that make up a merge can be seen (GitHub does not expose this at all). All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### The bosh Split (#201) The first goal of 307 was to split bosh into a package. After the patchers had been split out (#163) and basher turned into a package (#3), bosh was clearly the next big target. 1bdc253e362857dc8a2a484be60bbcccd7891d82 began the splitting, then it was continued by splitting out the 'messages' code backing the PM Archives tab in 906858fe14c730ba797711a855933d996d573e2f. As a small interlude, 5b14b499e17c48917a472ee9ed426b3c7afd8985 then removed the PM Archives tab entirely (#221) due to the maintenance burden it had turned into. 682d2134c5e9f69b057cc9fb6e69e13767f5e7de continued the bosh split by ripping out: - The face transfer code (used for moving NPC/player faces between saves and mods) into faces.py - The OMODs code (used for unpacking OMODs, an old, ugly mod format used by OBMM) into omods.py - And the BAIN converter code (used for BCFs, aka BAIN Conversion Files - small files containing instructions that can be used to tell BAIN how to repackage an archive into a format it can recognize) into converters.py 318e66a9ad3a096c316decc751bc76d6c3b5b858 and 75c48e654c21feccd52cd851f2b502fd7c41c45f moved some bosh contents into parsers.py and bass.py - most importantly, the very commonly used 'dirs' and 'settings' constants. ea242af573952328168759b9c9c9ff1c01ac681c moved various miscellaneous things mostly related to plugins and LOOT into a new mods_metadata.py file. Coinciding with the INI refactoring (see 'INIs' section below), 6d2a1fa7bcb15f3d5fe787b5475e904e79809253 moved the INI handling code into bosh/ini_files.py. Similarly, afb7058a8e0d8a4567d43ec8f6f8828cafc34b83 moved BAIN code into bosh/bain.py as part of BAIN refactoring (see 'BAIN' section below). A last few commits splitting out the saves and mergeability checking into their own files (bosh/_mergeability.py, bosh/_saves.py and bosh/save_headers.py) were made in 347c552c8bbd3506a5bacd6c678a92e1275c8e06 and aa82a7af6fbcc6ce24ac1a940d3c2d4f665d626b, closing the issue for now. bosh/__init__.py is still quite big (3427 lines), but splitting further is far from trivial. ----------------------------------------------------------------------- ### Fallout 4 (#251) On November 10 of 2015, Fallout 4 was released - and less than three weeks later a branch by @lojack5 adding basic support for it was merged in e85506af4fbd28c4617b46c9ef0832c6c6570ecb0. Unfortunately, not all game merges in 307 were this speedy ;) Some more improvements and fixes landed in 134d7c63839de3daa52108f7ce9cc02fcc928df8 by @Sharlikran. Fallout 4 support was barebones then and is still barebones now. The real followup work here will come in 308/309 (see #525 and #482). ----------------------------------------------------------------------- ### Dropped Support For Older Settings Files (#253) Various bits of backwards compatibility cruft had accumulated over the course of 306 (and even earlier) to keep pre-306 settings working. 307 dropped all this and instead shows an error message if pre-306 settings are loaded in 307, telling the user to resave them in 306 first so that they can be upgraded to the new format. All this was done in a single merge, 8ceef20f269aedb18dbffe17063fac1cb7ccc0f4. With how big 307 has become, it is probably no surprise that it too has by now managed to accumulate a metric ton of backwards compatibility hacks. Cleanup for those will follow in 308/309. ----------------------------------------------------------------------- ### env.py (#258) In the ongoing quest for native Linux support (#243), encapsulating OS-specific code (i.e. Windows-specific code) in its own module is an important step. This was done in 646df2ca063c2dff03cb2185e7e2969fdea065a5. We're still not really there yet and env.py is, for the most part, still a Windows-specific module. The end goal would be to have two separate backends from env.py, so that it imports from one of them based on the OS it is running on. ----------------------------------------------------------------------- ### BAIN (#219) 307 also marks the beginning of the grueling task that is refactoring BAIN. Still nowhere near done, but it has become manageable. Much more work to be done in 308/309. Some of the important targets were: refreshSizeCrcDate was a big method that handled scanning directories for their contents. The problem was that it was basically two methods in one and implemented things that only make sense in one specific directory, namely the Data folder (e.g. empty directory removal). 20e41b3cfc38c71e8baae6e34778b964ca809533 refactored refreshSizeCrcDate to prepare it for 5ecf4000f10383e3018b4a4febbf8a8f13164a9e, where it was split into two methods (_refresh_from_project_dir and _refresh_from_data_dir). refreshDataSizeCrc is one of the central parts of BAIN. It scans a package for its contents, applies skips, remaps documentation to the 'Docs' folder, queries and caches CRCs, etc. ea79bb4fdfe2318d84952ba63fec2fb9b2a7e33a and 5fef1e3924bce2edcf19ff41e469c2a846d851bb tackled this behemoth, reducing its size significantly. Various later commits like 54844fbe1ea4ddfd48128e61f4232439431aac45 and c48112237b9953effb74b4206707394f225f035b touched this one up even further, bringing its final size in 307 to 210 lines (down from 348 lines in 306). 5d435bb05b9e09340eef9a71a402ac018abcb573 and 5ebbf4e8e3646efa8019b9cbccb144be9bc0ead9 focused on centralizing and refactoring BAIN refreshes, most notably the irefresh method. b109a4d08f21e023a30d5c0bf4ed45def0ae26e1 made BAIN use the modInfos cache to avoid recalculating CRCs that we already calculated for the Mods tab, giving a minor speedup - but mostly the idea here is we want to read the Data folder once and then delegate the files to the various FileInfos based on name and/or extension. This is a first step in that direction (see also #353 and #265). b1711b1b52fe76f0463b47fab825058f0f3bc41e was an important merge addressing case insensitive string comparisons. BAIN makes heavy use of this to keep track of files in the Data folder, each package, external changes, etc. So a central dictionary that made case-insensitive comparisons of its string keys was necessary - which is exactly what was introduced in this merge, as bolt.CIstr and bolt.LowerDict. 545cad09d29cf4b17ed470caba974829f5876d7f refactored the conflict detection and reporting algorithms to, well, separate them in the first place. They sat in a single big method called getConflictReport which came in at 120 lines pre-refactoring. In addition, the fact that this method returned a single string made it impossible to improve the GUI for conflicts (a goal in 309) without parsing the string - the string we just constructed from in-memory objects. The new getConflictReport is 55 lines long, much more readable and makes use of a new find_conflicts method that can be used to retrieve the actual conflicts as in-memory objects to work on. ----------------------------------------------------------------------- ### basher Package Followups (#163) Began in 306, finished here. daafdd5e76746833afe4eba496aa4afac41ff439 dropped the ancient Tank class for good, curing the flickering on the Mods tab in the process (#179). More work done in b7ffca085feb4afe93e48cdafb68521fd9402899. a28fcfce1f59482d1e701504b738a8df26a9bfe4 is a joint merge by @Utumno and @DianaNites refactoring the mod export/import links, which were pasta-filled mess. More prerequisite work landed in 10703a9e2982ae28db60af72146de8d8dca9c324 The topic was finally laid to rest with fb37a8bf110b6953c07a61a470d05b73ab03eb17 and 3bf6006d842942214694547a3e4c2b2fccb71dee. Every tab now has neatly separated tab panels, UILists and details panels. The API could still be better (especially considering how many new tabs we have planned for 308+ - see e.g. #233, #456, #50, etc.), but it's a far cry from the situation in 305 and 306. ----------------------------------------------------------------------- ### Skyrim Patchers (#151) Another principal goal of 307 is getting Skyrim patcher support as close to Oblivion as possible. Some prerequisites were merged in 3a2d396b2a9cf2ecd2525006a4e9960f3b4c85ff and 1e18dad44b97bf80b115de0c7175d14df07f2c3f (mostly records). The first real patcher porting then happened in 9599368dee429f4b63f116d25945beae73d838db. ----------------------------------------------------------------------- ### Load Order (#295 and #309) Load order handling is a complex beast, to say the least. Not only is handling all the edge cases difficult, but there are three different methods that the games use for implementing load order (four if you count Morrowind, which we don't support - yet ;)). Wrye Bash originally used the BOSS API for managing load order. This was changed in 1bf84f3e3b246195b93d9715e2d1891decc47354 to instead use the dedicated libloadorder. Even libloadorder itself proved problematic however: - Adding support for new games was tough and quickly turned into 'adding more clauses into if-else chains'. - The API generally made no effort to keep actives order and load order together - which became untenable in Fallout 4, where the two are inextricably linked. - Wrye Bash needs to read all the plugins anyways (e.g. CRC, ESM flag, etc.) and keeps that information cached in bosh.modInfos. libloadorder was reading it all again which, on top of being inelegant, was thrown away performance - syscalls are *not* cheap! - It's a DLL - bad for git and won't work on Linux (#243) So, in one big merge, 66d7b4d695289f6dc29142f94a6004494e3306bd replaced libloadorder entirely with a new API written in pure Python. This enabled many new features such as locking the load order in all games (bc7e5de47f3ecf31d1290d436940a37a7b6eb0cf), automatic backups whenever we make a fix to the LO (38817bb5ad6e222c4fdbde270d10341b2371bdb5), etc. ----------------------------------------------------------------------- ### Patchers (#312 and #461) Porting patchers to newer games is going to become harder and harder unless we refactor them to make it easier. Patcher code wasn't *terrible* per se and was fairly isolated from the rest of the code (apart from its tight connection to records code - see 'Records refactoring' section below), but a lot of it was copy-pasted and hence difficult to understand and expand - plus bugs often had to be fixed in upwards of 10 places. Adding a new patcher easily required hundreds of lines of pasta. This is still an ongoing goal, with much more to come in 308. 30eda2dd5c987648a11fbe01b8ee1b6c56d7c1ac was an early merge, containing refactoring on other more or less related things. 122784f5cf4d737bbb5943e9bd395d2bfc53c43a then moved config handling (i.e. which patchers are active and which sources they are operating on) to basher. The idea is to have the patchers operating only on a list of sources - not only for elegance and simplicity, but also to make them testable (see the 'CI & Tests' section below). Heavyweight refactoring began in 66d7b4ef3f8959e180f0029874c73e40de2a5a52 and f8bfdc4c5894ca2832aa5aef6515d70f52df110f, which introduced the _SimpleImporter class to deduplicate a lot of copy-pasted implementation code. Keep your eyes on this class, it proved to be a good idea ;) eb110497c3ce7b9d72df5a6565a0c7efffbc9a28 and 604ebd31d2f20d26c06ab2d043f114f7e0abc26b went in a different direction, by using the work that had already been done on refactoring the patchers to port many of the Oblivion-specific patchers over to FO3, FNV and Skyrim, as well as add some entirely new patchers that had been commonly requested. Of course, this involved a bunch of refactoring too, mostly moving implicit constants from all over the patcher code into game/*/constants.py, which will make it much easier to port the patchers in the future (e.g. to Fallout 4, see #482). 10e680cbb347b203f935c042145a5fc308b5f4c8 returned back to good old-fashioned refactoring by decoupling the config/GUI side of the patchers entirely from the model side (i.e. the code that actually implements patcher behavior). Previously, patchers were linked together by importing the patcher implementations in the GUI and using them as mixins. This led to lots of weird, hard-to-debug code that crippled the IDE's ability to perform static analysis. For example, the implementation of the leveled list patcher would use if not self.remove_empty_sublists: return to skip the 'empty sublist removal' part of the patcher if the checkbox for this was not checked in the GUI. However, the IDE had no way of knowing that that variable actually existed, since it came from the GUI side of the code via a mixin. After many failed attempts to devise a base class for both the model and GUI side of patchers, @Utumno instead realized that a much cleaner design would be to have no mixins. Instead, each patcher's GUI panel now has a class variable called patcher_type, which it sets to the model class that gets imported from the `patcher` package. This allowed us to drop tons of boilerplate code and make the resulting code much cleaner and easier to understand, but most importantly it acted as a springboard for further refactoring. Most notably, 9697b64abc00beaed04e950af20c8116db15151a split our importers.py file into four files: _cbash_importers.py, _shared.py, mergers.py and preservers.py. The key insight here was that we can split our importers nicely into two types: preservers simply carry forward the last value(s) from a tagged mod, while mergers merge values from all tagged mods based on the tags those mods have applied. This resulted in several hundred lines of duplicate patcher code being chopped off due to us absorbing many preservers back into the base class. It will also make it much easier to drop the CBash patchers, since they are now in a separate file altogether (see the 'CBash Deprecation' section below for more information on this). One last merge worth mentioning here is 426db77ee71c939bb785b94eab7d4281f0f1fa26. It is a highly WIP attempt to tame the mess that is parsers.py by devising a proper base class. The savings so far do look promising, but CSV reading and writing are ugly warts that still stand in the way. Plus the base class might be too complicated - right now it has six different knobs that tweak its behavior, and it's not even clear if using it for the all parsers is feasible. Still, we had to merge since previous betas came out with the plugin export/import commands this merge ports to Skyrim. Much more will follow here in 308 - most notably an upcoming refactoring of tweaks that will make them *much* faster and drop ~1200 lines of duplicate code. ----------------------------------------------------------------------- ### INIs (#247 and #326) INI handling was spotty at best. Random unicode tracebacks kept showing up, the INI Edits tab was one of the last big performance hogs and the default INI tweaks being files in the Mopy folder led to confusion (at least one mod had a 'Mopy\INI Tweaks' structure and was supposed to be installed *into the Mopy folder*). The first step was df2fcc5f95bc7bc2613a14cb996671f3b82dd2db, a series of smaller refactorings and fixups to make the tab's code more manageable. 1d4c23a037f6845f5dfebaf1e5e005f8f121a63b began the work on performance by introducing the proof of concept for a cache, while 58c47d562f43773c5b017531176b8869138869bf attacked the refresh APIs used by the INI Edits tab and significantly reduced the number of syscalls it made. The default INI tweaks were finally dealt with in 656127c645070d35d31423e014b692621ff94015 by hardcoding them into Mopy/bash/game/*/default_tweaks.py. No more tampering with the tweaks, no more Mopy\INI Tweaks folder to confuse users, fewer loose files packaged, simplified INI refresh, better performance due to fewer syscalls... ef21c5bb0a4f23d737dde3b22af36dcaf4f9b58b contained some more work on centralizing the 'apply a tweak' logic and fixing a longstanding issue where INIs would have Unix line endings written out, even on Windows. The LowerDict introduced during BAIN refactoring (see 'BAIN' section above) also turned out to be very useful for INIs: a9112d6761e47b1fc41aca53df1add5bdd41ad79 used it to rewrite core parts of ini_files.py for performance and readability. bb6c8bd2e7ac22d1218cbad90e404b9739dba7bc reworked the handling of INI encodings based on a central principle: work with unicode and stripped newlines internally, encode/decode and add/remove newlines at IO boundaries. ----------------------------------------------------------------------- ### wxPython (#190, #15 and #488) A war that started before living memory and will continue until long after we're all gone - or will it? Actually, we're very close to winning this conflict for good! wxPython was all over the place in 305. 306 improved the situation significantly, but 307 puts even those efforts to shame: 305: 2260 usages (balt: 381, basher: 1586) 306: 1112 usages (balt: 418, basher: 494) 307: 207 usages (balt: 191, basher: 8) *8* direct usages in basher, down from 494! Let's see how we got there: - 114c83729aa50045cac4f381912007826d8048bf: Utumno vs wx. Utumno lost, of course, but wx usages did go down from 1075 to 923. Mostly accomplished by moving common code to balt. - c7f5f5085acf1ec6e55f0048c256f7a0ebc3f367: Down to 902, and some progress was made towards wxPython 3. - 69c7f9f679f48df8cf7562442e80c9129f10924a: wx.lib.iewin was an ugly beast that was binding Wrye Bash to the comtypes dependency. Unfortunately, dropping it required upgrading to wxPython 3 (see below), so this merge simply centralized the iewin import for a future removal. - 531679d37d6c50cfc030b9c462f44250bcfab7a0: A small merge containing backwards-compatible changes that brought us closer to wxPython 3. - 050391ca22d7c8451390cbff6fd150ab0b9bcabd: The upgrade to wxPython 3. Introduced several significant architectural achievements: - Dropping the comtypes dependency by rewriting our HTML rendering code to use WebView instead of wx.lib.iewin. - Removing bolt's locale-related behavior on import that made importing it dangerous - it's been encapsulated in a new top-level module, localize.py. - Rewriting a lot of the very early boot process - see also the 'Boot' section below. - f9e46eed670b3d2805b1570fc62daf9cca12ce79: The big one. An absolutely enormous joint merge by @Utumno, @Infernio and @nycz introducing a new package, gui, that truly encapsulates wxPython. nycz wrote the first version of the code back in 2017, most importantly the layouts code that encapsulates wxPython's sizers in a declarative API. We then devised an event handling framework that enabled us to hunt down a lot of *implicit* wxPython usages. These are much harder and nastier to track because they can't simply be regexed. Thankfully PEP8 will be able to help us here, since all of wxPython uses PascalCase for its methods, while we're using snake_case for all new code. The result is a reduction down to 218 direct wx usages outside of gui. - 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: After upgrading to wxPython 3, the next goal was upgrading to wxPython 4. This is also the first release of wxPython that supports Python 3, making this a significant step in the direction of py3 support (#460). - eb86a4cb35fc24b58288a6b3e917ed9350a02e23: In preparation for finalizing and merging the FOMOD support (see 'FOMODs' section below), a bit more de-wx'ing happened, mostly on radio buttons and the splash screen. ----------------------------------------------------------------------- ### BSAs (#339 and #338) Same story as libloadorder. We were using a binary, libbsa, to do it. This was thrown away performance (we already read and cached the BSAs in bosh.bsaInfos), had no Linux support, etc. Its replacement, bsa_files.py, was introduced in b199a7bd5eec69ec0852f86d6e3629784e6b90ce, then used to support strings files packed into BSAs in c4f12d56d1273f40096abe45b6803d9265b1e75e and finished in 8ce81bfc9b5dc4643988e54471fab3e9b6a1f72d. With BSA handling code now taking shape in the form of bsa_files.py, having a second class arbitrarily handling a few things with entirely different (and much uglier) code would be a bad idea - so d69f3e82c3afe4b6461b6ccce49a210bedd28dd6 dropped the ancient BsaFile. There were still some unimplemented parts of the BSA format: - TES3 format: Added in 4ed5bd8c4ed9a67f872f109c660cf0afeb6ddca5, also in preparation of the POC Morrowind support we have in 307 (see the 'Morrowind' section below). - Compressed BSAs: Added in a6e11c4601d788e0e25c17361d92eaa659e59768, including both lz4-compressed ones for SSE and zlib-compressed ones for all previous games. - FO4 DX10 format: Added in 903b6d11d1451855951cce608c0ce8fe6a23743f. Currently unused, since we only use BSA extraction for strings files, which the DX10 format can't contain (it may only contain textures, for which it is specifically built and optimized). - Writing: We can read and extract everything from Morrowind to FO76 now, but have no support for altering and writing out BSAs yet. This will be a goal in 308/309 (see note below). All this work on BSAs acts as a prerequisite for the BSAs tab we want to enable and expand in 308/309 (it already exists in the codebase, but is very unfinished at the moment) - see #233. ----------------------------------------------------------------------- ### FileInfo(s) (#336) At the heart of each tab sits a DataStore subclass. This provides the UIList (i.e. what you see on the left side of each tab) with the data it should show. Most tabs (all but People and Installers) then have TableFileInfos in the hierarchy, and all but INIInfos (which backs the INI Edits tab, unsurprisingly) then have FileInfos in its hierarchy. These FileInfos use FileInfo classes to represent the files that are going to be shown in the list. These APIs are not *bad*, but they're not *good* either. Refactoring this is an ongoing goal, with lots of work done in 307. b199a7bd5eec69ec0852f86d6e3629784e6b90ce devised a common API for representing a tracked file with caching called AFile and used it for the new BSA API (see 'BSAs' section above). Some further work on freezing the AFile API and making FileInfo use it happened in 11f6769f641e80552b6f2d6e3c25d49a85e66c63, e3064942119c504786cbe6befafd8a0aa38bec2e and a0ae08c42fdf47688870387a6f6296de0551bfc2. Cosaves got a lot of work done in 307, bringing them from pre-alpha at best to a solid beta API (see the 'Cosaves' section below). Screenshots also got reworked to use the FileInfo(s) APIs in e007a5308d509fcd5e123d566b2b3c24e228edaa and 9a317f62cc8f7f807581a0a5f5453637d5d8b5ff. ----------------------------------------------------------------------- ### Skyrim SE (#347) In a join merge by @Arthmoor, @Utumno and @Sharlikran, Wrye Bash got SSE support added: 76b8abd5437042dd5b1f1e4505a651211d5523e8. See the 'ESLs' section below for some of the following challenges with SSE support. Of course, patcher support was spotty at first too - we ported all Skyrim patchers to it in c076e9fe1f5a27746a11b91ba9e0e1b43b80a915. One more merge worth mentioning here is 064d5021e068260cc99225be29d8d651b0761c61, which sped up startup in all games, but most notably in SSE. Our reference setup we used for testing (with ~200 saves) went from 10s down to 2s. ----------------------------------------------------------------------- ### Boot (#373 and #390) The boot phase was nothing short of a mess. There was no clear guiding principle of what is initialized when, leading to hard to debug problems. Unexpected errors could take down Wrye Bash for good without any way to tell what the problem even was. Restoring settings wasn't working at all. This is still somewhat in flux just due to how complex the boot phase is, but it's definitely gotten better. 0e3ef608e2906afb3405b009c622c632caa21dab began the process by centralizing the wxPython import. In 6060a157be13372ff2a8e09c8de9878c16190f2a, @D4id4los rewrote core parts of the boot procedure to gracefully handle and show errors, even when not in debug mode - making fixing startup errors encountered by users much easier. Restoring settings was addressed in 17f2266525e5095f8c068804fe3cc3471c9bac1a. In short, when restoring settings, we would override the settings we just tried to restore due to our atexit hook firing immediately afterwards. Instead of hacking away at it with monkey patches, @Utumno carefully reworked the boot sequence to clearly lay out what gets initialized when, breaking barb's dependency on the rest of Wrye Bash (balt, bosh, bush, etc.) in the process and adding a new top-level module, initialization.py, to better encapsulate init procedures. The locale mess that bolt did on import was addressed during the wxPython 3 upgrade in 050391ca22d7c8451390cbff6fd150ab0b9bcabd. This also resulted in a much more well-documented early boot process, including setting up the BashBugDump and bolt.depring much earlier, allowing us to use it to consistently log during the entire boot phase. ----------------------------------------------------------------------- ### Cosaves (#437) xSE (i.e. the script extenders - OBSE, SKSE, etc.) create cosaves for each save you make. Wrye Bash originally only needed these for its master remapping feature to not break things, but over the course of 307 we've come to use them to display save masters with ESLs in them (since those saves store two separate lists, an accurate master list is only possible by looking at the PLGN chunk in the cosave). They present a unique challenge in that each cosave is attached to a regular save, and all operations on that save need to respect the cosave. That means renaming, deleting, backing up, etc. need to not just apply to the main save, but also to its cosave, if it has one. 0a300af01a85bb3535d3194203fd34cc0d3e9b29 introduced the initial API for this, bosh/cosaves.py. It was mostly just a collection of code from various parts of bosh (mostly _saves.py), and as such was difficult to understand, maintain and extend. 822c0bd16e5788d1811449810f34edfee16d77a1 then refactored it (the commit looks like a rewrite, but was actually a gigantic refactoring comprising 100+ commits that had to get squashed down to a single one in order to not break dev) for maintainability and to bring Pluggy (an ancient cosave format in Oblivion) support into the cosave hierarchy. It also added support for saves with ESL masters, as mentioned above. Finally, e4dc76995703f16ee97fb07d495b6db635a2c6a8 reworked our handling of cosaves to be much more robust ----------------------------------------------------------------------- ### ESLs (#382 and #429) ESLs are a new type of plugin file introduced with SSE and FO4. They present a unique challenge in that they can bypass the usual 256 plugins limit, and as such stress-test many central assumption in any tool that tries to support them. Our APIs stood well to the test however, with d507111773d459e41468ac835955fe88915e622e only having to make minimal changes to add initial support. Due to the limited understanding of ESLs at the time, we were very conservative in what we allowed users to do with them. There was no way to verify ESL flags, add and remove them, and the Bashed Patch excluded ESLs completely. That was fixed in 547565a32f8d1d4a2944984c8c29092834f4fff6, a join merge by @Sharlikran, @Utumno and @Infernio. With it, Wrye Bash gained the ability to add the ESL flag to ESL-capable mods, importing from ESLs into the Bashed Patch was reenabled and load order operations for ESL-flagged ESPs were fixed. Once again, we were quite conservative in implementing the ESL flagging in Wrye Bash. For all record types we had not decoded yet, we simply failed the verification and told people to use xEdit to check instead. c137405418360063b6d767e7179c31fa94936cbc changed that to use a generic method that does not rely on our record definitions, since the only thing we actually have to care about are the headers of all records in the file. The contents of those records do not matter. This made ESL-flagging both faster and completely accurate for all record types in both SSE and FO4. ----------------------------------------------------------------------- ### Game Handling (#358) Along the way, especially after adding initial ESL suport (see 'ESLs' section above), it became clear that Wrye Bash's game handling would become a big issue that needed addressing. Adding support for a new game involved dozens of edits all over the codebase due to fsName checks and copy-pasting and editing a big constants.template file. This cripped the IDE's static analysis, since it couldn't check that any given game constant existed, let alone had the right type. We were also importing way too much for each game (e.g. all the constants, the default tweaks, the vanilla files, etc.), when we should really just import them for the one game we're actually managing. Thrown away performance and memory, plus just plain inelegant. In a joint merge by @Utumno and @GandaG, 11fa0f6a71ca8420071e76357b89f5d3e221c904, the game constants were moved from module-level into classes, allowing them to be inherited. This got rid of tons of duplicate code (1229 insertions(+), 1824 deletions(-)) and made adding both a new game and a new game constant easier and less error-prone. Some more work to move game-specific constants out of random files and into the game/*/constants.py files they belong into happened in fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, along with some fixes in 06d6a6bdaa925f379ffc1f05d0fa5977057ac739 and 57d3a621a08f4852dc5d5cc36878db9286351579. 4050e60bd372494c46860c85fa15c1701abcc5ae devised a way for us to avoid importing the constants, default tweaks, etc. for every game, while ddda9393d9b08a132c4a346fbdd8cc84454bccbd and 0fecde47b73d3204735c28b37260b7af8a01f700 finally finished off the last few constants outside game/*/constants.py, closing this issue for the time being. ----------------------------------------------------------------------- ### Fallout 3 & New Vegas (#150 and #468) Not originally planned to be part of 307, but after it was accidentally included in Beta 3, we had to merge it: 0f06e4fd306684aafccd2764b47a66d2205d10fc @valda originally ported Wrye Bash to FO3/FNV as Wrye Flash. Efforts to backport the changes to WB had been dragging along since forever, so the main thing we learned from this merge was that leaving games to rot around in branches is a *terrible* idea. Better to have the WIP code on dev without explicitly providing support, as leaving it on a branch makes it accumulate subtle bugs from refactoring extremely quickly. Some work on synchronizing the FO3 and FNV versions happened in the form of 54b8e614acd0841460f550d33d23340824f06e31 - since FNV is a vastly more popular game when it comes to modding, many of the improvements that valda made to the NV version of Wrye Flash did not make it back to the FO3 version. With us being based on a single codebase, doing that is much easier - eventually culminating in the FNV constants being entirely deduplicated in 3f96076501d5c260af856dac1f6475c21aa53a6e, e68b7af1eb9a3eed96b72cda067be82d86e067c6 and dccd28c70ffb857ccc70265a5ca22ca718d8e442 so that they are based on the FO3 ones, meaning that adding e.g. a new patcher or bash tag to FO3 will automatically add it to FNV as well. We're almost at feature parity with valda's version now - only the race patcher is missing from our version. This will be addressed in 308/309. On the other hand, we support several patchers and tags that valda's version doesn't, on top of tons of other features and bugfixes (see, for example, the rest of this commit message ;)). ----------------------------------------------------------------------- ### Readmes (#432 and #464) 307 includes a significant reworking of the readmes, courtesy of @FelesNoctis in 493c76b38c760d31757657a5b9bcf70f196d1dcd. It was later followed up with more edits for maintainability and to update the screenshots: see 18969116f7547148bf7b66196f91accd1225b465 and TODO ----------------------------------------------------------------------- ### Wizards (#446, #445, #444, #436 and #189) Several issues related to wizards were fixed (see e.g. 169d8347c1e4f3a3f6d696d4a660f89987f6bffc, 1f71bf335b85276566c12db43b53097842e05981 and 30f698520be938cd3cb6aa950cf979bc5468edb6). We also refactored the code quite a bit and deprecated the old 'Espm' versions of keywords and functions in favor of new 'Plugin' versions (more intuitive, easier to spell and remember, and more accurate with the advent of ESLs) in 3426384083bd5d7c61d42730ed7fa9c5629bc2db. Finally, 9244f8536ff0e4b780e3190cc40a032771310f4c added a new wizard function that had been requested a long time ago, enabling wizards to alter their behavior based on a plugin's load order. There is still a lot to do on wizards. For a start, the format is not formally defined - and the parser that acts as a reference is quite buggy (e.g. `Note thisIsAString` will print out 'thisIsAString', because the parser gets confused about its token states and accidentally treats 'thisIsAString' as a string). See https://github.com/Infernio/wizparse for my POC attempt at defining a formal grammar based on ANTLR that other mod managers will also be able to use. This is low priority, but will be continued in 309. Additionally, the wizard GUI presents a significant challenge in de-wx'ing (see 'wxPython' section above) and has a lot of duplication with the new FOMOD GUI (see 'FOMODs' section below). ----------------------------------------------------------------------- ### Python 3 (#460) We officially started the process of porting Wrye Bash over to Python 3 in September 2019, seeing as Python 2 has reached its end of life. This has turned out to be nothing short of a giant can of worms. Wrye Bash makes heavy use of bytestrings, so simply letting 2to3 run over the codebase would be disastrous - we'd be fixing unicode/bytes tracebacks for the next few months and getting no actual work done. In addition, our policy of having no breaking commits on dev means that an eventual Python 3 port will have to be a single commit, which makes bisecting useless. So the result is that we need that py3 commit to be as small as possible. With that goal in mind, a lot of prerequisite work that brings us closer to a py3 port without breaking py2 has landed: - 660ecbb81f49d478bdc8e4a8905e328d1daf9dca: py3 has no 'ur' prefix for strings since the one in py2 wasn't actually a 'raw' prefix: >>> print(ur'\u03B3') γ So dropping this one from the codebase was necessary. This commit just dropped all usages in strings that didn't actually have backslashes or were autogenerated paths (i.e. vanilla_files). - d43ad244170e2110a6daca7d5febed4020550247: This commit by @syntaxaire finished off the 'ur' removal mentioned above. - 5a98eb4c025651f4e9366db2a7d488ec2068f1fc: cmp and __cmp__ do not exist in py3. For the most part, we just had to implement rich comparisons. - fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, cae844b9dde8af014b09a1cb24af2348d5620058 and 6db5b8b59e28bc46a9d42e966d31007e113c59e6: Changing old-style classes to new-style ones work fine, except when the class is used as a mixin with a new-style one that uses __slots__. That can lead to nasty layout conflicts, as seen in the first of these three commits. - 8e201c49bbc809da89b1bda1d269f4cb7619dfc0: Our codebase included an ancient version of chardet (1.0.1 from 2008) due to a single manual edit that was needed to make it avoid returning the EUC-TW encodings that Python doesn't support. We dropped it in favor of the PyPI version, and addressed the EUC-TW problem in 60d0c29dbae91c12c1f7825df9f4e8e243ca09d2. - d8d03ca39e1e9f85250fd014cabcc2a65945e5e7, 197b6a2de78acd723f9d747fd6751fd2c68cf944 and 659e5b696be5083b9bef0d39356acc30ab46b5a4: Long integers don't exist in py3 due to its int type having no max size. So we needed to drop all 'L' postfixes and usages of sys.maxint. - 664f1722a53c91794f192e343936dfd34b8e86a8: Fixes for various issues encountered during an experimental run of 2to3 by @lojack5. - 33eac7624971ecd22e1f65ff5e47bc71ca175dbc: Merge by @GandaG addressing various py3 issues like print, moved stdlib modules, old exception syntax and usage of local absolute imports. - 050391ca22d7c8451390cbff6fd150ab0b9bcabd and 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: We were stuck on wxPython 2.8 for a long time, but the first version of wxPython that actually has py3 support is wxPython 4. These two commits (as well as tons of prerequisite refactoring, see the 'wxPython' section above) cleared that blocker for good. The Python 3 port is one of the primary 308 goals, along with patcher refactoring (#312) and records refactoring (#480), on which it is blocked (due to the aforementioned heavy bytestrings usage, which those refactorings will help us isolate and encapsulate). ----------------------------------------------------------------------- ### Build Scripts (#415) An enormous productivity gain for developers came in the form of @GandaG's reworking of build scripts in a1b5bfaa40fdbe04549ba3775106ffdff471e62e and 5f31a2adf39a607db282f49bd43e66c992a1baad. The ancient package_for_release.py has been replaced with a sleek new build.py script that does everything you need to do to build Wrye Bash in a single invocation. On top of that, Ganda also dropped tons of weird legacy things the build scripts did, like using ResHacker.exe to set the Wrye Bash icon - more binaries gone <3 ----------------------------------------------------------------------- ### Enderal (#433) Support for the Steam release of Enderal: Forgotten Stories, a total conversion mod for Skyrim was added in c2d73965fba4d0a82bb95f7cbe13b7f5dbcc0155. This was a fairly simple game to suport since it is pretty much just a pre-modded version of Skyrim LE. Of course, the work on refactoring game handling is the reason why we had so few problems with this merge. Once again, we left this game too long on a branch, meaning it began to accumulate bugs. That necessitated a fixup commit almost immediately in 3afa217d987c8c4ab86341ed8a8ec906b099767a. For all future game merges, we resolved to merge more quickly, as long as adding support for the game doesn't break any other code. ----------------------------------------------------------------------- ### Records (#480) After all the above, there were a few spots left in the codebase that needed *heavy* refactoring: records, patchers, saves (*not* save headers, the Oblivion-specific save editing code) and BAIN. Since the records and patchers code are very closely intertwined, they need to be attacked in tandem (refactoring the patchers is sort of a 'top-down' approach, while refactoring the records is a 'bottom-up' approach to the same problem). See the 'Patchers' section above for more information on that refactoring. The whole shebang began in ecac15d01dc5c89463ad47aab74260abfbea4167 and 3a3e9c935f5c1a798211eb0eaed0a0dd76a9af24, which were mostly just random commits improving some record definitions. Heavyweight refactoring began in 134433fde71534fc09e357ad64c696194f51a8eb, which moved an awful lot of records code into brec by creating new tools for defining record definitions. The result is a massive reduction in code size: 6787 insertions(+), 9581 deletions(-) ..and a very nice situation where almost the entire *implementation* of PBash sits in brec, while the (almost) purely declarative definitions sit in game/*/records.py. Unfortunately, this bloated brec to 3000+ lines and made it much harder to tell which classes belonged together. This was addressed in 28c11cb934056790e2a07703a9a09b4a5de8aa48, a huge merge that split brec into a package, added OBME support to PBash, sped up plugin loading by using AOT construction of struct.Struct instances instead of struct.pack/unpack and implemented merging of all record types in Oblivion. That's right, PBash can now merge everything CBash can. See the 'CBash Deprecation' section below for more information. After reading both the 'Python 3' and 'Patchers' sections, you should already know what's coming here: much more in 308. There's already a large refactoring ('part 2.5' of #480) that just needs some testing before it can be merged, and @Utumno is working on a 'part 3' of #480 that will seriously turn some parts of the records code on its head. ----------------------------------------------------------------------- ### Usability and Accessibility Wrye Bash has a (not entirely undeserved ;)) reputation for being difficult for newcomers to get started with (in UX terms, we'd say its out-of-box experience is bad). While this is obviously a big goal that we're nowhere close to solving today (really, someone with actual UX experience would be needed on the team), 307 does include some work towards both this goal and the goal of making Wrye Bash accessible to everyone, regardless of disabilities. - 1bf488a10a4c9fedb2737e4b2eee86c484f7b93d: The ability to jump to a plugin's matching installer from the Mods tab has been added (#53). - 3d7b9816ec0e2b7d666a7bead0cd45db5347aed7 by @fireundubh added the ability to jump to the matching plugin when a master is double-clicked in a masterlist (#311). - 261a1029a78e2cae773386c4e41f496a05f018c0 and f4987d1d0db38e4379f6470f67c37b982192817f by @BeermotorWB and @MacSplody trimmed the jungle that was the package context menu on the Installers tab by moving the more rarely used commands into submenus. - f65112bea111157558f78f056b550a7f689752a3 added the ability to jump to a plugin from the Plugin Filter on the Installers tab. - 92691409567ce020c6958325ee8c4bd8c9e820cc made the 'Sort By', 'Columns' and 'File' submenus on each tab consistent by putting them in the same places (they were in seemingly random positions on the context menus before). - 206ffbd9b09a0b8107ac5dacd9b0b3f9c2d1bfc2 by @warmfrost85 allowed users to get a preview of what Clean Data and Sync From Data are going to do, as well as the ability to use that preview to change which files the commands will affect. - e5685f3e87abd0bce199fe619509a9648ce2db89, also by @warmfrost85, allowed Sync From Data to work with archives. Previously if you wanted to, say, clean a plugin and sync it back into its archive, you would have to either do it manually in 7zip or use the workaround of unpacking the archive to a project, then using Sync From Data and finally packing the project back into an archive. Now you can just do it all in one go by using Sync From Data directly on the archive. - b427bbd383688a2f0fd55266b040cdc6ddf7c590 and a2297331b6571b6989bf1ea3b1c253a85c834448 increased the contrast on all our checkbox images to pass WCAG AAA guidelines, to make it much easier for people with weak eyesight to use Wrye Bash. In the future, we want to allow people to customize the colors of the checkboxes so that colorblind people can make use of them too (see #511). - edbe5e51d294ed0706478a9f8896d1e170d76058 added a menubar to improve discoverability of Wrye Bash's column context menus, as well as making it possible to access them purely using a keyboard. Previously you would have had to right click one of the columns to access these commands and options, which is a bit arcane and doesn't work for people who can't use a mouse at all. The same merge also reworked our half-baked settings menu that was implemented as a list of popup options when the tiny gear icon is clicked to instead be a proper settings dialog. Not only is this much easier to navigate, it will scale far better for our future needs (e.g. extensions). ----------------------------------------------------------------------- ### Morrowind (#479) Morrowind is not officially supported in 307. We've added some (very) WIP code in 45ed499de6b5564756867ce4b586ed8f4acb6c1f, enough to install mods and manage load order, but nowhere close to the featureset that Wrye Mash sports. The main purpose of this code is to act as a sort of regression test, making sure that we won't introduce anything breaking Morrowind support in the future (e.g. during a refactoring). However, adding full Morrowind support on par with Wrye Mash won't be a goal for quite a while (309+ at least). There's a lot of research and refactoring to get into before we can approach that. Thankfully, Elminster has began adding Morrowind support to xEdit as well, so we may soon have all the tools and documentation we need to write a modern version of Wrye Mash's features. ----------------------------------------------------------------------- ### Skyrim VR & Fallout 4 VR (#401 and #454) The situation here is the same as with Morrowind, but for different reasons. None of us developers own VR hardware, so we have no way to actually test if Wrye Bash isn't breaking everything on these games. Which is why there is no official support yet. Still, @nallar and @nephatrine contributed some initial code in 8273f37e431dfebcfbf114d2fee8cf8365dec889 and b2418eb47007a110fb3e255190d79304837a85f1 that seems to be working fine, judging by the fact that we're apparently included on at least one Skyrim VR guide already. And again, including this semi-supported game in our codebase means we are much less likely to break it in the future (not to mention that each game we add generally leads us down a rabbit hole of refactoring to make Wrye Bash more flexible and game-agnostic, which is definitely a good thing - one of our long-term goals is adding support for games outside the 'Bethesda sphere'). ----------------------------------------------------------------------- ### FOMODs (#380) One of the most commonly requested features in Wrye Bash's entire history. FOMODs are a fairly terrible mod format due to their incredibly loose nature. You can place files pretty much anywhere in the package, and there is no specification. That means mod managers are free to implement whatever they want - as long as it vaguely works like NMM's installer did, it's an FOMOD installer. Still, this format has become very common in Skyrim and Fallout 4, so we should support it. Huge thanks to @GandaG for contributing the initial version in fa48b1d26bcaf3e9a49b96376df56e2c5b425146, including a full FOMOD parser, which was a showstopper before. While all other parts of FOMOD support (including the GUI, the command itself and BAIN integration) have been rewritten since then (see below), the parser is still pretty much just as Ganda left it. Due to BAIN being built around mods being, well, *structured*, it wasn't a great fit for FOMOD support at first. We eventually came up with a way to hide all the ugliness of FOMODs from BAIN and just present it with the clearly structured resulting list of files to install in a big merge (54844fbe1ea4ddfd48128e61f4232439431aac45) that also contained a bunch of GUI improvements. It turns out that BAIN wasn't *actually* that bad a fit for FOMODs after all: it already had machinery in place to map one path in an package to a different path in the Data folder, since that's necessary for its 'root heuristic', remapping of docs into the `Docs` folder and the rarely used plugin remapping feature of the Plugin Filter. By hooking into the right part of refreshDataSizeCrc, we were able to use this feature to map the ugly mess that an FOMOD can be to a final list of neat paths. BAIN can now work with this regular list of paths, and when it does need to deal with the real package it will look them back up in the mapping. Of course all the BAIN refactoring that went into 307 (see 'BAIN' section above) is responsible for making this possible :) We still don't recognize all FOMODs, but to get any better would require extensive BAIN refactoring - which just so happens to be a 309 goal. ----------------------------------------------------------------------- ### CI & Tests (#474, #508) After two of our betas needed point releases immediately afterwards to correct blatantly obvious errors, we decided it was time to seriously push for a CI service that will build and test each Wrye Bash commit. 40285cbab414e3b7fcbcffc33c0be816e68d13f1 added the first version of this using GitHub Actions. The test suite is still very limited (only cosaves and a few parts of bolt are extensively tested) and there are tons of open questions - for example, we're currently using a very hacky way to change bush.game to fake having restarted Wrye Bash with a different game selected. While this works fine for the few tests we have right now, it *will not* scale, especially not once we get to testing bosh and patcher (which are the ones we really *want* to test). Expanding on this will be another important goal in 308/309. ----------------------------------------------------------------------- ### Nehrim (#514) Support for the Steam release of Nehrim: At Fate's Edge was added in 1bdcd9df2cd9cee164645b74beec1f513aa24129. Unlike Enderal, Nehrim is not installed as a separate game. Instead, the launcher backs up your Oblivion installation and allows you to switch between Nehrim and Oblivion by simply swapping the two folders around. That means we instead check which game is currently installed in your Oblivion folder and launch Wrye Bash for that. That meant a lot of refactoring, including some ugly warts that will need profiles (#250) to fully resolve, but it also means we were able to drop our hacky half-baked support for the manual Nehrim version. One particular thing about this merge that is worth highlighting is that we merged it *instantly*. As soon as the branch was finished and somewhat tested, it landed on dev. The reasoning was twofold: 1. We wanted to merge a huge records refactoring branch afterwards, and did not want to put others through resolving conflicts for it all the time. 2. FO3, FNV and Enderal have taught us that letting games sit around on branches is a terrible idea, since they quickly accumulate subtle bugs. ----------------------------------------------------------------------- ### CBash Deprecation (#520) With all the refactoring that has happened in 307, CBash has been completely left behind in the dust. PBash can now merge all record types (#516), has functional OBME support (#515) and supports several tags and patchers that CBash does not (#461). Additionally, CBash is starting to become both a maintenance burden and a roadblock on the way to refactoring and the patchers and therefore the upgrade to Python 3. 209d884469581248d8ca97954bcb4d05c8ef0d61 officially deprecates CBash. In fact, the whole reason we are putting out the 307 release *right now* is so we can get on with removing CBash for good in 308 ;) ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Utumno, @Infernio, @Sharlikran, @GandaG, @lojack5, @nycz, @BeermotorWB, @leandor, @syntaxaire, @fireundubh, @Ortham, @warmfrost85, @Arthmoor, @D4id4los, @MacSplody, @saebel, @nephatrine, @nallar, @llde, @FelesNoctis, @DianaNites, @valda and many more that GitHub's contribution tracker doesn't list.
…nity members] 307 is a gigantic release that has been cooking for almost five years at this point. In fact, it is even bigger than 306: 305 -> 306: 48,748 additions and 44,482 deletions 306 -> 307: 172,916 additions and 111,125 deletions Where 306 was mostly about refactoring the codebase to save it from becoming inoperable, 307 is all about using the newly refactored codebase to design new features, support more games and, of course, continue the everlasting refactoring war. The following notes are an attempt to summarize every important thing that's happened in 307, grouped as topics in a semi-chronological order. Do note that this is mostly a futile effort due to the 2000+ commits and 280000+ lines worth of changes that make up 307's development (and the fact that I only joined the project two thirds of the way through 307 :P). It is also highly recommended to read this on GitHub, with gitk open in the background. That way you can click on issue numbers to open them, while pasting commit SHAs into gitk, where the individual commits that make up a merge can be seen (GitHub does not expose this at all). All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### The bosh Split (#201) The first goal of 307 was to split bosh into a package. After the patchers had been split out (#163) and basher turned into a package (#3), bosh was clearly the next big target. 1bdc253e362857dc8a2a484be60bbcccd7891d82 began the splitting, then it was continued by splitting out the 'messages' code backing the PM Archives tab in 906858fe14c730ba797711a855933d996d573e2f. As a small interlude, 5b14b499e17c48917a472ee9ed426b3c7afd8985 then removed the PM Archives tab entirely (#221) due to the maintenance burden it had turned into. 682d2134c5e9f69b057cc9fb6e69e13767f5e7de continued the bosh split by ripping out: - The face transfer code (used for moving NPC/player faces between saves and mods) into faces.py - The OMODs code (used for unpacking OMODs, an old, ugly mod format used by OBMM) into omods.py - And the BAIN converter code (used for BCFs, aka BAIN Conversion Files - small files containing instructions that can be used to tell BAIN how to repackage an archive into a format it can recognize) into converters.py 318e66a9ad3a096c316decc751bc76d6c3b5b858 and 75c48e654c21feccd52cd851f2b502fd7c41c45f moved some bosh contents into parsers.py and bass.py - most importantly, the very commonly used 'dirs' and 'settings' constants. ea242af573952328168759b9c9c9ff1c01ac681c moved various miscellaneous things mostly related to plugins and LOOT into a new mods_metadata.py file. Coinciding with the INI refactoring (see 'INIs' section below), 6d2a1fa7bcb15f3d5fe787b5475e904e79809253 moved the INI handling code into bosh/ini_files.py. Similarly, afb7058a8e0d8a4567d43ec8f6f8828cafc34b83 moved BAIN code into bosh/bain.py as part of BAIN refactoring (see 'BAIN' section below). A last few commits splitting out the saves and mergeability checking into their own files (bosh/_mergeability.py, bosh/_saves.py and bosh/save_headers.py) were made in 347c552c8bbd3506a5bacd6c678a92e1275c8e06 and aa82a7af6fbcc6ce24ac1a940d3c2d4f665d626b, closing the issue for now. bosh/__init__.py is still quite big (3427 lines), but splitting further is far from trivial. ----------------------------------------------------------------------- ### Fallout 4 (#251) On November 10 of 2015, Fallout 4 was released - and less than three weeks later a branch by @lojack5 adding basic support for it was merged in e85506af4fbd28c4617b46c9ef0832c6c6570ecb0. Unfortunately, not all game merges in 307 were this speedy ;) Some more improvements and fixes landed in 134d7c63839de3daa52108f7ce9cc02fcc928df8 by @Sharlikran. Fallout 4 support was barebones then and is still barebones now. The real followup work here will come in 308/309 (see #525 and #482). ----------------------------------------------------------------------- ### Dropped Support For Older Settings Files (#253) Various bits of backwards compatibility cruft had accumulated over the course of 306 (and even earlier) to keep pre-306 settings working. 307 dropped all this and instead shows an error message if pre-306 settings are loaded in 307, telling the user to resave them in 306 first so that they can be upgraded to the new format. All this was done in a single merge, 8ceef20f269aedb18dbffe17063fac1cb7ccc0f4. With how big 307 has become, it is probably no surprise that it too has by now managed to accumulate a metric ton of backwards compatibility hacks. Cleanup for those will follow in 308/309. ----------------------------------------------------------------------- ### env.py (#258) In the ongoing quest for native Linux support (#243), encapsulating OS-specific code (i.e. Windows-specific code) in its own module is an important step. This was done in 646df2ca063c2dff03cb2185e7e2969fdea065a5. We're still not really there yet and env.py is, for the most part, still a Windows-specific module. The end goal would be to have two separate backends from env.py, so that it imports from one of them based on the OS it is running on. ----------------------------------------------------------------------- ### BAIN (#219) 307 also marks the beginning of the grueling task that is refactoring BAIN. Still nowhere near done, but it has become manageable. Much more work to be done in 308/309. Some of the important targets were: refreshSizeCrcDate was a big method that handled scanning directories for their contents. The problem was that it was basically two methods in one and implemented things that only make sense in one specific directory, namely the Data folder (e.g. empty directory removal). 20e41b3cfc38c71e8baae6e34778b964ca809533 refactored refreshSizeCrcDate to prepare it for 5ecf4000f10383e3018b4a4febbf8a8f13164a9e, where it was split into two methods (_refresh_from_project_dir and _refresh_from_data_dir). refreshDataSizeCrc is one of the central parts of BAIN. It scans a package for its contents, applies skips, remaps documentation to the 'Docs' folder, queries and caches CRCs, etc. ea79bb4fdfe2318d84952ba63fec2fb9b2a7e33a and 5fef1e3924bce2edcf19ff41e469c2a846d851bb tackled this behemoth, reducing its size significantly. Various later commits like 54844fbe1ea4ddfd48128e61f4232439431aac45 and c48112237b9953effb74b4206707394f225f035b touched this one up even further, bringing its final size in 307 to 210 lines (down from 348 lines in 306). 5d435bb05b9e09340eef9a71a402ac018abcb573 and 5ebbf4e8e3646efa8019b9cbccb144be9bc0ead9 focused on centralizing and refactoring BAIN refreshes, most notably the irefresh method. b109a4d08f21e023a30d5c0bf4ed45def0ae26e1 made BAIN use the modInfos cache to avoid recalculating CRCs that we already calculated for the Mods tab, giving a minor speedup - but mostly the idea here is we want to read the Data folder once and then delegate the files to the various FileInfos based on name and/or extension. This is a first step in that direction (see also #353 and #265). b1711b1b52fe76f0463b47fab825058f0f3bc41e was an important merge addressing case insensitive string comparisons. BAIN makes heavy use of this to keep track of files in the Data folder, each package, external changes, etc. So a central dictionary that made case-insensitive comparisons of its string keys was necessary - which is exactly what was introduced in this merge, as bolt.CIstr and bolt.LowerDict. 545cad09d29cf4b17ed470caba974829f5876d7f refactored the conflict detection and reporting algorithms to, well, separate them in the first place. They sat in a single big method called getConflictReport which came in at 120 lines pre-refactoring. In addition, the fact that this method returned a single string made it impossible to improve the GUI for conflicts (a goal in 309) without parsing the string - the string we just constructed from in-memory objects. The new getConflictReport is 55 lines long, much more readable and makes use of a new find_conflicts method that can be used to retrieve the actual conflicts as in-memory objects to work on. ----------------------------------------------------------------------- ### basher Package Followups (#163) Began in 306, finished here. daafdd5e76746833afe4eba496aa4afac41ff439 dropped the ancient Tank class for good, curing the flickering on the Mods tab in the process (#179). More work done in b7ffca085feb4afe93e48cdafb68521fd9402899. a28fcfce1f59482d1e701504b738a8df26a9bfe4 is a joint merge by @Utumno and @DianaNites refactoring the mod export/import links, which were pasta-filled mess. More prerequisite work landed in 10703a9e2982ae28db60af72146de8d8dca9c324 The topic was finally laid to rest with fb37a8bf110b6953c07a61a470d05b73ab03eb17 and 3bf6006d842942214694547a3e4c2b2fccb71dee. Every tab now has neatly separated tab panels, UILists and details panels. The API could still be better (especially considering how many new tabs we have planned for 308+ - see e.g. #233, #456, #50, etc.), but it's a far cry from the situation in 305 and 306. ----------------------------------------------------------------------- ### Skyrim Patchers (#151) Another principal goal of 307 is getting Skyrim patcher support as close to Oblivion as possible. Some prerequisites were merged in 3a2d396b2a9cf2ecd2525006a4e9960f3b4c85ff and 1e18dad44b97bf80b115de0c7175d14df07f2c3f (mostly records). The first real patcher porting then happened in 9599368dee429f4b63f116d25945beae73d838db. ----------------------------------------------------------------------- ### Load Order (#295 and #309) Load order handling is a complex beast, to say the least. Not only is handling all the edge cases difficult, but there are three different methods that the games use for implementing load order (four if you count Morrowind, which we don't support - yet ;)). Wrye Bash originally used the BOSS API for managing load order. This was changed in 1bf84f3e3b246195b93d9715e2d1891decc47354 to instead use the dedicated libloadorder. Even libloadorder itself proved problematic however: - Adding support for new games was tough and quickly turned into 'adding more clauses into if-else chains'. - The API generally made no effort to keep actives order and load order together - which became untenable in Fallout 4, where the two are inextricably linked. - Wrye Bash needs to read all the plugins anyways (e.g. CRC, ESM flag, etc.) and keeps that information cached in bosh.modInfos. libloadorder was reading it all again which, on top of being inelegant, was thrown away performance - syscalls are *not* cheap! - It's a DLL - bad for git and won't work on Linux (#243) So, in one big merge, 66d7b4d695289f6dc29142f94a6004494e3306bd replaced libloadorder entirely with a new API written in pure Python. This enabled many new features such as locking the load order in all games (bc7e5de47f3ecf31d1290d436940a37a7b6eb0cf), automatic backups whenever we make a fix to the LO (38817bb5ad6e222c4fdbde270d10341b2371bdb5), etc. ----------------------------------------------------------------------- ### Patchers (#312 and #461) Porting patchers to newer games is going to become harder and harder unless we refactor them to make it easier. Patcher code wasn't *terrible* per se and was fairly isolated from the rest of the code (apart from its tight connection to records code - see 'Records refactoring' section below), but a lot of it was copy-pasted and hence difficult to understand and expand - plus bugs often had to be fixed in upwards of 10 places. Adding a new patcher easily required hundreds of lines of pasta. This is still an ongoing goal, with much more to come in 308. 30eda2dd5c987648a11fbe01b8ee1b6c56d7c1ac was an early merge, containing refactoring on other more or less related things. 122784f5cf4d737bbb5943e9bd395d2bfc53c43a then moved config handling (i.e. which patchers are active and which sources they are operating on) to basher. The idea is to have the patchers operating only on a list of sources - not only for elegance and simplicity, but also to make them testable (see the 'CI & Tests' section below). Heavyweight refactoring began in 66d7b4ef3f8959e180f0029874c73e40de2a5a52 and f8bfdc4c5894ca2832aa5aef6515d70f52df110f, which introduced the _SimpleImporter class to deduplicate a lot of copy-pasted implementation code. Keep your eyes on this class, it proved to be a good idea ;) eb110497c3ce7b9d72df5a6565a0c7efffbc9a28 and 604ebd31d2f20d26c06ab2d043f114f7e0abc26b went in a different direction, by using the work that had already been done on refactoring the patchers to port many of the Oblivion-specific patchers over to FO3, FNV and Skyrim, as well as add some entirely new patchers that had been commonly requested. Of course, this involved a bunch of refactoring too, mostly moving implicit constants from all over the patcher code into game/*/constants.py, which will make it much easier to port the patchers in the future (e.g. to Fallout 4, see #482). 10e680cbb347b203f935c042145a5fc308b5f4c8 returned back to good old-fashioned refactoring by decoupling the config/GUI side of the patchers entirely from the model side (i.e. the code that actually implements patcher behavior). Previously, patchers were linked together by importing the patcher implementations in the GUI and using them as mixins. This led to lots of weird, hard-to-debug code that crippled the IDE's ability to perform static analysis. For example, the implementation of the leveled list patcher would use if not self.remove_empty_sublists: return to skip the 'empty sublist removal' part of the patcher if the checkbox for this was not checked in the GUI. However, the IDE had no way of knowing that that variable actually existed, since it came from the GUI side of the code via a mixin. After many failed attempts to devise a base class for both the model and GUI side of patchers, @Utumno instead realized that a much cleaner design would be to have no mixins. Instead, each patcher's GUI panel now has a class variable called patcher_type, which it sets to the model class that gets imported from the `patcher` package. This allowed us to drop tons of boilerplate code and make the resulting code much cleaner and easier to understand, but most importantly it acted as a springboard for further refactoring. Most notably, 9697b64abc00beaed04e950af20c8116db15151a split our importers.py file into four files: _cbash_importers.py, _shared.py, mergers.py and preservers.py. The key insight here was that we can split our importers nicely into two types: preservers simply carry forward the last value(s) from a tagged mod, while mergers merge values from all tagged mods based on the tags those mods have applied. This resulted in several hundred lines of duplicate patcher code being chopped off due to us absorbing many preservers back into the base class. It will also make it much easier to drop the CBash patchers, since they are now in a separate file altogether (see the 'CBash Deprecation' section below for more information on this). One last merge worth mentioning here is 426db77ee71c939bb785b94eab7d4281f0f1fa26. It is a highly WIP attempt to tame the mess that is parsers.py by devising a proper base class. The savings so far do look promising, but CSV reading and writing are ugly warts that still stand in the way. Plus the base class might be too complicated - right now it has six different knobs that tweak its behavior, and it's not even clear if using it for the all parsers is feasible. Still, we had to merge since previous betas came out with the plugin export/import commands this merge ports to Skyrim. Much more will follow here in 308 - most notably an upcoming refactoring of tweaks that will make them *much* faster and drop ~1200 lines of duplicate code. ----------------------------------------------------------------------- ### INIs (#247 and #326) INI handling was spotty at best. Random unicode tracebacks kept showing up, the INI Edits tab was one of the last big performance hogs and the default INI tweaks being files in the Mopy folder led to confusion (at least one mod had a 'Mopy\INI Tweaks' structure and was supposed to be installed *into the Mopy folder*). The first step was df2fcc5f95bc7bc2613a14cb996671f3b82dd2db, a series of smaller refactorings and fixups to make the tab's code more manageable. 1d4c23a037f6845f5dfebaf1e5e005f8f121a63b began the work on performance by introducing the proof of concept for a cache, while 58c47d562f43773c5b017531176b8869138869bf attacked the refresh APIs used by the INI Edits tab and significantly reduced the number of syscalls it made. The default INI tweaks were finally dealt with in 656127c645070d35d31423e014b692621ff94015 by hardcoding them into Mopy/bash/game/*/default_tweaks.py. No more tampering with the tweaks, no more Mopy\INI Tweaks folder to confuse users, fewer loose files packaged, simplified INI refresh, better performance due to fewer syscalls... ef21c5bb0a4f23d737dde3b22af36dcaf4f9b58b contained some more work on centralizing the 'apply a tweak' logic and fixing a longstanding issue where INIs would have Unix line endings written out, even on Windows. The LowerDict introduced during BAIN refactoring (see 'BAIN' section above) also turned out to be very useful for INIs: a9112d6761e47b1fc41aca53df1add5bdd41ad79 used it to rewrite core parts of ini_files.py for performance and readability. bb6c8bd2e7ac22d1218cbad90e404b9739dba7bc reworked the handling of INI encodings based on a central principle: work with unicode and stripped newlines internally, encode/decode and add/remove newlines at IO boundaries. ----------------------------------------------------------------------- ### wxPython (#190, #15 and #488) A war that started before living memory and will continue until long after we're all gone - or will it? Actually, we're very close to winning this conflict for good! wxPython was all over the place in 305. 306 improved the situation significantly, but 307 puts even those efforts to shame: 305: 2260 usages (balt: 381, basher: 1586) 306: 1112 usages (balt: 418, basher: 494) 307: 207 usages (balt: 191, basher: 8) *8* direct usages in basher, down from 494! Let's see how we got there: - 114c83729aa50045cac4f381912007826d8048bf: Utumno vs wx. Utumno lost, of course, but wx usages did go down from 1075 to 923. Mostly accomplished by moving common code to balt. - c7f5f5085acf1ec6e55f0048c256f7a0ebc3f367: Down to 902, and some progress was made towards wxPython 3. - 69c7f9f679f48df8cf7562442e80c9129f10924a: wx.lib.iewin was an ugly beast that was binding Wrye Bash to the comtypes dependency. Unfortunately, dropping it required upgrading to wxPython 3 (see below), so this merge simply centralized the iewin import for a future removal. - 531679d37d6c50cfc030b9c462f44250bcfab7a0: A small merge containing backwards-compatible changes that brought us closer to wxPython 3. - 050391ca22d7c8451390cbff6fd150ab0b9bcabd: The upgrade to wxPython 3. Introduced several significant architectural achievements: - Dropping the comtypes dependency by rewriting our HTML rendering code to use WebView instead of wx.lib.iewin. - Removing bolt's locale-related behavior on import that made importing it dangerous - it's been encapsulated in a new top-level module, localize.py. - Rewriting a lot of the very early boot process - see also the 'Boot' section below. - f9e46eed670b3d2805b1570fc62daf9cca12ce79: The big one. An absolutely enormous joint merge by @Utumno, @Infernio and @nycz introducing a new package, gui, that truly encapsulates wxPython. nycz wrote the first version of the code back in 2017, most importantly the layouts code that encapsulates wxPython's sizers in a declarative API. We then devised an event handling framework that enabled us to hunt down a lot of *implicit* wxPython usages. These are much harder and nastier to track because they can't simply be regexed. Thankfully PEP8 will be able to help us here, since all of wxPython uses PascalCase for its methods, while we're using snake_case for all new code. The result is a reduction down to 218 direct wx usages outside of gui. - 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: After upgrading to wxPython 3, the next goal was upgrading to wxPython 4. This is also the first release of wxPython that supports Python 3, making this a significant step in the direction of py3 support (#460). - eb86a4cb35fc24b58288a6b3e917ed9350a02e23: In preparation for finalizing and merging the FOMOD support (see 'FOMODs' section below), a bit more de-wx'ing happened, mostly on radio buttons and the splash screen. ----------------------------------------------------------------------- ### BSAs (#339 and #338) Same story as libloadorder. We were using a binary, libbsa, to do it. This was thrown away performance (we already read and cached the BSAs in bosh.bsaInfos), had no Linux support, etc. Its replacement, bsa_files.py, was introduced in b199a7bd5eec69ec0852f86d6e3629784e6b90ce, then used to support strings files packed into BSAs in c4f12d56d1273f40096abe45b6803d9265b1e75e and finished in 8ce81bfc9b5dc4643988e54471fab3e9b6a1f72d. With BSA handling code now taking shape in the form of bsa_files.py, having a second class arbitrarily handling a few things with entirely different (and much uglier) code would be a bad idea - so d69f3e82c3afe4b6461b6ccce49a210bedd28dd6 dropped the ancient BsaFile. There were still some unimplemented parts of the BSA format: - TES3 format: Added in 4ed5bd8c4ed9a67f872f109c660cf0afeb6ddca5, also in preparation of the POC Morrowind support we have in 307 (see the 'Morrowind' section below). - Compressed BSAs: Added in a6e11c4601d788e0e25c17361d92eaa659e59768, including both lz4-compressed ones for SSE and zlib-compressed ones for all previous games. - FO4 DX10 format: Added in 903b6d11d1451855951cce608c0ce8fe6a23743f. Currently unused, since we only use BSA extraction for strings files, which the DX10 format can't contain (it may only contain textures, for which it is specifically built and optimized). - Writing: We can read and extract everything from Morrowind to FO76 now, but have no support for altering and writing out BSAs yet. This will be a goal in 308/309 (see note below). All this work on BSAs acts as a prerequisite for the BSAs tab we want to enable and expand in 308/309 (it already exists in the codebase, but is very unfinished at the moment) - see #233. ----------------------------------------------------------------------- ### FileInfo(s) (#336) At the heart of each tab sits a DataStore subclass. This provides the UIList (i.e. what you see on the left side of each tab) with the data it should show. Most tabs (all but People and Installers) then have TableFileInfos in the hierarchy, and all but INIInfos (which backs the INI Edits tab, unsurprisingly) then have FileInfos in its hierarchy. These FileInfos use FileInfo classes to represent the files that are going to be shown in the list. These APIs are not *bad*, but they're not *good* either. Refactoring this is an ongoing goal, with lots of work done in 307. b199a7bd5eec69ec0852f86d6e3629784e6b90ce devised a common API for representing a tracked file with caching called AFile and used it for the new BSA API (see 'BSAs' section above). Some further work on freezing the AFile API and making FileInfo use it happened in 11f6769f641e80552b6f2d6e3c25d49a85e66c63, e3064942119c504786cbe6befafd8a0aa38bec2e and a0ae08c42fdf47688870387a6f6296de0551bfc2. Cosaves got a lot of work done in 307, bringing them from pre-alpha at best to a solid beta API (see the 'Cosaves' section below). Screenshots also got reworked to use the FileInfo(s) APIs in e007a5308d509fcd5e123d566b2b3c24e228edaa and 9a317f62cc8f7f807581a0a5f5453637d5d8b5ff. ----------------------------------------------------------------------- ### Skyrim SE (#347) In a join merge by @Arthmoor, @Utumno and @Sharlikran, Wrye Bash got SSE support added: 76b8abd5437042dd5b1f1e4505a651211d5523e8. See the 'ESLs' section below for some of the following challenges with SSE support. Of course, patcher support was spotty at first too - we ported all Skyrim patchers to it in c076e9fe1f5a27746a11b91ba9e0e1b43b80a915. One more merge worth mentioning here is 064d5021e068260cc99225be29d8d651b0761c61, which sped up startup in all games, but most notably in SSE. Our reference setup we used for testing (with ~200 saves) went from 10s down to 2s. ----------------------------------------------------------------------- ### Boot (#373 and #390) The boot phase was nothing short of a mess. There was no clear guiding principle of what is initialized when, leading to hard to debug problems. Unexpected errors could take down Wrye Bash for good without any way to tell what the problem even was. Restoring settings wasn't working at all. This is still somewhat in flux just due to how complex the boot phase is, but it's definitely gotten better. 0e3ef608e2906afb3405b009c622c632caa21dab began the process by centralizing the wxPython import. In 6060a157be13372ff2a8e09c8de9878c16190f2a, @D4id4los rewrote core parts of the boot procedure to gracefully handle and show errors, even when not in debug mode - making fixing startup errors encountered by users much easier. Restoring settings was addressed in 17f2266525e5095f8c068804fe3cc3471c9bac1a. In short, when restoring settings, we would override the settings we just tried to restore due to our atexit hook firing immediately afterwards. Instead of hacking away at it with monkey patches, @Utumno carefully reworked the boot sequence to clearly lay out what gets initialized when, breaking barb's dependency on the rest of Wrye Bash (balt, bosh, bush, etc.) in the process and adding a new top-level module, initialization.py, to better encapsulate init procedures. The locale mess that bolt did on import was addressed during the wxPython 3 upgrade in 050391ca22d7c8451390cbff6fd150ab0b9bcabd. This also resulted in a much more well-documented early boot process, including setting up the BashBugDump and bolt.depring much earlier, allowing us to use it to consistently log during the entire boot phase. ----------------------------------------------------------------------- ### Cosaves (#437) xSE (i.e. the script extenders - OBSE, SKSE, etc.) create cosaves for each save you make. Wrye Bash originally only needed these for its master remapping feature to not break things, but over the course of 307 we've come to use them to display save masters with ESLs in them (since those saves store two separate lists, an accurate master list is only possible by looking at the PLGN chunk in the cosave). They present a unique challenge in that each cosave is attached to a regular save, and all operations on that save need to respect the cosave. That means renaming, deleting, backing up, etc. need to not just apply to the main save, but also to its cosave, if it has one. 0a300af01a85bb3535d3194203fd34cc0d3e9b29 introduced the initial API for this, bosh/cosaves.py. It was mostly just a collection of code from various parts of bosh (mostly _saves.py), and as such was difficult to understand, maintain and extend. 822c0bd16e5788d1811449810f34edfee16d77a1 then refactored it (the commit looks like a rewrite, but was actually a gigantic refactoring comprising 100+ commits that had to get squashed down to a single one in order to not break dev) for maintainability and to bring Pluggy (an ancient cosave format in Oblivion) support into the cosave hierarchy. It also added support for saves with ESL masters, as mentioned above. Finally, e4dc76995703f16ee97fb07d495b6db635a2c6a8 reworked our handling of cosaves to be much more robust ----------------------------------------------------------------------- ### ESLs (#382 and #429) ESLs are a new type of plugin file introduced with SSE and FO4. They present a unique challenge in that they can bypass the usual 256 plugins limit, and as such stress-test many central assumption in any tool that tries to support them. Our APIs stood well to the test however, with d507111773d459e41468ac835955fe88915e622e only having to make minimal changes to add initial support. Due to the limited understanding of ESLs at the time, we were very conservative in what we allowed users to do with them. There was no way to verify ESL flags, add and remove them, and the Bashed Patch excluded ESLs completely. That was fixed in 547565a32f8d1d4a2944984c8c29092834f4fff6, a join merge by @Sharlikran, @Utumno and @Infernio. With it, Wrye Bash gained the ability to add the ESL flag to ESL-capable mods, importing from ESLs into the Bashed Patch was reenabled and load order operations for ESL-flagged ESPs were fixed. Once again, we were quite conservative in implementing the ESL flagging in Wrye Bash. For all record types we had not decoded yet, we simply failed the verification and told people to use xEdit to check instead. c137405418360063b6d767e7179c31fa94936cbc changed that to use a generic method that does not rely on our record definitions, since the only thing we actually have to care about are the headers of all records in the file. The contents of those records do not matter. This made ESL-flagging both faster and completely accurate for all record types in both SSE and FO4. ----------------------------------------------------------------------- ### Game Handling (#358) Along the way, especially after adding initial ESL suport (see 'ESLs' section above), it became clear that Wrye Bash's game handling would become a big issue that needed addressing. Adding support for a new game involved dozens of edits all over the codebase due to fsName checks and copy-pasting and editing a big constants.template file. This cripped the IDE's static analysis, since it couldn't check that any given game constant existed, let alone had the right type. We were also importing way too much for each game (e.g. all the constants, the default tweaks, the vanilla files, etc.), when we should really just import them for the one game we're actually managing. Thrown away performance and memory, plus just plain inelegant. In a joint merge by @Utumno and @GandaG, 11fa0f6a71ca8420071e76357b89f5d3e221c904, the game constants were moved from module-level into classes, allowing them to be inherited. This got rid of tons of duplicate code (1229 insertions(+), 1824 deletions(-)) and made adding both a new game and a new game constant easier and less error-prone. Some more work to move game-specific constants out of random files and into the game/*/constants.py files they belong into happened in fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, along with some fixes in 06d6a6bdaa925f379ffc1f05d0fa5977057ac739 and 57d3a621a08f4852dc5d5cc36878db9286351579. 4050e60bd372494c46860c85fa15c1701abcc5ae devised a way for us to avoid importing the constants, default tweaks, etc. for every game, while ddda9393d9b08a132c4a346fbdd8cc84454bccbd and 0fecde47b73d3204735c28b37260b7af8a01f700 finally finished off the last few constants outside game/*/constants.py, closing this issue for the time being. ----------------------------------------------------------------------- ### Fallout 3 & New Vegas (#150 and #468) Not originally planned to be part of 307, but after it was accidentally included in Beta 3, we had to merge it: 0f06e4fd306684aafccd2764b47a66d2205d10fc @valda originally ported Wrye Bash to FO3/FNV as Wrye Flash. Efforts to backport the changes to WB had been dragging along since forever, so the main thing we learned from this merge was that leaving games to rot around in branches is a *terrible* idea. Better to have the WIP code on dev without explicitly providing support, as leaving it on a branch makes it accumulate subtle bugs from refactoring extremely quickly. Some work on synchronizing the FO3 and FNV versions happened in the form of 54b8e614acd0841460f550d33d23340824f06e31 - since FNV is a vastly more popular game when it comes to modding, many of the improvements that valda made to the NV version of Wrye Flash did not make it back to the FO3 version. With us being based on a single codebase, doing that is much easier - eventually culminating in the FNV constants being entirely deduplicated in 3f96076501d5c260af856dac1f6475c21aa53a6e, e68b7af1eb9a3eed96b72cda067be82d86e067c6 and dccd28c70ffb857ccc70265a5ca22ca718d8e442 so that they are based on the FO3 ones, meaning that adding e.g. a new patcher or bash tag to FO3 will automatically add it to FNV as well. We're almost at feature parity with valda's version now - only the race patcher is missing from our version. This will be addressed in 308/309. On the other hand, we support several patchers and tags that valda's version doesn't, on top of tons of other features and bugfixes (see, for example, the rest of this commit message ;)). ----------------------------------------------------------------------- ### Readmes (#432 and #464) 307 includes a significant reworking of the readmes, courtesy of @FelesNoctis in 493c76b38c760d31757657a5b9bcf70f196d1dcd. It was later followed up with more edits for maintainability and to update the screenshots: see 18969116f7547148bf7b66196f91accd1225b465 and TODO ----------------------------------------------------------------------- ### Wizards (#446, #445, #444, #436 and #189) Several issues related to wizards were fixed (see e.g. 169d8347c1e4f3a3f6d696d4a660f89987f6bffc, 1f71bf335b85276566c12db43b53097842e05981 and 30f698520be938cd3cb6aa950cf979bc5468edb6). We also refactored the code quite a bit and deprecated the old 'Espm' versions of keywords and functions in favor of new 'Plugin' versions (more intuitive, easier to spell and remember, and more accurate with the advent of ESLs) in 3426384083bd5d7c61d42730ed7fa9c5629bc2db. Finally, 9244f8536ff0e4b780e3190cc40a032771310f4c added a new wizard function that had been requested a long time ago, enabling wizards to alter their behavior based on a plugin's load order. There is still a lot to do on wizards. For a start, the format is not formally defined - and the parser that acts as a reference is quite buggy (e.g. `Note thisIsAString` will print out 'thisIsAString', because the parser gets confused about its token states and accidentally treats 'thisIsAString' as a string). See https://github.com/Infernio/wizparse for my POC attempt at defining a formal grammar based on ANTLR that other mod managers will also be able to use. This is low priority, but will be continued in 309. Additionally, the wizard GUI presents a significant challenge in de-wx'ing (see 'wxPython' section above) and has a lot of duplication with the new FOMOD GUI (see 'FOMODs' section below). ----------------------------------------------------------------------- ### Python 3 (#460) We officially started the process of porting Wrye Bash over to Python 3 in September 2019, seeing as Python 2 has reached its end of life. This has turned out to be nothing short of a giant can of worms. Wrye Bash makes heavy use of bytestrings, so simply letting 2to3 run over the codebase would be disastrous - we'd be fixing unicode/bytes tracebacks for the next few months and getting no actual work done. In addition, our policy of having no breaking commits on dev means that an eventual Python 3 port will have to be a single commit, which makes bisecting useless. So the result is that we need that py3 commit to be as small as possible. With that goal in mind, a lot of prerequisite work that brings us closer to a py3 port without breaking py2 has landed: - 660ecbb81f49d478bdc8e4a8905e328d1daf9dca: py3 has no 'ur' prefix for strings since the one in py2 wasn't actually a 'raw' prefix: >>> print(ur'\u03B3') γ So dropping this one from the codebase was necessary. This commit just dropped all usages in strings that didn't actually have backslashes or were autogenerated paths (i.e. vanilla_files). - d43ad244170e2110a6daca7d5febed4020550247: This commit by @syntaxaire finished off the 'ur' removal mentioned above. - 5a98eb4c025651f4e9366db2a7d488ec2068f1fc: cmp and __cmp__ do not exist in py3. For the most part, we just had to implement rich comparisons. - fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, cae844b9dde8af014b09a1cb24af2348d5620058 and 6db5b8b59e28bc46a9d42e966d31007e113c59e6: Changing old-style classes to new-style ones work fine, except when the class is used as a mixin with a new-style one that uses __slots__. That can lead to nasty layout conflicts, as seen in the first of these three commits. - 8e201c49bbc809da89b1bda1d269f4cb7619dfc0: Our codebase included an ancient version of chardet (1.0.1 from 2008) due to a single manual edit that was needed to make it avoid returning the EUC-TW encodings that Python doesn't support. We dropped it in favor of the PyPI version, and addressed the EUC-TW problem in 60d0c29dbae91c12c1f7825df9f4e8e243ca09d2. - d8d03ca39e1e9f85250fd014cabcc2a65945e5e7, 197b6a2de78acd723f9d747fd6751fd2c68cf944 and 659e5b696be5083b9bef0d39356acc30ab46b5a4: Long integers don't exist in py3 due to its int type having no max size. So we needed to drop all 'L' postfixes and usages of sys.maxint. - 664f1722a53c91794f192e343936dfd34b8e86a8: Fixes for various issues encountered during an experimental run of 2to3 by @lojack5. - 33eac7624971ecd22e1f65ff5e47bc71ca175dbc: Merge by @GandaG addressing various py3 issues like print, moved stdlib modules, old exception syntax and usage of local absolute imports. - 050391ca22d7c8451390cbff6fd150ab0b9bcabd and 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: We were stuck on wxPython 2.8 for a long time, but the first version of wxPython that actually has py3 support is wxPython 4. These two commits (as well as tons of prerequisite refactoring, see the 'wxPython' section above) cleared that blocker for good. The Python 3 port is one of the primary 308 goals, along with patcher refactoring (#312) and records refactoring (#480), on which it is blocked (due to the aforementioned heavy bytestrings usage, which those refactorings will help us isolate and encapsulate). ----------------------------------------------------------------------- ### Build Scripts (#415) An enormous productivity gain for developers came in the form of @GandaG's reworking of build scripts in a1b5bfaa40fdbe04549ba3775106ffdff471e62e and 5f31a2adf39a607db282f49bd43e66c992a1baad. The ancient package_for_release.py has been replaced with a sleek new build.py script that does everything you need to do to build Wrye Bash in a single invocation. On top of that, Ganda also dropped tons of weird legacy things the build scripts did, like using ResHacker.exe to set the Wrye Bash icon - more binaries gone <3 ----------------------------------------------------------------------- ### Enderal (#433) Support for the Steam release of Enderal: Forgotten Stories, a total conversion mod for Skyrim was added in c2d73965fba4d0a82bb95f7cbe13b7f5dbcc0155. This was a fairly simple game to suport since it is pretty much just a pre-modded version of Skyrim LE. Of course, the work on refactoring game handling is the reason why we had so few problems with this merge. Once again, we left this game too long on a branch, meaning it began to accumulate bugs. That necessitated a fixup commit almost immediately in 3afa217d987c8c4ab86341ed8a8ec906b099767a. For all future game merges, we resolved to merge more quickly, as long as adding support for the game doesn't break any other code. ----------------------------------------------------------------------- ### Records (#480) After all the above, there were a few spots left in the codebase that needed *heavy* refactoring: records, patchers, saves (*not* save headers, the Oblivion-specific save editing code) and BAIN. Since the records and patchers code are very closely intertwined, they need to be attacked in tandem (refactoring the patchers is sort of a 'top-down' approach, while refactoring the records is a 'bottom-up' approach to the same problem). See the 'Patchers' section above for more information on that refactoring. The whole shebang began in ecac15d01dc5c89463ad47aab74260abfbea4167 and 3a3e9c935f5c1a798211eb0eaed0a0dd76a9af24, which were mostly just random commits improving some record definitions. Heavyweight refactoring began in 134433fde71534fc09e357ad64c696194f51a8eb, which moved an awful lot of records code into brec by creating new tools for defining record definitions. The result is a massive reduction in code size: 6787 insertions(+), 9581 deletions(-) ..and a very nice situation where almost the entire *implementation* of PBash sits in brec, while the (almost) purely declarative definitions sit in game/*/records.py. Unfortunately, this bloated brec to 3000+ lines and made it much harder to tell which classes belonged together. This was addressed in 28c11cb934056790e2a07703a9a09b4a5de8aa48, a huge merge that split brec into a package, added OBME support to PBash, sped up plugin loading by using AOT construction of struct.Struct instances instead of struct.pack/unpack and implemented merging of all record types in Oblivion. That's right, PBash can now merge everything CBash can. See the 'CBash Deprecation' section below for more information. After reading both the 'Python 3' and 'Patchers' sections, you should already know what's coming here: much more in 308. There's already a large refactoring ('part 2.5' of #480) that just needs some testing before it can be merged, and @Utumno is working on a 'part 3' of #480 that will seriously turn some parts of the records code on its head. ----------------------------------------------------------------------- ### Usability and Accessibility Wrye Bash has a (not entirely undeserved ;)) reputation for being difficult for newcomers to get started with (in UX terms, we'd say its out-of-box experience is bad). While this is obviously a big goal that we're nowhere close to solving today (really, someone with actual UX experience would be needed on the team), 307 does include some work towards both this goal and the goal of making Wrye Bash accessible to everyone, regardless of disabilities. - 1bf488a10a4c9fedb2737e4b2eee86c484f7b93d: The ability to jump to a plugin's matching installer from the Mods tab has been added (#53). - 3d7b9816ec0e2b7d666a7bead0cd45db5347aed7 by @fireundubh added the ability to jump to the matching plugin when a master is double-clicked in a masterlist (#311). - 261a1029a78e2cae773386c4e41f496a05f018c0 and f4987d1d0db38e4379f6470f67c37b982192817f by @BeermotorWB and @MacSplody trimmed the jungle that was the package context menu on the Installers tab by moving the more rarely used commands into submenus. - f65112bea111157558f78f056b550a7f689752a3 added the ability to jump to a plugin from the Plugin Filter on the Installers tab. - 92691409567ce020c6958325ee8c4bd8c9e820cc made the 'Sort By', 'Columns' and 'File' submenus on each tab consistent by putting them in the same places (they were in seemingly random positions on the context menus before). - 206ffbd9b09a0b8107ac5dacd9b0b3f9c2d1bfc2 by @warmfrost85 allowed users to get a preview of what Clean Data and Sync From Data are going to do, as well as the ability to use that preview to change which files the commands will affect. - e5685f3e87abd0bce199fe619509a9648ce2db89, also by @warmfrost85, allowed Sync From Data to work with archives. Previously if you wanted to, say, clean a plugin and sync it back into its archive, you would have to either do it manually in 7zip or use the workaround of unpacking the archive to a project, then using Sync From Data and finally packing the project back into an archive. Now you can just do it all in one go by using Sync From Data directly on the archive. - b427bbd383688a2f0fd55266b040cdc6ddf7c590 and a2297331b6571b6989bf1ea3b1c253a85c834448 increased the contrast on all our checkbox images to pass WCAG AAA guidelines, to make it much easier for people with weak eyesight to use Wrye Bash. In the future, we want to allow people to customize the colors of the checkboxes so that colorblind people can make use of them too (see #511). - edbe5e51d294ed0706478a9f8896d1e170d76058 added a menubar to improve discoverability of Wrye Bash's column context menus, as well as making it possible to access them purely using a keyboard. Previously you would have had to right click one of the columns to access these commands and options, which is a bit arcane and doesn't work for people who can't use a mouse at all. The same merge also reworked our half-baked settings menu that was implemented as a list of popup options when the tiny gear icon is clicked to instead be a proper settings dialog. Not only is this much easier to navigate, it will scale far better for our future needs (e.g. extensions). ----------------------------------------------------------------------- ### Morrowind (#479) Morrowind is not officially supported in 307. We've added some (very) WIP code in 45ed499de6b5564756867ce4b586ed8f4acb6c1f, enough to install mods and manage load order, but nowhere close to the featureset that Wrye Mash sports. The main purpose of this code is to act as a sort of regression test, making sure that we won't introduce anything breaking Morrowind support in the future (e.g. during a refactoring). However, adding full Morrowind support on par with Wrye Mash won't be a goal for quite a while (309+ at least). There's a lot of research and refactoring to get into before we can approach that. Thankfully, Elminster has began adding Morrowind support to xEdit as well, so we may soon have all the tools and documentation we need to write a modern version of Wrye Mash's features. ----------------------------------------------------------------------- ### Skyrim VR & Fallout 4 VR (#401 and #454) The situation here is the same as with Morrowind, but for different reasons. None of us developers own VR hardware, so we have no way to actually test if Wrye Bash isn't breaking everything on these games. Which is why there is no official support yet. Still, @nallar and @nephatrine contributed some initial code in 8273f37e431dfebcfbf114d2fee8cf8365dec889 and b2418eb47007a110fb3e255190d79304837a85f1 that seems to be working fine, judging by the fact that we're apparently included on at least one Skyrim VR guide already. And again, including this semi-supported game in our codebase means we are much less likely to break it in the future (not to mention that each game we add generally leads us down a rabbit hole of refactoring to make Wrye Bash more flexible and game-agnostic, which is definitely a good thing - one of our long-term goals is adding support for games outside the 'Bethesda sphere'). ----------------------------------------------------------------------- ### FOMODs (#380) One of the most commonly requested features in Wrye Bash's entire history. FOMODs are a fairly terrible mod format due to their incredibly loose nature. You can place files pretty much anywhere in the package, and there is no specification. That means mod managers are free to implement whatever they want - as long as it vaguely works like NMM's installer did, it's an FOMOD installer. Still, this format has become very common in Skyrim and Fallout 4, so we should support it. Huge thanks to @GandaG for contributing the initial version in fa48b1d26bcaf3e9a49b96376df56e2c5b425146, including a full FOMOD parser, which was a showstopper before. While all other parts of FOMOD support (including the GUI, the command itself and BAIN integration) have been rewritten since then (see below), the parser is still pretty much just as Ganda left it. Due to BAIN being built around mods being, well, *structured*, it wasn't a great fit for FOMOD support at first. We eventually came up with a way to hide all the ugliness of FOMODs from BAIN and just present it with the clearly structured resulting list of files to install in a big merge (54844fbe1ea4ddfd48128e61f4232439431aac45) that also contained a bunch of GUI improvements. It turns out that BAIN wasn't *actually* that bad a fit for FOMODs after all: it already had machinery in place to map one path in an package to a different path in the Data folder, since that's necessary for its 'root heuristic', remapping of docs into the `Docs` folder and the rarely used plugin remapping feature of the Plugin Filter. By hooking into the right part of refreshDataSizeCrc, we were able to use this feature to map the ugly mess that an FOMOD can be to a final list of neat paths. BAIN can now work with this regular list of paths, and when it does need to deal with the real package it will look them back up in the mapping. Of course all the BAIN refactoring that went into 307 (see 'BAIN' section above) is responsible for making this possible :) We still don't recognize all FOMODs, but to get any better would require extensive BAIN refactoring - which just so happens to be a 309 goal. ----------------------------------------------------------------------- ### CI & Tests (#474, #508) After two of our betas needed point releases immediately afterwards to correct blatantly obvious errors, we decided it was time to seriously push for a CI service that will build and test each Wrye Bash commit. 40285cbab414e3b7fcbcffc33c0be816e68d13f1 added the first version of this using GitHub Actions. The test suite is still very limited (only cosaves and a few parts of bolt are extensively tested) and there are tons of open questions - for example, we're currently using a very hacky way to change bush.game to fake having restarted Wrye Bash with a different game selected. While this works fine for the few tests we have right now, it *will not* scale, especially not once we get to testing bosh and patcher (which are the ones we really *want* to test). Expanding on this will be another important goal in 308/309. ----------------------------------------------------------------------- ### Nehrim (#514) Support for the Steam release of Nehrim: At Fate's Edge was added in 1bdcd9df2cd9cee164645b74beec1f513aa24129. Unlike Enderal, Nehrim is not installed as a separate game. Instead, the launcher backs up your Oblivion installation and allows you to switch between Nehrim and Oblivion by simply swapping the two folders around. That means we instead check which game is currently installed in your Oblivion folder and launch Wrye Bash for that. That meant a lot of refactoring, including some ugly warts that will need profiles (#250) to fully resolve, but it also means we were able to drop our hacky half-baked support for the manual Nehrim version. One particular thing about this merge that is worth highlighting is that we merged it *instantly*. As soon as the branch was finished and somewhat tested, it landed on dev. The reasoning was twofold: 1. We wanted to merge a huge records refactoring branch afterwards, and did not want to put others through resolving conflicts for it all the time. 2. FO3, FNV and Enderal have taught us that letting games sit around on branches is a terrible idea, since they quickly accumulate subtle bugs. ----------------------------------------------------------------------- ### CBash Deprecation (#520) With all the refactoring that has happened in 307, CBash has been completely left behind in the dust. PBash can now merge all record types (#516), has functional OBME support (#515) and supports several tags and patchers that CBash does not (#461). Additionally, CBash is starting to become both a maintenance burden and a roadblock on the way to refactoring and the patchers and therefore the upgrade to Python 3. 209d884469581248d8ca97954bcb4d05c8ef0d61 officially deprecates CBash. In fact, the whole reason we are putting out the 307 release *right now* is so we can get on with removing CBash for good in 308 ;) ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Utumno, @Infernio, @Sharlikran, @GandaG, @lojack5, @nycz, @BeermotorWB, @leandor, @syntaxaire, @fireundubh, @Ortham, @warmfrost85, @Arthmoor, @D4id4los, @MacSplody, @saebel, @nephatrine, @nallar, @llde, @FelesNoctis, @DianaNites, @valda and many more that GitHub's contribution tracker doesn't list.
This fixes the focus bug, at the cost of making the progress dialog uglier (i.e. not native). Mostly doing this because the native dialog seems to have gotten even worse on wx4.1.
@Utumno, @lojack5, @D4id4los: I was just on the wxPython site, and I noticed that they don't have their 2.8 releases listed any more. While the binaries are still available on Sourceforge and through the Wrye Python packages, this might be a good indication that we should move to wxPython 3.0. Is this a goal?
The changelog looks fairly detailed, and so it probably wouldn't be too much work. I'm not familiar with wxPython, but I thought wxWidgets 3.0 was a massive improvement over wxWidgets 2.8 (which I plain refused to use).
While IMHO the ideal goal would be to move to Python 3.x and wxPython's Project Phoenix, AFAIK the latter isn't ready for stable use, but Project Phoenix will still be based on wxWidgets 3.0, so should we ever move to it, this would still be a good first step.
The text was updated successfully, but these errors were encountered: