Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BAIN Wizards: General Improvements #446

Closed
11 tasks done
Infernio opened this issue Jul 23, 2019 · 11 comments
Closed
11 tasks done

BAIN Wizards: General Improvements #446

Infernio opened this issue Jul 23, 2019 · 11 comments
Assignees
Labels
A-docs Area: Documentation (Everything in the Docs folder) A-wizards Area: Wizards (belt.py and ScriptParser.py) C-enhancement Category: Enhancement, a request to add or enhance a feature M-relnotes Misc: Issue should be listed in the version history for its milestone
Milestone

Comments

@Infernio
Copy link
Member

Infernio commented Jul 23, 2019

While looking through belt.py and the BAIN Wizard documentation for investigating issues 444 and 445, I noticed that two functions were missing documentation. Opening this issue as a TODO list for me as I go through and fix all the problems I found (and so I can nicely refer to this issue in commit messages).

Constants

  • The documentation lies when it says that wizards only define two constants, True and False, they also define a constant called SubPackages. It's even used in many examples further down the page.

Functions

  • DisableINILine - disables a setting by commenting out the line in the INI file. These tweaks don't seem to work in WB right now though.
  • EndExec - internal function, marks the ending of an Exec function's contents. Calling EndExec manually is an error (should still have documentation to this effect though).

Operators

  • Did you know that wizards support incrementing or decrementing variables? I didn't either, because the documentation never mentions it.
    • Turns out that the increment / decrement operators were supposed to act as postfix ones, but someone made a mistake and they actually act as prefix operators. I'll just change the specification to keep backwards compatibility.
  • The operators need some better documentation. When I see ^= I assume bitwise negation, but looking at the source code, it's actually exponentiation.
  • The in operator needs an actual explanation of what it does (with an example).
  • The dot operator basically assumes that you're familiar with OO to understand it. An example would help (starting to notice a pattern here? 😛 ).
  • An implementation of %= and % for modulo exists, but is commented out. Might be worth reviving?

Random Notes

  • Everything mentioning Espm should probably be deprecated in favor of new keywords / functions with Plugin instead, since .esl files are a thing now - and there may be other types in the future (obviously we'll keep the old versions around indefinitely, similar to how CompareOblivionVersion was handled, since an absolute ton of wizards would break otherwise).
  • The string[start:stop:step] syntax in Python isn't called indexing, it's called slicing. Indexing is only when you use a single number in the brackets (e.g. string[target]).
@Infernio Infernio added this to the 307 milestone Jul 23, 2019
@Infernio Infernio self-assigned this Jul 23, 2019
@Infernio Infernio added the C-enhancement Category: Enhancement, a request to add or enhance a feature label Jul 23, 2019
@Utumno
Copy link
Member

Utumno commented Jul 23, 2019

DisableINILine - seems to set a setting in an INI file to an empty string?

should probably just prepend a ;-.

Historically @lojack5 said that this (or the Scriptparser) was the first python he wrote so there are many not-so-pythonic idioms. That being said if you really want to dig in there that's ok. I am busy lately but if I do find some time would like to look into the wx3/p3 switch as p2 is going out of service soon.

@Infernio
Copy link
Member Author

@Utumno That seems to be exactly what it does. However, it looks like WB can't actually apply those tweaks :P
Try executing this wizard, and then try applying the resulting tweaks:

; test BAIN Wizard Installation Script
DisableINILine("Skyrim.ini", "General", "iPresentInterval")
DisableINILine("Skyrim.ini", "General", "iMadeUpSetting")

@Infernio
Copy link
Member Author

So, fun fact, BAIN wizards are Turing-complete.
Here's an emulator that doubles as a proof, because why the hell not: https://github.com/Infernio/bainwiz-turing

@Utumno
Copy link
Member

Utumno commented Jul 24, 2019

Try executing this wizard, and then try applying the resulting tweaks:

Away from windoz machine - so the theory is that there is a "deleted line" concept in wrye bash ini handling - that's basically half-broken, I am sure. Lo was the wizard/ini guy, but the ini part has been rewritten since. I never tested "deleted" though - let me see... start from here:

reDeletedSetting = re.compile(ur';-\s*(\w.*?)\s*(;.*$|=.*$|$)',re.U)

Avoid renames till the very end while refactoring, does help reviewing - the ini code uses mixed style, PEP8 one is mine, ci prefix stands for case insensitive, AFile api is nearing beta.

@Infernio
Copy link
Member Author

Documentation updates are done on inf-wiz-docs now. I'm splitting any refactoring etc. into a different branch since the documentation changes are harmless (i.e. they can't possibly break dev).

@Utumno
Copy link
Member

Utumno commented Jul 28, 2019 via email

@Utumno
Copy link
Member

Utumno commented Jul 29, 2019

Had a very brief look in your branch - great work. Some random points:

  • plugin is ok but better not float around outside of an element of other keywords, as it used to be a bane (not sure if anymore though).
  • one of the most annoying things with belt is that is imported in bash/bash.py as it sets a codebox static to WryeParser. Well if don't deprecate all this maybe it's time to move this out of there, it's super ugly (plus files doing stuff on import is always a smell). Be warned that I kind of tried but it was far from simple IIRC.
  • maybe (up to you) not add the newline between one liner functions? Really appreciated you zapped the ones inside the code :)

@alphaniner
Copy link

That seems to be exactly what it does. However, it looks like WB can't actually apply those tweaks :P

And if used to disable lines of an OBSE-type .ini without also editing the .ini, get_ini_type_and_decoding (in bosh/ini_files.py) raises BoltError because the resulting tweak contains only comments (which are effectively ignored by the function).

@Utumno
Copy link
Member

Utumno commented Jul 30, 2019

@alphaniner can you python? Maybe hash up some unit tests? :P

@alphaniner
Copy link

I would if I could. I know enough python to troubleshoot some bugs. I have only a basic understanding of unit testing and no idea how to apply it to a project like this.

@alphaniner
Copy link

Back on topic, it would be nice if the example version string for CompareWBVersion were appropriate for WB instead of SE.

If possible for CompareGEVersion too, but that's probably not practical if there are GEs for multiple games (I only know of OBGE).

Infernio added a commit that referenced this issue Jul 31, 2019
Documents two previously undocumented functions, one constant and two
operators. On top of that, it also adds or improves many examples and
rewrites a few parts of the documentation significantly. Under #446.

Ugly, squashed notes follow:

Minor improvements to EditINI docs

Added types, some formatting

Added docs for DisableINILine

Add docs for EndExec

Can't be manually called, but should still have documentation in case
someone stumbles over it in error messages / code / logs / whatever one
day.

Mathematical operators: rewrite section

Adds explanations and actual examples. Especially necessary since two of
the operators (+ and *) are overloaded for strings, but no documentation
existed of what e.g. 2 * "foo" or "foo" + "bar" did.

Compound assignment: Improve docs

Actually mentions the sementaics of each of them now.

Swap order of Mathematical operators and Assignment operators

The compound assignment operators should probably only be introduced
after the mathematical ones, since that will make understanding them
much easier for someone who is new to programming.

Add docs for SubPackages constant

BAIN actually defines three constants: True, False and SubPackages.

Add docs for increment and decrement operators

Improve example for 'in' operator

Add example for dot operator

Improve indexing section

Split it into indexing and slicing, since they're separate terms.
Slicing is an extension of indexing, but we should still mention them
separately.

Show actual version examples for GE and WB
Infernio added a commit that referenced this issue Jul 31, 2019
Documents two previously undocumented functions, one constant and two
operators. On top of that, it also adds or improves many examples and
rewrites a few parts of the documentation significantly. Under #446.

Ugly, squashed notes follow:

Minor improvements to EditINI docs

Added types, some formatting

Added docs for DisableINILine

Add docs for EndExec

Can't be manually called, but should still have documentation in case
someone stumbles over it in error messages / code / logs / whatever one
day.

Mathematical operators: rewrite section

Adds explanations and actual examples. Especially necessary since two of
the operators (+ and *) are overloaded for strings, but no documentation
existed of what e.g. 2 * "foo" or "foo" + "bar" did.

Compound assignment: Improve docs

Actually mentions the sementaics of each of them now.

Swap order of Mathematical operators and Assignment operators

The compound assignment operators should probably only be introduced
after the mathematical ones, since that will make understanding them
much easier for someone who is new to programming.

Add docs for SubPackages constant

BAIN actually defines three constants: True, False and SubPackages.

Add docs for increment and decrement operators

Improve example for 'in' operator

Add example for dot operator

Improve indexing section

Split it into indexing and slicing, since they're separate terms.
Slicing is an extension of indexing, but we should still mention them
separately.

Show actual version examples for GE and WB
Infernio added a commit that referenced this issue Jul 31, 2019
Squashed version of the following commits:
1. Unify EditINI and DisableINILine implementations
2. Disallow all manual EndExec calls
3. Make inc/dec operator implementations more obvious
4. belt: Fix case-insensitive 'in' operator not working

None of these have much impact and some (specifically the EndExec and
case insensitive in) commits are pretty important fixes.
Under #446.

Ugly, squashed notes follow below:

Unify EditINI and DisableINILine implementations

Pretty much the exact same code. Also simplified the implementation.
Would be a diff with a negative line count, but I added a docstring to
the combined implementation.

Disallow all manual EndExec calls

This is supposed to be an internal function. However, without this
commit, it is actually possible to call it like this:

  Exec("EndExec(1)\nAnythingHere\nReturn")

This doesn't really cause harm, but it's definitely not intended, since
all other EndExec calls are disallowed (note how the 'exploit' above has
to use Return to skip the autogenerated EndExec statement). Therefore,
let's make that strange case above illegal as well.

This technically changes behavior, but if anyone used this in a wizard,
I'd have to question their sanity.

Make inc/dec operator implementations more obvious

The previous code accidentally implemented pre-inc/dec operators
correctly instead of the post-inc/dec operators it wanted to implement.
Since we've now created a specification that calls for pre-operators,
let's make the implementation more obviously correct.

belt: Fix case-insensitive 'in' operator not working

Didn't actually return anything when both operands were strings.
Infernio added a commit that referenced this issue Sep 16, 2019
Mopy/bash/cint.py:
Drop CBash 0.5.0 support entirely

Still had a random try in the middle of cint, when there's absolutely no
point - we can't support CBash 0.5.0 anymore, since all of the functions
have been renamed to start with a cb_, so even the fallback functions
won't exist, leading to cryptic error messages.

Instead, we now check if the CBash DLL is too old with
env.get_file_version() and log a more informative error if that is the
case.

Was originally thought to be the solution for #457, but that issue
turned out to be unrelated. Still a worthwhile commit for dropping a
print and improving the error message.

Mopy/bash/bolt.py:
Mopy/bash/mods_metadta.py:
Fix WryeText display of new tab links

Previously, it kept the exclamation mark if no alternate title was
defined for the link. Also added a better name for the xEdit cleaning
link.

Mopy/bash/basher/dialogs.py:
Fix Import Face dialog crashing

Since this line is unchanged from 9e6bbd9,
I can only conclude that wxPython must have changed behavior somewhere
along the way without anyone noticing. The line is clearly incorrect
though, we were allocating a sizer with space for 6 elements and shoving
10 in.

Note that this is *not* due to wx3 - wx2.8 exhibits the same behavior.

scripts/mktaglist.py:
Regression fix for SSE taglist generation

The taglist dir for SkyrimSE was renamed from 'Skyrim Special Edition'
to 'SkyrimSE' in 3542194.

However, the mktaglist script was not updated to account for these
changes. Whoops.

Mopy/bash/bosh/cosaves.py:
cosaves: Fixup for pluggy cosaves

Missed this when I fixed the xSE ones not discarding their lightly
loaded chunks before loading the full cosave.

Threw in a minor formatting change right below it, wrapping those
comments properly.

Under #437

Mopy/bash/ScriptParser.py:
ScriptParser: Fix broken arg count errors

When redoing some screenshots, I tried to reproduce the following error message:

  An error occurred in the wizard script:
  Line 15:	    Else 'What am I doing here?'
  Error:	(Line 15, Column 33): Too many arguments to Keyword 'Else':  got 1, expected 0.

However, I only managed to create the following:

  An error occurred in the wizard script:
  Line 15:	    Else 'What am I doing here?'
  Error:	(Line 15, Column 33): Too many arguments to Keyword 'self.text':  got 1, expected 0.

Turns out that the error reporting here has been broken since
40e3faa back in 2011.
Also wrapped the code I touched.

Under #446
@Infernio Infernio changed the title BAIN Wizards: Documentation Improvements BAIN Wizards: General Improvements Sep 29, 2019
@Infernio Infernio added the A-wizards Area: Wizards (belt.py and ScriptParser.py) label Mar 1, 2020
@Infernio Infernio added the A-docs Area: Documentation (Everything in the Docs folder) label Jul 19, 2020
Infernio added a commit that referenced this issue Sep 16, 2020
307 is a gigantic release that has been cooking for almost five years
at this point. In fact, it is even bigger than 306:

 305 -> 306: 48,748 additions and 44,482 deletions
 306 -> 307: 172,916 additions and 111,125 deletions

Where 306 was mostly about refactoring the codebase to save it from
becoming inoperable, 307 is all about using the newly refactored
codebase to design new features, support more games and, of course,
continue the everlasting refactoring war.

The following notes are an attempt to summarize every important thing
that's happened in 307, grouped as topics in a semi-chronological
order. Do note that this is mostly a futile effort due to the 2000+
commits and 280000+ lines worth of changes that make up 307's
development (and the fact that I only joined the project two thirds of
the way through 307 :P).

It is also highly recommended to read this on GitHub, with gitk open
in the background. That way you can click on issue numbers to open
them, while pasting commit SHAs into gitk, where the individual
commits that make up a merge can be seen (GitHub does not expose
this at all).

All mentioned commits are authored by @Utumno or @Infernio unless
otherwise mentioned.

-----------------------------------------------------------------------

### The bosh Split (#201)

The first goal of 307 was to split bosh into a package. After the
patchers had been split out (#163) and basher turned into a
package (#3), bosh was clearly the next big target.
1bdc253e362857dc8a2a484be60bbcccd7891d82 began the splitting, then
it was continued by splitting out the 'messages' code backing the
PM Archives tab in 906858fe14c730ba797711a855933d996d573e2f.

As a small interlude, 5b14b499e17c48917a472ee9ed426b3c7afd8985 then
removed the PM Archives tab entirely (#221) due to the maintenance
burden it had turned into.

682d2134c5e9f69b057cc9fb6e69e13767f5e7de continued the bosh split by
ripping out:
 - The face transfer code (used for moving NPC/player faces between
   saves and mods) into faces.py
 - The OMODs code (used for unpacking OMODs, an old, ugly mod format
   used by OBMM) into omods.py
 - And the BAIN converter code (used for BCFs, aka BAIN Conversion
   Files - small files containing instructions that can be used to
   tell BAIN how to repackage an archive into a format it can
   recognize) into converters.py

318e66a9ad3a096c316decc751bc76d6c3b5b858 and
75c48e654c21feccd52cd851f2b502fd7c41c45f moved some bosh contents into
parsers.py and bass.py - most importantly, the very commonly used
'dirs' and 'settings' constants.

ea242af573952328168759b9c9c9ff1c01ac681c moved various miscellaneous
things mostly related to plugins and LOOT into a new mods_metadata.py
file. 

Coinciding with the INI refactoring (see 'INIs' section below),
6d2a1fa7bcb15f3d5fe787b5475e904e79809253 moved the INI handling code
into bosh/ini_files.py. Similarly,
afb7058a8e0d8a4567d43ec8f6f8828cafc34b83 moved BAIN code into
bosh/bain.py as part of BAIN refactoring (see 'BAIN' section below).

A last few commits splitting out the saves and mergeability checking
into their own files (bosh/_mergeability.py, bosh/_saves.py and
bosh/save_headers.py) were made in
347c552c8bbd3506a5bacd6c678a92e1275c8e06 and
aa82a7af6fbcc6ce24ac1a940d3c2d4f665d626b, closing the issue for now.
bosh/__init__.py is still quite big (3427 lines), but splitting further
is far from trivial.

-----------------------------------------------------------------------

### Fallout 4 (#251)

On November 10 of 2015, Fallout 4 was released - and less than three
weeks later a branch by @lojack5 adding basic support for it was merged
in e85506af4fbd28c4617b46c9ef0832c6c6570ecb0. Unfortunately, not all
game merges in 307 were this speedy ;)
Some more improvements and fixes landed in
134d7c63839de3daa52108f7ce9cc02fcc928df8 by @Sharlikran.

Fallout 4 support was barebones then and is still barebones now. The
real followup work here will come in 308/309 (see #525 and #482).

-----------------------------------------------------------------------

### Dropped Support For Older Settings Files (#253)

Various bits of backwards compatibility cruft had accumulated over the
course of 306 (and even earlier) to keep pre-306 settings working. 307
dropped all this and instead shows an error message if pre-306 settings
are loaded in 307, telling the user to resave them in 306 first so that
they can be upgraded to the new format. All this was done in a single
merge, 8ceef20f269aedb18dbffe17063fac1cb7ccc0f4.

With how big 307 has become, it is probably no surprise that it too has
by now managed to accumulate a metric ton of backwards compatibility
hacks. Cleanup for those will follow in 308/309.

-----------------------------------------------------------------------

### env.py (#258)

In the ongoing quest for native Linux support (#243), encapsulating
OS-specific code (i.e. Windows-specific code) in its own module is
an important step. This was done in
646df2ca063c2dff03cb2185e7e2969fdea065a5. We're still not really there
yet and env.py is, for the most part, still a Windows-specific module.
The end goal would be to have two separate backends from env.py, so
that it imports from one of them based on the OS it is running on.

-----------------------------------------------------------------------

### BAIN (#219)

307 also marks the beginning of the grueling task that is refactoring
BAIN. Still nowhere near done, but it has become manageable. Much more
work to be done in 308/309. Some of the important targets were:

refreshSizeCrcDate was a big method that handled scanning directories
for their contents. The problem was that it was basically two methods
in one and implemented things that only make sense in one specific
directory, namely the Data folder (e.g. empty directory removal).
20e41b3cfc38c71e8baae6e34778b964ca809533 refactored refreshSizeCrcDate
to prepare it for 5ecf4000f10383e3018b4a4febbf8a8f13164a9e, where it
was split into two methods (_refresh_from_project_dir and
_refresh_from_data_dir).

refreshDataSizeCrc is one of the central parts of BAIN. It scans a
package for its contents, applies skips, remaps documentation to the
'Docs' folder, queries and caches CRCs, etc.
ea79bb4fdfe2318d84952ba63fec2fb9b2a7e33a and
5fef1e3924bce2edcf19ff41e469c2a846d851bb tackled this behemoth,
reducing its size significantly. Various later commits like
54844fbe1ea4ddfd48128e61f4232439431aac45 and
c48112237b9953effb74b4206707394f225f035b touched this one up even
further, bringing its final size in 307 to 210 lines (down from 348
lines in 306).

5d435bb05b9e09340eef9a71a402ac018abcb573 and
5ebbf4e8e3646efa8019b9cbccb144be9bc0ead9 focused on centralizing and
refactoring BAIN refreshes, most notably the irefresh method.

b109a4d08f21e023a30d5c0bf4ed45def0ae26e1 made BAIN use the modInfos
cache to avoid recalculating CRCs that we already calculated for the
Mods tab, giving a minor speedup - but mostly the idea here is we want
to read the Data folder once and then delegate the files to the various
FileInfos based on name and/or extension. This is a first step in that
direction (see also #353 and #265).

b1711b1b52fe76f0463b47fab825058f0f3bc41e was an important merge
addressing case insensitive string comparisons. BAIN makes heavy use of
this to keep track of files in the Data folder, each package, external
changes, etc. So a central dictionary that made case-insensitive
comparisons of its string keys was necessary - which is exactly what
was introduced in this merge, as bolt.CIstr and bolt.LowerDict.

545cad09d29cf4b17ed470caba974829f5876d7f refactored the conflict
detection and reporting algorithms to, well, separate them in the first
place. They sat in a single big method called getConflictReport which
came in at 120 lines pre-refactoring. In addition, the fact that this
method returned a single string made it impossible to improve the GUI
for conflicts (a goal in 309) without parsing the string - the string
we just constructed from in-memory objects. The new getConflictReport
is 55 lines long, much more readable and makes use of a new
find_conflicts method that can be used to retrieve the actual conflicts
as in-memory objects to work on.

-----------------------------------------------------------------------

### basher Package Followups (#163)

Began in 306, finished here. daafdd5e76746833afe4eba496aa4afac41ff439
dropped the ancient Tank class for good, curing the flickering on the
Mods tab in the process (#179).
More work done in b7ffca085feb4afe93e48cdafb68521fd9402899.

a28fcfce1f59482d1e701504b738a8df26a9bfe4 is a joint merge by @Utumno
and @DianaNites refactoring the mod export/import links, which were
pasta-filled mess. More prerequisite work landed in
10703a9e2982ae28db60af72146de8d8dca9c324

The topic was finally laid to rest with
fb37a8bf110b6953c07a61a470d05b73ab03eb17 and
3bf6006d842942214694547a3e4c2b2fccb71dee. Every tab now has neatly
separated tab panels, UILists and details panels. The API could still
be better (especially considering how many new tabs we have planned
for 308+ - see e.g. #233, #456, #50, etc.), but it's a far cry from
the situation in 305 and 306.

-----------------------------------------------------------------------

### Skyrim Patchers (#151)

Another principal goal of 307 is getting Skyrim patcher support as
close to Oblivion as possible.

Some prerequisites were merged in
3a2d396b2a9cf2ecd2525006a4e9960f3b4c85ff and
1e18dad44b97bf80b115de0c7175d14df07f2c3f (mostly records).
The first real patcher porting then happened in
9599368dee429f4b63f116d25945beae73d838db.

-----------------------------------------------------------------------

### Load Order (#295 and #309)

Load order handling is a complex beast, to say the least. Not only is
handling all the edge cases difficult, but there are three different
methods that the games use for implementing load order (four if you
count Morrowind, which we don't support - yet ;)).

Wrye Bash originally used the BOSS API for managing load order. This
was changed in 1bf84f3e3b246195b93d9715e2d1891decc47354 to instead use
the dedicated libloadorder. Even libloadorder itself proved problematic
however:

 - Adding support for new games was tough and quickly turned into
   'adding more clauses into if-else chains'.
 - The API generally made no effort to keep actives order and load
   order together - which became untenable in Fallout 4, where the two
   are inextricably linked.
 - Wrye Bash needs to read all the plugins anyways (e.g. CRC, ESM flag,
   etc.) and keeps that information cached in bosh.modInfos.
   libloadorder was reading it all again which, on top of being
   inelegant, was thrown away performance - syscalls are *not* cheap!
 - It's a DLL - bad for git and won't work on Linux (#243)

So, in one big merge, 66d7b4d695289f6dc29142f94a6004494e3306bd replaced
libloadorder entirely with a new API written in pure Python. This
enabled many new features such as locking the load order in all games
(bc7e5de47f3ecf31d1290d436940a37a7b6eb0cf), automatic backups whenever
we make a fix to the LO (38817bb5ad6e222c4fdbde270d10341b2371bdb5),
etc.

-----------------------------------------------------------------------

### Patchers (#312 and #461)

Porting patchers to newer games is going to become harder and harder
unless we refactor them to make it easier. Patcher code wasn't
*terrible* per se and was fairly isolated from the rest of the code
(apart from its tight connection to records code - see 'Records
refactoring' section below), but a lot of it was copy-pasted and hence
difficult to understand and expand - plus bugs often had to be fixed in
upwards of 10 places. Adding a new patcher easily required hundreds of
lines of pasta. This is still an ongoing goal, with much more to come
in 308.

30eda2dd5c987648a11fbe01b8ee1b6c56d7c1ac was an early merge, containing
refactoring on other more or less related things.
122784f5cf4d737bbb5943e9bd395d2bfc53c43a then moved config handling
(i.e. which patchers are active and which sources they are operating
on) to basher. The idea is to have the patchers operating only on a
list of sources - not only for elegance and simplicity, but also to
make them testable (see the 'CI & Tests' section below).

Heavyweight refactoring began in
66d7b4ef3f8959e180f0029874c73e40de2a5a52 and
f8bfdc4c5894ca2832aa5aef6515d70f52df110f, which introduced the
_SimpleImporter class to deduplicate a lot of copy-pasted
implementation code. Keep your eyes on this class, it proved to be a
good idea ;)

eb110497c3ce7b9d72df5a6565a0c7efffbc9a28 and
604ebd31d2f20d26c06ab2d043f114f7e0abc26b went in a different direction,
by using the work that had already been done on refactoring the
patchers to port many of the Oblivion-specific patchers over to FO3,
FNV and Skyrim, as well as add some entirely new patchers that had
been commonly requested. Of course, this involved a bunch of
refactoring too, mostly moving implicit constants from all over the
patcher code into game/*/constants.py, which will make it much easier
to port the patchers in the future (e.g. to Fallout 4, see #482).

10e680cbb347b203f935c042145a5fc308b5f4c8 returned back to good
old-fashioned refactoring by decoupling the config/GUI side of the
patchers entirely from the model side (i.e. the code that actually
implements patcher behavior). Previously, patchers were linked
together by importing the patcher implementations in the GUI and using
them as mixins. This led to lots of weird, hard-to-debug code that
crippled the IDE's ability to perform static analysis. For example, the
implementation of the leveled list patcher would use

  if not self.remove_empty_sublists: return

to skip the 'empty sublist removal' part of the patcher if the checkbox
for this was not checked in the GUI. However, the IDE had no way of
knowing that that variable actually existed, since it came from the GUI
side of the code via a mixin. After many failed attempts to devise a
base class for both the model and GUI side of patchers, @Utumno instead
realized that a much cleaner design would be to have no mixins.
Instead, each patcher's GUI panel now has a class variable called
patcher_type, which it sets to the model class that gets imported from
the `patcher` package. This allowed us to drop tons of boilerplate code
and make the resulting code much cleaner and easier to understand, but
most importantly it acted as a springboard for further refactoring.

Most notably, 9697b64abc00beaed04e950af20c8116db15151a split our
importers.py file into four files: _cbash_importers.py, _shared.py,
mergers.py and preservers.py. The key insight here was that we can
split our importers nicely into two types: preservers simply carry
forward the last value(s) from a tagged mod, while mergers merge values
from all tagged mods based on the tags those mods have applied.

This resulted in several hundred lines of duplicate patcher code being
chopped off due to us absorbing many preservers back into the base
class. It will also make it much easier to drop the CBash patchers,
since they are now in a separate file altogether (see the 'CBash
Deprecation' section below for more information on this).

One last merge worth mentioning here is
426db77ee71c939bb785b94eab7d4281f0f1fa26. It is a highly WIP attempt to
tame the mess that is parsers.py by devising a proper base class. The
savings so far do look promising, but CSV reading and writing are ugly
warts that still stand in the way. Plus the base class might be too
complicated - right now it has six different knobs that tweak its
behavior, and it's not even clear if using it for the all parsers is
feasible. Still, we had to merge since previous betas came out with
the plugin export/import commands this merge ports to Skyrim.

Much more will follow here in 308 - most notably an upcoming
refactoring of tweaks that will make them *much* faster and drop ~1200
lines of duplicate code.

-----------------------------------------------------------------------

### INIs (#247 and #326)

INI handling was spotty at best. Random unicode tracebacks kept showing
up, the INI Edits tab was one of the last big performance hogs and the
default INI tweaks being files in the Mopy folder led to confusion (at
least one mod had a 'Mopy\INI Tweaks' structure and was supposed to be
installed *into the Mopy folder*).

The first step was df2fcc5f95bc7bc2613a14cb996671f3b82dd2db, a series
of smaller refactorings and fixups to make the tab's code more
manageable.

1d4c23a037f6845f5dfebaf1e5e005f8f121a63b began the work on performance
by introducing the proof of concept for a cache, while
58c47d562f43773c5b017531176b8869138869bf attacked the refresh APIs used
by the INI Edits tab and significantly reduced the number of syscalls
it made.

The default INI tweaks were finally dealt with in
656127c645070d35d31423e014b692621ff94015 by hardcoding them into
Mopy/bash/game/*/default_tweaks.py. No more tampering with the tweaks,
no more Mopy\INI Tweaks folder to confuse users, fewer loose files
packaged, simplified INI refresh, better performance due to fewer
syscalls...

ef21c5bb0a4f23d737dde3b22af36dcaf4f9b58b contained some more work on
centralizing the 'apply a tweak' logic and fixing a longstanding issue
where INIs would have Unix line endings written out, even on Windows.

The LowerDict introduced during BAIN refactoring (see 'BAIN' section
above) also turned out to be very useful for INIs:
a9112d6761e47b1fc41aca53df1add5bdd41ad79 used it to rewrite core parts
of ini_files.py for performance and readability.

bb6c8bd2e7ac22d1218cbad90e404b9739dba7bc reworked the handling of INI
encodings based on a central principle: work with unicode and stripped
newlines internally, encode/decode and add/remove newlines at IO
boundaries.

-----------------------------------------------------------------------

### wxPython (#190, #15 and #488)

A war that started before living memory and will continue until long
after we're all gone - or will it? Actually, we're very close to
winning this conflict for good!

wxPython was all over the place in 305. 306 improved the situation
significantly, but 307 puts even those efforts to shame:

 305: 2260 usages (balt: 381, basher: 1586)
 306: 1112 usages (balt: 418, basher: 494)
 307: 207  usages (balt: 191, basher: 8)

*8* direct usages in basher, down from 494! Let's see how we got there:

 - 114c83729aa50045cac4f381912007826d8048bf: Utumno vs wx. Utumno lost,
   of course, but wx usages did go down from 1075 to 923. Mostly
   accomplished by moving common code to balt.
 - c7f5f5085acf1ec6e55f0048c256f7a0ebc3f367: Down to 902, and some
   progress was made towards wxPython 3.
 - 69c7f9f679f48df8cf7562442e80c9129f10924a: wx.lib.iewin was an ugly
   beast that was binding Wrye Bash to the comtypes dependency.
   Unfortunately, dropping it required upgrading to wxPython 3 (see
   below), so this merge simply centralized the iewin import for a
   future removal.
 - 531679d37d6c50cfc030b9c462f44250bcfab7a0: A small merge containing
   backwards-compatible changes that brought us closer to wxPython 3.
 - 050391ca22d7c8451390cbff6fd150ab0b9bcabd: The upgrade to wxPython 3.
   Introduced several significant architectural achievements:
     - Dropping the comtypes dependency by rewriting our HTML rendering
       code to use WebView instead of wx.lib.iewin.
     - Removing bolt's locale-related behavior on import that made
       importing it dangerous - it's been encapsulated in a new
       top-level module, localize.py.
     - Rewriting a lot of the very early boot process - see also the
       'Boot' section below.
 - f9e46eed670b3d2805b1570fc62daf9cca12ce79: The big one. An absolutely
   enormous joint merge by @Utumno, @Infernio and @nycz introducing a
   new package, gui, that truly encapsulates wxPython. nycz wrote the
   first version of the code back in 2017, most importantly the layouts
   code that encapsulates wxPython's sizers in a declarative API. We
   then devised an event handling framework that enabled us to hunt
   down a lot of *implicit* wxPython usages. These are much harder and
   nastier to track because they can't simply be regexed. Thankfully
   PEP8 will be able to help us here, since all of wxPython uses
   PascalCase for its methods, while we're using snake_case for all new
   code. The result is a reduction down to 218 direct wx usages outside
   of gui.
 - 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: After upgrading to
   wxPython 3, the next goal was upgrading to wxPython 4. This is also
   the first release of wxPython that supports Python 3, making this a
   significant step in the direction of py3 support (#460).
 - eb86a4cb35fc24b58288a6b3e917ed9350a02e23: In preparation for
   finalizing and merging the FOMOD support (see 'FOMODs' section
   below), a bit more de-wx'ing happened, mostly on radio buttons and
   the splash screen.

-----------------------------------------------------------------------

### BSAs (#339 and #338)

Same story as libloadorder. We were using a binary, libbsa, to do it.
This was thrown away performance (we already read and cached the BSAs
in bosh.bsaInfos), had no Linux support, etc.

Its replacement, bsa_files.py, was introduced in
b199a7bd5eec69ec0852f86d6e3629784e6b90ce, then used to support strings
files packed into BSAs in c4f12d56d1273f40096abe45b6803d9265b1e75e and
finished in 8ce81bfc9b5dc4643988e54471fab3e9b6a1f72d.

With BSA handling code now taking shape in the form of bsa_files.py,
having a second class arbitrarily handling a few things with entirely
different (and much uglier) code would be a bad idea - so
d69f3e82c3afe4b6461b6ccce49a210bedd28dd6 dropped the ancient BsaFile.

There were still some unimplemented parts of the BSA format:
  - TES3 format: Added in 4ed5bd8c4ed9a67f872f109c660cf0afeb6ddca5,
    also in preparation of the POC Morrowind support we have in 307 (see
    the 'Morrowind' section below).
  - Compressed BSAs: Added in a6e11c4601d788e0e25c17361d92eaa659e59768,
    including both lz4-compressed ones for SSE and zlib-compressed ones
    for all previous games.
  - FO4 DX10 format: Added in 903b6d11d1451855951cce608c0ce8fe6a23743f.
    Currently unused, since we only use BSA extraction for strings
    files, which the DX10 format can't contain (it may only contain
    textures, for which it is specifically built and optimized).
  - Writing: We can read and extract everything from Morrowind to FO76
    now, but have no support for altering and writing out BSAs yet.
    This will be a goal in 308/309 (see note below).

All this work on BSAs acts as a prerequisite for the BSAs tab we want
to enable and expand in 308/309 (it already exists in the codebase, but
is very unfinished at the moment) - see #233.

-----------------------------------------------------------------------

### FileInfo(s) (#336)

At the heart of each tab sits a DataStore subclass. This provides the
UIList (i.e. what you see on the left side of each tab) with the data
it should show. Most tabs (all but People and Installers) then have
TableFileInfos in the hierarchy, and all but INIInfos (which backs the
INI Edits tab, unsurprisingly) then have FileInfos in its hierarchy.
These FileInfos use FileInfo classes to represent the files that are
going to be shown in the list.

These APIs are not *bad*, but they're not *good* either. Refactoring
this is an ongoing goal, with lots of work done in 307.

b199a7bd5eec69ec0852f86d6e3629784e6b90ce devised a common API for
representing a tracked file with caching called AFile and used it for
the new BSA API (see 'BSAs' section above). Some further work on
freezing the AFile API and making FileInfo use it happened in
11f6769f641e80552b6f2d6e3c25d49a85e66c63,
e3064942119c504786cbe6befafd8a0aa38bec2e and
a0ae08c42fdf47688870387a6f6296de0551bfc2.

Cosaves got a lot of work done in 307, bringing them from pre-alpha at
best to a solid beta API (see the 'Cosaves' section below). Screenshots
also got reworked to use the FileInfo(s) APIs in
e007a5308d509fcd5e123d566b2b3c24e228edaa and
9a317f62cc8f7f807581a0a5f5453637d5d8b5ff.

-----------------------------------------------------------------------

### Skyrim SE (#347)

In a join merge by @Arthmoor, @Utumno and @Sharlikran, Wrye Bash got
SSE support added: 76b8abd5437042dd5b1f1e4505a651211d5523e8.

See the 'ESLs' section below for some of the following challenges with
SSE support. Of course, patcher support was spotty at first too - we
ported all Skyrim patchers to it in
c076e9fe1f5a27746a11b91ba9e0e1b43b80a915.

One more merge worth mentioning here is
064d5021e068260cc99225be29d8d651b0761c61, which sped up startup in all
games, but most notably in SSE. Our reference setup we used for testing
(with ~200 saves) went from 10s down to 2s.

-----------------------------------------------------------------------

### Boot (#373 and #390)

The boot phase was nothing short of a mess. There was no clear guiding
principle of what is initialized when, leading to hard to debug
problems. Unexpected errors could take down Wrye Bash for good without
any way to tell what the problem even was. Restoring settings wasn't
working at all. This is still somewhat in flux just due to how complex
the boot phase is, but it's definitely gotten better.

0e3ef608e2906afb3405b009c622c632caa21dab began the process by
centralizing the wxPython import. In
6060a157be13372ff2a8e09c8de9878c16190f2a, @D4id4los rewrote core parts
of the boot procedure to gracefully handle and show errors, even when
not in debug mode - making fixing startup errors encountered by users
much easier.

Restoring settings was addressed in
17f2266525e5095f8c068804fe3cc3471c9bac1a. In short, when restoring
settings, we would override the settings we just tried to restore due
to our atexit hook firing immediately afterwards. Instead of hacking
away at it with monkey patches, @Utumno carefully reworked the boot
sequence to clearly lay out what gets initialized when, breaking barb's
dependency on the rest of Wrye Bash (balt, bosh, bush, etc.) in the
process and adding a new top-level module, initialization.py, to better
encapsulate init procedures.

The locale mess that bolt did on import was addressed during the
wxPython 3 upgrade in 050391ca22d7c8451390cbff6fd150ab0b9bcabd. This
also resulted in a much more well-documented early boot process,
including setting up the BashBugDump and bolt.depring much earlier,
allowing us to use it to consistently log during the entire boot phase.

-----------------------------------------------------------------------

### Cosaves (#437)

xSE (i.e. the script extenders - OBSE, SKSE, etc.) create cosaves for
each save you make. Wrye Bash originally only needed these for its
master remapping feature to not break things, but over the course of
307 we've come to use them to display save masters with ESLs in them
(since those saves store two separate lists, an accurate master list is
only possible by looking at the PLGN chunk in the cosave). They present
a unique challenge in that each cosave is attached to a regular save,
and all operations on that save need to respect the cosave. That means
renaming, deleting, backing up, etc. need to not just apply to the
main save, but also to its cosave, if it has one.

0a300af01a85bb3535d3194203fd34cc0d3e9b29 introduced the initial API for
this, bosh/cosaves.py. It was mostly just a collection of code from
various parts of bosh (mostly _saves.py), and as such was difficult to
understand, maintain and extend.

822c0bd16e5788d1811449810f34edfee16d77a1 then refactored it (the commit
looks like a rewrite, but was actually a gigantic refactoring
comprising 100+ commits that had to get squashed down to a single one
in order to not break dev) for maintainability and to bring Pluggy
(an ancient cosave format in Oblivion) support into the cosave
hierarchy. It also added support for saves with ESL masters, as
mentioned above.

Finally, e4dc76995703f16ee97fb07d495b6db635a2c6a8 reworked our handling
of cosaves to be much more robust 

-----------------------------------------------------------------------

### ESLs (#382 and #429)

ESLs are a new type of plugin file introduced with SSE and FO4. They
present a unique challenge in that they can bypass the usual 256
plugins limit, and as such stress-test many central assumption in any
tool that tries to support them. Our APIs stood well to the test
however, with d507111773d459e41468ac835955fe88915e622e only having to
make minimal changes to add initial support.

Due to the limited understanding of ESLs at the time, we were very
conservative in what we allowed users to do with them. There was no way
to verify ESL flags, add and remove them, and the Bashed Patch excluded
ESLs completely. That was fixed in
547565a32f8d1d4a2944984c8c29092834f4fff6, a join merge by @Sharlikran,
@Utumno and @Infernio. With it, Wrye Bash gained the ability to add the
ESL flag to ESL-capable mods, importing from ESLs into the Bashed Patch
was reenabled and load order operations for ESL-flagged ESPs were
fixed.

Once again, we were quite conservative in implementing the ESL flagging
in Wrye Bash. For all record types we had not decoded yet, we simply
failed the verification and told people to use xEdit to check instead.
c137405418360063b6d767e7179c31fa94936cbc changed that to use a generic
method that does not rely on our record definitions, since the only
thing we actually have to care about are the headers of all records in
the file. The contents of those records do not matter. This made
ESL-flagging both faster and completely accurate for all record types
in both SSE and FO4.

-----------------------------------------------------------------------

### Game Handling (#358)

Along the way, especially after adding initial ESL suport (see 'ESLs'
section above), it became clear that Wrye Bash's game handling would
become a big issue that needed addressing. Adding support for a new
game involved dozens of edits all over the codebase due to fsName
checks and copy-pasting and editing a big constants.template file.
This cripped the IDE's static analysis, since it couldn't check that
any given game constant existed, let alone had the right type. We were
also importing way too much for each game (e.g. all the constants, the
default tweaks, the vanilla files, etc.), when we should really just
import them for the one game we're actually managing. Thrown away
performance and memory, plus just plain inelegant.

In a joint merge by @Utumno and @GandaG,
11fa0f6a71ca8420071e76357b89f5d3e221c904, the game constants were moved
from module-level into classes, allowing them to be inherited. This got
rid of tons of duplicate code (1229 insertions(+), 1824 deletions(-))
and made adding both a new game and a new game constant easier and less
error-prone.

Some more work to move game-specific constants out of random files and
into the game/*/constants.py files they belong into happened in
fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, along with some fixes in
06d6a6bdaa925f379ffc1f05d0fa5977057ac739 and
57d3a621a08f4852dc5d5cc36878db9286351579.

4050e60bd372494c46860c85fa15c1701abcc5ae devised a way for us to avoid
importing the constants, default tweaks, etc. for every game, while
ddda9393d9b08a132c4a346fbdd8cc84454bccbd and
0fecde47b73d3204735c28b37260b7af8a01f700 finally finished off the last
few constants outside game/*/constants.py, closing this issue for the
time being.

-----------------------------------------------------------------------

### Fallout 3 & New Vegas (#150 and #468)

Not originally planned to be part of 307, but after it was accidentally
included in Beta 3, we had to merge it:
0f06e4fd306684aafccd2764b47a66d2205d10fc

@valda originally ported Wrye Bash to FO3/FNV as Wrye Flash. Efforts to
backport the changes to WB had been dragging along since forever, so
the main thing we learned from this merge was that leaving games to rot
around in branches is a *terrible* idea. Better to have the WIP code on
dev without explicitly providing support, as leaving it on a branch
makes it accumulate subtle bugs from refactoring extremely quickly.

Some work on synchronizing the FO3 and FNV versions happened in the
form of 54b8e614acd0841460f550d33d23340824f06e31 - since FNV is a
vastly more popular game when it comes to modding, many of the
improvements that valda made to the NV version of Wrye Flash did not
make it back to the FO3 version. With us being based on a single
codebase, doing that is much easier - eventually culminating in the FNV
constants being entirely deduplicated in
3f96076501d5c260af856dac1f6475c21aa53a6e,
e68b7af1eb9a3eed96b72cda067be82d86e067c6 and
dccd28c70ffb857ccc70265a5ca22ca718d8e442 so that they are based on the
FO3 ones, meaning that adding e.g. a new patcher or bash tag to FO3
will automatically add it to FNV as well.

We're almost at feature parity with valda's version now - only the race
patcher is missing from our version. This will be addressed in 308/309.
On the other hand, we support several patchers and tags that valda's
version doesn't, on top of tons of other features and bugfixes (see,
for example, the rest of this commit message ;)).

-----------------------------------------------------------------------

### Readmes (#432 and #464)

307 includes a significant reworking of the readmes, courtesy of
@FelesNoctis in 493c76b38c760d31757657a5b9bcf70f196d1dcd. It was later
followed up with more edits for maintainability and to update the
screenshots: see 18969116f7547148bf7b66196f91accd1225b465 and TODO

-----------------------------------------------------------------------

### Wizards (#446, #445, #444, #436 and #189)

Several issues related to wizards were fixed (see e.g.
169d8347c1e4f3a3f6d696d4a660f89987f6bffc,
1f71bf335b85276566c12db43b53097842e05981 and
30f698520be938cd3cb6aa950cf979bc5468edb6).

We also refactored the code quite a bit and deprecated the old 'Espm'
versions of keywords and functions in favor of new 'Plugin' versions
(more intuitive, easier to spell and remember, and more accurate with
the advent of ESLs) in 3426384083bd5d7c61d42730ed7fa9c5629bc2db.
Finally, 9244f8536ff0e4b780e3190cc40a032771310f4c added a new wizard
function that had been requested a long time ago, enabling wizards to
alter their behavior based on a plugin's load order.

There is still a lot to do on wizards. For a start, the format is not
formally defined - and the parser that acts as a reference is quite
buggy (e.g. `Note thisIsAString` will print out 'thisIsAString',
because the parser gets confused about its token states and
accidentally treats 'thisIsAString' as a string). See
https://github.com/Infernio/wizparse for my POC attempt at defining a
formal grammar based on ANTLR that other mod managers will also be able
to use. This is low priority, but will be continued in 309.
Additionally, the wizard GUI presents a significant challenge in
de-wx'ing (see 'wxPython' section above) and has a lot of duplication
with the new FOMOD GUI (see 'FOMODs' section below).

-----------------------------------------------------------------------

### Python 3 (#460)

We officially started the process of porting Wrye Bash over to Python 3
in September 2019, seeing as Python 2 has reached its end of life. This
has turned out to be nothing short of a giant can of worms. Wrye Bash
makes heavy use of bytestrings, so simply letting 2to3 run over the
codebase would be disastrous - we'd be fixing unicode/bytes tracebacks
for the next few months and getting no actual work done. In addition,
our policy of having no breaking commits on dev means that an eventual
Python 3 port will have to be a single commit, which makes bisecting
useless. So the result is that we need that py3 commit to be as small
as possible. With that goal in mind, a lot of prerequisite work that
brings us closer to a py3 port without breaking py2 has landed:

 - 660ecbb81f49d478bdc8e4a8905e328d1daf9dca: py3 has no 'ur' prefix for
   strings since the one in py2 wasn't actually a 'raw' prefix:

     >>> print(ur'\u03B3')
     γ

   So dropping this one from the codebase was necessary. This commit
   just dropped all usages in strings that didn't actually have
   backslashes or were autogenerated paths (i.e. vanilla_files).
 - d43ad244170e2110a6daca7d5febed4020550247: This commit by @syntaxaire
   finished off the 'ur' removal mentioned above.
 - 5a98eb4c025651f4e9366db2a7d488ec2068f1fc: cmp and __cmp__ do not
   exist in py3. For the most part, we just had to implement rich
   comparisons.
 - fa74a1b7a61d9b3150f0d2b171145e171f2d27e5,
   cae844b9dde8af014b09a1cb24af2348d5620058 and
   6db5b8b59e28bc46a9d42e966d31007e113c59e6: Changing old-style classes
   to new-style ones work fine, except when the class is used as a
   mixin with a new-style one that uses __slots__. That can lead to
   nasty layout conflicts, as seen in the first of these three commits.
 - 8e201c49bbc809da89b1bda1d269f4cb7619dfc0: Our codebase included an
   ancient version of chardet (1.0.1 from 2008) due to a single manual
   edit that was needed to make it avoid returning the EUC-TW encodings
   that Python doesn't support. We dropped it in favor of the PyPI
   version, and addressed the EUC-TW problem in
   60d0c29dbae91c12c1f7825df9f4e8e243ca09d2.
 - d8d03ca39e1e9f85250fd014cabcc2a65945e5e7,
   197b6a2de78acd723f9d747fd6751fd2c68cf944 and
   659e5b696be5083b9bef0d39356acc30ab46b5a4: Long integers don't exist
   in py3 due to its int type having no max size. So we needed to drop
   all 'L' postfixes and usages of sys.maxint.
 - 664f1722a53c91794f192e343936dfd34b8e86a8: Fixes for various issues
   encountered during an experimental run of 2to3 by @lojack5.
 - 33eac7624971ecd22e1f65ff5e47bc71ca175dbc: Merge by @GandaG
   addressing various py3 issues like print, moved stdlib modules, old
   exception syntax and usage of local absolute imports.
 - 050391ca22d7c8451390cbff6fd150ab0b9bcabd and
   22de7ff9e804b2bcaa8a819922f4b5100b86d80f: We were stuck on wxPython
   2.8 for a long time, but the first version of wxPython that actually
   has py3 support is wxPython 4. These two commits (as well as tons of
   prerequisite refactoring, see the 'wxPython' section above) cleared
   that blocker for good.

The Python 3 port is one of the primary 308 goals, along with patcher
refactoring (#312) and records refactoring (#480), on which it is
blocked (due to the aforementioned heavy bytestrings usage, which
those refactorings will help us isolate and encapsulate).

-----------------------------------------------------------------------

### Build Scripts (#415)

An enormous productivity gain for developers came in the form of
@GandaG's reworking of build scripts in
a1b5bfaa40fdbe04549ba3775106ffdff471e62e and
5f31a2adf39a607db282f49bd43e66c992a1baad. The ancient
package_for_release.py has been replaced with a sleek new build.py
script that does everything you need to do to build Wrye Bash in a
single invocation. On top of that, Ganda also dropped tons of weird
legacy things the build scripts did, like using ResHacker.exe to set
the Wrye Bash icon - more binaries gone <3

-----------------------------------------------------------------------

### Enderal (#433)

Support for the Steam release of Enderal: Forgotten Stories, a total
conversion mod for Skyrim was added in
c2d73965fba4d0a82bb95f7cbe13b7f5dbcc0155. This was a fairly simple game
to suport since it is pretty much just a pre-modded version of Skyrim
LE. Of course, the work on refactoring game handling is the reason why
we had so few problems with this merge. Once again, we left this game
too long on a branch, meaning it began to accumulate bugs. That
necessitated a fixup commit almost immediately in
3afa217d987c8c4ab86341ed8a8ec906b099767a. For all future game merges,
we resolved to merge more quickly, as long as adding support for the
game doesn't break any other code.

-----------------------------------------------------------------------

### Records (#480)

After all the above, there were a few spots left in the codebase that
needed *heavy* refactoring: records, patchers, saves (*not* save
headers, the Oblivion-specific save editing code) and BAIN. Since
the records and patchers code are very closely intertwined, they need
to be attacked in tandem (refactoring the patchers is sort of a
'top-down' approach, while refactoring the records is a 'bottom-up'
approach to the same problem). See the 'Patchers' section above for
more information on that refactoring.

The whole shebang began in ecac15d01dc5c89463ad47aab74260abfbea4167 and
3a3e9c935f5c1a798211eb0eaed0a0dd76a9af24, which were mostly just random
commits improving some record definitions.

Heavyweight refactoring began in
134433fde71534fc09e357ad64c696194f51a8eb, which moved an awful lot of
records code into brec by creating new tools for defining record
definitions. The result is a massive reduction in code size:

  6787 insertions(+), 9581 deletions(-)

..and a very nice situation where almost the entire *implementation* of
PBash sits in brec, while the (almost) purely declarative definitions
sit in game/*/records.py. Unfortunately, this bloated brec to 3000+
lines and made it much harder to tell which classes belonged together.

This was addressed in 28c11cb934056790e2a07703a9a09b4a5de8aa48, a huge
merge that split brec into a package, added OBME support to PBash, sped
up plugin loading by using AOT construction of struct.Struct instances
instead of struct.pack/unpack and implemented merging of all record
types in Oblivion. That's right, PBash can now merge everything CBash
can. See the 'CBash Deprecation' section below for more information.

After reading both the 'Python 3' and 'Patchers' sections, you should
already know what's coming here: much more in 308. There's already a
large refactoring ('part 2.5' of #480) that just needs some testing
before it can be merged, and @Utumno is working on a 'part 3' of #480
that will seriously turn some parts of the records code on its head.

-----------------------------------------------------------------------

### Usability and Accessibility

Wrye Bash has a (not entirely undeserved ;)) reputation for being
difficult for newcomers to get started with (in UX terms, we'd say its
out-of-box experience is bad). While this is obviously a big goal that
we're nowhere close to solving today (really, someone with actual UX
experience would be needed on the team), 307 does include some work
towards both this goal and the goal of making Wrye Bash accessible to
everyone, regardless of disabilities.

 - 1bf488a10a4c9fedb2737e4b2eee86c484f7b93d: The ability to jump to a
   plugin's matching installer from the Mods tab has been added (#53).
 - 3d7b9816ec0e2b7d666a7bead0cd45db5347aed7 by @fireundubh added the
   ability to jump to the matching plugin when a master is
   double-clicked in a masterlist (#311).
 - 261a1029a78e2cae773386c4e41f496a05f018c0 and
   f4987d1d0db38e4379f6470f67c37b982192817f by @BeermotorWB and
   @MacSplody trimmed the jungle that was the package context menu on
   the Installers tab by moving the more rarely used commands into
   submenus.
 - f65112bea111157558f78f056b550a7f689752a3 added the ability to jump
   to a plugin from the Plugin Filter on the Installers tab.
 - 92691409567ce020c6958325ee8c4bd8c9e820cc made the 'Sort By',
   'Columns' and 'File' submenus on each tab consistent by putting them
   in the same places (they were in seemingly random positions on the
   context menus before).
 - 206ffbd9b09a0b8107ac5dacd9b0b3f9c2d1bfc2 by @warmfrost85 allowed
   users to get a preview of what Clean Data and Sync From Data are
   going to do, as well as the ability to use that preview to change
   which files the commands will affect.
 - e5685f3e87abd0bce199fe619509a9648ce2db89, also by @warmfrost85,
   allowed Sync From Data to work with archives. Previously if you
   wanted to, say, clean a plugin and sync it back into its archive,
   you would have to either do it manually in 7zip or use the
   workaround of unpacking the archive to a project, then using Sync
   From Data and finally packing the project back into an archive. Now
   you can just do it all in one go by using Sync From Data directly
   on the archive.
 - b427bbd383688a2f0fd55266b040cdc6ddf7c590 and
   a2297331b6571b6989bf1ea3b1c253a85c834448 increased the contrast on
   all our checkbox images to pass WCAG AAA guidelines, to make it much
   easier for people with weak eyesight to use Wrye Bash. In the
   future, we want to allow people to customize the colors of the
   checkboxes so that colorblind people can make use of them too
   (see #511).
 - edbe5e51d294ed0706478a9f8896d1e170d76058 added a menubar to improve
   discoverability of Wrye Bash's column context menus, as well as
   making it possible to access them purely using a keyboard.
   Previously you would have had to right click one of the columns to
   access these commands and options, which is a bit arcane and doesn't
   work for people who can't use a mouse at all. The same merge also
   reworked our half-baked settings menu that was implemented as a list
   of popup options when the tiny gear icon is clicked to instead be a
   proper settings dialog. Not only is this much easier to navigate, it
   will scale far better for our future needs (e.g. extensions).

-----------------------------------------------------------------------

### Morrowind (#479)

Morrowind is not officially supported in 307. We've added some (very)
WIP code in 45ed499de6b5564756867ce4b586ed8f4acb6c1f, enough to install
mods and manage load order, but nowhere close to the featureset that
Wrye Mash sports. The main purpose of this code is to act as a sort of
regression test, making sure that we won't introduce anything breaking
Morrowind support in the future (e.g. during a refactoring).

However, adding full Morrowind support on par with Wrye Mash won't be a
goal for quite a while (309+ at least). There's a lot of research and
refactoring to get into before we can approach that. Thankfully,
Elminster has began adding Morrowind support to xEdit as well, so we
may soon have all the tools and documentation we need to write a modern
version of Wrye Mash's features.

-----------------------------------------------------------------------

### Skyrim VR & Fallout 4 VR (#401 and #454)

The situation here is the same as with Morrowind, but for different
reasons. None of us developers own VR hardware, so we have no way to
actually test if Wrye Bash isn't breaking everything on these games.
Which is why there is no official support yet. Still, @nallar and
@nephatrine contributed some initial code in
8273f37e431dfebcfbf114d2fee8cf8365dec889 and
b2418eb47007a110fb3e255190d79304837a85f1 that seems to be working fine,
judging by the fact that we're apparently included on at least one
Skyrim VR guide already. And again, including this semi-supported game
in our codebase means we are much less likely to break it in the
future (not to mention that each game we add generally leads us down a
rabbit hole of refactoring to make Wrye Bash more flexible and
game-agnostic, which is definitely a good thing - one of our long-term
goals is adding support for games outside the 'Bethesda sphere').

-----------------------------------------------------------------------

### FOMODs (#380)

One of the most commonly requested features in Wrye Bash's entire
history. FOMODs are a fairly terrible mod format due to their
incredibly loose nature. You can place files pretty much anywhere in
the package, and there is no specification. That means mod managers are
free to implement whatever they want - as long as it vaguely works like
NMM's installer did, it's an FOMOD installer. Still, this format has
become very common in Skyrim and Fallout 4, so we should support it.

Huge thanks to @GandaG for contributing the initial version in
fa48b1d26bcaf3e9a49b96376df56e2c5b425146, including a full FOMOD
parser, which was a showstopper before. While all other parts of
FOMOD support (including the GUI, the command itself and BAIN
integration) have been rewritten since then (see below), the parser is
still pretty much just as Ganda left it.

Due to BAIN being built around mods being, well, *structured*, it
wasn't a great fit for FOMOD support at first. We eventually came up
with a way to hide all the ugliness of FOMODs from BAIN and just
present it with the clearly structured resulting list of files to
install in a big merge (54844fbe1ea4ddfd48128e61f4232439431aac45) that
also contained a bunch of GUI improvements.

It turns out that BAIN wasn't *actually* that bad a fit for FOMODs
after all: it already had machinery in place to map one path in an
package to a different path in the Data folder, since that's necessary
for its 'root heuristic', remapping of docs into the `Docs` folder and
the rarely used plugin remapping feature of the Plugin Filter. By
hooking into the right part of refreshDataSizeCrc, we were able to use
this feature to map the ugly mess that an FOMOD can be to a final list
of neat paths. BAIN can now work with this regular list of paths, and
when it does need to deal with the real package it will look them back
up in the mapping. Of course all the BAIN refactoring that went into
307 (see 'BAIN' section above) is responsible for making this
possible :)

We still don't recognize all FOMODs, but to get any better would
require extensive BAIN refactoring - which just so happens to be a 309
goal.

-----------------------------------------------------------------------

### CI & Tests (#474, #508)

After two of our betas needed point releases immediately afterwards to
correct blatantly obvious errors, we decided it was time to seriously
push for a CI service that will build and test each Wrye Bash commit.
40285cbab414e3b7fcbcffc33c0be816e68d13f1 added the first version of
this using GitHub Actions.

The test suite is still very limited (only cosaves and a few parts of
bolt are extensively tested) and there are tons of open questions - for
example, we're currently using a very hacky way to change bush.game to
fake having restarted Wrye Bash with a different game selected. While
this works fine for the few tests we have right now, it *will not*
scale, especially not once we get to testing bosh and patcher (which
are the ones we really *want* to test). Expanding on this will be
another important goal in 308/309.

-----------------------------------------------------------------------

### Nehrim (#514)

Support for the Steam release of Nehrim: At Fate's Edge was added in
1bdcd9df2cd9cee164645b74beec1f513aa24129. Unlike Enderal, Nehrim is
not installed as a separate game. Instead, the launcher backs up your
Oblivion installation and allows you to switch between Nehrim and
Oblivion by simply swapping the two folders around. That means we
instead check which game is currently installed in your Oblivion folder
and launch Wrye Bash for that.

That meant a lot of refactoring, including some ugly warts that will
need profiles (#250) to fully resolve, but it also means we were able
to drop our hacky half-baked support for the manual Nehrim version. One
particular thing about this merge that is worth highlighting is that we
merged it *instantly*. As soon as the branch was finished and somewhat
tested, it landed on dev. The reasoning was twofold:

 1. We wanted to merge a huge records refactoring branch afterwards,
    and did not want to put others through resolving conflicts for it
    all the time.
 2. FO3, FNV and Enderal have taught us that letting games sit around
    on branches is a terrible idea, since they quickly accumulate
    subtle bugs.

-----------------------------------------------------------------------

### CBash Deprecation (#520)

With all the refactoring that has happened in 307, CBash has been
completely left behind in the dust. PBash can now merge all record
types (#516), has functional OBME support (#515) and supports several
tags and patchers that CBash does not (#461). Additionally, CBash is
starting to become both a maintenance burden and a roadblock on the way
to refactoring and the patchers and therefore the upgrade to Python 3.

209d884469581248d8ca97954bcb4d05c8ef0d61 officially deprecates CBash.
In fact, the whole reason we are putting out the 307 release *right
now* is so we can get on with removing CBash for good in 308 ;)

-----------------------------------------------------------------------

Massive thanks to everyone who contributed to this release, including:

@Utumno, @Infernio, @Sharlikran, @GandaG, @lojack5, @nycz,
@BeermotorWB, @leandor, @syntaxaire, @fireundubh, @Ortham,
@warmfrost85, @Arthmoor, @D4id4los, @MacSplody, @saebel, @nephatrine,
@nallar, @llde, @FelesNoctis, @DianaNites, @valda and many more that
GitHub's contribution tracker doesn't list.
Infernio added a commit that referenced this issue Sep 16, 2020
…nity members]

307 is a gigantic release that has been cooking for almost five years
at this point. In fact, it is even bigger than 306:

 305 -> 306: 48,748 additions and 44,482 deletions
 306 -> 307: 172,916 additions and 111,125 deletions

Where 306 was mostly about refactoring the codebase to save it from
becoming inoperable, 307 is all about using the newly refactored
codebase to design new features, support more games and, of course,
continue the everlasting refactoring war.

The following notes are an attempt to summarize every important thing
that's happened in 307, grouped as topics in a semi-chronological
order. Do note that this is mostly a futile effort due to the 2000+
commits and 280000+ lines worth of changes that make up 307's
development (and the fact that I only joined the project two thirds of
the way through 307 :P).

It is also highly recommended to read this on GitHub, with gitk open
in the background. That way you can click on issue numbers to open
them, while pasting commit SHAs into gitk, where the individual
commits that make up a merge can be seen (GitHub does not expose
this at all).

All mentioned commits are authored by @Utumno or @Infernio unless
otherwise mentioned.

-----------------------------------------------------------------------

### The bosh Split (#201)

The first goal of 307 was to split bosh into a package. After the
patchers had been split out (#163) and basher turned into a
package (#3), bosh was clearly the next big target.
1bdc253e362857dc8a2a484be60bbcccd7891d82 began the splitting, then
it was continued by splitting out the 'messages' code backing the
PM Archives tab in 906858fe14c730ba797711a855933d996d573e2f.

As a small interlude, 5b14b499e17c48917a472ee9ed426b3c7afd8985 then
removed the PM Archives tab entirely (#221) due to the maintenance
burden it had turned into.

682d2134c5e9f69b057cc9fb6e69e13767f5e7de continued the bosh split by
ripping out:
 - The face transfer code (used for moving NPC/player faces between
   saves and mods) into faces.py
 - The OMODs code (used for unpacking OMODs, an old, ugly mod format
   used by OBMM) into omods.py
 - And the BAIN converter code (used for BCFs, aka BAIN Conversion
   Files - small files containing instructions that can be used to
   tell BAIN how to repackage an archive into a format it can
   recognize) into converters.py

318e66a9ad3a096c316decc751bc76d6c3b5b858 and
75c48e654c21feccd52cd851f2b502fd7c41c45f moved some bosh contents into
parsers.py and bass.py - most importantly, the very commonly used
'dirs' and 'settings' constants.

ea242af573952328168759b9c9c9ff1c01ac681c moved various miscellaneous
things mostly related to plugins and LOOT into a new mods_metadata.py
file. 

Coinciding with the INI refactoring (see 'INIs' section below),
6d2a1fa7bcb15f3d5fe787b5475e904e79809253 moved the INI handling code
into bosh/ini_files.py. Similarly,
afb7058a8e0d8a4567d43ec8f6f8828cafc34b83 moved BAIN code into
bosh/bain.py as part of BAIN refactoring (see 'BAIN' section below).

A last few commits splitting out the saves and mergeability checking
into their own files (bosh/_mergeability.py, bosh/_saves.py and
bosh/save_headers.py) were made in
347c552c8bbd3506a5bacd6c678a92e1275c8e06 and
aa82a7af6fbcc6ce24ac1a940d3c2d4f665d626b, closing the issue for now.
bosh/__init__.py is still quite big (3427 lines), but splitting further
is far from trivial.

-----------------------------------------------------------------------

### Fallout 4 (#251)

On November 10 of 2015, Fallout 4 was released - and less than three
weeks later a branch by @lojack5 adding basic support for it was merged
in e85506af4fbd28c4617b46c9ef0832c6c6570ecb0. Unfortunately, not all
game merges in 307 were this speedy ;)
Some more improvements and fixes landed in
134d7c63839de3daa52108f7ce9cc02fcc928df8 by @Sharlikran.

Fallout 4 support was barebones then and is still barebones now. The
real followup work here will come in 308/309 (see #525 and #482).

-----------------------------------------------------------------------

### Dropped Support For Older Settings Files (#253)

Various bits of backwards compatibility cruft had accumulated over the
course of 306 (and even earlier) to keep pre-306 settings working. 307
dropped all this and instead shows an error message if pre-306 settings
are loaded in 307, telling the user to resave them in 306 first so that
they can be upgraded to the new format. All this was done in a single
merge, 8ceef20f269aedb18dbffe17063fac1cb7ccc0f4.

With how big 307 has become, it is probably no surprise that it too has
by now managed to accumulate a metric ton of backwards compatibility
hacks. Cleanup for those will follow in 308/309.

-----------------------------------------------------------------------

### env.py (#258)

In the ongoing quest for native Linux support (#243), encapsulating
OS-specific code (i.e. Windows-specific code) in its own module is
an important step. This was done in
646df2ca063c2dff03cb2185e7e2969fdea065a5. We're still not really there
yet and env.py is, for the most part, still a Windows-specific module.
The end goal would be to have two separate backends from env.py, so
that it imports from one of them based on the OS it is running on.

-----------------------------------------------------------------------

### BAIN (#219)

307 also marks the beginning of the grueling task that is refactoring
BAIN. Still nowhere near done, but it has become manageable. Much more
work to be done in 308/309. Some of the important targets were:

refreshSizeCrcDate was a big method that handled scanning directories
for their contents. The problem was that it was basically two methods
in one and implemented things that only make sense in one specific
directory, namely the Data folder (e.g. empty directory removal).
20e41b3cfc38c71e8baae6e34778b964ca809533 refactored refreshSizeCrcDate
to prepare it for 5ecf4000f10383e3018b4a4febbf8a8f13164a9e, where it
was split into two methods (_refresh_from_project_dir and
_refresh_from_data_dir).

refreshDataSizeCrc is one of the central parts of BAIN. It scans a
package for its contents, applies skips, remaps documentation to the
'Docs' folder, queries and caches CRCs, etc.
ea79bb4fdfe2318d84952ba63fec2fb9b2a7e33a and
5fef1e3924bce2edcf19ff41e469c2a846d851bb tackled this behemoth,
reducing its size significantly. Various later commits like
54844fbe1ea4ddfd48128e61f4232439431aac45 and
c48112237b9953effb74b4206707394f225f035b touched this one up even
further, bringing its final size in 307 to 210 lines (down from 348
lines in 306).

5d435bb05b9e09340eef9a71a402ac018abcb573 and
5ebbf4e8e3646efa8019b9cbccb144be9bc0ead9 focused on centralizing and
refactoring BAIN refreshes, most notably the irefresh method.

b109a4d08f21e023a30d5c0bf4ed45def0ae26e1 made BAIN use the modInfos
cache to avoid recalculating CRCs that we already calculated for the
Mods tab, giving a minor speedup - but mostly the idea here is we want
to read the Data folder once and then delegate the files to the various
FileInfos based on name and/or extension. This is a first step in that
direction (see also #353 and #265).

b1711b1b52fe76f0463b47fab825058f0f3bc41e was an important merge
addressing case insensitive string comparisons. BAIN makes heavy use of
this to keep track of files in the Data folder, each package, external
changes, etc. So a central dictionary that made case-insensitive
comparisons of its string keys was necessary - which is exactly what
was introduced in this merge, as bolt.CIstr and bolt.LowerDict.

545cad09d29cf4b17ed470caba974829f5876d7f refactored the conflict
detection and reporting algorithms to, well, separate them in the first
place. They sat in a single big method called getConflictReport which
came in at 120 lines pre-refactoring. In addition, the fact that this
method returned a single string made it impossible to improve the GUI
for conflicts (a goal in 309) without parsing the string - the string
we just constructed from in-memory objects. The new getConflictReport
is 55 lines long, much more readable and makes use of a new
find_conflicts method that can be used to retrieve the actual conflicts
as in-memory objects to work on.

-----------------------------------------------------------------------

### basher Package Followups (#163)

Began in 306, finished here. daafdd5e76746833afe4eba496aa4afac41ff439
dropped the ancient Tank class for good, curing the flickering on the
Mods tab in the process (#179).
More work done in b7ffca085feb4afe93e48cdafb68521fd9402899.

a28fcfce1f59482d1e701504b738a8df26a9bfe4 is a joint merge by @Utumno
and @DianaNites refactoring the mod export/import links, which were
pasta-filled mess. More prerequisite work landed in
10703a9e2982ae28db60af72146de8d8dca9c324

The topic was finally laid to rest with
fb37a8bf110b6953c07a61a470d05b73ab03eb17 and
3bf6006d842942214694547a3e4c2b2fccb71dee. Every tab now has neatly
separated tab panels, UILists and details panels. The API could still
be better (especially considering how many new tabs we have planned
for 308+ - see e.g. #233, #456, #50, etc.), but it's a far cry from
the situation in 305 and 306.

-----------------------------------------------------------------------

### Skyrim Patchers (#151)

Another principal goal of 307 is getting Skyrim patcher support as
close to Oblivion as possible.

Some prerequisites were merged in
3a2d396b2a9cf2ecd2525006a4e9960f3b4c85ff and
1e18dad44b97bf80b115de0c7175d14df07f2c3f (mostly records).
The first real patcher porting then happened in
9599368dee429f4b63f116d25945beae73d838db.

-----------------------------------------------------------------------

### Load Order (#295 and #309)

Load order handling is a complex beast, to say the least. Not only is
handling all the edge cases difficult, but there are three different
methods that the games use for implementing load order (four if you
count Morrowind, which we don't support - yet ;)).

Wrye Bash originally used the BOSS API for managing load order. This
was changed in 1bf84f3e3b246195b93d9715e2d1891decc47354 to instead use
the dedicated libloadorder. Even libloadorder itself proved problematic
however:

 - Adding support for new games was tough and quickly turned into
   'adding more clauses into if-else chains'.
 - The API generally made no effort to keep actives order and load
   order together - which became untenable in Fallout 4, where the two
   are inextricably linked.
 - Wrye Bash needs to read all the plugins anyways (e.g. CRC, ESM flag,
   etc.) and keeps that information cached in bosh.modInfos.
   libloadorder was reading it all again which, on top of being
   inelegant, was thrown away performance - syscalls are *not* cheap!
 - It's a DLL - bad for git and won't work on Linux (#243)

So, in one big merge, 66d7b4d695289f6dc29142f94a6004494e3306bd replaced
libloadorder entirely with a new API written in pure Python. This
enabled many new features such as locking the load order in all games
(bc7e5de47f3ecf31d1290d436940a37a7b6eb0cf), automatic backups whenever
we make a fix to the LO (38817bb5ad6e222c4fdbde270d10341b2371bdb5),
etc.

-----------------------------------------------------------------------

### Patchers (#312 and #461)

Porting patchers to newer games is going to become harder and harder
unless we refactor them to make it easier. Patcher code wasn't
*terrible* per se and was fairly isolated from the rest of the code
(apart from its tight connection to records code - see 'Records
refactoring' section below), but a lot of it was copy-pasted and hence
difficult to understand and expand - plus bugs often had to be fixed in
upwards of 10 places. Adding a new patcher easily required hundreds of
lines of pasta. This is still an ongoing goal, with much more to come
in 308.

30eda2dd5c987648a11fbe01b8ee1b6c56d7c1ac was an early merge, containing
refactoring on other more or less related things.
122784f5cf4d737bbb5943e9bd395d2bfc53c43a then moved config handling
(i.e. which patchers are active and which sources they are operating
on) to basher. The idea is to have the patchers operating only on a
list of sources - not only for elegance and simplicity, but also to
make them testable (see the 'CI & Tests' section below).

Heavyweight refactoring began in
66d7b4ef3f8959e180f0029874c73e40de2a5a52 and
f8bfdc4c5894ca2832aa5aef6515d70f52df110f, which introduced the
_SimpleImporter class to deduplicate a lot of copy-pasted
implementation code. Keep your eyes on this class, it proved to be a
good idea ;)

eb110497c3ce7b9d72df5a6565a0c7efffbc9a28 and
604ebd31d2f20d26c06ab2d043f114f7e0abc26b went in a different direction,
by using the work that had already been done on refactoring the
patchers to port many of the Oblivion-specific patchers over to FO3,
FNV and Skyrim, as well as add some entirely new patchers that had
been commonly requested. Of course, this involved a bunch of
refactoring too, mostly moving implicit constants from all over the
patcher code into game/*/constants.py, which will make it much easier
to port the patchers in the future (e.g. to Fallout 4, see #482).

10e680cbb347b203f935c042145a5fc308b5f4c8 returned back to good
old-fashioned refactoring by decoupling the config/GUI side of the
patchers entirely from the model side (i.e. the code that actually
implements patcher behavior). Previously, patchers were linked
together by importing the patcher implementations in the GUI and using
them as mixins. This led to lots of weird, hard-to-debug code that
crippled the IDE's ability to perform static analysis. For example, the
implementation of the leveled list patcher would use

  if not self.remove_empty_sublists: return

to skip the 'empty sublist removal' part of the patcher if the checkbox
for this was not checked in the GUI. However, the IDE had no way of
knowing that that variable actually existed, since it came from the GUI
side of the code via a mixin. After many failed attempts to devise a
base class for both the model and GUI side of patchers, @Utumno instead
realized that a much cleaner design would be to have no mixins.
Instead, each patcher's GUI panel now has a class variable called
patcher_type, which it sets to the model class that gets imported from
the `patcher` package. This allowed us to drop tons of boilerplate code
and make the resulting code much cleaner and easier to understand, but
most importantly it acted as a springboard for further refactoring.

Most notably, 9697b64abc00beaed04e950af20c8116db15151a split our
importers.py file into four files: _cbash_importers.py, _shared.py,
mergers.py and preservers.py. The key insight here was that we can
split our importers nicely into two types: preservers simply carry
forward the last value(s) from a tagged mod, while mergers merge values
from all tagged mods based on the tags those mods have applied.

This resulted in several hundred lines of duplicate patcher code being
chopped off due to us absorbing many preservers back into the base
class. It will also make it much easier to drop the CBash patchers,
since they are now in a separate file altogether (see the 'CBash
Deprecation' section below for more information on this).

One last merge worth mentioning here is
426db77ee71c939bb785b94eab7d4281f0f1fa26. It is a highly WIP attempt to
tame the mess that is parsers.py by devising a proper base class. The
savings so far do look promising, but CSV reading and writing are ugly
warts that still stand in the way. Plus the base class might be too
complicated - right now it has six different knobs that tweak its
behavior, and it's not even clear if using it for the all parsers is
feasible. Still, we had to merge since previous betas came out with
the plugin export/import commands this merge ports to Skyrim.

Much more will follow here in 308 - most notably an upcoming
refactoring of tweaks that will make them *much* faster and drop ~1200
lines of duplicate code.

-----------------------------------------------------------------------

### INIs (#247 and #326)

INI handling was spotty at best. Random unicode tracebacks kept showing
up, the INI Edits tab was one of the last big performance hogs and the
default INI tweaks being files in the Mopy folder led to confusion (at
least one mod had a 'Mopy\INI Tweaks' structure and was supposed to be
installed *into the Mopy folder*).

The first step was df2fcc5f95bc7bc2613a14cb996671f3b82dd2db, a series
of smaller refactorings and fixups to make the tab's code more
manageable.

1d4c23a037f6845f5dfebaf1e5e005f8f121a63b began the work on performance
by introducing the proof of concept for a cache, while
58c47d562f43773c5b017531176b8869138869bf attacked the refresh APIs used
by the INI Edits tab and significantly reduced the number of syscalls
it made.

The default INI tweaks were finally dealt with in
656127c645070d35d31423e014b692621ff94015 by hardcoding them into
Mopy/bash/game/*/default_tweaks.py. No more tampering with the tweaks,
no more Mopy\INI Tweaks folder to confuse users, fewer loose files
packaged, simplified INI refresh, better performance due to fewer
syscalls...

ef21c5bb0a4f23d737dde3b22af36dcaf4f9b58b contained some more work on
centralizing the 'apply a tweak' logic and fixing a longstanding issue
where INIs would have Unix line endings written out, even on Windows.

The LowerDict introduced during BAIN refactoring (see 'BAIN' section
above) also turned out to be very useful for INIs:
a9112d6761e47b1fc41aca53df1add5bdd41ad79 used it to rewrite core parts
of ini_files.py for performance and readability.

bb6c8bd2e7ac22d1218cbad90e404b9739dba7bc reworked the handling of INI
encodings based on a central principle: work with unicode and stripped
newlines internally, encode/decode and add/remove newlines at IO
boundaries.

-----------------------------------------------------------------------

### wxPython (#190, #15 and #488)

A war that started before living memory and will continue until long
after we're all gone - or will it? Actually, we're very close to
winning this conflict for good!

wxPython was all over the place in 305. 306 improved the situation
significantly, but 307 puts even those efforts to shame:

 305: 2260 usages (balt: 381, basher: 1586)
 306: 1112 usages (balt: 418, basher: 494)
 307: 207  usages (balt: 191, basher: 8)

*8* direct usages in basher, down from 494! Let's see how we got there:

 - 114c83729aa50045cac4f381912007826d8048bf: Utumno vs wx. Utumno lost,
   of course, but wx usages did go down from 1075 to 923. Mostly
   accomplished by moving common code to balt.
 - c7f5f5085acf1ec6e55f0048c256f7a0ebc3f367: Down to 902, and some
   progress was made towards wxPython 3.
 - 69c7f9f679f48df8cf7562442e80c9129f10924a: wx.lib.iewin was an ugly
   beast that was binding Wrye Bash to the comtypes dependency.
   Unfortunately, dropping it required upgrading to wxPython 3 (see
   below), so this merge simply centralized the iewin import for a
   future removal.
 - 531679d37d6c50cfc030b9c462f44250bcfab7a0: A small merge containing
   backwards-compatible changes that brought us closer to wxPython 3.
 - 050391ca22d7c8451390cbff6fd150ab0b9bcabd: The upgrade to wxPython 3.
   Introduced several significant architectural achievements:
     - Dropping the comtypes dependency by rewriting our HTML rendering
       code to use WebView instead of wx.lib.iewin.
     - Removing bolt's locale-related behavior on import that made
       importing it dangerous - it's been encapsulated in a new
       top-level module, localize.py.
     - Rewriting a lot of the very early boot process - see also the
       'Boot' section below.
 - f9e46eed670b3d2805b1570fc62daf9cca12ce79: The big one. An absolutely
   enormous joint merge by @Utumno, @Infernio and @nycz introducing a
   new package, gui, that truly encapsulates wxPython. nycz wrote the
   first version of the code back in 2017, most importantly the layouts
   code that encapsulates wxPython's sizers in a declarative API. We
   then devised an event handling framework that enabled us to hunt
   down a lot of *implicit* wxPython usages. These are much harder and
   nastier to track because they can't simply be regexed. Thankfully
   PEP8 will be able to help us here, since all of wxPython uses
   PascalCase for its methods, while we're using snake_case for all new
   code. The result is a reduction down to 218 direct wx usages outside
   of gui.
 - 22de7ff9e804b2bcaa8a819922f4b5100b86d80f: After upgrading to
   wxPython 3, the next goal was upgrading to wxPython 4. This is also
   the first release of wxPython that supports Python 3, making this a
   significant step in the direction of py3 support (#460).
 - eb86a4cb35fc24b58288a6b3e917ed9350a02e23: In preparation for
   finalizing and merging the FOMOD support (see 'FOMODs' section
   below), a bit more de-wx'ing happened, mostly on radio buttons and
   the splash screen.

-----------------------------------------------------------------------

### BSAs (#339 and #338)

Same story as libloadorder. We were using a binary, libbsa, to do it.
This was thrown away performance (we already read and cached the BSAs
in bosh.bsaInfos), had no Linux support, etc.

Its replacement, bsa_files.py, was introduced in
b199a7bd5eec69ec0852f86d6e3629784e6b90ce, then used to support strings
files packed into BSAs in c4f12d56d1273f40096abe45b6803d9265b1e75e and
finished in 8ce81bfc9b5dc4643988e54471fab3e9b6a1f72d.

With BSA handling code now taking shape in the form of bsa_files.py,
having a second class arbitrarily handling a few things with entirely
different (and much uglier) code would be a bad idea - so
d69f3e82c3afe4b6461b6ccce49a210bedd28dd6 dropped the ancient BsaFile.

There were still some unimplemented parts of the BSA format:
  - TES3 format: Added in 4ed5bd8c4ed9a67f872f109c660cf0afeb6ddca5,
    also in preparation of the POC Morrowind support we have in 307 (see
    the 'Morrowind' section below).
  - Compressed BSAs: Added in a6e11c4601d788e0e25c17361d92eaa659e59768,
    including both lz4-compressed ones for SSE and zlib-compressed ones
    for all previous games.
  - FO4 DX10 format: Added in 903b6d11d1451855951cce608c0ce8fe6a23743f.
    Currently unused, since we only use BSA extraction for strings
    files, which the DX10 format can't contain (it may only contain
    textures, for which it is specifically built and optimized).
  - Writing: We can read and extract everything from Morrowind to FO76
    now, but have no support for altering and writing out BSAs yet.
    This will be a goal in 308/309 (see note below).

All this work on BSAs acts as a prerequisite for the BSAs tab we want
to enable and expand in 308/309 (it already exists in the codebase, but
is very unfinished at the moment) - see #233.

-----------------------------------------------------------------------

### FileInfo(s) (#336)

At the heart of each tab sits a DataStore subclass. This provides the
UIList (i.e. what you see on the left side of each tab) with the data
it should show. Most tabs (all but People and Installers) then have
TableFileInfos in the hierarchy, and all but INIInfos (which backs the
INI Edits tab, unsurprisingly) then have FileInfos in its hierarchy.
These FileInfos use FileInfo classes to represent the files that are
going to be shown in the list.

These APIs are not *bad*, but they're not *good* either. Refactoring
this is an ongoing goal, with lots of work done in 307.

b199a7bd5eec69ec0852f86d6e3629784e6b90ce devised a common API for
representing a tracked file with caching called AFile and used it for
the new BSA API (see 'BSAs' section above). Some further work on
freezing the AFile API and making FileInfo use it happened in
11f6769f641e80552b6f2d6e3c25d49a85e66c63,
e3064942119c504786cbe6befafd8a0aa38bec2e and
a0ae08c42fdf47688870387a6f6296de0551bfc2.

Cosaves got a lot of work done in 307, bringing them from pre-alpha at
best to a solid beta API (see the 'Cosaves' section below). Screenshots
also got reworked to use the FileInfo(s) APIs in
e007a5308d509fcd5e123d566b2b3c24e228edaa and
9a317f62cc8f7f807581a0a5f5453637d5d8b5ff.

-----------------------------------------------------------------------

### Skyrim SE (#347)

In a join merge by @Arthmoor, @Utumno and @Sharlikran, Wrye Bash got
SSE support added: 76b8abd5437042dd5b1f1e4505a651211d5523e8.

See the 'ESLs' section below for some of the following challenges with
SSE support. Of course, patcher support was spotty at first too - we
ported all Skyrim patchers to it in
c076e9fe1f5a27746a11b91ba9e0e1b43b80a915.

One more merge worth mentioning here is
064d5021e068260cc99225be29d8d651b0761c61, which sped up startup in all
games, but most notably in SSE. Our reference setup we used for testing
(with ~200 saves) went from 10s down to 2s.

-----------------------------------------------------------------------

### Boot (#373 and #390)

The boot phase was nothing short of a mess. There was no clear guiding
principle of what is initialized when, leading to hard to debug
problems. Unexpected errors could take down Wrye Bash for good without
any way to tell what the problem even was. Restoring settings wasn't
working at all. This is still somewhat in flux just due to how complex
the boot phase is, but it's definitely gotten better.

0e3ef608e2906afb3405b009c622c632caa21dab began the process by
centralizing the wxPython import. In
6060a157be13372ff2a8e09c8de9878c16190f2a, @D4id4los rewrote core parts
of the boot procedure to gracefully handle and show errors, even when
not in debug mode - making fixing startup errors encountered by users
much easier.

Restoring settings was addressed in
17f2266525e5095f8c068804fe3cc3471c9bac1a. In short, when restoring
settings, we would override the settings we just tried to restore due
to our atexit hook firing immediately afterwards. Instead of hacking
away at it with monkey patches, @Utumno carefully reworked the boot
sequence to clearly lay out what gets initialized when, breaking barb's
dependency on the rest of Wrye Bash (balt, bosh, bush, etc.) in the
process and adding a new top-level module, initialization.py, to better
encapsulate init procedures.

The locale mess that bolt did on import was addressed during the
wxPython 3 upgrade in 050391ca22d7c8451390cbff6fd150ab0b9bcabd. This
also resulted in a much more well-documented early boot process,
including setting up the BashBugDump and bolt.depring much earlier,
allowing us to use it to consistently log during the entire boot phase.

-----------------------------------------------------------------------

### Cosaves (#437)

xSE (i.e. the script extenders - OBSE, SKSE, etc.) create cosaves for
each save you make. Wrye Bash originally only needed these for its
master remapping feature to not break things, but over the course of
307 we've come to use them to display save masters with ESLs in them
(since those saves store two separate lists, an accurate master list is
only possible by looking at the PLGN chunk in the cosave). They present
a unique challenge in that each cosave is attached to a regular save,
and all operations on that save need to respect the cosave. That means
renaming, deleting, backing up, etc. need to not just apply to the
main save, but also to its cosave, if it has one.

0a300af01a85bb3535d3194203fd34cc0d3e9b29 introduced the initial API for
this, bosh/cosaves.py. It was mostly just a collection of code from
various parts of bosh (mostly _saves.py), and as such was difficult to
understand, maintain and extend.

822c0bd16e5788d1811449810f34edfee16d77a1 then refactored it (the commit
looks like a rewrite, but was actually a gigantic refactoring
comprising 100+ commits that had to get squashed down to a single one
in order to not break dev) for maintainability and to bring Pluggy
(an ancient cosave format in Oblivion) support into the cosave
hierarchy. It also added support for saves with ESL masters, as
mentioned above.

Finally, e4dc76995703f16ee97fb07d495b6db635a2c6a8 reworked our handling
of cosaves to be much more robust 

-----------------------------------------------------------------------

### ESLs (#382 and #429)

ESLs are a new type of plugin file introduced with SSE and FO4. They
present a unique challenge in that they can bypass the usual 256
plugins limit, and as such stress-test many central assumption in any
tool that tries to support them. Our APIs stood well to the test
however, with d507111773d459e41468ac835955fe88915e622e only having to
make minimal changes to add initial support.

Due to the limited understanding of ESLs at the time, we were very
conservative in what we allowed users to do with them. There was no way
to verify ESL flags, add and remove them, and the Bashed Patch excluded
ESLs completely. That was fixed in
547565a32f8d1d4a2944984c8c29092834f4fff6, a join merge by @Sharlikran,
@Utumno and @Infernio. With it, Wrye Bash gained the ability to add the
ESL flag to ESL-capable mods, importing from ESLs into the Bashed Patch
was reenabled and load order operations for ESL-flagged ESPs were
fixed.

Once again, we were quite conservative in implementing the ESL flagging
in Wrye Bash. For all record types we had not decoded yet, we simply
failed the verification and told people to use xEdit to check instead.
c137405418360063b6d767e7179c31fa94936cbc changed that to use a generic
method that does not rely on our record definitions, since the only
thing we actually have to care about are the headers of all records in
the file. The contents of those records do not matter. This made
ESL-flagging both faster and completely accurate for all record types
in both SSE and FO4.

-----------------------------------------------------------------------

### Game Handling (#358)

Along the way, especially after adding initial ESL suport (see 'ESLs'
section above), it became clear that Wrye Bash's game handling would
become a big issue that needed addressing. Adding support for a new
game involved dozens of edits all over the codebase due to fsName
checks and copy-pasting and editing a big constants.template file.
This cripped the IDE's static analysis, since it couldn't check that
any given game constant existed, let alone had the right type. We were
also importing way too much for each game (e.g. all the constants, the
default tweaks, the vanilla files, etc.), when we should really just
import them for the one game we're actually managing. Thrown away
performance and memory, plus just plain inelegant.

In a joint merge by @Utumno and @GandaG,
11fa0f6a71ca8420071e76357b89f5d3e221c904, the game constants were moved
from module-level into classes, allowing them to be inherited. This got
rid of tons of duplicate code (1229 insertions(+), 1824 deletions(-))
and made adding both a new game and a new game constant easier and less
error-prone.

Some more work to move game-specific constants out of random files and
into the game/*/constants.py files they belong into happened in
fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, along with some fixes in
06d6a6bdaa925f379ffc1f05d0fa5977057ac739 and
57d3a621a08f4852dc5d5cc36878db9286351579.

4050e60bd372494c46860c85fa15c1701abcc5ae devised a way for us to avoid
importing the constants, default tweaks, etc. for every game, while
ddda9393d9b08a132c4a346fbdd8cc84454bccbd and
0fecde47b73d3204735c28b37260b7af8a01f700 finally finished off the last
few constants outside game/*/constants.py, closing this issue for the
time being.

-----------------------------------------------------------------------

### Fallout 3 & New Vegas (#150 and #468)

Not originally planned to be part of 307, but after it was accidentally
included in Beta 3, we had to merge it:
0f06e4fd306684aafccd2764b47a66d2205d10fc

@valda originally ported Wrye Bash to FO3/FNV as Wrye Flash. Efforts to
backport the changes to WB had been dragging along since forever, so
the main thing we learned from this merge was that leaving games to rot
around in branches is a *terrible* idea. Better to have the WIP code on
dev without explicitly providing support, as leaving it on a branch
makes it accumulate subtle bugs from refactoring extremely quickly.

Some work on synchronizing the FO3 and FNV versions happened in the
form of 54b8e614acd0841460f550d33d23340824f06e31 - since FNV is a
vastly more popular game when it comes to modding, many of the
improvements that valda made to the NV version of Wrye Flash did not
make it back to the FO3 version. With us being based on a single
codebase, doing that is much easier - eventually culminating in the FNV
constants being entirely deduplicated in
3f96076501d5c260af856dac1f6475c21aa53a6e,
e68b7af1eb9a3eed96b72cda067be82d86e067c6 and
dccd28c70ffb857ccc70265a5ca22ca718d8e442 so that they are based on the
FO3 ones, meaning that adding e.g. a new patcher or bash tag to FO3
will automatically add it to FNV as well.

We're almost at feature parity with valda's version now - only the race
patcher is missing from our version. This will be addressed in 308/309.
On the other hand, we support several patchers and tags that valda's
version doesn't, on top of tons of other features and bugfixes (see,
for example, the rest of this commit message ;)).

-----------------------------------------------------------------------

### Readmes (#432 and #464)

307 includes a significant reworking of the readmes, courtesy of
@FelesNoctis in 493c76b38c760d31757657a5b9bcf70f196d1dcd. It was later
followed up with more edits for maintainability and to update the
screenshots: see 18969116f7547148bf7b66196f91accd1225b465 and TODO

-----------------------------------------------------------------------

### Wizards (#446, #445, #444, #436 and #189)

Several issues related to wizards were fixed (see e.g.
169d8347c1e4f3a3f6d696d4a660f89987f6bffc,
1f71bf335b85276566c12db43b53097842e05981 and
30f698520be938cd3cb6aa950cf979bc5468edb6).

We also refactored the code quite a bit and deprecated the old 'Espm'
versions of keywords and functions in favor of new 'Plugin' versions
(more intuitive, easier to spell and remember, and more accurate with
the advent of ESLs) in 3426384083bd5d7c61d42730ed7fa9c5629bc2db.
Finally, 9244f8536ff0e4b780e3190cc40a032771310f4c added a new wizard
function that had been requested a long time ago, enabling wizards to
alter their behavior based on a plugin's load order.

There is still a lot to do on wizards. For a start, the format is not
formally defined - and the parser that acts as a reference is quite
buggy (e.g. `Note thisIsAString` will print out 'thisIsAString',
because the parser gets confused about its token states and
accidentally treats 'thisIsAString' as a string). See
https://github.com/Infernio/wizparse for my POC attempt at defining a
formal grammar based on ANTLR that other mod managers will also be able
to use. This is low priority, but will be continued in 309.
Additionally, the wizard GUI presents a significant challenge in
de-wx'ing (see 'wxPython' section above) and has a lot of duplication
with the new FOMOD GUI (see 'FOMODs' section below).

-----------------------------------------------------------------------

### Python 3 (#460)

We officially started the process of porting Wrye Bash over to Python 3
in September 2019, seeing as Python 2 has reached its end of life. This
has turned out to be nothing short of a giant can of worms. Wrye Bash
makes heavy use of bytestrings, so simply letting 2to3 run over the
codebase would be disastrous - we'd be fixing unicode/bytes tracebacks
for the next few months and getting no actual work done. In addition,
our policy of having no breaking commits on dev means that an eventual
Python 3 port will have to be a single commit, which makes bisecting
useless. So the result is that we need that py3 commit to be as small
as possible. With that goal in mind, a lot of prerequisite work that
brings us closer to a py3 port without breaking py2 has landed:

 - 660ecbb81f49d478bdc8e4a8905e328d1daf9dca: py3 has no 'ur' prefix for
   strings since the one in py2 wasn't actually a 'raw' prefix:

     >>> print(ur'\u03B3')
     γ

   So dropping this one from the codebase was necessary. This commit
   just dropped all usages in strings that didn't actually have
   backslashes or were autogenerated paths (i.e. vanilla_files).
 - d43ad244170e2110a6daca7d5febed4020550247: This commit by @syntaxaire
   finished off the 'ur' removal mentioned above.
 - 5a98eb4c025651f4e9366db2a7d488ec2068f1fc: cmp and __cmp__ do not
   exist in py3. For the most part, we just had to implement rich
   comparisons.
 - fa74a1b7a61d9b3150f0d2b171145e171f2d27e5,
   cae844b9dde8af014b09a1cb24af2348d5620058 and
   6db5b8b59e28bc46a9d42e966d31007e113c59e6: Changing old-style classes
   to new-style ones work fine, except when the class is used as a
   mixin with a new-style one that uses __slots__. That can lead to
   nasty layout conflicts, as seen in the first of these three commits.
 - 8e201c49bbc809da89b1bda1d269f4cb7619dfc0: Our codebase included an
   ancient version of chardet (1.0.1 from 2008) due to a single manual
   edit that was needed to make it avoid returning the EUC-TW encodings
   that Python doesn't support. We dropped it in favor of the PyPI
   version, and addressed the EUC-TW problem in
   60d0c29dbae91c12c1f7825df9f4e8e243ca09d2.
 - d8d03ca39e1e9f85250fd014cabcc2a65945e5e7,
   197b6a2de78acd723f9d747fd6751fd2c68cf944 and
   659e5b696be5083b9bef0d39356acc30ab46b5a4: Long integers don't exist
   in py3 due to its int type having no max size. So we needed to drop
   all 'L' postfixes and usages of sys.maxint.
 - 664f1722a53c91794f192e343936dfd34b8e86a8: Fixes for various issues
   encountered during an experimental run of 2to3 by @lojack5.
 - 33eac7624971ecd22e1f65ff5e47bc71ca175dbc: Merge by @GandaG
   addressing various py3 issues like print, moved stdlib modules, old
   exception syntax and usage of local absolute imports.
 - 050391ca22d7c8451390cbff6fd150ab0b9bcabd and
   22de7ff9e804b2bcaa8a819922f4b5100b86d80f: We were stuck on wxPython
   2.8 for a long time, but the first version of wxPython that actually
   has py3 support is wxPython 4. These two commits (as well as tons of
   prerequisite refactoring, see the 'wxPython' section above) cleared
   that blocker for good.

The Python 3 port is one of the primary 308 goals, along with patcher
refactoring (#312) and records refactoring (#480), on which it is
blocked (due to the aforementioned heavy bytestrings usage, which
those refactorings will help us isolate and encapsulate).

-----------------------------------------------------------------------

### Build Scripts (#415)

An enormous productivity gain for developers came in the form of
@GandaG's reworking of build scripts in
a1b5bfaa40fdbe04549ba3775106ffdff471e62e and
5f31a2adf39a607db282f49bd43e66c992a1baad. The ancient
package_for_release.py has been replaced with a sleek new build.py
script that does everything you need to do to build Wrye Bash in a
single invocation. On top of that, Ganda also dropped tons of weird
legacy things the build scripts did, like using ResHacker.exe to set
the Wrye Bash icon - more binaries gone <3

-----------------------------------------------------------------------

### Enderal (#433)

Support for the Steam release of Enderal: Forgotten Stories, a total
conversion mod for Skyrim was added in
c2d73965fba4d0a82bb95f7cbe13b7f5dbcc0155. This was a fairly simple game
to suport since it is pretty much just a pre-modded version of Skyrim
LE. Of course, the work on refactoring game handling is the reason why
we had so few problems with this merge. Once again, we left this game
too long on a branch, meaning it began to accumulate bugs. That
necessitated a fixup commit almost immediately in
3afa217d987c8c4ab86341ed8a8ec906b099767a. For all future game merges,
we resolved to merge more quickly, as long as adding support for the
game doesn't break any other code.

-----------------------------------------------------------------------

### Records (#480)

After all the above, there were a few spots left in the codebase that
needed *heavy* refactoring: records, patchers, saves (*not* save
headers, the Oblivion-specific save editing code) and BAIN. Since
the records and patchers code are very closely intertwined, they need
to be attacked in tandem (refactoring the patchers is sort of a
'top-down' approach, while refactoring the records is a 'bottom-up'
approach to the same problem). See the 'Patchers' section above for
more information on that refactoring.

The whole shebang began in ecac15d01dc5c89463ad47aab74260abfbea4167 and
3a3e9c935f5c1a798211eb0eaed0a0dd76a9af24, which were mostly just random
commits improving some record definitions.

Heavyweight refactoring began in
134433fde71534fc09e357ad64c696194f51a8eb, which moved an awful lot of
records code into brec by creating new tools for defining record
definitions. The result is a massive reduction in code size:

  6787 insertions(+), 9581 deletions(-)

..and a very nice situation where almost the entire *implementation* of
PBash sits in brec, while the (almost) purely declarative definitions
sit in game/*/records.py. Unfortunately, this bloated brec to 3000+
lines and made it much harder to tell which classes belonged together.

This was addressed in 28c11cb934056790e2a07703a9a09b4a5de8aa48, a huge
merge that split brec into a package, added OBME support to PBash, sped
up plugin loading by using AOT construction of struct.Struct instances
instead of struct.pack/unpack and implemented merging of all record
types in Oblivion. That's right, PBash can now merge everything CBash
can. See the 'CBash Deprecation' section below for more information.

After reading both the 'Python 3' and 'Patchers' sections, you should
already know what's coming here: much more in 308. There's already a
large refactoring ('part 2.5' of #480) that just needs some testing
before it can be merged, and @Utumno is working on a 'part 3' of #480
that will seriously turn some parts of the records code on its head.

-----------------------------------------------------------------------

### Usability and Accessibility

Wrye Bash has a (not entirely undeserved ;)) reputation for being
difficult for newcomers to get started with (in UX terms, we'd say its
out-of-box experience is bad). While this is obviously a big goal that
we're nowhere close to solving today (really, someone with actual UX
experience would be needed on the team), 307 does include some work
towards both this goal and the goal of making Wrye Bash accessible to
everyone, regardless of disabilities.

 - 1bf488a10a4c9fedb2737e4b2eee86c484f7b93d: The ability to jump to a
   plugin's matching installer from the Mods tab has been added (#53).
 - 3d7b9816ec0e2b7d666a7bead0cd45db5347aed7 by @fireundubh added the
   ability to jump to the matching plugin when a master is
   double-clicked in a masterlist (#311).
 - 261a1029a78e2cae773386c4e41f496a05f018c0 and
   f4987d1d0db38e4379f6470f67c37b982192817f by @BeermotorWB and
   @MacSplody trimmed the jungle that was the package context menu on
   the Installers tab by moving the more rarely used commands into
   submenus.
 - f65112bea111157558f78f056b550a7f689752a3 added the ability to jump
   to a plugin from the Plugin Filter on the Installers tab.
 - 92691409567ce020c6958325ee8c4bd8c9e820cc made the 'Sort By',
   'Columns' and 'File' submenus on each tab consistent by putting them
   in the same places (they were in seemingly random positions on the
   context menus before).
 - 206ffbd9b09a0b8107ac5dacd9b0b3f9c2d1bfc2 by @warmfrost85 allowed
   users to get a preview of what Clean Data and Sync From Data are
   going to do, as well as the ability to use that preview to change
   which files the commands will affect.
 - e5685f3e87abd0bce199fe619509a9648ce2db89, also by @warmfrost85,
   allowed Sync From Data to work with archives. Previously if you
   wanted to, say, clean a plugin and sync it back into its archive,
   you would have to either do it manually in 7zip or use the
   workaround of unpacking the archive to a project, then using Sync
   From Data and finally packing the project back into an archive. Now
   you can just do it all in one go by using Sync From Data directly
   on the archive.
 - b427bbd383688a2f0fd55266b040cdc6ddf7c590 and
   a2297331b6571b6989bf1ea3b1c253a85c834448 increased the contrast on
   all our checkbox images to pass WCAG AAA guidelines, to make it much
   easier for people with weak eyesight to use Wrye Bash. In the
   future, we want to allow people to customize the colors of the
   checkboxes so that colorblind people can make use of them too
   (see #511).
 - edbe5e51d294ed0706478a9f8896d1e170d76058 added a menubar to improve
   discoverability of Wrye Bash's column context menus, as well as
   making it possible to access them purely using a keyboard.
   Previously you would have had to right click one of the columns to
   access these commands and options, which is a bit arcane and doesn't
   work for people who can't use a mouse at all. The same merge also
   reworked our half-baked settings menu that was implemented as a list
   of popup options when the tiny gear icon is clicked to instead be a
   proper settings dialog. Not only is this much easier to navigate, it
   will scale far better for our future needs (e.g. extensions).

-----------------------------------------------------------------------

### Morrowind (#479)

Morrowind is not officially supported in 307. We've added some (very)
WIP code in 45ed499de6b5564756867ce4b586ed8f4acb6c1f, enough to install
mods and manage load order, but nowhere close to the featureset that
Wrye Mash sports. The main purpose of this code is to act as a sort of
regression test, making sure that we won't introduce anything breaking
Morrowind support in the future (e.g. during a refactoring).

However, adding full Morrowind support on par with Wrye Mash won't be a
goal for quite a while (309+ at least). There's a lot of research and
refactoring to get into before we can approach that. Thankfully,
Elminster has began adding Morrowind support to xEdit as well, so we
may soon have all the tools and documentation we need to write a modern
version of Wrye Mash's features.

-----------------------------------------------------------------------

### Skyrim VR & Fallout 4 VR (#401 and #454)

The situation here is the same as with Morrowind, but for different
reasons. None of us developers own VR hardware, so we have no way to
actually test if Wrye Bash isn't breaking everything on these games.
Which is why there is no official support yet. Still, @nallar and
@nephatrine contributed some initial code in
8273f37e431dfebcfbf114d2fee8cf8365dec889 and
b2418eb47007a110fb3e255190d79304837a85f1 that seems to be working fine,
judging by the fact that we're apparently included on at least one
Skyrim VR guide already. And again, including this semi-supported game
in our codebase means we are much less likely to break it in the
future (not to mention that each game we add generally leads us down a
rabbit hole of refactoring to make Wrye Bash more flexible and
game-agnostic, which is definitely a good thing - one of our long-term
goals is adding support for games outside the 'Bethesda sphere').

-----------------------------------------------------------------------

### FOMODs (#380)

One of the most commonly requested features in Wrye Bash's entire
history. FOMODs are a fairly terrible mod format due to their
incredibly loose nature. You can place files pretty much anywhere in
the package, and there is no specification. That means mod managers are
free to implement whatever they want - as long as it vaguely works like
NMM's installer did, it's an FOMOD installer. Still, this format has
become very common in Skyrim and Fallout 4, so we should support it.

Huge thanks to @GandaG for contributing the initial version in
fa48b1d26bcaf3e9a49b96376df56e2c5b425146, including a full FOMOD
parser, which was a showstopper before. While all other parts of
FOMOD support (including the GUI, the command itself and BAIN
integration) have been rewritten since then (see below), the parser is
still pretty much just as Ganda left it.

Due to BAIN being built around mods being, well, *structured*, it
wasn't a great fit for FOMOD support at first. We eventually came up
with a way to hide all the ugliness of FOMODs from BAIN and just
present it with the clearly structured resulting list of files to
install in a big merge (54844fbe1ea4ddfd48128e61f4232439431aac45) that
also contained a bunch of GUI improvements.

It turns out that BAIN wasn't *actually* that bad a fit for FOMODs
after all: it already had machinery in place to map one path in an
package to a different path in the Data folder, since that's necessary
for its 'root heuristic', remapping of docs into the `Docs` folder and
the rarely used plugin remapping feature of the Plugin Filter. By
hooking into the right part of refreshDataSizeCrc, we were able to use
this feature to map the ugly mess that an FOMOD can be to a final list
of neat paths. BAIN can now work with this regular list of paths, and
when it does need to deal with the real package it will look them back
up in the mapping. Of course all the BAIN refactoring that went into
307 (see 'BAIN' section above) is responsible for making this
possible :)

We still don't recognize all FOMODs, but to get any better would
require extensive BAIN refactoring - which just so happens to be a 309
goal.

-----------------------------------------------------------------------

### CI & Tests (#474, #508)

After two of our betas needed point releases immediately afterwards to
correct blatantly obvious errors, we decided it was time to seriously
push for a CI service that will build and test each Wrye Bash commit.
40285cbab414e3b7fcbcffc33c0be816e68d13f1 added the first version of
this using GitHub Actions.

The test suite is still very limited (only cosaves and a few parts of
bolt are extensively tested) and there are tons of open questions - for
example, we're currently using a very hacky way to change bush.game to
fake having restarted Wrye Bash with a different game selected. While
this works fine for the few tests we have right now, it *will not*
scale, especially not once we get to testing bosh and patcher (which
are the ones we really *want* to test). Expanding on this will be
another important goal in 308/309.

-----------------------------------------------------------------------

### Nehrim (#514)

Support for the Steam release of Nehrim: At Fate's Edge was added in
1bdcd9df2cd9cee164645b74beec1f513aa24129. Unlike Enderal, Nehrim is
not installed as a separate game. Instead, the launcher backs up your
Oblivion installation and allows you to switch between Nehrim and
Oblivion by simply swapping the two folders around. That means we
instead check which game is currently installed in your Oblivion folder
and launch Wrye Bash for that.

That meant a lot of refactoring, including some ugly warts that will
need profiles (#250) to fully resolve, but it also means we were able
to drop our hacky half-baked support for the manual Nehrim version. One
particular thing about this merge that is worth highlighting is that we
merged it *instantly*. As soon as the branch was finished and somewhat
tested, it landed on dev. The reasoning was twofold:

 1. We wanted to merge a huge records refactoring branch afterwards,
    and did not want to put others through resolving conflicts for it
    all the time.
 2. FO3, FNV and Enderal have taught us that letting games sit around
    on branches is a terrible idea, since they quickly accumulate
    subtle bugs.

-----------------------------------------------------------------------

### CBash Deprecation (#520)

With all the refactoring that has happened in 307, CBash has been
completely left behind in the dust. PBash can now merge all record
types (#516), has functional OBME support (#515) and supports several
tags and patchers that CBash does not (#461). Additionally, CBash is
starting to become both a maintenance burden and a roadblock on the way
to refactoring and the patchers and therefore the upgrade to Python 3.

209d884469581248d8ca97954bcb4d05c8ef0d61 officially deprecates CBash.
In fact, the whole reason we are putting out the 307 release *right
now* is so we can get on with removing CBash for good in 308 ;)

-----------------------------------------------------------------------

Massive thanks to everyone who contributed to this release, including:

@Utumno, @Infernio, @Sharlikran, @GandaG, @lojack5, @nycz,
@BeermotorWB, @leandor, @syntaxaire, @fireundubh, @Ortham,
@warmfrost85, @Arthmoor, @D4id4los, @MacSplody, @saebel, @nephatrine,
@nallar, @llde, @FelesNoctis, @DianaNites, @valda and many more that
GitHub's contribution tracker doesn't list.
@Infernio Infernio added the M-relnotes Misc: Issue should be listed in the version history for its milestone label Nov 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-docs Area: Documentation (Everything in the Docs folder) A-wizards Area: Wizards (belt.py and ScriptParser.py) C-enhancement Category: Enhancement, a request to add or enhance a feature M-relnotes Misc: Issue should be listed in the version history for its milestone
Projects
None yet
Development

No branches or pull requests

3 participants