Increasing performance for large .bsp files

[snake-biscuits]

The sheer cost of interpretting some large .bsp lumps can massively hurt performance reading various .bsp files

One serious source of this is SpecialLumpClasses, specifically the ValveBsp/RespawnBsp GAME_LUMP

Apex Legends Game Lumps can easily contain hundreds of thousands of static props

>>> import cProfile, pstats
>>> cProfile.run("; ".join(["import bsp_tool",
...     "bsp = bsp_tool.load_bsp('.../ApexLegends/season3/3dec19/maps/mp_rr_desertlands_64k_x_64k.bsp')",
...     "del bsp;", "s3we.prof")
>>> pstats.Stats("s3we.prof").sort_stats("cumulative").print_stats(20)
Thu Jan 26 17:50:42 2023    s3we.prof

         370309403 function calls (370309380 primitive calls) in 187.542 seconds

   Ordered by: cumulative time
   List reduced from 251 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      2/1    0.000    0.000  187.542  187.542 {built-in method builtins.exec}
        1    0.000    0.000  187.542  187.542 <string>:1(<module>)
        1    0.000    0.000  187.542  187.542 bsp_tool/__init__.py:48(load_bsp)
        1    0.000    0.000  187.541  187.541 bsp_tool/respawn.py:158(__init__)
        1    0.000    0.000  187.541  187.541 bsp_tool/valve.py:23(__init__)
        1    0.000    0.000  187.541  187.541 bsp_tool/base.py:26(__init__)
        1    0.007    0.007  187.538  187.538 bsp_tool/respawn.py:163(_preload)
        1    0.000    0.000  172.700  172.700 bsp_tool/lumps/__init__.py:350(__init__)
        1    0.000    0.000  172.478  172.478 bsp_tool/branches/respawn/apex_legends.py:517(ApexSPRP)
        1    0.798    0.798  172.478  172.478 bsp_tool/branches/respawn/titanfall2.py:304(__init__)
   456847   10.352    0.000  171.665    0.000 bsp_tool/branches/base.py:173(from_tuple)
  3198052    7.984    0.000   94.685    0.000 bsp_tool/bsp_tool/branches/base.py:359(from_tuple)
  3198052   29.631    0.000   81.329    0.000 bsp_tool/branches/base.py:237(__init__)
  4111746   13.142    0.000   45.065    0.000 bsp_tool/branches/base.py:606(split_format)
   456847   12.274    0.000   45.013    0.000 bsp_tool/branches/base.py:68(__init__)
 31066700   20.109    0.000   24.750    0.000 bsp_tool/branches/base.py:316(__setattr__)
 15537157    6.274    0.000   19.127    0.000 /usr/lib/python3.9/re.py:188(match)
 14619688    4.465    0.000   16.369    0.000 {built-in method builtins.setattr}
        6   11.212    1.869   12.818    2.136 bsp_tool/branches/shared.py:48(__init__)
 10050875    7.883    0.000   12.130    0.000 bsp_tool/branches/base.py:587(mapping_length)

Deferring the load time cost of SPRP GameLumps would be incredibly easy
bsp_tool.lump.BspLump could be used to defer reading & interpretting data within the lump
It's not like we check every lump has 100% valid data anyway (only size checks, no NaN / invalid enum checks)

see #68 for points on why this should apply to reading from file, and not necessarily the creation of new lumps
also see #23 to understand current issues w/ BspLump

[snake-biscuits]

re.match also makes the list here, along side StaticProp initialisation
Likely coming from shared.Entities.__init__ parsing the entity lumps

We might be able to avoid fully parsing ENTITIES by checking the raw lump string for { & } (curly brace) indices
Pre-emptively slicing up the lump for later parsing (indexing by integer or slice should be quite fast & low cost)
Though bsp.ENTITIES.search() would then end up parsing the entire lump on first call (which is the more common indexing)

Again, I would like to defer costs as much as possible so users only pay for what they use.
SpecialLumpClasses in their current implementation do not follow this philosophy.
We should really take this approach when building .from_bytes methods for SpecialLumpClasses (see #68)
But we will first need to resolve #23 for the sake of stability (stability first, performance second)

[snake-biscuits]

After e9cd7ea
We no longer create ApexSPRP when loading a .bsp!
Loading is also much faster!
This only applies to respawn.titanfall2 & respawn.apex_legends rn

>>> import cProfile, pstats
>>> cProfile.run("; ".join(["import bsp_tool",
...     "bsp = bsp_tool.load_bsp('.../ApexLegends/season3/3dec19/maps/mp_rr_desertlands_64k_x_64k.bsp')",
...     "del bsp;", "s3we.prof")
>>> pstats.Stats("s3we.prof").sort_stats("cumulative").print_stats(20)
Tue May  9 14:51:47 2023    s3we.prof

         5877186 function calls (5877138 primitive calls) in 3.456 seconds

   Ordered by: cumulative time
   List reduced from 271 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      2/1    0.000    0.000    3.456    3.456 {built-in method builtins.exec}
        1    0.000    0.000    3.456    3.456 <string>:1(<module>)
        1    0.000    0.000    3.456    3.456 bsp_tool\__init__.py:48(load_bsp)
        1    0.000    0.000    2.759    2.759 bsp_tool\respawn.py:158(__init__)
        1    0.000    0.000    2.759    2.759 bsp_tool\valve.py:23(__init__)
        1    0.000    0.000    2.759    2.759 bsp_tool\base.py:26(__init__)
        1    0.001    0.001    2.759    2.759 bsp_tool\respawn.py:163(_preload)
        6    0.508    0.085    2.106    0.351 bsp_tool\branches\shared.py:68(from_bytes)
   950037    0.348    0.000    1.023    0.000 Python310\lib\re.py:187(match)
       68    0.654    0.010    0.654    0.010 {built-in method nt.stat}
        2    0.000    0.000    0.644    0.322 Python310\lib\genericpath.py:16(exists)
  1355876    0.390    0.000    0.564    0.000 Python310\lib\re.py:288(_compile)
      128    0.000    0.000    0.510    0.004 bsp_tool\valve.py:26(_preload_lump)
   405707    0.156    0.000    0.480    0.000 Python310\lib\re.py:197(search)
      148    0.458    0.003    0.458    0.003 {method 'read' of '_io.BufferedReader' objects}
   950037    0.279    0.000    0.279    0.000 {method 'match' of 're.Pattern' objects}
        1    0.000    0.000    0.249    0.249 bsp_tool\lumps.py:354(__init__)
        7    0.218    0.031    0.218    0.031 {built-in method io.open}
  1357273    0.174    0.000    0.174    0.000 {built-in method builtins.isinstance}
   405707    0.157    0.000    0.157    0.000 {method 'search' of 're.Pattern' objects}


<pstats.Stats object at 0x0000025E5712E8C0>
>>> quit()

[snake-biscuits]

MegaTest duration reduced from 14hrs to 1hr
Big success

Comparing the cProfile runs, the load time have improved by 50x (takes ~1/50th of the time)
With some luck, we won't need to make any new performance fixes for a while
(So long as the BspLump._changes cache doesn't get too messy)