Optimize .hdr loading and RGB9E5 conversion

[BlueCube3310]

Improves .hdr file loading/importing times by doing the following:

• changing the RGBF - RGB9E5 conversion code to the one from https://github.com/microsoft/DirectX-Graphics-Samples/blob/master/MiniEngine/Core/Color.cpp,
• directly converting low-range RGBE8 to RGB9E5, from https://github.com/george-steel/rgbe-rs/blob/main/src/types.rs,
• more efficiently loading RLE-compressed data from the disk.

Results:
Optimized x64 build
OS: Windows 10 64-bit
CPU: AMD Ryzen 9 5900X 12-Core
RAM: 64 GB DDR4

Loading only:

master	PR	Image
7354 ms	329 ms	https://polyhaven.com/a/symmetrical_garden_02 - 8K
6901 ms	283 ms	https://polyhaven.com/a/zwartkops_straight_sunset - 8K
1865 ms	78 ms	https://polyhaven.com/a/cobblestone_street_night - 4K
114 ms	4 ms	https://polyhaven.com/a/st_peters_square_night - 1K

On average, this PR loads data ~25x faster.

Whole import:

master w mips	PR w mips	master no mips	PR no mips	Image
12318 ms	3227 ms	9268 ms	2227 ms	https://polyhaven.com/a/symmetrical_garden_02 - 8K
11845 ms	3240 ms	8884 ms	2212 ms	https://polyhaven.com/a/zwartkops_straight_sunset - 8K
3190 ms	820 ms	2380 ms	602 ms	https://polyhaven.com/a/cobblestone_street_night - 4K
204 ms	69 ms	142 ms	32 ms	https://polyhaven.com/a/st_peters_square_night - 1K

On average, the import time for images with mipmaps is ~3.5x faster. Without mipmaps, it is ~4x faster.
The engine also generates mipmaps, saves the data to the import directory and reloads it again, which is bottlenecked by the disk's read/write speeds (The project was on an HDD).

TODO:

• verify whether the converted values are accurate,
• add more accurate performance benchmarks.

[fire]

Can you post your samples as png / openexr converted from the your test cases to compare?

[BlueCube3310]

Here are the samples in .exr format: hdrsamples.zip

I'll upload png comparisons later.

[BlueCube3310]

master	PR	diff

Image sources:
https://polyhaven.com/a/crystal_falls - Greg Zaal - CC0
https://polyhaven.com/a/st_peters_square_night - Andreas Mischok - CC0
https://polyhaven.com/a/table_mountain_1 - Greg Zaal - CC0

[clayjohn]

The code looks great and the performance improvements are very exciting. I assume most of the performance benefit comes from not reading 8 bits at a time?

[BlueCube3310]

The code looks great and the performance improvements are very exciting. I assume most of the performance benefit comes from not reading 8 bits at a time?

Interestingly, the RGBE conversion makes the biggest difference. The fast low-range optimization had the biggest impact on load speeds.

[clayjohn]

The code looks great and the performance improvements are very exciting. I assume most of the performance benefit comes from not reading 8 bits at a time?

Interestingly, the RGBE conversion makes the biggest difference. The fast low-range optimization had the biggest impact on load speeds.

Wow, that is really surprising! But I guess it is more than just a few extra instructions, the old path moves data around a lot more too.

[Calinou]

Tested locally, it works as expected. The imported HDR panoramas look pretty much identical to before (I can't tell a difference with ACES tonemapping and whitepoint set to 6 in Forward+).

Benchmark

PC specifications

• CPU: Intel Core i9-13900K
• GPU: NVIDIA GeForce RTX 4090
• RAM: 64 GB (2×32 GB DDR5-5800 C30)
• SSD: Solidigm P44 Pro 2 TB
• OS: Linux (Fedora 40)

Using an optimized editor build (optimize=speed lto=full). .godot/ folder is removed between each iteration.

Here's the time taken to open the editor, import the 4 HDRIs linked in OP with lossless compression and exit the editor. The times shown here include editor startup and shutdown, so the import process is actually more than 1.45× faster compared to before (probably at least 3× faster in practice).

Benchmark 1: bin/godot.linuxbsd.editor.x86_64.master ~/Documents/Godot/test_hdri_import/project.godot --import
  Time (mean ± σ):      7.856 s ±  0.402 s    [User: 11.515 s, System: 1.181 s]
  Range (min … max):    7.326 s …  8.333 s    10 runs
 
Benchmark 2: bin/godot.linuxbsd.editor.x86_64 ~/Documents/Godot/test_hdri_import/project.godot --import
  Time (mean ± σ):      5.399 s ±  0.203 s    [User: 5.504 s, System: 1.076 s]
  Range (min … max):    5.121 s …  5.698 s    10 runs
 
Summary
  bin/godot.linuxbsd.editor.x86_64 ~/Documents/Godot/test_hdri_import/project.godot --import ran
    1.45 ± 0.09 times faster than bin/godot.linuxbsd.editor.x86_64.master ~/Documents/Godot/test_hdri_import/project.godot --import

Stripped binary size is 8 KB smaller with this PR compared to master.

[akien-mga]

Thanks!