EXT_hdr_textures

[MiiBond]

I'm writing an extension that adds the ability to reference high dynamic range images from within glTF and I'd like some feedback. For context, we (Adobe) are interested in using this in conjunction with the KHR_environments extension being discussed #946. I suppose it's also possible to use these as emissive textures in your materials.

There are two main possibilities for HDR storage:

1. An existing HDR file format like .hdr, .exr or .heif
2. Packing HDR data into something that glTF already supports; .png.

Option 1 Pros:

1. Best quality
2. Doesn't require shoe-horning data that the format wasn't designed for.

Option 2 Pros:

1. Smaller download size
2. Potentially significantly smaller in-memory size (though this depends on the encoding and runtime choices)
3. glTF already understands PNG.
4. Can be combined with MSFT_dds_texture extension and future format extensions to encode HDR is other file types.

I'm leaning more towards option 2 but, really, we could define an extension that supports both. To encourage adoption, however, I'm going to suggest that we keep the number of possibilities low.

"images": [ 
  { 
    "uri": "my_hdr_image.png", 
    "mimeType": "image/png", 
    "extensions": { 
      "EXT_hdr_textures": { 
        "encoding": "RGBD", 
        "maxValue": 256 // optional? We can certainly decide on recommended values for RGBM and RGBD
      } 
    } 
  } 
] 

I'm thinking of limiting the encoding types to just RGBM and RGBD. RGBE and LogLuv are also possible, and are higher-quality, but they don't interpolate well so need to be unpacked to float textures in memory to be useful. This makes them okay delivery formats but eat up a lot of GPU memory. RGBM and RGBD also don't interpolate correctly, of course, but it's far less noticeable and, in my experience, isn't a major concern.

Another possible hit against RGBE and LogLuv is that they look garbled when just viewed straight-up in the browser. Is that something we should be concerned about? I believe SketchFab used to (and maybe still does) use RGBE and that's also what I've used in the past. Though, my personal choice landed on RGBE because I found that RGBM lacked the range I needed. I've later learned that I probably wasn't using it correctly as it's important that the packing and unpacking happen in gamma space, not linear.

Referencing actual HDR formats like .hdr (which uses RGBE packing, I believe), .exr or even .heif could be possible but it would require a difference structure to the extension as "uri" would need to be inside the extension. These images can't be viewed natively by the browser so maybe that's a hit against them. Thoughts?

The RGBM and RGBD encodings obviously require that the referenced images have 4-channels. This is currently just PNG and DDS using the existing MSFT extension but leaves the door open to other format extensions.

Further reading:
http://graphicrants.blogspot.com/2009/04/rgbm-color-encoding.html
http://iwasbeingirony.blogspot.com/2010/06/difference-between-rgbm-and-rgbd.html
http://lousodrome.net/blog/light/tag/rgbm/

[abwood]

Thanks for starting this discussion. I won't pretend that I'm an expert in HDR images; I've gone as far as loading HDR (rgbE encoded) images and dynamically generated the prefiltered mipmapped specular maps for IBL on PBR materials (such as demonstrated here: https://learnopengl.com/PBR/IBL/Specular-IBL). I haven't used RGBM or RGBD first hand. The tooling for .hdr images seems to be pretty good (e.g., Gimp 2.10 can export to .hdr and OpenEXR, stb_image supports loading of .hdr). There is also a lot of content available natively in the HDR format, which I would say is another pro for Option 1.

In my opinion, viewing HDR images in the browser shouldn't matter; HDR quality and performance are the primary concern. It would be helpful to see some side by side comparisons of rgbE encoded HDR images vs rgbM/D packed into a PNG to see the differences in quality and image footprint. With standard formats already available for HDR, we should have good reasons to be using nonstandard encodings in PNGs.

With a prefiltered specular mipmapped texture common to IBL, do you see RGBM or RGBD being sufficient enough for trilinear filtering given the questionable interpolation quality? Or will clients ultimately have to decode into a float texture anyway before using for IBL on PBR materials? In other words, do you see this as merely a delivery format or will a RGBM png be the runtime format?

[MiiBond]

With a prefiltered specular mipmapped texture common to IBL, do you see RGBM or RGBD being sufficient enough for trilinear filtering given the questionable interpolation quality?

My suggestion was that their interpolation quality was sufficient that they should be able to remain encoded as RGBA8 in memory. However, yes, we need plenty of testing to prove that out. RGBE definitely can't be interpolated without suffering severe issues so it's out as a runtime format. If delivery was the only concern, however, it would have been my pick.

[lexaknyazev]

The work on supporting additional texture features and formats is happening in other threads.