feat(Rendering): Add support for halfFloat 3D textures #2046

sedghi · 2021-08-27T22:05:56Z

Recently half float precision for 2D textures was added, this PR will try to add it for the 3D textures too.

Data used is a full body CT from TCIA (download .vti from here).

run npm run example -- VolumeMapperBlendModes with the data

Before

2021-08-27.17-21-22.mp4

Chrome task manager before

After

2021-08-27.17-21-22.mp4

Chrome task manager after

PR and Code Checklist

semantic-release commit messages
Run npm run reformat to have correctly formatted code

Context

Changes

Documentation and TypeScript definitions were updated to match those changes

Results

Testing

This change adds or fixes unit tests
All tests complete without errors on the following environment:
- vtk.js: latest master
- OS: macOS 10.15.7
- Browser: Chrome 92.0.4515.159

CC'ing @swederik

Sources/Rendering/OpenGL/Texture/index.js

martinken · 2021-08-30T18:08:12Z

This is awesome, just had two comments on the code to review then I think it is good to go.

sedghi · 2021-08-30T18:19:21Z

This is awesome, just had two comments on the code to review then I think it is good to go.

Thanks @martinken for reviewing this. I will apply your comments today.

I have one question though, this PR clearly reduces the GPU memory consumption (task manager screenshots); I was wondering about the CPU memory and how it is not getting that dramatic decrease? I checked the arrays feeding to texImage3D and they are not Float32 anymore as expected, but is there any other place in vtk that need to be changed for the CPU memory to also take benefit of this?

martinken · 2021-08-31T13:49:57Z

I have no clue on memory issues in JS. It looks like you are saving about 200 MB on both CPU and GPU which seems reasonable to me and definitely a win.

martinken · 2021-08-31T13:50:02Z

+1

martinken · 2021-08-31T13:50:31Z

If you want me to merge just let me know. LGTM

sedghi · 2021-08-31T14:40:00Z

@martinken Thanks again for your review, yes please merge it

martinken · 2021-08-31T16:14:16Z

Sources/Rendering/OpenGL/Texture/index.js

+        case VtkDataTypes.SHORT:
+          return model.context.HALF_FLOAT;
+        case VtkDataTypes.UNSIGNED_SHORT:
+          return model.context.HALF_FLOAT;


OK I spoke too soon. We need to think about this section right here. The issue is that float can fully represent short and unsigned short while half float cannot.

Short has 15 bits of precision versus about 11 in half float. Unsigned short has 16 bits versus half floats 11. So while we can stuff short and short and unsigned short into half float in webgl2 we are losing precision. So we are saving memory at the cost of precision. So it is a trade off. Any thoughts on this?

Thanks for pointing this out. I dug deeper into the half float standard, and according to Wikipedia section "Precision limitations on integer values" :

Integers between 0 and 2048 can be exactly represented (and also between −2048 and 0)

Integers between 2048 and 4096 round to a multiple of 2 (even number)

Integers between 4096 and 8192 round to a multiple of 4

Integers between 8192 and 16384 round to a multiple of 8

Integers between 16384 and 32768 round to a multiple of 16

Integers between 32768 and 65519 round to a multiple of 32

Since at the time we convert the buffer, we should know the min and max, one approach can be to convert by default if the min and max are in the range that can be exactly represented (-2048 to 2048), and be Optional (by flag, default false) if otherwise?

Actually there is another extension EXT_texture_norm16, but it is totally unclear to me if it makes the texture filterable. Any comment on this extension to enable non-Float32 3D textures that are filterable? The detail of this extension is here, check Table 8.13

Per that table it would be filterable (TF == Texture Filterable) so it would work. However I'm not sure what systems have that extension. I checked on Chrome/Firefox on windows and neither reported it

I like the idea of both checking the data range and having a setting. The setting could be on the volume mapper and named preferSizeOverAccuracy or something like that.

Sounds good, I'll add the preferSizeOverAccuracy for this PR.

Regarding the EXT_texture_norm16, I have it on my Chrome on Mac. I think it is GPU dependant, I checked mine on WebGL report website
Currently I'm working on a demo to utilize EXT_texture_norm16, but I haven't had any luck so far, which is why I moved to half_float. However, I've begun a conversation with one of the developers of this extension, and I'll keep you updated, and hopefully create another PR later when the problem is resolved with EXT_texture_norm16.

Since is no native support for 16 bit texture in webGL as far as I know, the options are

use float32 (memory inefficient)

use norm16Texture (not widely available)

feat(Rendering): Add support for 16bit normalized 3D texture formats #2058

use half float (at the cost of inaccuracies in rendering, but almost widely available)

In WebGPU as I understand 16 bit texture is natively supported so vtk.js should not do that in webGPU. Otherwise, I can't think of any other solution.

I guess you can put it under a flag to do half float when memory is the end goal, but i thought I added that here

fix(imageMapper): not exact half float should be under a flag #2786

Int16 and Uint16 textures are supported by WebGL so I didn't understand why all this was needed
But I found out that even if they are supported but they are not filterable, which means that we can only use nearest neighbor interpolation

Do you think that using software interpolation would be a good idea?
It would cost 8 unfiltered texture sampling and 7 call to mix but we don't have to do expensive operations to convert the volume, or loose memory

They are not filterable correct. I mean I love to see software filtering on 16bits, but do you think doing 8 unfiltered texture sampling and 7 call to mix would be cheap? It might be, but if the fps is like 15 after I think we should let the user decide with some flags. But I love your idea

I made this small shader in ShaderToy to benchmark trilinear sampling in the fragment shader
It looks like trilinear interpolation in software is around 8.5 times slower than in hardware, but I don't know how many samples are made per pixel in the volume mapper
I suppose that it depends on the options, for example if the shadows are enabled

Sources/Rendering/OpenGL/Texture/index.js

sedghi · 2021-09-13T19:31:21Z

@martinken I have edited the PR to include the preferSizeOverAccuracy flag by checking the min and max of the data and a fallback to Float32.
Everything seems to work fine, except the additiveIntensity blend mode. Is it supposed to work for halfFloat too?

Update on `EXT_texture_norm16` extension

My demo is now working fine using EXT_texture_norm16 here (https://jsfiddle.net/sedghi/j5vnb7Lc/2/)

On MacOS, however, there is a problem in Chromium that prevents correct rendering. I reported it and the bug is confirmed, and seems to be resolved soon (https://bugs.chromium.org/p/chromium/issues/detail?id=1246379 ).

martinken · 2021-09-16T14:31:52Z

@sedghi Let me know when you want me to take another review of it. The basic gist here all looks good, I will take another careful look at it once you think it is ready for it.

sedghi · 2021-09-16T14:37:29Z

@martinken This PR is ready for your review. Please go ahead.
I'm opening another PR for the EXT_texture_norm16 right now.

martinken · 2021-09-23T07:48:43Z

LGTM, @sedghi can you rebase on master and squash the commits? Then we should be good to merge.

martinken · 2021-09-23T07:54:55Z

I'll take a look at the additiveIntensity

martinken · 2021-09-23T08:16:01Z

additive looks OK to me, I ran the volumemapperblendmodes example. Is there something you are seeing that doesn't look correct?

martinken · 2021-09-23T08:34:13Z

Oddly enough we are counting on webgl2 supporting linear float textures and yet looking at the spec I don't think it is required. SO we are just getting lucky I guess.

sedghi · 2021-09-28T03:46:14Z

@martinken I did rebase and squash. Are you running the VolumemapperBlendModes with the mapper.setPreferSizeOverAccuracy(true)? The headsq has maximum of 3926 so the flag is needed.

The problem I can see for the additiveBlend is that it renders empty compared to without the setPreferSizeOverAccuracy flag

martinken · 2021-09-29T12:59:39Z

Ahh OK let me give that a try

martinken · 2021-09-29T13:45:17Z

The issue is the ipScalarRange code. It is written to go from a range of 0 to 1 and when using half float textures the texture values are coming back with a larger range (the actual range of the data). The trick is that due to texture limitations we have to encode data into many texture formats, so the texture class has a shift/scale factor it computes that reflects how the texture map values map to the original dataset. (see line 1217 of Texture). When using halfFloat those values have to change. I pushed a commit with a fix for it that also makes it much faster when changing values.

martinken · 2021-09-29T13:46:08Z

I just happened to be in that code yesterday on the WebGPU side so the timing was fortunate :-)

martinken · 2021-09-29T13:46:35Z

Please give my change a look/test and see if it works for you/makes sense.

ipScalarRange is a 0-1 value in texture values which with different texture formats may require different scaling/offsets. This commit fixes it's use for halfFloat textures and also changes how it is implemented to be done with uniforms which are quick to change, instead of changing the shader program which is expensive.

sedghi · 2021-10-03T21:02:34Z

@martinken Thank you for correcting the additive blend and providing such a thorough explanation of the root cause of the issue; I really appreciate it.
It worked well when I tried it. I also rebased on top of the master branch.

github-actions · 2021-10-04T13:42:52Z

🎉 This PR is included in version 19.6.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

jourdain requested a review from martinken August 27, 2021 22:28

martinken reviewed Aug 30, 2021

View reviewed changes

Sources/Rendering/OpenGL/Texture/index.js Outdated Show resolved Hide resolved

martinken reviewed Aug 30, 2021

View reviewed changes

Sources/Rendering/OpenGL/Texture/index.js Outdated Show resolved Hide resolved

martinken reviewed Aug 31, 2021

View reviewed changes

sedghi commented Sep 13, 2021

View reviewed changes

Sources/Rendering/OpenGL/Texture/index.js Show resolved Hide resolved

sedghi force-pushed the feat/texImage3D-halfFloat branch from c11774c to e3d5bbf Compare September 22, 2021 13:43

sedghi force-pushed the feat/texImage3D-halfFloat branch 2 times, most recently from 30fa1ad to a6024d2 Compare September 28, 2021 03:42

sedghi and others added 2 commits October 3, 2021 16:57

feat(Rendering): add support for halfFloat 3D textures

4d3a37b

sedghi force-pushed the feat/texImage3D-halfFloat branch from 40f8265 to eb116f5 Compare October 3, 2021 21:00

martinken merged commit e575df6 into Kitware:master Oct 4, 2021

github-actions bot added the released Automated label label Oct 4, 2021

sedghi mentioned this pull request Mar 10, 2022

LabelmapOutline test fails on Chrome headless #2309

Open

sedghi mentioned this pull request Apr 12, 2023

fix(imageMapper): not exact half float should be under a flag #2786

Merged

5 tasks

sedghi mentioned this pull request Sep 19, 2023

feat/16 bit texture add OHIF/Viewers#3662

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(Rendering): Add support for halfFloat 3D textures #2046

feat(Rendering): Add support for halfFloat 3D textures #2046

sedghi commented Aug 27, 2021 •

edited

Loading

martinken commented Aug 30, 2021

sedghi commented Aug 30, 2021

martinken commented Aug 31, 2021

martinken commented Aug 31, 2021

martinken commented Aug 31, 2021

sedghi commented Aug 31, 2021

martinken Aug 31, 2021

sedghi Sep 1, 2021

sedghi Sep 1, 2021

martinken Sep 1, 2021

sedghi Sep 2, 2021

sedghi Mar 12, 2024

sedghi Mar 12, 2024 •

edited

Loading

bruyeret Mar 13, 2024

sedghi Mar 13, 2024

bruyeret Mar 13, 2024

sedghi commented Sep 13, 2021

martinken commented Sep 16, 2021

sedghi commented Sep 16, 2021

martinken commented Sep 23, 2021

martinken commented Sep 23, 2021

martinken commented Sep 23, 2021

martinken commented Sep 23, 2021

sedghi commented Sep 28, 2021

martinken commented Sep 29, 2021

martinken commented Sep 29, 2021

martinken commented Sep 29, 2021

martinken commented Sep 29, 2021

sedghi commented Oct 3, 2021

github-actions bot commented Oct 4, 2021

feat(Rendering): Add support for halfFloat 3D textures #2046

feat(Rendering): Add support for halfFloat 3D textures #2046

Conversation

sedghi commented Aug 27, 2021 • edited Loading

Before

After

PR and Code Checklist

Context

Changes

Results

Testing

martinken commented Aug 30, 2021

sedghi commented Aug 30, 2021

martinken commented Aug 31, 2021

martinken commented Aug 31, 2021

martinken commented Aug 31, 2021

sedghi commented Aug 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sedghi Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sedghi commented Sep 13, 2021

Update on EXT_texture_norm16 extension

martinken commented Sep 16, 2021

sedghi commented Sep 16, 2021

martinken commented Sep 23, 2021

martinken commented Sep 23, 2021

martinken commented Sep 23, 2021

martinken commented Sep 23, 2021

sedghi commented Sep 28, 2021

martinken commented Sep 29, 2021

martinken commented Sep 29, 2021

martinken commented Sep 29, 2021

martinken commented Sep 29, 2021

sedghi commented Oct 3, 2021

github-actions bot commented Oct 4, 2021

sedghi commented Aug 27, 2021 •

edited

Loading

sedghi Mar 12, 2024 •

edited

Loading

Update on `EXT_texture_norm16` extension