Resize textures for iOS in low material quality mode. #5437

takahirox · 2022-05-16T21:10:09Z

Note: This change description is outdated. See #5437 (comment) and the following comments for update.

Resize huge textures up to 512x512 (Update: I changed to 1024x1024) for low material quality mode on HubsMeshBasicMaterial creation, for

Better reliability on iOS devices. Currently browsers on iOS can crash with huge textures [iOS] The page is refreshed when uploading certain objects from Sketchfab #4669
Better performance. Users choose the low material quality mode for better performance. But fetching huge textures may be costly.
Shorter main thread blocking time. It seems the main thread blocking time caused by texture image resize + smaller texture upload than the one caused by huge texture upload.

As written in #5295 (comment) I want to start with resizeing textures only for low material quality mode on HubsMeshBasicMaterial creation for simplicity and quickly fixing #4669. I hope we can think of further texture resizing optimization later if needed.

takahirox · 2022-05-18T17:29:04Z

Visual quality with resized 512x512 textures looks worse than I expected. Perhaps it isn't acceptable even for low material quality mode.

Ideally it's the best if we come up with a technique that can resize the textures while keeping the quality. (We need smoothing and/or uv adjustment?)

But what I really want to quickly fix is #4669 , I want to think of better texture resizing later if needed.

Instead I'm thinking of relaxing the maximum texture size for the low quality mode, like from 512x512 (all textures must be small enough) to 1024x1024 or 2048x2048 (too huge textures must be avoided).

takahirox · 2022-05-18T17:59:14Z

Changed from 512x512 to 1024x1024

Looks ok so far in this scene (tho I'm not really sure what textures in the scene are resized).

takahirox · 2022-06-03T22:57:05Z

We internally discussed in the team. Resized textures visual quality may not be acceptable especially if scene authors tune well and it's much worse than author's expectation.

But we also agree that assets from Sketchfab may not be optimized.

So we are thinking of resizing textures only the assets imported from Sketchfab so far.

takahirox · 2022-06-07T18:52:24Z

In today's team meeting in Hubs, I encountered the unexpected page refresh problem on my iPhoneX.

Probably no assets is imported from Sketchfab. And I can enter the room if no one there. I speculate some avatars were not optimized well.

So... I'm thinking what we should do for the problem. Ideally we create a simple test reproducing the problem and push the iOS or Safari team to fix the problem. Until the problem will be fixed on iOS or Safari end, what should the workaround be?

src/utils/material-utils.js

takahirox · 2022-06-09T19:34:59Z

Let me switch this PR to draft for now because I started to worry if this feature or configuration is good because

The resized texture visual quality is worse than my expectation
Even if textures are resized only for assets from Sketchfab the problem can still happen
This PR forces the texture resize in low material quality mode. But we might want to think of adding a new preference max texture size.

takahirox · 2022-06-28T02:00:07Z

Rethinking the issue because I encountered the problem on an iOS device again...

I'm thinking if we can resize huge textures to 2K textures in the low material quality mode only on iOS devices (regardless of whether assets from Sketchfab or not).

Some thoughts.

Crashing problem on iOS is a serious problem. For example if someone places an asset with huge textures all the iOS devices in the room can crash. We may need a workaround.
We already reported the root issue to webkit dev. We may be able to expect that the problem can be fixed on webkit end. The workaround will be removed sooner or later. So I don't want to add a complex workaround.
Not distinguishing assets whether from Sketchfab or not can make the workaround simpler.
Resizing to 2K textures may not be a perfect solution. But it should metigate the problem. It would be good as workaround.
Resizing textures can affect visual quality. But (I hope) 2K may be big enough. And doing it only on iOS devices can limit the devices affected by the workaround.
At least, a bit worse visual quality should be better than broken app.

wsxiaoys · 2022-06-28T04:47:35Z

Adding a few cases this PR won't solve (but still causing problem on mobile iOS) for consideration:

Large resolution image implemented by a-image still causes crash.
Large resolution pdf will not render (black plane)

takahirox · 2022-06-28T13:59:52Z

Then resizing textures in HubsTextureLoader may be better? I assume all textures are loaded via it, but not sure if it's true.

takahirox · 2022-06-29T01:06:36Z

Updated the PR to force to resize huge textures to 2K textures in HubsTextureLoader in the low material quality mode on iOS devices.

Browsers on iOS devices can crash if huge or many textures are used in a room. Perhaps huge VRAM usage may cause it. We already reported it to the webkit devs. Until the root issue is fixed on iOS end, we resize huge texture images in the low material quality mode as workaround. It isn't a perfect solution but it should mitigate the problem. Also disable image bitmap for iOS because texture image resize + image bitmap can still even cause the problem. Disabling it seemt to further mitigate the problem.

netpro2k

Still not in love with doing this but like you said I guess its better than the app crashing... I think the issue mentioned in the thread about a-image no longer applies since that has since been removed from our aframe, so everything should be going through our texture loader.

While reviewing I dug into ThreeJS since I remember that it had texture resizing code. It does resize if textures are > gl.MAX_TEXTURE_SIZE. Another options for doing this patch would be to override WebGLCapabilities in our fork to report a lower MAX_TEXTURE_SIZE on iOS. Not sure if I like that solution better but figured i would mention it as an option.

I also wonder what the typical MAX_TEXTURE_SIZE is on an iOS device, since we would have already been resizing anything larger than that anyway.

netpro2k · 2022-06-29T04:01:07Z

src/loaders/HubsTextureLoader.js

+  const newWidth = Math.round(image.width * conversionRatio);
+  const newHeight = Math.round(image.height * conversionRatio);
+
+  const canvas = document.createElement("canvas");


We may want to reuse this cavnas/context and just resize it instead of making anew one for each texture. (I noticed the code in ThreeJS for resizing textures does this).

canvas is set to texture.image. If we reuse canvas, doesn't the update affect all the textures having resized images?

Ah yeah, since we aren't directly uploading it to the GPU we can't reuse it.

takahirox · 2022-06-29T05:04:15Z

I also wonder what the typical MAX_TEXTURE_SIZE is on an iOS device,

16K as far as I know

takahirox · 2022-06-29T20:27:56Z

I guess its better than the app crashing...

Yes, it should be much better. Actually I rarely be able to join our meeting and meetup rooms on my iOS devices due to this problem. And 2K may be big enouch. It may not affect many assets.

While reviewing I dug into ThreeJS since I remember that it had texture resizing code. It does resize if textures are > gl.MAX_TEXTURE_SIZE. Another options for doing this patch would be to override WebGLCapabilities in our fork to report a lower MAX_TEXTURE_SIZE on iOS.

Hm, honestly I don't really agree with that option. (But thanks for mentioning the option anyways.) I don't think we should really rely on Three.js internal implementation. It's an internal design. The behavior might be able to change in future revisions without any notice (internal change may not appear in the migration guide.) Overriding capabilities might have similar risk. In the future revision it might cause unexpected side effects.

Resizing image code is just about 50 lines. It may not be a big deal. I think having the 50 lines in our code is less problem and better maintainability.

If we want to avoid duplicated code with Three.js core, the best option may suggest to Three.js upstream for exposing the resize dunction.

wsxiaoys · 2022-07-01T03:04:10Z

Still not in love with doing this but like you said I guess its better than the app crashing... I think the issue mentioned in the thread about a-image no longer applies since that has since been removed from our aframe, so everything should be going through our texture loader.

Great to know a-image is removed!

while it seems media-pdf is still loading texture by its own. It won't cause crash on iOS device, but the content become black due to large dimension

takahirox · 2022-07-01T03:15:10Z

If I remember correctly, the black texture problem was caused on iOS with image bitmap. But I heard it has been fixed in the latest iOS. (I have bad memory. Sorry if I'm wrong.) Do you use the latest iOS? Even if it can happen in the latest iOS, disabling image bitmap may avoid or mitigate the problem.

wsxiaoys · 2022-07-01T04:24:38Z

I'm on iPhone 12 Pro Max 15.5.

I haven't tried disabling image bitmap, but this MR fixed the pdf rendering issue for me on an 1920x1080 resolution pdf: #5554

takahirox · 2022-07-01T06:12:29Z

Sounds like the same issue as

But the root issue seems to have been fixed on iOS end. If you can still reproduce the problem, first we should report to the iOS team.

Can you make and share a room that you can reproduce the problem?

takahirox added the optimization label May 16, 2022

takahirox force-pushed the ResizeTextures branch from 61a835a to 072a4ab Compare May 18, 2022 17:47

takahirox mentioned this pull request May 24, 2022

[iOS] The page is refreshed when uploading certain objects from Sketchfab #4669

Open

wsxiaoys reviewed Jun 8, 2022

View reviewed changes

src/utils/material-utils.js Outdated Show resolved Hide resolved

takahirox marked this pull request as draft June 9, 2022 18:39

takahirox force-pushed the ResizeTextures branch from 072a4ab to 1bd0082 Compare June 29, 2022 01:03

takahirox marked this pull request as ready for review June 29, 2022 01:05

takahirox force-pushed the ResizeTextures branch from 1bd0082 to ea921d0 Compare June 29, 2022 03:00

takahirox changed the title ~~Resize textures for low material quality mode.~~ Resize textures for iOS in low material quality mode. Jun 29, 2022

netpro2k reviewed Jun 29, 2022

View reviewed changes

takahirox removed the optimization label Jun 29, 2022

takahirox merged commit 778e02c into master Jul 19, 2022

takahirox deleted the ResizeTextures branch July 19, 2022 21:37

takahirox mentioned this pull request Jul 19, 2022

Consider to resize textures for low-end devices #5295

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resize textures for iOS in low material quality mode. #5437

Resize textures for iOS in low material quality mode. #5437

takahirox commented May 16, 2022 •

edited

Loading

takahirox commented May 18, 2022 •

edited

Loading

takahirox commented May 18, 2022

takahirox commented Jun 3, 2022

takahirox commented Jun 7, 2022

takahirox commented Jun 9, 2022

takahirox commented Jun 28, 2022

wsxiaoys commented Jun 28, 2022

takahirox commented Jun 28, 2022

takahirox commented Jun 29, 2022

netpro2k left a comment

netpro2k Jun 29, 2022

takahirox Jun 29, 2022

netpro2k Jul 19, 2022

takahirox commented Jun 29, 2022

takahirox commented Jun 29, 2022

wsxiaoys commented Jul 1, 2022

takahirox commented Jul 1, 2022 •

edited

Loading

wsxiaoys commented Jul 1, 2022

takahirox commented Jul 1, 2022

Resize textures for iOS in low material quality mode. #5437

Resize textures for iOS in low material quality mode. #5437

Conversation

takahirox commented May 16, 2022 • edited Loading

takahirox commented May 18, 2022 • edited Loading

takahirox commented May 18, 2022

takahirox commented Jun 3, 2022

takahirox commented Jun 7, 2022

takahirox commented Jun 9, 2022

takahirox commented Jun 28, 2022

wsxiaoys commented Jun 28, 2022

takahirox commented Jun 28, 2022

takahirox commented Jun 29, 2022

netpro2k left a comment

Choose a reason for hiding this comment

netpro2k Jun 29, 2022

Choose a reason for hiding this comment

takahirox Jun 29, 2022

Choose a reason for hiding this comment

netpro2k Jul 19, 2022

Choose a reason for hiding this comment

takahirox commented Jun 29, 2022

takahirox commented Jun 29, 2022

wsxiaoys commented Jul 1, 2022

takahirox commented Jul 1, 2022 • edited Loading

wsxiaoys commented Jul 1, 2022

takahirox commented Jul 1, 2022

takahirox commented May 16, 2022 •

edited

Loading

takahirox commented May 18, 2022 •

edited

Loading

takahirox commented Jul 1, 2022 •

edited

Loading