-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[webgl] Texture to tensor API. #4376
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you Ann! It would be great to provide an example or examples of usage in the doccomment of createTensorFromTexture. As someone who is not very familiar with the use case, I have below questions:
- Is the use case to reuse a texture for other purpose, e.g. rendering, or is the use case to reuse the data in the texture for other purpose? What's the benefit, is it save data downloading time, is it reusing the canvas?
- Is the tensor created from the texture used as input for model or a copy of the output of model?
- Who owns the texture and who should dispose the texture, is it tfjs or the application or both (like registered in two places)?
- What does a comprehensive usage of this api look like?
It would be great to get an answer from the doccomment.
Reviewable status: 0 of 1 approvals obtained (waiting on @annxingyuan, @jinjingforever, @lina128, @pyu10055, and @tafsiri)
tfjs-backend-webgl/src/backend_webgl_test.ts, line 153 at r1 (raw file):
'WEBGL_RENDER_FLOAT32_ENABLED', webglRenderF32EnabledFlagSaved); tf.env().set('WEBGL_PACK', webglPackedFlagSaved); tf.engine().startScope();
endScope?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the feedback Ping and Na. Please take another look.
Ping - regarding your question about physical shape requirements - I added some tests wherein physicalShape !== logicalShape, and I added a description in the docs about the constraints for physical shape. Let me know if any of it is unclear.
Na - regarding your questions about usage - I added some more detail to the docs - please let me know if any of it is still unclear.
Reviewable status: 0 of 1 approvals obtained (waiting on @jinjingforever, @lina128, @pyu10055, and @tafsiri)
tfjs-backend-webgl/src/backend_webgl_test.ts, line 153 at r1 (raw file):
Previously, lina128 (Na Li) wrote…
endScope?
Done
tfjs-backend-webgl/src/webgl.ts, line 45 at r1 (raw file):
Previously, pyu10055 (Ping Yu) wrote…
for these three params internalFormat, textureFormat and textureType, are there enum or type to restrict the values?
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the doccomment Ann, LGTM. I have another question, is the second use case in the issue (#3937) (post-processing after tfjs computation, keep everything in gpu) fulfilled or will there be a separate PR for that or is that not feasible? #3937
Reviewable status: complete! 1 of 1 approvals obtained (waiting on @jinjingforever, @lina128, @pyu10055, and @tafsiri)
Hi Na - regarding the second use case of post-processing in TF.js without leaving the GPU, this is possible today as long as the model returns tensors. For example the faceLandmarksDetection model returns its prediction as a JS array by default, but it can also return its prediction as a tensor if the user passes in In this case, the user can retrieve the texture backing the tensor by calling |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the context Ann, this is great!
Reviewable status: complete! 1 of 1 approvals obtained (waiting on @jinjingforever, @lina128, @pyu10055, and @tafsiri)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For tensor output, how do user get the texture handle for the tensor?
Reviewed 4 of 6 files at r1, 3 of 3 files at r2.
Reviewable status: complete! 2 of 1 approvals obtained (waiting on @jinjingforever, @lina128, and @tafsiri)
@pyu10055 - users can retrieve the texture backing a tensor like so:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cool, is this documented somewhere, if not, would be good to add to jsdoc of this method.
Reviewable status: complete! 2 of 1 approvals obtained (waiting on @jinjingforever, @lina128, and @tafsiri)
@pyu10055 good idea - done. |
@pyu10055 Hey Ping - sorry just getting back to this PR now. I'm trying to remember the changes we discussed making to this PR. Correct me if I'm wrong, but were we discussing adding functionality to Let me know if you agree, or if there was something else we wanted to change about this PR. Otherwise I will merge it! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you Ann, I think this is fine. One question on texture retrieval for post processing, do users need to do unpacking specifically? If so, how does the users know if the tensor is packed in the texture? Thanks!
Reviewable status: complete! 2 of 1 approvals obtained (waiting on @jinjingforever, @lina128, and @tafsiri)
Hi @pyu10055 - this change should not affect the API for downloading textures. But to answer your question, the |
@annxingyuan thank you for this! We had planned to work on it but haven't been able to allocate time toward it thus far. Seeing it move into main hopefully prompts deeper interest here :) |
@annxingyuan This is great !!! thanks a lot |
This reverts commit 65dcfcc.
cc @lina128 |
sorry @pyu100552, I has moved my question to #3937. |
This fixes #4080
This fixes #3937
Changes
tf.webgl.createTensorFromTexture
that allows users to create tensors from textures.gpgpu_util
that enumerate the texture properties for different environments (WebGL 1 vs 2, Float32 supported vs disabled, etc.)To see the logs from the Cloud Build CI, please join either our discussion or announcement mailing list.
This change is