tfjs-models: make pose-detection deno compatible #6746
Comments
|
@andykais Thank you for trying tfjs in deno. If they are both false, then you need a new platform file to expose the platform specific APIs (i.e. fetch) If you are interested in contributing, it would be great to add the flag and platform file to TFJS repo in order to support deno better. |
|
that prints I may be able to get away with the browser platform if I can somehow force that. Deno implements most browser apis. Is it possible to manually force the platform? If not, can you at least point me to where it is detected? |
|
yeah so looking at the browser_platform.ts file, this should function fine in deno, if we can force it https://github.com/tensorflow/tfjs/blob/master/tfjs-core/src/platforms/platform_browser.ts. IndexedDB is not implemented in deno, but that appears to be an optional requirement for the browser platform. Still havent found where we detect the platform. I imagine the indexedDB is used to cache the models? |
|
Ok I see. Here it is tfjs/tfjs-core/src/device_util.ts Line 60 in a1e5aea window.document != null. In deno, window.document is undefined.
So actually, I think this could be possibly be solved by simply removing the |
andykais
changed the title
|
I think it is possible to make deno looks like browser, but conceptually it is not a browser. If you want to hack the API to testing it out, it might be easier to inject a mock |
|
I guess what I am describing is "duck typing", which to a degree, is what you already are checking. You have a check that is essentially: if it looks like a browser, treat it like a browser. All I am suggesting is to not check browser apis that your library does not use. Alright testing it out, it doesnt actually appear to set ;(window as any).document = {} // polyfill for browser detection
import * as tf from 'https://cdn.skypack.dev/@tensorflow/tfjs'
import 'https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu'
import * as poseDetection from 'https://cdn.skypack.dev/@tensorflow-models/pose-detection';
// initialize tensorflow
await tf.setBackend('webgpu')
await tf.ready()
console.log('isBrowser:', tf.device_util.isBrowser())
console.log('IS_BROWSER:', tf.env().getBool('IS_BROWSER'));this logs either Actually, this doesnt seem to work either. ;(window as any).document = {} // polyfill for browser detection
import * as tf from 'https://cdn.skypack.dev/@tensorflow/tfjs'
tf.env().set('IS_BROWSER', true)
import 'https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu'
import * as poseDetection from 'https://cdn.skypack.dev/@tensorflow-models/pose-detection';
// initialize tensorflow
await tf.setBackend('webgpu')
await tf.ready()
console.log('isBrowser:', tf.device_util.isBrowser())
console.log('IS_BROWSER:', tf.env().getBool('IS_BROWSER'));
const detectorConfig = {
modelType: poseDetection.movenet.modelType.SINGLEPOSE_LIGHTNING,
}
const detector = await poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, detectorConfig)it still fails with I suppose theres more debugging to do |
|
The reason might be the env ;(window as any).document = {} // polyfill for browser detection
import * as tf from 'https://cdn.skypack.dev/@tensorflow/tfjs'
tf.env().set('IS_BROWSER', true)
import {PlatformBrowser} from 'https://cdn.skypack.dev/@tensorflow/tfjs-core/dist/platforms/platform_browser'
tf.env().setPlatform('browser', new PlatformBrowser());
import 'https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu'
import * as poseDetection from 'https://cdn.skypack.dev/@tensorflow-models/pose-detection';
// initialize tensorflow
await tf.setBackend('webgpu')
await tf.ready()
console.log('isBrowser:', tf.device_util.isBrowser())
console.log('IS_BROWSER:', tf.env().getBool('IS_BROWSER'));
const detectorConfig = {
modelType: poseDetection.movenet.modelType.SINGLEPOSE_LIGHTNING,
}
const detector = await poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, detectorConfig) |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you. |
|
@haoyunfeix I tried out your example and was able to at least get tensorflowjs to register the platform as deno. Going one step further to estimate a pose however, yields this error: import * as jpegts from "https://deno.land/x/jpegts@1.1/mod.ts";
;(window as any).document = {} // polyfill for browser detection
import * as tf from 'https://cdn.skypack.dev/@tensorflow/tfjs'
tf.env().set('IS_BROWSER', true)
import {PlatformBrowser} from 'https://cdn.skypack.dev/@tensorflow/tfjs-core/dist/platforms/platform_browser'
tf.env().setPlatform('browser', new PlatformBrowser());
import 'https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu'
import * as poseDetection from 'https://cdn.skypack.dev/@tensorflow-models/pose-detection';
// initialize tensorflow
await tf.setBackend('webgpu')
await tf.ready()
console.log('isBrowser:', tf.device_util.isBrowser())
console.log('IS_BROWSER:', tf.env().getBool('IS_BROWSER'));
const detectorConfig = {
modelType: poseDetection.movenet.modelType.MULTIPOSE_LIGHTNING,
}
const detector = await poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, detectorConfig)
const file = await Deno.readFile('./sample.jpg')
const image = await jpegts.decode(file)
const poses = await detector.estimatePoses(image);
console.log(poses)removing the webgpu import and the line |
This op was added two weeks ago. Please try the latest webgpu (0.0.1-alpha.13) npm package, or you can build your own from the source. Thanks. |
|
@andykais, @qjia7 is right, please use latest code, I tried your pasted code with I thought it maybe a WebGPU implement bug in deno. Due to WebGPU spec and webidl spec, I've submitted a bug to deno upstream denoland/deno#15702 |
|
thanks @haoyunfeix, I did a |
|
It's by design I think, if your Although in our case, the exception happened in |
|
I built Deno from source using my fork in PR denoland/deno#15853 and the backend check passes. I can do But when I tried to decode the model outputs via Full stack trace (click to open)I don't mind making more PRs just to push this forward, but I really need help in pinning down the issue in tfjs. SidenoteIn my Apple M1, the contents of [
"depth-clip-control",
"depth32float-stencil8",
"texture-compression-bc",
"texture-compression-etc2",
"texture-compression-astc",
"indirect-first-instance",
"shader-f16",
"mappable-primary-buffers",
"texture-binding-array",
"storage-resource-binding-array",
"sampled-texture-and-storage-buffer-array-non-uniform-indexing",
"uniform-buffer-and-storage-buffer-texture-non-uniform-indexing",
"address-mode-clamp-to-border",
"texture-adapter-specific-format-features",
"vertex-writable-storage",
"clear-texture"
] |
|
that seems like its possibly a binding issue, it might be helpful to share a minimal repro code snippet @vicary. This looks like the source line https://github.com/tensorflow/tfjs/blob/master/tfjs-core/src/engine.ts#L424 |
|
@andykais Thanks for the hint. It turns out to be me prematurely disposing a tensor before the promise is resolved, we could definitely use a better error message but that's for another issue. I can gladly say that my Deno fork basically works for all the use cases I have. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you. |
|
Closing as stale. Please @mention us if this needs more attention. |
|
tensorflow is not working with the current script in the latest version of deno. Running exactly the script described above: import * as jpegts from "https://deno.land/x/jpegts@1.1/mod.ts";
;(window as any).document = {} // polyfill for browser detection
import * as tf from 'https://cdn.skypack.dev/@tensorflow/tfjs'
tf.env().set('IS_BROWSER', true)
import {PlatformBrowser} from 'https://cdn.skypack.dev/@tensorflow/tfjs-core/dist/platforms/platform_browser'
tf.env().setPlatform('browser', new PlatformBrowser());
import 'https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu'
import * as poseDetection from 'https://cdn.skypack.dev/@tensorflow-models/pose-detection';
// initialize tensorflow
await tf.setBackend('webgpu')
await tf.ready()
console.log('isBrowser:', tf.device_util.isBrowser())
console.log('IS_BROWSER:', tf.env().getBool('IS_BROWSER'));
const detectorConfig = {
modelType: poseDetection.movenet.modelType.MULTIPOSE_LIGHTNING,
}
const detector = await poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, detectorConfig)
const file = await Deno.readFile('./sample.jpg')
const image = await jpegts.decode(file)
const poses = await detector.estimatePoses(image);
console.log(poses)runs, but has an empty output of poses ( Attempting to update the script to use npm specifiers looks like this: import * as jpegts from "https://deno.land/x/jpegts@1.1/mod.ts";
;(window as any).document = {} // polyfill for browser detection
import * as tf from 'npm:@tensorflow/tfjs'
tf.env().set('IS_BROWSER', true)
import {PlatformBrowser} from 'npm:@tensorflow/tfjs-core/dist/platforms/platform_browser'
tf.env().setPlatform('browser', new PlatformBrowser());
import 'npm:@tensorflow/tfjs-backend-webgpu'
import * as poseDetection from 'npm:@tensorflow-models/pose-detection';
// initialize tensorflow
await tf.setBackend('webgpu')
await tf.ready()
console.log('isBrowser:', tf.device_util.isBrowser())
console.log('IS_BROWSER:', tf.env().getBool('IS_BROWSER'));
const detectorConfig = {
modelType: poseDetection.movenet.modelType.MULTIPOSE_LIGHTNING,
}
const detector = await poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, detectorConfig)
const file = await Deno.readFile('./sample.jpg')
const image = await jpegts.decode(file)
const poses = await detector.estimatePoses(image);
console.log(poses)but this gives the following error: I havent been able to discover the cause of that error, since it appears that the source line specified is not actually where the error is occurring @google-ml-butler unstale |
|
update, this is a minimally reproducable example that causes the SyntaxError: import * as jpegts from "https://deno.land/x/jpegts@1.1/mod.ts";
;(window as any).document = {} // polyfill for browser detection
import * as tf from 'npm:@tensorflow/tfjs'
tf.env().set('IS_BROWSER', true)deno --version output: |
|
@andykais For your first piece of code
There're 2 known issues in naga program as mentioned in #6842 (comment), it means generated compute shaders were not available for naga when using upstream WebGPU backend(https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu). |
|
Awesome, glad to hear this is an active issue. @haoyunfeix if you have the ability to reopen this issue could you do that? |
|
As mentioned above, the empty output is caused by invalid shaders for naga. I attached the sample code and rebuilt webgpu js file, please try as you like. You can also try with fix_6746 branch to build a local package for yourself. |
in shader's global scope FIXES tensorflow#6746
…s global scope FIXES BUG: tensorflow#6746 Deno uses Naga for wgsl compilation, but Naga currently uses let for global constants(will be fixed in gfx-rs/naga#1829). This PR helps WebGPU to run pose-detection models on Deno by removing global constants in shaders. Address comments
* [webgpu] Use numbers directly instead of `const variables` in shader's global scope FIXES BUG: #6746 Deno uses Naga for wgsl compilation, but Naga currently uses let for global constants(will be fixed in gfx-rs/naga#1829). This PR helps WebGPU to run pose-detection models on Deno by removing global constants in shaders.
|
@andykais shader problems fixed with #7193, please look forward to the next version(should be https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu@0.0.1-alpha.17, not published yet) can be runnable on your demo. |
|
Closing as stale. Please @mention us if this needs more attention. |
|
just sharing back the working code with the most recent fix: import * as jpegts from "https://deno.land/x/jpegts@1.1/mod.ts";
;(window as any).document = {} // polyfill for browser detection
import * as tf from 'https://cdn.skypack.dev/@tensorflow/tfjs'
import 'https://cdn.skypack.dev/@tensorflow/tfjs-backend-webgpu@0.0.1-alpha.17'
tf.env().set('IS_BROWSER', true)
import {PlatformBrowser} from 'https://cdn.skypack.dev/@tensorflow/tfjs-core/dist/platforms/platform_browser'
tf.env().setPlatform('browser', new PlatformBrowser());
import * as poseDetection from 'npm:@tensorflow-models/pose-detection';
// initialize tensorflow
await tf.setBackend('webgpu')
await tf.ready()
tf.env().set('WEBGPU_DEFERRED_SUBMIT_BATCH_SIZE', 1);
// console.log('isBrowser:', tf.device_util.isBrowser())
// console.log('IS_BROWSER:', tf.env().getBool('IS_BROWSER'));
// console.log(tf.getBackend());
const detectorConfig = {
modelType: poseDetection.movenet.modelType.MULTIPOSE_LIGHTNING,
}
const detector = await poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, detectorConfig)
const file = await Deno.readFile('./sample.jpg')
const image = await jpegts.decode(file)
const poses = await detector.estimatePoses(image);
console.log(poses)I am still not able to use npm specifiers for really any of these modules, I assume thats because not all the relevant pieces have been published (though the problematic webgpu backend code has been published on npm https://www.npmjs.com/package/@tensorflow/tfjs-backend-webgpu/v/0.0.1-alpha.17) |
|
@andykais Thanks for the feedback! Your pasted code works on my machine, env: And did you try to reload your cache? => |
…7193) * [webgpu] Use numbers directly instead of `const variables` in shader's global scope FIXES BUG: tensorflow#6746 Deno uses Naga for wgsl compilation, but Naga currently uses let for global constants(will be fixed in gfx-rs/naga#1829). This PR helps WebGPU to run pose-detection models on Deno by removing global constants in shaders.
|
Deno latest version 1.30.3 hits an error when loading webgpu backend, refs to denoland/deno#17869 |
|
the script I have above works on deno latest version 1.31.1 (the webgpu backend issue was solved in the deno repo). However, I still cannot use npm specifiers, only skypack imports. A very tiny repro looks like this: ;(window as any).document = {} // polyfill for browser detection
import * as tf from 'npm:@tensorflow/tfjs@4.2.0'
tf.env().set('IS_BROWSER', true)
|
I've the same. @andykais Did you find any solution? |
I am attempting to get tensorflowjs running within deno. A basic script like the following will work:
I can also load up a model, specifically over http, using the
file://protocol actually still triggers a fetch, which deno fails on.However, I would like to be able to reuse some of the code in tfjs-models, since there is a decent amount of code that makes the output from the models useful. This is the following code snippet I want to get working:
System information MacOS M1
v3.14.0Describe the feature and the current behavior/state.
tfjs throws an error
Will this change the current api? How?
yes, adding a third option to the
createDetectormethod. Example:Who will benefit with this feature?
anyone porting tensoflowjs to deno trying to use the pose detection
Any Other info.
deno --versionThe text was updated successfully, but these errors were encountered: