tfjs-node does not implement kernel FlipLeftRight

[vladmandic]

Both BlazeFace and HandDetector models work just fine in NodeJS using tensorflow backend

The issue is in how post-processing is done in JS code which uses FlipLeftRight kernel func to reorient input for better second-pass detection.

So if you use tfjs examples, you'll run into this issue

Error: Kernel 'FlipLeftRight' not registered for backend 'tensorflow'

Unfortunately, those models are internally used by several others as well such as FaceMesh and HandSkeleton which makes list of affected models quite long.

Request is to implement FlipLeftRight in tfjs-node - expectation is to have feature parity between main backends.

Also, is there a matrix of implemented features per backend?

Environment: TFJS 3.6.0 on Ubuntu 20.04

[rthadur]

@vladmandic can you please check for a similar issue here

[vladmandic]

There is no version conflict and I'm running latest TFJS 2.6.0. Just to confirm, I created a simple test that only has tfjs-node listed in package.json, zero other libraries so there is nothing to have conflict with.

And I did note the issue #3883 that clearly lists that forwardFunc is not implemented in tensorflow kernel

And it's easy to check:

nm -D node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow.so | grep forwardFunc

function is not present in tensorflow.so shared library which is the backend for tfjs-node (unless it's supposed to use a fallback defined in @tensorflow/tfjs-core/src/engine.ts).

[tafsiri]

I'm going to rename this issue, because the underlying issue is that as mentioned in #3883 blazeface and handpose run kernels that are not implemented in the node.js backend. The error message is a bit confusing to that can be improved as well. I'll file a separate issue for that.

[vladmandic]

i understand the need for better error message, but will this feature request to actually implement kernel function stay open and eventually be completed?

[tafsiri]

I'll try and get an update on this.

[Nuzzlet]

I get
(node:15556) UnhandledPromiseRejectionWarning: Error: Backend name 'webgl' not found in registry

with node-gpu on Windows 10

[vladmandic]

i've tested with tfjs 3.0.0 and error message is improved:

Error: Kernel 'FlipLeftRight' not registered for backend 'tensorflow'

now the remaining part is to actually implement function FlipLeftRight for NodeJS (backend: tensorflow).

@rthadur @tafsiri and plans for implementing this? the issue is open and not even assigned.

[rthadur]

Usually feature request are prioritized by the team and are assigned later , hopefully it will get picked up soon. I am tagging this for contributions welcome if any one interested.

cc @pyu10055 @lina128 @mattsoulanille @jinjingforever

[DCtheTall]

@rthadur I am interested in implementing this if you're still accepting external contributors.

[rthadur]

@DCtheTall please go-ahead , thanks for your interest.

[DCtheTall]

@rthadur you're welcome! I was wondering if there is a place in tfjs-node which would be a good example of how you typically implement new kernel functions for Node.JS. Just an example of what to go off of should be enough for me to hack something together for a PR

[jinjingforever]

Thank you @DCtheTall! This PR might be a good example of adding missing ops to node JS, and this doc might also be helpful. Really appreciate it!

[DCtheTall]

@jinjingforever thank you for the links. This should be all I need to get started!

[DCtheTall]

@rthadur @jinjingforever I have opened a #5261 which implements this function for Node.JS. Lmk if anything else needs to be done.

[jinjingforever]

Thank you @DCtheTall for the PR!

[rthadur]

This change will be available in next release , thank you so much @DCtheTall for your contribution.

[jinjingforever]

Sorry, thanks to @mattsoulanille, looks like the underlying TF C API (1.15) doesn't support this op. It is not supported in the latest version (2.5) either.

What is the right approach to support this kind of ops? (@pyu10055 @lina128 @mattsoulanille). Thanks!

[lina128]

Hi @jinjingforever, I see that there's a ScaleAndTranslate in the c API: https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/scale-and-translate

Can we use that for FlipLeftAndRight?

[lina128]

If ImageProjectiveTransformV3 is available in the c API, it's even better, it's easier to express flip with an Affine transformation matrix: https://www.tensorflow.org/api_docs/python/tf/raw_ops/ImageProjectiveTransformV3

[jinjingforever]

Thank you @lina128! I checked the code and looks like the scale factors have to be greater than 0. So it probably cannot be used to flip an image. And looks like ImageProjectiveTransformV3 is not available in the C API.

Maybe we could use "reverse" to do the flip, i.e. reverse on the 'H' dimension: tf.reverse(image, [2])?

Also, there might be a bug in the current tfjs flipLeftRight implementations that might need to be fixed first. Will discuss with @lina128 offline.

[lina128]

reverse sounds good to me.

[DCtheTall]

@lina128 thanks for the reply. I'd be happy to assist if I can with any other work needed to reland my original PR

[jinjingforever]

Thanks @DCtheTall! I am working on a fix for the existing implementations in other backends and related tests. After that is merged, it would be great if you could help refactor your PR to use "reverse" to implement flipLeftRight in node.

Really appreciate it!

[DCtheTall]

@jinjingforever SGTM. Let me know when your change has merged and I can refactor my PR accordingly

[jinjingforever]

Hi @DCtheTall, I've merged my PR. Now you can try implementing flipLeftRight using reverse. The PR also updated the tests. You can test your changes locally by running yarn test in the tfjs-node directory.

Thank you! Really appreciate your help!

[DCtheTall]

@jinjingforever @lina128 I just uploaded a PR that implements the kernel using reverse. Let me know if this is what you had in mind 😄