[wasm] Add relu, relu6 kernels.

[annxingyuan]

Changes

• Add relu, relu6 kernels to wasm backend.

To see the logs from the Cloud Build CI, please join either our discussion or announcement mailing list.

---

This change is 

[dsmilkov]

Reviewed 8 of 8 files at r1.
Reviewable status: complete! 1 of 1 approvals obtained (waiting on @annxingyuan and @dsmilkov)

---

tfjs-backend-wasm/src/setup_test.ts, line 52 at r1 (raw file):

      'gradient',           // Not yet implemented.
      'valueAndGradients',  // Not yet implemented.
      'fused',              // Not yet implemented.

curious what tests are behind this fused bit?

---

tfjs-backend-wasm/src/cc/kernels/Relu.cc, line 25 at r1 (raw file):

namespace {
inline float oper(const float val) { return val < 0. ? 0. : val; }

can you add a TODO, here and in Relu6 below, to replace these relu and relu6 with xnnpack_clamp. The reason why that is better in the long run is because xnnpack will take care of calling SIMD instructions/acceleration and multi-threading

---

tfjs-backend-wasm/src/cc/kernels/Relu6.cc, line 26 at r1 (raw file):

namespace {
float max = 6.;

since this is a hard-coded constant, rename to kMax (the k prefix tells the reader it's a constant). See https://google.github.io/styleguide/cppguide.html#Constant_Names

[dsmilkov]

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @annxingyuan and @dsmilkov)

---

tfjs-backend-wasm/src/cc/kernels/Relu.cc, line 25 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

can you add a TODO, here and in Relu6 below, to replace these relu and relu6 with xnnpack_clamp. The reason why that is better in the long run is because xnnpack will take care of calling SIMD instructions/acceleration and multi-threading

cc @Maratyszcza Curious does xnnpack clamp do something special to accelerate? does it use wasm simd instrincis if enabled?

[dsmilkov]

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @annxingyuan)

---

tfjs-backend-wasm/src/cc/kernels/Relu6.cc, line 26 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

since this is a hard-coded constant, rename to kMax (the k prefix tells the reader it's a constant). See https://google.github.io/styleguide/cppguide.html#Constant_Names

also it's important to mark it as constant, otherwise the compiler can't inline it

[annxingyuan]

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @dsmilkov)

---

tfjs-backend-wasm/src/setup_test.ts, line 52 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

curious what tests are behind this fused bit?

Changed to be more specific about what to exclude.

---

tfjs-backend-wasm/src/cc/kernels/Relu.cc, line 25 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

cc @Maratyszcza Curious does xnnpack clamp do something special to accelerate? does it use wasm simd instrincis if enabled?

Done

---

tfjs-backend-wasm/src/cc/kernels/Relu6.cc, line 26 at r1 (raw file):

Previously, dsmilkov (Daniel Smilkov) wrote…

also it's important to mark it as constant, otherwise the compiler can't inline it

Done

[dsmilkov]

Reviewed 3 of 3 files at r2.
Reviewable status: complete! 1 of 1 approvals obtained