Align API with TF

[dsmilkov]

Aligns the backend API and functionality (NaN propagation, dtype strictness, kernel signatures) with TensorFlow Python.

• Remove backend.minPool since TF doesn't have it.
• Remove normRegion param in localResponseNormalization kernel since TF doesn't support it.
• Remove leakyRelu, prelu and preluDer from the backend and implement using higher-level ops, aligning with TF Python.
• Make backend.multinomial take logits instead of probabilities and normalized: boolean param for backwards compatibility.
• argMin and argMax take single axis: number instead of axes: number[]
• Change eluDer(x: T): T signature to eluDer(dy: T, y: T): T to align with TF.
• Change NaN behavior of max/avgPool and conv2d to align with TF.
• Change avgPool out of bounds (padding) behavior to align with TF
• Require indices in oneHot and gather to be of dtype int32

Fixes tensorflow/tfjs#195

---

This change is 

[manrajgrover]

@dsmilkov I just saw this. Would you be fixing tensorflow/tfjs#195 in this PR?

[dsmilkov]

Nice observation! Yes, those will be removed as part of that PR

[nsthorat]

---

Reviewed 9 of 18 files at r1, 8 of 14 files at r2.
Review status: 17 of 24 files reviewed at latest revision, all discussions resolved, some commit checks failed.

---

src/index.ts, line 68 at r2 (raw file):

  gpgpu_util
};
export {WebGLTimingInfo} from './kernels/backend_webgl';

put this under webgl

---

src/kernels/webgl/pool_gpu.ts, line 108 at r2 (raw file):

        'minMaxValue[0], minMaxValue[1]), minMaxValue[2]), minMaxValue[3])';
    if (poolType === 'avg') {
      returnValue = `avgValue / count`;

why keep an accumulator around when you know this statically?

---

src/ops/lrn.ts, line 44 at r2 (raw file):

  static localResponseNormalization<T extends Tensor3D|Tensor4D>(
      x: T, radius = 5, bias = 1, alpha = 1, beta = 0.5,
      normRegion: 'acrossChannels'|'withinChannel' = 'acrossChannels'): T {

this will break caffe

can you just throw for the TF backend? I'd like to not break API if possible. Going forward we can reject these changes.

---

src/ops/multinomial_test.ts, line 31 at r2 (raw file):

    const probs = tf.tensor1d([0.5, 0.5]);
    const seed: number = null;
    const normalized = true;

the API still allows normalized, but we now don't have coverage

---

src/ops/pool.ts, line 165 at r2 (raw file):

  @doc({heading: 'Operations', subheading: 'Convolution'})
  @operation
  static minPool<T extends Tensor3D|Tensor4D>(

hmm... I depend on this for some channel-normalization stuff in the activation visualization demos. What about just throwing in the backend? Alternatively, we could think about finding a way to expose the WebGL backend directly so we can call a WebGL kernel that does this.

---

Comments from Reviewable

[dsmilkov]

Thanks for the fast review!

---

Review status: 17 of 24 files reviewed at latest revision, 5 unresolved discussions, some commit checks failed.

---

src/index.ts, line 68 at r2 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

put this under webgl

Can't do it since it's a type.

---

src/kernels/webgl/pool_gpu.ts, line 108 at r2 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

why keep an accumulator around when you know this statically?

to align with TF avgPool behavior where padded values are ignored (different than treating them as 0)

---

src/ops/lrn.ts, line 44 at r2 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

this will break caffe

can you just throw for the TF backend? I'd like to not break API if possible. Going forward we can reject these changes.

Chatted offline. LRN is rarely used and we are keeping the default behavior.

---

src/ops/multinomial_test.ts, line 31 at r2 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

the API still allows normalized, but we now don't have coverage

Chatted offline. We have webgl and cpu coverage, and there is something we can do later on for other backends.

---

src/ops/pool.ts, line 165 at r2 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

hmm... I depend on this for some channel-normalization stuff in the activation visualization demos. What about just throwing in the backend? Alternatively, we could think about finding a way to expose the WebGL backend directly so we can call a WebGL kernel that does this.

We can use -maxPool(-x) to simulate minPool

---

Comments from Reviewable