Add EPSILON to the environment flag

[dsmilkov]

The motivation for this change is debugging numerical issues with Adam (and other optimizers) on phones.

• Added backend.floatPrecision() which does feature testing using abs(epsilon) > 0 (epsilon is a small number) to determine if the webgl is using 16bit or 32bit floats. This test works on Android and iPhones pointing to 16bit arithmetic.
• Added new environment flag EPSILON which points to the smallest positive number used to make ops numerically stable (e.g. div(x + eps), log(x + eps), etc). The value depends on backend.floatPrecision().
• TEST_EPSILON now depends on backend.floatPrecision() as well. This makes unit tests pass on Android devices as well.
• Optimizers are now using ENV.get('EPSILON')
• Converted couple tests from describe to describeWithFlags since they call expectArraysClose() which needs a backend in order to determine TEST_EPSILON
• Removed readPixels() call in feature testing for whether webgl can download from float textures and used hasExtension(gl, 'WEBGL_color_buffer_float') as a test instead. This removes the WEBGL_ERROR message in Safari.
• Updated texture usage from RENDER to UPLOAD whenever we read from a texture. Without this, re-upload on iPhone causes an error where render texType is float16 but upload texType is float32. See the added unit test.

@caisq, a followup would be to have K.epsilon in tfjs-layers point to ENV.get('EPSILON') and do further testing on the iris example with the Adam Optimizer (e.g. there is a call to log(x) in categorical cross entropy which can be stabilized by calling log(x+eps) instead.)

BUG

---

This change is 

[nsthorat]

Reviewed 6 of 11 files at r1, 10 of 10 files at r3.
Reviewable status: 0 of 1 approvals obtained (waiting on @dsmilkov, @nsthorat, and @caisq)

---

src/environment_util.ts, line 165 at r3 (raw file):

  gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.FLOAT, new Float32Array(4));

  const readPixelsNoError = gl.getError() === gl.NO_ERROR;

If I remember correctly this was purposeful, can you just make sure iOS works?

[dsmilkov]

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @dsmilkov and @caisq)

---

src/environment_util.ts, line 165 at r3 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

If I remember correctly this was purposeful, can you just make sure iOS works?

iOS works because the the above added check if (!hasExtension(gl, 'WEBGL_color_buffer_float')) { which fails on iOS.