Layers flaky tests with incorrect math results

[mattsoulanille]

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Debian
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow.js installed from (npm or script link):
• TensorFlow.js version (use command below):
• Browser version: Chrome 102
• Tensorflow.js Converter Version:

Describe the current behavior
Running layers tests rarely results in flaky math failures like the following.

Chrome 102.0.5005.115 (Linux x86_64) LayersModel.fit() with classWeight cpu {} One output, multi-class, sparse encoding FAILED
	Expected 3.070847272872925 to be close to 5.3727.
	    at <Jasmine>
	    at ../../tfjs-layers/src/engine/training_weighted_test.ts:224:37 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:45710:43
	    at <Jasmine>
	    at fulfilled (tfjs-layers/src/tfjs-layers_test_bundle.js:41:27)

Chrome 102.0.5005.115 (Linux x86_64) LayersModel.fit() with classWeight cpu {} One output, multi-class, one-hot encoding FAILED
	Expected 3.0704801082611084 to be close to 5.3727.
	    at <Jasmine>
	    at ../../tfjs-layers/src/engine/training_weighted_test.ts:82:37 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:45593:43
	    at <Jasmine>
	    at fulfilled (tfjs-layers/src/tfjs-layers_test_bundle.js:41:27)

Describe the expected behavior
No math errors.

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/CodePen/any notebook.

Unfortunately, I have no idea how to reproduce this issue. It seems to rarely happen in CI and locally. I also don't know the seed that was used in the test that failed because we don't log it. I'll create a PR that logs the seed, and we can see if it's caused by test order.

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

layers_flakiness.txt

[mattsoulanille]

I've discovered a consistent way to reproduce this issue.

First, focus the One output, multi-class, sparse encoding spec with fit and run bazel run //tfjs-layers:tfjs-layers_webgl2_test. Open localhost and click debug. This should run the cpu test and then the webgl test for the spec (2 tests total), and these tests should pass. If the order is backwards, try another seed.

Then, in the browser results page, click the webgl version of the test to run only it. This test should fail with a numerical error.

FAILED LayersModel.fit() with classWeight webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} One output, multi-class, sparse encoding
debug.js:6 Skipped 10 tests
debug.js:21 Expected 3.070847272872925 to be close to 5.3727.

[Linchenn]

CC @Linchenn

[mattsoulanille]

This has been temporarily fixed by #6878, which set a known good seed for jasmine layers tests, but the underlying error is still present.

[gaikwadrahul8]

Hi, @mattsoulanille

Apologize for the delayed response and I see this PR #6878 got merged so May I know have we taken care the underlying error or not now ? if yes could you please close this issue if we have already taken care of this issue ? Thank you!

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[mattsoulanille]

Won't fix, since the frozen seed seems to work fine.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No