Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trying to run a subset of tests seems to just run them all #8164

Open
hchiam opened this issue Feb 1, 2024 · 1 comment
Open

trying to run a subset of tests seems to just run them all #8164

hchiam opened this issue Feb 1, 2024 · 1 comment
Assignees
@gaikwadrahul8
Labels
comp:core type:bug Something isn't working

Comments

@hchiam
Copy link

hchiam commented Feb 1, 2024

i ran tensorflow/tfjs in a github codespace and followed https://github.com/tensorflow/tfjs/blob/master/DEVELOPMENT.md#testing
and i'm trying to run a subset of tests, but the output doesn't seem to change when i run commands like yarn test --//:grep=multinomial

yarn
yarn lint
cd tfjs-core
yarn

then running this:

yarn test
click to expand output for yarn test 
yarn run v1.22.19
$ bazel test :tests --test_output=all
INFO: Analyzed 6 targets (1 packages loaded, 158 targets configured).
INFO: Found 6 test targets...
FAIL: //tfjs-core:tfjs-core_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_test/test.log)
INFO: From Testing //tfjs-core:tfjs-core_test:
==================== Test output for //tfjs-core:tfjs-core_test:




Running karma tests
-----------------------------------------------------------------------------
version     : 6.4.0
pwd         : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/142/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/tfjs-core_test.runfiles/tfjs
conf        : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/142/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/tfjs-core_test.runfiles/tfjs/tfjs-core/tfjs-core_test.conf.js
node_options:

Running with arguments 
Your karma configuration specifies 'frameworks' which will be overwritten by Bazel
01 02 2024 04:12:31.536:WARN [filelist]: All files matched by "/home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/142/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/tfjs-core_test.runfiles/tfjs/tfjs-core/src/tfjs-core_test_bundle.js" were excluded or matched by prior matchers.
01 02 2024 04:12:31.929:INFO [karma-server]: Karma v6.4.0 server started at http://localhost:9876/
01 02 2024 04:12:31.930:INFO [launcher]: Launching browsers chrome_no_sandbox with concurrency unlimited
01 02 2024 04:12:31.936:INFO [launcher]: Starting browser ChromeHeadless
01 02 2024 04:12:31.937:ERROR [launcher]: No binary for ChromeHeadless browser on your platform.
  Please, set "CHROME_BIN" env variable.
================================================================================
FAIL: //tfjs-core/src:worker_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/src/worker_test/test.log)
INFO: From Testing //tfjs-core/src:worker_test:
==================== Test output for //tfjs-core/src:worker_test:




Running karma tests
-----------------------------------------------------------------------------
version     : 6.4.0
pwd         : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/143/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/src/worker_test.runfiles/tfjs
conf        : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/143/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/src/worker_test.runfiles/tfjs/tfjs-core/src/worker_test.conf.js
node_options:

Running with arguments 
Your karma configuration specifies 'frameworks' which will be overwritten by Bazel
01 02 2024 04:12:32.583:INFO [karma-server]: Karma v6.4.0 server started at http://localhost:9877/
01 02 2024 04:12:32.584:INFO [launcher]: Launching browsers chrome_no_sandbox with concurrency unlimited
01 02 2024 04:12:32.587:INFO [launcher]: Starting browser ChromeHeadless
01 02 2024 04:12:32.588:ERROR [launcher]: No binary for ChromeHeadless browser on your platform.
  Please, set "CHROME_BIN" env variable.
================================================================================
INFO: Elapsed time: 4.962s, Critical Path: 3.67s
INFO: 5 processes: 1 internal, 4 linux-sandbox.
INFO: Build completed, 2 tests FAILED, 5 total actions
PASSED: //tfjs-core:test_snippets_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/test_snippets_test/test.log)
INFO: From Testing //tfjs-core:test_snippets_test
==================== Test output for //tfjs-core:test_snippets_test:
Parsed and evaluated 192 snippets successfully.
================================================================================
PASSED: //tfjs-core:tfjs-core_async_backends_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_async_backends_test/test.log)
INFO: From Testing //tfjs-core:tfjs-core_async_backends_test
==================== Test output for //tfjs-core:tfjs-core_async_backends_test:
Started
...................................................................................................**...................................................................*.......*......................................................................................................................................................................................................................................................................................................................................................................................**............................................................................................................................................*.......*........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Pending:

1) IS_BROWSER isBrowser: true
  Temporarily disabled with xit


2) IS_BROWSER isBrowser: false
  Temporarily disabled with xit


3) kernel_registry test-async-cpu {} errors when registering the same kernel twice
  Temporarily disabled with xit


4) gradient registry test-async-cpu {} warning when registering the same gradient twice
  Temporarily disabled with xit


5) IS_BROWSER isBrowser: true
  Temporarily disabled with xit


6) IS_BROWSER isBrowser: false
  Temporarily disabled with xit


7) kernel_registry test-async-cpu {} errors when registering the same kernel twice
  Temporarily disabled with xit


8) gradient registry test-async-cpu {} warning when registering the same gradient twice
  Temporarily disabled with xit

4534 specs, 0 failures, 8 pending specs
Finished in 9.266 seconds
================================================================================
PASSED: //tfjs-core:tfjs-core_node_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_node_test/test.log)
INFO: From Testing //tfjs-core:tfjs-core_node_test
==================== Test output for //tfjs-core:tfjs-core_node_test:
Started
.....................................Relu                       0.2787399999797344ms    1D 4                    4       x: 1D 4 
.....RealDiv                    0.4682040000334382ms    1D 4                    4       a: 1D 4 b: 1D 4 
.Pow                            0.12125599943101406ms   1D 2                    2       a: 1D 2 b: 1D 2 
....*....................................................**............................................................................................................................................*.......*.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Pending:

1) debug on cpu {} A x B
  Temporarily disabled with xit


2) IS_BROWSER isBrowser: true
  Temporarily disabled with xit


3) IS_BROWSER isBrowser: false
  Temporarily disabled with xit


4) kernel_registry cpu {} errors when registering the same kernel twice
  Temporarily disabled with xit


5) gradient registry cpu {} warning when registering the same gradient twice
  Temporarily disabled with xit

4132 specs, 0 failures, 5 pending specs
Finished in 8.293 seconds
================================================================================
PASSED: //tfjs-core/src:worker_node_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/src/worker_node_test/test.log)
INFO: From Testing //tfjs-core/src:worker_node_test
==================== Test output for //tfjs-core/src:worker_node_test:
Randomized with seed 72723
Started

============================
Hi, looks like you are running TensorFlow.js in Node.js. To speed things up dramatically, install our node backend, visit https://github.com/tensorflow/tfjs-node for more details. 
============================
.


1 spec, 0 failures
Finished in 1.278 seconds
Randomized with seed 72723 (jasmine --random=true --seed=72723)
================================================================================
//tfjs-core:test_snippets_test                                  (cached) PASSED in 5.3s
//tfjs-core:tfjs-core_async_backends_test                       (cached) PASSED in 12.5s
//tfjs-core:tfjs-core_node_test                                 (cached) PASSED in 10.2s
//tfjs-core/src:worker_node_test                                (cached) PASSED in 2.5s
//tfjs-core:tfjs-core_test                                               FAILED in 2.1s
  /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_test/test.log
//tfjs-core/src:worker_test                                              FAILED in 2.4s
  /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/src/worker_test/test.log

Executed 2 out of 6 tests: 4 tests pass and 2 fail locally.
INFO: Build completed, 2 tests FAILED, 5 total actions
error Command failed with exit code 3.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

or running this:

yarn test --//:grep=multinomial
click to expand output for yarn test --//:grep=multinomial 
yarn run v1.22.19
$ bazel test :tests --test_output=all --//:grep=multinomial
INFO: Build option --//:grep has changed, discarding analysis cache.
INFO: Analyzed 6 targets (1 packages loaded, 14020 targets configured).
INFO: Found 6 test targets...
FAIL: //tfjs-core:tfjs-core_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_test/test.log)
INFO: From Testing //tfjs-core:tfjs-core_test:
==================== Test output for //tfjs-core:tfjs-core_test:




Running karma tests
-----------------------------------------------------------------------------
version     : 6.4.0
pwd         : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/147/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/tfjs-core_test.runfiles/tfjs
conf        : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/147/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/tfjs-core_test.runfiles/tfjs/tfjs-core/tfjs-core_test.conf.js
node_options:

Running with arguments --grep=multinomial
Your karma configuration specifies 'frameworks' which will be overwritten by Bazel
01 02 2024 04:15:04.434:WARN [filelist]: All files matched by "/home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/147/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/tfjs-core_test.runfiles/tfjs/tfjs-core/src/tfjs-core_test_bundle.js" were excluded or matched by prior matchers.
01 02 2024 04:15:04.721:INFO [karma-server]: Karma v6.4.0 server started at http://localhost:9876/
01 02 2024 04:15:04.722:INFO [launcher]: Launching browsers chrome_no_sandbox with concurrency unlimited
01 02 2024 04:15:04.726:INFO [launcher]: Starting browser ChromeHeadless
01 02 2024 04:15:04.727:ERROR [launcher]: No binary for ChromeHeadless browser on your platform.
  Please, set "CHROME_BIN" env variable.
================================================================================
FAIL: //tfjs-core/src:worker_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/src/worker_test/test.log)
INFO: From Testing //tfjs-core/src:worker_test:
==================== Test output for //tfjs-core/src:worker_test:




Running karma tests
-----------------------------------------------------------------------------
version     : 6.4.0
pwd         : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/146/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/src/worker_test.runfiles/tfjs
conf        : /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/sandbox/linux-sandbox/146/execroot/tfjs/bazel-out/k8-fastbuild/bin/tfjs-core/src/worker_test.runfiles/tfjs/tfjs-core/src/worker_test.conf.js
node_options:

Running with arguments --grep=multinomial
Your karma configuration specifies 'frameworks' which will be overwritten by Bazel
01 02 2024 04:15:05.356:INFO [karma-server]: Karma v6.4.0 server started at http://localhost:9877/
01 02 2024 04:15:05.357:INFO [launcher]: Launching browsers chrome_no_sandbox with concurrency unlimited
01 02 2024 04:15:05.361:INFO [launcher]: Starting browser ChromeHeadless
01 02 2024 04:15:05.361:ERROR [launcher]: No binary for ChromeHeadless browser on your platform.
  Please, set "CHROME_BIN" env variable.
================================================================================
INFO: Elapsed time: 5.742s, Critical Path: 3.52s
INFO: 5 processes: 1 internal, 4 linux-sandbox.
INFO: Build completed, 2 tests FAILED, 5 total actions
PASSED: //tfjs-core:test_snippets_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/test_snippets_test/test.log)
INFO: From Testing //tfjs-core:test_snippets_test
==================== Test output for //tfjs-core:test_snippets_test:
Parsed and evaluated 192 snippets successfully.
================================================================================
PASSED: //tfjs-core:tfjs-core_async_backends_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_async_backends_test/test.log)
INFO: From Testing //tfjs-core:tfjs-core_async_backends_test
==================== Test output for //tfjs-core:tfjs-core_async_backends_test:
Started
...................................................................................................**...................................................................*.......*......................................................................................................................................................................................................................................................................................................................................................................................**............................................................................................................................................*.......*........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Pending:

1) IS_BROWSER isBrowser: true
  Temporarily disabled with xit


2) IS_BROWSER isBrowser: false
  Temporarily disabled with xit


3) kernel_registry test-async-cpu {} errors when registering the same kernel twice
  Temporarily disabled with xit


4) gradient registry test-async-cpu {} warning when registering the same gradient twice
  Temporarily disabled with xit


5) IS_BROWSER isBrowser: true
  Temporarily disabled with xit


6) IS_BROWSER isBrowser: false
  Temporarily disabled with xit


7) kernel_registry test-async-cpu {} errors when registering the same kernel twice
  Temporarily disabled with xit


8) gradient registry test-async-cpu {} warning when registering the same gradient twice
  Temporarily disabled with xit

4534 specs, 0 failures, 8 pending specs
Finished in 9.266 seconds
================================================================================
PASSED: //tfjs-core:tfjs-core_node_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_node_test/test.log)
INFO: From Testing //tfjs-core:tfjs-core_node_test
==================== Test output for //tfjs-core:tfjs-core_node_test:
Started
.....................................Relu                       0.2787399999797344ms    1D 4                    4       x: 1D 4 
.....RealDiv                    0.4682040000334382ms    1D 4                    4       a: 1D 4 b: 1D 4 
.Pow                            0.12125599943101406ms   1D 2                    2       a: 1D 2 b: 1D 2 
....*....................................................**............................................................................................................................................*.......*.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Pending:

1) debug on cpu {} A x B
  Temporarily disabled with xit


2) IS_BROWSER isBrowser: true
  Temporarily disabled with xit


3) IS_BROWSER isBrowser: false
  Temporarily disabled with xit


4) kernel_registry cpu {} errors when registering the same kernel twice
  Temporarily disabled with xit


5) gradient registry cpu {} warning when registering the same gradient twice
  Temporarily disabled with xit

4132 specs, 0 failures, 5 pending specs
Finished in 8.293 seconds
================================================================================
PASSED: //tfjs-core/src:worker_node_test (see /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/src/worker_node_test/test.log)
INFO: From Testing //tfjs-core/src:worker_node_test
==================== Test output for //tfjs-core/src:worker_node_test:
Randomized with seed 72723
Started

============================
Hi, looks like you are running TensorFlow.js in Node.js. To speed things up dramatically, install our node backend, visit https://github.com/tensorflow/tfjs-node for more details. 
============================
.


1 spec, 0 failures
Finished in 1.278 seconds
Randomized with seed 72723 (jasmine --random=true --seed=72723)
================================================================================
//tfjs-core:test_snippets_test                                  (cached) PASSED in 5.3s
//tfjs-core:tfjs-core_async_backends_test                       (cached) PASSED in 12.5s
//tfjs-core:tfjs-core_node_test                                 (cached) PASSED in 10.2s
//tfjs-core/src:worker_node_test                                (cached) PASSED in 2.5s
//tfjs-core:tfjs-core_test                                               FAILED in 1.7s
  /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/tfjs-core_test/test.log
//tfjs-core/src:worker_test                                              FAILED in 2.4s
  /home/codespace/.cache/bazel/_bazel_codespace/3157473a0cc0695b2f14ebf7f06f1a75/execroot/tfjs/bazel-out/k8-fastbuild/testlogs/tfjs-core/src/worker_test/test.log

Executed 2 out of 6 tests: 4 tests pass and 2 fail locally.
INFO: Build completed, 2 tests FAILED, 5 total actions
error Command failed with exit code 3.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The two outputs are mostly the same, and both still say Executed 2 out of 6 tests: 4 tests pass and 2 fail locally.

i also tried this:

"test1": "bazel test :tests --//:grep=multinomial",
yarn test1

but got similar results. not sure what i'm doing wrong so far
@hchiam hchiam added the type:bug Something isn't working label Feb 1, 2024
@gaikwadrahul8 gaikwadrahul8 self-assigned this Feb 1, 2024
@hchiam hchiam mentioned this issue Feb 1, 2024
Open
@hchiam
Copy link
Author

hchiam commented Feb 1, 2024 •
edited

For reference: I also linked and posted this question in the new TF discussion forum
https://discuss.tensorflow.org/t/how-do-i-run-a-subset-of-tests/22351

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gaikwadrahul8 gaikwadrahul8

Labels
comp:core type:bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hchiam @gaikwadrahul8