Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remote coverage does not have a configuration #19781

Open
stevebarrau opened this issue Oct 10, 2023 · 6 comments
Open

Remote coverage does not have a configuration #19781

stevebarrau opened this issue Oct 10, 2023 · 6 comments
Assignees
@c-mita
Labels
coverage team-Rules-API API for writing rules/aspects: providers, runfiles, actions, artifacts type: bug untriaged

Comments

@stevebarrau
Copy link
Contributor

Description of the bug:

I am trying to get coverage to work in our RBE cluster.

Collecting the coverage works locally with:

$ bazel coverage -s --combined_report=lcov //python: trivial_test
SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system]
(cd XXX/execroot/federation_example && \
  exec env - \
    JAVA_RUNFILES=bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/remote_coverage_tools/Main.runfiles \
    PYTHON_RUNFILES=bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/remote_coverage_tools/Main.runfiles \
  bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat')
# Configuration: system
INFO: LCOV coverage report is located at /private/var/tmp/_bazel_steve/988dbe4dd5ee602e299c1f37c1a3d6b2/execroot/federation_example/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
INFO: From Coverage report generation:
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: Found 1 tracefiles.
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
INFO: Parsing file bazel-out/darwin-fastbuild/testlogs/python/trivial_test/coverage.dat
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: No gcov json file found.
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Target //python: trivial_test up-to-date:
  bazel-bin/python/trivial_test
INFO: Elapsed time: 45.101s, Critical Path: 4.29s
INFO: 15 processes: 4 disk cache hit, 10 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 15 total actions
//python: trivial_test                                            (cached) PASSED in 4.0s
  XXX/execroot/federation_example/bazel-out/darwin-fastbuild/testlogs/python/trivial_test/coverage.dat

Executed 0 out of 1 test: 1 test passes.

Running the same with RBE:

$ bazel-with-our-rbe coverage --combined_report=lcov  --experimental_fetch_all_coverage_outputs --experimental_split_coverage_postprocessing -s //python:trivial_test
....
SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system]
(cd XXX/execroot/federation_example && \
  exec env - \
    JAVA_RUNFILES=bazel-out/k8-opt-exec-BAC2A3E1/bin/external/remote_coverage_tools/Main.runfiles \
    PYTHON_RUNFILES=bazel-out/k8-opt-exec-BAC2A3E1/bin/external/remote_coverage_tools/Main.runfiles \
  bazel-out/k8-opt-exec-BAC2A3E1/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat')
# Configuration: system
ERROR: <builtin>: Coverage report generation failed: (Exit 34): FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
java.io.IOException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:244)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1499)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:269)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:244)
	at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:301)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:152)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:112)
	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:46)
	at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:152)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:961)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1128)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1086)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:160)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:93)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:519)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:827)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:323)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:161)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:562)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:365)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:648)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:178)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:149)
	at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:523)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:147)
	... 28 more
Target //python:trivial_test up-to-date:
  bazel-bin/python/trivial_test
INFO: Elapsed time: 1.548s, Critical Path: 0.67s
INFO: 3 processes: 1 remote cache hit, 2 internal.
FAILED: Build did NOT complete successfully
//python: trivial_test                                            (cached) PASSED in 1.0s
  XXX/bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat

Executed 0 out of 1 test: 1 test passes.
All tests passed but there were other errors during the build.

I am puzzled by the Configuration: system bit. IIUC the CoverageReportAction should spawn a BasicSpawn with a configuration. In our RBE implementation this means we need to manually specify the exec properties using --remote_default_platform_properties to get this to work remotely. We would like to avoid using remote_default_platform_properties as a workaround given this probably creates situations where if a spawn does not have a platform set, it will run them incorrectly.

Is this the expected behavior or is there an oversight is setting the configuration for the CoverageReportAction? It feels like either the BasicSpawn needs to inherit a platform (e.g. from the host platform or the default exec platform), or there needs to be a way to specify the coverage action's flag (e.g. a --coverage_platform flat or similar)?

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

# python/trivial.py
def foo():
    return 1
# python/trivial_test.py
import unittest

from python import trivial

class TrivialTest(unittest.TestCase):
    def test_trivial(self):
        self.assertEqual(1, trivial.foo())

Use rules_python and a wrapper for unittest to get XML outputs.

Which operating system are you running Bazel on?

macOS 12.6.3

What is the output of bazel info release?

release 6.3.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

I looked for coverage configuration to no avail.

Any other information, logs, or outputs that you want to share?

No response
@illicitonion
Copy link
Contributor

@coeuvre and @fmeum - I think you've been working at the intersection of coverage + remote execution(e.g. #16556) - any thoughts on how the merger action should be getting a platform?

@fmeum
Copy link
Collaborator

fmeum commented Oct 10, 2023 •
edited
Loading

Could you check whether #19784 fixes this? It's pretty hacky though and untested (Bazel CI passes with it). It also requires Skymeld.

@iancha1992 iancha1992 added the team-Configurability platforms, toolchains, cquery, select(), config transitions label Oct 10, 2023
@stevebarrau
Copy link
Contributor Author

I checked on 7b28c63.

The Configuration is still "system" but now our RBE platform is getting picked up in the Execution platform. We looked into how execProperties are set based on the platform and found a code path where only remoteExecutionProperties are used, and not execProperties. I filed a separate PR with the fix here: #19792

This separate PR + the changes in this PR makes coverage pass in our use case.

$ bazel-with-our-rbe coverage -s --combined_report=lcov //python:trivial_test
...
SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: CoverageReport]
(cd XXX/execroot/federation_example && \
  exec env - \
    JAVA_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \
    PYTHON_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \
  bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat')
# Configuration: system
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
ERROR: <builtin>: Coverage report generation failed: (Exit 34): FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
java.io.IOException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:241)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1490)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:286)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:259)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:156)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:116)
	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
	at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:140)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1135)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1052)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:553)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:852)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:331)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:169)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:175)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:146)
	at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:526)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:144)
	... 26 more
Target //python:trivial_test up-to-date:
  bazel-bin/python/trivial_test
INFO: Elapsed time: 248.538s, Critical Path: 119.85s
INFO: 15 processes: 3 remote cache hit, 11 internal, 1 remote.
ERROR: Build did NOT complete successfully
//python:trivial_test                                                     PASSED in 1.4s
  XXX/execroot/federation_example/bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat

Executed 1 out of 1 test: 1 test passes.
All tests passed but there were other errors during the build.
Full log 
$ cat .bazelrc | rg coverage
coverage:remote --experimental_fetch_all_coverage_outputs
coverage:remote --experimental_split_coverage_postprocessing
$ bazel-with-our-rbe info release
development version
$ bazel-with-our-rbe coverage -s --combined_report=lcov //python:trivial_test
INFO: Invocation ID: 0a2f4994-9ef5-498a-a4fd-4ef313cd9bbf
INFO: Using default value for --instrumentation_filter: "^//python[/:]".
INFO: Override the above default with --instrumentation_filter
INFO: Analyzed target //python:trivial_test (134 packages loaded, 14289 targets configured).
SUBCOMMAND: # @rules_java_builtin//toolchains:platformclasspath [action 'JavaToolchainCompileClasses external/rules_java_builtin/toolchains/platformclasspath_classes [for tool]', configuration: 09XXX61, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: JavaToolchainCompileClasses]
(cd XXX/execroot/federation_example && \
  exec env - \
  external/jdk_17_linux/bin/javac -source 8 -target 8 -Xlint:-options -d bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/rules_java_builtin/toolchains/platformclasspath_classes external/rules_java_builtin/toolchains/DumpPlatformClassPath.java)
# Configuration: 09XXX61
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
SUBCOMMAND: # @rules_java_builtin//toolchains:platformclasspath [action 'JavaToolchainCompileBootClasspath external/rules_java_builtin/toolchains/platformclasspath.jar [for tool]', configuration: 09XXX61, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: JavaToolchainCompileBootClasspath]
(cd XXX/execroot/federation_example && \
  exec env - \
  external/jdk_17_linux/bin/java -XX:+IgnoreUnrecognizedVMOptions '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.platform=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' -cp bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/rules_java_builtin/toolchains/platformclasspath_classes DumpPlatformClassPath bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/rules_java_builtin/toolchains/platformclasspath.jar external/jdk_17_linux)
# Configuration: 09XXX61
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
SUBCOMMAND: # @remote_coverage_tools//:Main [action 'Building external/remote_coverage_tools/Main.jar () [for tool]', configuration: 09XXX61, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: Javac]
(cd XXX/execroot/federation_example && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/bin:/usr/bin:/usr/local/bin \
  external/jdk_17_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.resources=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '-Dsun.io.useCanonCaches=false' -XX:-CompactStrings -Xlog:disable '-Xlog:all=warning:stderr:uptime,level,tags' -jar external/remote_java_tools/java_tools/JavaBuilder_deploy.jar @bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.jar-0.params @bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.jar-1.params)
# Configuration: 09XXX61
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
SUBCOMMAND: # //python:trivial_test [action 'Testing //python:trivial_test', configuration: a3XXX79a, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: TestRunner]
(cd XXX/execroot/federation_example && \
  exec env - \
    BAZEL_CC_COVERAGE_TOOL=GCOV \
    BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \
    CC_CODE_COVERAGE_SCRIPT=external/bazel_tools/tools/test/collect_cc_coverage.sh \
    COVERAGE_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/_coverage \
    COVERAGE_MANIFEST=bazel-out/k8-fastbuild/bin/python/trivial_test.instrumented_files \
    COVERAGE_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat \
    EXPERIMENTAL_SPLIT_XML_GENERATION=1 \
    IS_COVERAGE_SPAWN=0 \
    JAVA_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    LCOV_MERGER=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main \
    PATH=/bin:/usr/bin:/usr/sbin \
    PYTHON_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    RUNFILES_DIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    RUNTEST_PRESERVE_CWD=1 \
    RUN_UNDER_RUNFILES=1 \
    SPLIT_COVERAGE_POST_PROCESSING=1 \
    TEST_BINARY=python/trivial_test \
    TEST_INFRASTRUCTURE_FAILURE_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.infrastructure_failure \
    TEST_LOGSPLITTER_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.raw_splitlogs/test.splitlogs \
    TEST_PREMATURE_EXIT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.exited_prematurely \
    TEST_SIZE=small \
    TEST_SRCDIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    TEST_TARGET=//python:trivial_test \
    TEST_TIMEOUT=300 \
    TEST_TMPDIR=_tmp/ef05b1d4578920cbc6e63775f3dc0d1d \
    TEST_UNDECLARED_OUTPUTS_ANNOTATIONS=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/ANNOTATIONS \
    TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest \
    TEST_UNDECLARED_OUTPUTS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs \
    TEST_UNDECLARED_OUTPUTS_MANIFEST=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/MANIFEST \
    TEST_UNDECLARED_OUTPUTS_ZIP=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs/outputs.zip \
    TEST_UNUSED_RUNFILES_LOG_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.unused_runfiles_log \
    TEST_WARNINGS_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.warnings \
    TEST_WORKSPACE=federation_example \
    TZ=UTC \
    XML_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.xml \
  external/bazel_tools/tools/test/test-setup.sh external/bazel_tools/tools/test/collect_coverage.sh python/trivial_test)
# Configuration: a3XXX79a
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
SUBCOMMAND: # //python:trivial_test [action 'Testing //python:trivial_test', configuration: a3XXX79a, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: TestRunner]
(cd XXX/execroot/federation_example && \
  exec env - \
    BAZEL_CC_COVERAGE_TOOL=GCOV \
    BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \
    CC_CODE_COVERAGE_SCRIPT=external/bazel_tools/tools/test/collect_cc_coverage.sh \
    COVERAGE_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/_coverage \
    COVERAGE_MANIFEST=bazel-out/k8-fastbuild/bin/python/trivial_test.instrumented_files \
    COVERAGE_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat \
    EXPERIMENTAL_SPLIT_XML_GENERATION=1 \
    IS_COVERAGE_SPAWN=1 \
    JAVA_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    LCOV_MERGER=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main \
    PATH=/bin:/usr/bin:/usr/sbin \
    PYTHON_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    RUNFILES_DIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    RUNTEST_PRESERVE_CWD=1 \
    RUN_UNDER_RUNFILES=1 \
    SPLIT_COVERAGE_POST_PROCESSING=1 \
    TEST_BINARY=python/trivial_test \
    TEST_INFRASTRUCTURE_FAILURE_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.infrastructure_failure \
    TEST_LOGSPLITTER_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.raw_splitlogs/test.splitlogs \
    TEST_NAME=//python:trivial_test \
    TEST_PREMATURE_EXIT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.exited_prematurely \
    TEST_SHARD_INDEX=0 \
    TEST_SIZE=small \
    TEST_SRCDIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \
    TEST_TARGET=//python:trivial_test \
    TEST_TIMEOUT=300 \
    TEST_TMPDIR=_tmp/ef05b1d4578920cbc6e63775f3dc0d1d \
    TEST_TOTAL_SHARDS=0 \
    TEST_UNDECLARED_OUTPUTS_ANNOTATIONS=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/ANNOTATIONS \
    TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest \
    TEST_UNDECLARED_OUTPUTS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs \
    TEST_UNDECLARED_OUTPUTS_MANIFEST=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/MANIFEST \
    TEST_UNDECLARED_OUTPUTS_ZIP=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs/outputs.zip \
    TEST_UNUSED_RUNFILES_LOG_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.unused_runfiles_log \
    TEST_WARNINGS_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.warnings \
    TEST_WORKSPACE=federation_example \
    TZ=UTC \
    XML_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.xml \
  external/bazel_tools/tools/test/collect_coverage.sh)
# Configuration: a3XXX79a
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: CoverageReport]
(cd XXX/execroot/federation_example && \
  exec env - \
    JAVA_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \
    PYTHON_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \
  bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat')
# Configuration: system
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
ERROR: <builtin>: Coverage report generation failed: (Exit 34): FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
java.io.IOException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:241)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1490)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:286)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:259)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:156)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:116)
	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
	at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:140)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1135)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1052)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:553)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:852)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:331)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:169)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:175)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:146)
	at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:526)
	at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:144)
	... 26 more
Target //python:trivial_test up-to-date:
  bazel-bin/python/trivial_test
INFO: Elapsed time: 248.538s, Critical Path: 119.85s
INFO: 15 processes: 3 remote cache hit, 11 internal, 1 remote.
ERROR: Build did NOT complete successfully
//python:trivial_test                                                     PASSED in 1.4s
  XXX/execroot/federation_example/bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat

Executed 1 out of 1 test: 1 test passes.
All tests passed but there were other errors during the build.

@stevebarrau
Copy link
Contributor Author

stevebarrau commented Oct 11, 2023 •
edited
Loading

Note: with this PR; setting remote_execution_properties causes the following NPE:

$ bazel-with-our-rbe -s --combined_report=lcov //python:trivial_test
Running host JVM under debugger (listening on TCP port 5005).
Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server (72354) after 10 seconds ...
INFO: Invocation ID: ee7ac809-1ca1-4904-82d4-e767829cb641
INFO: Using default value for --instrumentation_filter: "^//python[/:]".
INFO: Override the above default with --instrumentation_filter
ERROR: XXX/external/our_rbe/common/platforms/BUILD.bazel:40:9: in exec_properties attribute of platform rule @our_rbe//common/platforms:ubi9-x86_64: Platform contains both remote_execution_properties and exec_properties. Prefer exec_properties over the deprecated remote_execution_properties.
ERROR: XXX/external/our_rbe/common/platforms/BUILD.bazel:40:9: Analysis of target '@our_rbe//common/platforms:ubi9-x86_64' failed
Analyzing: target //python: trivial_test (7 packages loaded, 12 targets configured)
[1 / 1] checking cached actions
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.NullPointerException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903)
	at com.google.devtools.build.lib.skyframe.BuildResultListener.getHostPlatformInfo(BuildResultListener.java:122)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.analyzeAndExecuteTargets(SkyframeBuildView.java:732)
	at com.google.devtools.build.lib.analysis.BuildView.update(BuildView.java:293)
	at com.google.devtools.build.lib.buildtool.AnalysisAndExecutionPhaseRunner.runAnalysisAndExecutionPhase(AnalysisAndExecutionPhaseRunner.java:241)
	at com.google.devtools.build.lib.buildtool.AnalysisAndExecutionPhaseRunner.execute(AnalysisAndExecutionPhaseRunner.java:139)
	at com.google.devtools.build.lib.buildtool.BuildTool.buildTargetsWithMergedAnalysisExecution(BuildTool.java:305)
	at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:173)
	at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:510)
	at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:478)
	at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:163)
	at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:116)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:664)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:244)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:621)
	at io.grpc.Context$1.run(Context.java:566)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

@fmeum
Copy link
Collaborator

fmeum commented Oct 11, 2023

I can look into cleaning up the PR after yours has been merged, but I may not get to it this month.

@katre katre added team-Remote-Exec Issues and PRs for the Execution (Remote) team and removed team-Configurability platforms, toolchains, cquery, select(), config transitions labels Oct 20, 2023
@katre
Copy link
Member

katre commented Oct 20, 2023

This is not configurability.

@joeleba joeleba added team-Rules-API API for writing rules/aspects: providers, runfiles, actions, artifacts and removed team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Oct 31, 2023
@fmeum fmeum mentioned this issue Dec 19, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c-mita c-mita

Labels
coverage team-Rules-API API for writing rules/aspects: providers, runfiles, actions, artifacts type: bug untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@katre @illicitonion @c-mita @fmeum @joeleba @sgowroji @stevebarrau @Pavank1992 @iancha1992