Skip to content

Swarming: Tag metrics with swarming runtime.#5248

Merged
IvanBM18 merged 10 commits intomasterfrom
feature/swarming/improve_metrics
May 5, 2026
Merged

Swarming: Tag metrics with swarming runtime.#5248
IvanBM18 merged 10 commits intomasterfrom
feature/swarming/improve_metrics

Conversation

@IvanBM18
Copy link
Copy Markdown
Collaborator

@IvanBM18 IvanBM18 commented Apr 20, 2026

Currently if we run fuzzing sessions on swarming, the metrics dont seem to register those fuzzing hours correctly:
image
Note that the swarming fuzzing hours appear as empty: runtime=

  • This PR adds swarming as a possible runtime
  • Adds swarming in the error count metric
  • Now that theres no circular dependency between logs <-> environment modules, i removed some methods that were previously declared twice to avoid said circular dependency, now they are correctly imported from a single module
  • Also adds unit tests

Tests performed

Changes present in dev since before the last dev branch reset(which is around 22th of April).

Worth noting error logs in dev env:

Heres a list of logs that came to my attention:

There is a recurring PERMISSION_DENIED exception in the testcase_manager.py. It specifically points to "Missing or insufficient permissions" when calling the Datastore API via gRPC.

method: "_do_run_testcase_and_return_result_in_queue"
path: "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/bot/testcase_manager.py"
}
message: "Exception occurred while running run_testcase_and_return_result_in_queue.
Traceback (most recent call last):
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/_datastore_api.py", line 98, in rpc_call
    result = yield rpc
             ^^^^^^^^^
	status = StatusCode.PERMISSION_DENIED
	details = "Missing or insufficient permissions."
	debug_error_string = "UNKNOWN:Error received "
>

Not related since we are not making any changes nor to auth or to the data store api.

raise RuntimeError('Request to %s failed. Code: %d. Reason: %s' %
RuntimeError: Request to https://storage.googleapis.com....
<Error><Code>SignatureDoesNotMatch</Code><Message>Access denied.</Message><Details>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Details>

A weird error group, seems related to missing permissions or mismatch across service accounts, may be worth looking, or may come from other tests that someone is performing in dev

@IvanBM18 IvanBM18 self-assigned this Apr 20, 2026
@IvanBM18 IvanBM18 added the swarming Changes related to the clusterfuzz-swarming integration label Apr 20, 2026
Comment thread src/clusterfuzz/_internal/bot/tasks/utasks/__init__.py
Comment thread src/clusterfuzz/_internal/metrics/logs.py Outdated
Comment thread src/clusterfuzz/_internal/metrics/logs.py Outdated
@IvanBM18 IvanBM18 marked this pull request as ready for review April 28, 2026 04:14
@IvanBM18 IvanBM18 requested a review from a team as a code owner April 28, 2026 04:14
Comment thread src/clusterfuzz/_internal/system/environment.py Outdated
_default_extras = {}


def _is_running_on_k8s():
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we dont have a circular dependency we can start using the same method that its declared on environment.py

os.getenv('SERVER_SOFTWARE', '').startswith('Development/'))


def _is_running_on_app_engine():
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above:
Now that we dont have a circular dependency we can start using the same method that its declared on environment.py

@jardondiego
Copy link
Copy Markdown
Collaborator

It looks like we have failing tests. Can you take a look?

Copy link
Copy Markdown
Collaborator

@jardondiego jardondiego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Collaborator

@letitz letitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % nits.

Can you adjust the PR title? It does not describe what the change fixes. How about "Swarming: tag fuzzing hours metrics with correct runtime."?

Comment thread src/clusterfuzz/_internal/tests/core/system/environment_test.py Outdated
@IvanBM18 IvanBM18 changed the title Swarming: Fuzzing hours to be logged in metrics Swarming: Tag fuzzing hours metrics with correct runtime. Apr 30, 2026
@IvanBM18 IvanBM18 changed the title Swarming: Tag fuzzing hours metrics with correct runtime. Swarming: Tag fuzzing hours metrics with swarming runtime. Apr 30, 2026
@IvanBM18 IvanBM18 changed the title Swarming: Tag fuzzing hours metrics with swarming runtime. Swarming: Tag metrics with swarming runtime. Apr 30, 2026
Copy link
Copy Markdown
Collaborator

@javanlacerda javanlacerda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Single question I have is if with this changes, the fuzzing hours for swarming are with the swarming label?

Comment thread src/clusterfuzz/_internal/metrics/logs.py
@IvanBM18 IvanBM18 merged commit 7db57f1 into master May 5, 2026
10 of 11 checks passed
@IvanBM18 IvanBM18 deleted the feature/swarming/improve_metrics branch May 5, 2026 03:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

swarming Changes related to the clusterfuzz-swarming integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants