[Tracing] Create the events handler for sending bug notifications

[ViniciustCosta]

Background

In order to make ClusterFuzz execution less opaque to customers, we intend to send them notifications about the testcases lifecycle transitions. These transitions are traced by the events being emitted throughout the code and the notifications are done by adding comments in the testcase assigned buganizer issue (or any issue tracker).
b/420470379

Implementation

Implemented an event handler responsible for retrieving the testcase datastore entity from the event data and querying for the correspondent issue in the associated issue tracker. Then, using the issue object interface, adding a new comment in the assigned bug (if available) to notify the maintainers about clusterfuzz events.

Comment messages contain the event type and all the event data available (unset fields are omitted).

This handler was added to the default handlers used by events emit method. Thus, every emit call may generate a new issue comment if enabled.

Disable Notifications Config

This handler also allows defining a configuration in the project config to disable certain event types, or even specific tasks of an event type, from sending notifications. This is useful to enable customizations based on each project needs and avoid bothering our customers with unnecessary or unwanted event information. The config should look like this:

events:
  notification:
    enabled: true
    disabled_events:
      testcase_creation: true
      testcase_rejection:
        - analyze

Note that, for disabling tasks, it relies on the task name from the event metadata (usually retrieved from the CF_TASK_NAME env var).

Notifications Spam

There are some potential failure states that may result in spamming event notifications. For instance, these can be caused by stuck testcases that keep retrying failing tasks or by ClusterFuzz publishing repeated messages in task queues and executing the "same task" multiple times. This is not addressed for now and we may need a kind of throttling to avoid these, if they ever become a real problem. Also, this handler will be disabled until we are confident on the correctness of the events being emitted.

Tests

Unit tests

• Correctly parsing and disabling the notifications based on the events disabled config.
• Correctly handling the multiple fail cases that avoid emitting the issue notifications/comment (e.g., missing testcase, missing bug information in testcase, missing issue, etc.).
• Correctly executing the event handler's emit, which sends the issue notification for an event occurrence.
• Correctly configuring and retrieving the notification event handler in the top-level emit function.

Runtime test

To ensure the correct behavior at runtime, we opened an issue in the OSS-Fuzz issue tracker component and implemented a script (as a run_task.py) to be executed locally against the debugger using butler.py run. We ran impersonating the clusterfuzz appegine service account for the necessary permissions.

The script creates a generic testcase and uploads it to datastore, setting its properties and assigning it the debug issue (https://g-issues.oss-fuzz.com/issues/426517098). Then, it emits a testcase creation event and deletes the testcase afterwards.
Outcome:

• Issue comment:
  

• Event created in Datastore:
  

[vitorguidi]

lgtm, run this against a real testcase with the debugger to make sure it behaves as we expect at runtime