[core][enable_task_events] Options to disable task tracing on task/actor

[rickyyx]

Why are these changes needed?

There are requests to disable task events reporting at the actor/task level to avoid excessive data/communication:

• Closes [Core] A way to disable RAY_task_events_report_interval_ms for particular actor. #42076
• Closes [Core] Support not reporting task events for specified tasks. #42113

This PR will allow users to disable task events reporting (both task status events + profile events) for a task/actor through a enable_task_events=False flag on the remote for actor and task:

@ray.remote
def f():
    pass

# For task
f.options(enable_task_events=False).remote()

@ray.remote
class Actor():
    def g(self):
       pass


# For entire actor
a = Actor.options(enable_task_events=False).remote()

# Or for the specific actor task 
a.g.options(enable_task_events=False).remote()

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

Please update the related issue! Also why is this draft? (is this ready to be reviewd? )

[rickyyx]

Please update the related issue! Also why is this draft? (is this ready to be reviewd? )

Yeah, i would love to get a quick round of review for the high-level approach first.

[MissiontoMars]

Will the "_report_task_events" set for the actor task override the "_report_task_events" set during the creation of the actor?
For example, in the following code:

@ray.remote
class Actor():
    def g(self):
       pass


a = Actor.options(_report_task_events=False).remote()

a.g.options(_report_task_events=True).remote()

Does the actor task g report task events or not?
@rickyyx

[rickyyx]

Will the "_report_task_events" set for the actor task override the "_report_task_events" set during the creation of the actor? For example, in the following code:

@ray.remote
class Actor():
    def g(self):
       pass


a = Actor.options(_report_task_events=False).remote()

a.g.options(_report_task_events=True).remote()

Does the actor task g report task events or not? @rickyyx

Yes, an actor task level config would override the actor level config.

[rkooo567]

Approach lgtm

[rkooo567]

is this PR ready?

[rickyyx]

is this PR ready?

not yet - iterating on this.

[rickyyx]

not too sure about the default here.

[rickyyx]

Most of changes are just plumbing to add the option.

Major changes are in /core_worker for the task event buffer refactoring and changes.

[rkooo567]

In general LGTM. NIT questions

[rkooo567]

Approved given we choose the arguments discussed in api review

[rkooo567]

@rickyyx let's make sure this is merged asap. It is required for 2.10. (cc @jjyao)

I think the blocker is the doc approval.

[angelinalg]

Just some nits.

[rickyyx]

@rkooo567 can you take a quick look at the delta in the commits?

I made a few changes to make tests passed

• ActorHandle has to have this enforce_task_events flag as well otherwise remote actors would not work
• Add a global RAY_enable_task_events to enforce if task events would be reported from the entire ray cluster (rather than relying on RAY_task_events_report_interval) -> we could maybe refactor all task events related things into a config struct in the future.

[rkooo567]

ActorHandle has to have this enforce_task_events flag as well otherwise remote actors would not work

You mean enable_task_events right? This makes sense

Add a global RAY_enable_task_events to enforce if task events would be reported from the entire ray cluster (rather than relying on RAY_task_events_report_interval) -> we could maybe refactor all task events related things into a config struct in the future.

Hmm I don't understand why we need this?

[rickyyx]

Hmm I don't understand why we need this?

It's just a better name than the RAY_task_events_report_interval, and it provides a way to workers get default behaviour of if task events should be reported.

[rkooo567]

It's just a better name than the RAY_task_events_report_interval, and it provides a way to workers get default behaviour of if task events should be reported.

Hmm I feel like we should make it just orthogonal. It makes things confusing if we start coupling user-level config and system level config?

• True by default.
• False if you don't want
• although it is True, if user sets systems to not report task, it is disabled. (just use existing flag)

Btw, it is not a hard blocker.

[rickyyx]

It's just a better name than the RAY_task_events_report_interval, and it provides a way to workers get default behaviour of if task events should be reported.

Hmm I feel like we should make it just orthogonal. It makes things confusing if we start coupling user-level config and system level config?

• True by default.
• False if you don't want
• although it is True, if user sets systems to not report task, it is disabled. (just use existing flag)

Btw, it is not a hard blocker.

This is actually exactly how it's behaved in the PR, with the exception of not using existing flag (i.e. RAY_task_events_report_interval):
If users to set system config for not reporting, they would:

• In PR: set RAY_enable_tasks_events=False
• In master: set RAY_task_events_report_interval=0

I am also open to keeping with existing flags.

[rickyyx]

Updated with using existing flags.