[Tracing] TypeError: got an unexpected keyword argument '_ray_trace_ctx' when @ray.remote is used before ray.init

[ckw017]

What happened + What you expected to happen

Tracing fails when @ray.remote is used before ray.init

Versions / Dependencies

1.13.0, any python version, any OS. Looks like this has been broken since at least 1.6

Reproduction script

ray start --head --tracing-startup-hook=ray.util.tracing.setup_local_tmp_tracing:setup_tracing

import ray

@ray.remote
def f():
    return 4

ray.init(_tracing_startup_hook="ray.util.tracing.setup_local_tmp_tracing:setup_tracing")
print(ray.get(f.remote())) # This fails

Traceback (most recent call last):
  File "expensive_script_test.py", line 8, in <module>
    print(ray.get(f.remote()))
  File "/Users/cwong/anaconda3/envs/ray1130/lib/python3.7/site-packages/ray/remote_function.py", line 111, in _remote_proxy
    return self._remote(args=args, kwargs=kwargs, **self._default_options)
  File "/Users/cwong/anaconda3/envs/ray1130/lib/python3.7/site-packages/ray/util/tracing/tracing_helper.py", line 321, in _invocation_remote_span
    return method(self, args, kwargs, *_args, **_kwargs)
  File "/Users/cwong/anaconda3/envs/ray1130/lib/python3.7/site-packages/ray/remote_function.py", line 319, in _remote
    return invocation(args, kwargs)
  File "/Users/cwong/anaconda3/envs/ray1130/lib/python3.7/site-packages/ray/remote_function.py", line 288, in invocation
    self._function_signature, args, kwargs
  File "/Users/cwong/anaconda3/envs/ray1130/lib/python3.7/site-packages/ray/_private/signature.py", line 114, in flatten_args
    raise TypeError(str(exc)) from None
TypeError: got an unexpected keyword argument '_ray_trace_ctx'

import ray

ray.init(_tracing_startup_hook="ray.util.tracing.setup_local_tmp_tracing:setup_tracing")

@ray.remote
def f():
    return 4

print(ray.get(f.remote())) # this works

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[ckw017]

cc @simon-mo

[simon-mo]

noted. we probably don't have bandwidth to fix this atm. @ckw017 do you want to take a look?

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[ckw017]

Bump