[tune] AttributeError: 'NoneType' object has no attribute 'get_global_worker'

[vakker]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Ray installed from (source or binary): binary
• Ray version: 0.8.0.dev1
• Python version: 3.7
• Exact command to reproduce:

import ray
from ray import tune

ray.init()


def train_func(config, reporter):  # add a reporter arg
    accuracy = config["momentum"] * 2 + 0.3
    reporter(mean_accuracy=accuracy)  # report metrics


all_trials = tune.run(
    train_func,
    name="quick-start",
    stop={"mean_accuracy": 99},
    config={"momentum": tune.grid_search([0.1, 0.2])})

Describe the problem

When I run the script above (based on the examples from here), I get the following error at the end of each run:

Exception ignored in: <function ActorHandle.__del__ at 0x7f3aeaffc730>
Traceback (most recent call last):
  File "/home/user/miniconda3/lib/python3.7/site-packages/ray/actor.py", line 613, in __del__
AttributeError: 'NoneType' object has no attribute 'get_global_worker'

However, this is not happening for 0.7.0.dev2, the process exists normally in that case.

This might be related to #4467 .

[richardliaw]

Hm, somehow I'm unable to reproduce this (I'm using conda with python37 and the latest snapshot of Ray on AWS).

Usually this exception is just thrown because of some race condition and is harmless. Are you seeing this affect correctness?

[vakker]

That's weird. Are you using: https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev1-cp37-cp37m-manylinux1_x86_64.whl ?

I can also reproduce it in a Docker container that I'm using. I can trim it down and share it if that helps.

It does not affect me (or at least I cannot tell), I was just concerned that there might be an issue.

[heurainbow]

I have exactly the same issue. When the program exits, it pops with
Exception ignored in: <function ActorHandle.__del__ at 0x7fb8166f9ea0> Traceback (most recent call last): File "XXX/.local/lib/python3.7/site-packages/ray/actor.py", line 613, in __del__ AttributeError: 'NoneType' object has no attribute 'get_global_worker'

By the way, I am using https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev1-cp37-cp37m-manylinux1_x86_64.whl.

[vakker]

@richardliaw
Doing a bit of debugging it seems to be related to the Tensorboard logging.
I used the Dockerfile:

FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04

RUN apt-get update && apt-get install -y --no-install-recommends wget

RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && \
  bash ./miniconda.sh -b -p /opt/conda
ENV PATH "/opt/conda/bin:$PATH"

RUN conda install -y -c conda-forge tensorflow==1.13.1 pandas
RUN pip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev2-cp37-cp37m-manylinux1_x86_64.whl

WORKDIR /
COPY test.py .

with test.py (same as above):

import ray
from ray import tune

ray.init()


def train_func(config, reporter):  # add a reporter arg
    accuracy = config["momentum"] * 2 + 0.3
    reporter(mean_accuracy=accuracy)  # report metrics


all_trials = tune.run(
    train_func,
    name="quick-start",
    stop={"mean_accuracy": 99},
    config={"momentum": tune.grid_search([0.1, 0.2])})

Running python /test.py does give the error at the end, but running export RLLIB_TEST_NO_TF_IMPORT=1; python /test.py (that just disables the TB logging according to here) does not give the error.

Thanks

[jiahao87]

Same issue as well, using ray-0.8.1-cp36-cp36m-manylinux1_x86_64.whl. But if I did not pip install tensorflow in the cluster config file under setup_commands, the issue disappears.

setup_commands:
    # - pip install ray torch torchvision tensorflow tabulate tensorboard
    - pip install ray torch torchvision tabulate tensorboard

[richardliaw]

Hm, interesting. Anyways, we've removed Tensorboard (replaced with TensorboardX) from Tune, so this should go away now.

[ShamCondor]

Hm，I'm also removed tensorflow, so this issue go away now