[core] How to obtain memory occupied by object ref?

[Hoeze]

Hi, is there an easy way to get the memory occupied by some object ref?
E.g. ray.sizeof(ray.put(object))

[stephanie-wang]

There's no way to do this programmatically, but it would be an easy enough feature to add. Since ObjectRefs are also futures, we would have to return None or something similar if the size is unavailable, though.

[rkooo567]

Hey @Hoeze Can you also explain the use case here? Are you using it for debugging your Dask on Ray workload?

[Hoeze]

I tried to find out why my cross-validation method caused 30GB increase of memory usage.
My first guess was that the returned lightgbm models are large, so I wanted to check this.

Since ray has to track object sizes anyway, I was just curious whether it could tell me the object size.
For now, I joblib.dumped it to disk (the resulting file occupies then 4.7GB).

[rkooo567]

For now, you can achieve this using ray memory command (it is not programmable api tho). I can take a look if there’s a good way to support this.

[jmoralez]

Hi. I implemented this following what ray.dashboard.memory_utils.memory_summary does by getting the core worker stats and looking through the objectRefs and returning the objectSize of the one that matches the hex. Is that a good approach?

[rkooo567]

@jmoralez I think right now you can obtain the information using ray.experimental.get_object_locations. Note that this is an experimental API, and the semantics could be changed.

In [3]: @ray.remote 
   ...: def f(): 
   ...:     return 4 
   ...:                                                                                                               

In [4]: r = f.remote()                                                                                                

In [6]: ray.experimental.get_object_locations([r])                                                                    
Out[6]: 
{ObjectRef(a67dc375e60ddd1affffffffffffffffffffffff0100000001000000): {'node_ids': [],
  'object_size': 15}}

[mlguruz]

@jmoralez I think right now you can obtain the information using ray.experimental.get_object_locations. Note that this is an experimental API, and the semantics could be changed.

In [3]: @ray.remote 
   ...: def f(): 
   ...:     return 4 
   ...:                                                                                                               

In [4]: r = f.remote()                                                                                                

In [6]: ray.experimental.get_object_locations([r])                                                                    
Out[6]: 
{ObjectRef(a67dc375e60ddd1affffffffffffffffffffffff0100000001000000): {'node_ids': [],
  'object_size': 15}}

I'm actually running into this error running the example:

my ray version is 2.2.0

[rkooo567]

Hmm did you call ray.shutdown?

[mlguruz]

Hmm did you call ray.shutdown?

No This is exactly what I run:

ray://10.28.49.186:10001 <- this is the head node ip of AWS EC2 cluster

[rkooo567]

Ah, I think this feature is probably not supported by ray client https://docs.ray.io/en/master/cluster/running-applications/job-submission/ray-client.html.

We may fix this issue by ray 2.4.