Add ObjectSpace.memsize_of

[RoryO]

Hello, this is my first significant contribution. I hope the review is not a burden.

This implements the mri method of ObjectSpace.memsize_of. With mruby we care a great deal about tracking resource use therefore I feel this is useful. All of the code is new, none copied. The mruby structures differ from C Ruby structures so existing C Ruby code isn't useful.

One difference between this implementation and the mri one is an addition of a recurse parameter. Specifying ObjectSpace.memsize_of obj, recurse: true will recursively descend into collection members and instance variables. Do let me know if I should not include this option and I will remove it.

Additionally, I added a protocol for MRB_TT_DATA objects. If a data object implements the memsize method this code will call that method and use it's return value for the size total. This allows a chance for including data objects into memory calculations, a gap I noticed from mri.

[RoryO]

Here is an example of where it is useful exposing internal details. The memory size of a float is 0 with default no boxing, but is the size of an mrb_value when using word boxing.

[matz]

I missed that ObjectSpace.memsize_of does not recurse. It was originally designed to analyze single object allocation.
Do we really need recurring enhancement for mruby?

[RoryO]

If you wish I can remove it.

[matz]

I will do some fixes and refactoring to improve compatibility with CRuby. I am working on it.
Thank you!

[RoryO]

Please show me if there's anything I can do to help, including working on C Ruby as well. I do not wish to burden!

[matz]

Never mind. It's fun!

[shuujii]

I've noticed that ObjectSpace.memsize_of doesn't exist in JRuby and so on. Do we need to maintain incompatible features among Ruby implementations in the mruby repository?

ObjectSpace.memsize_of may not return an exact value, so I personally find it difficult to use. Therefore, I feel it's better to provide it as an external gem.

[matz]

Separating to a gem. Hmm. OK.

[RoryO]

ObjectSpace.memsize_of may not return an exact value

Can you provide some examples? I wanted to make it accurate along with matz' corrections. I think this is an important tool for an embedded project.

[matz]

I think he meant the result does not count alignment gaps nor allocation overhead, which are not easy to calculate.

[matz]

I made a separated gem for memsize_of. The diff is here:
https://gist.github.com/matz/0ece6c6fbeeb7609dd44e894f276ee7e
If I commit this change, we will lose @RoryO's name and history from the repository.
Maybe he can commit the above change as a PR.

[shuujii]

ObjectSpace.memsize_of may not return an exact value

Can you provide some examples? I wanted to make it accurate along with matz' corrections. I think this is an important tool for an embedded project.

Here are some examples. I'm not sure what value users typically expect or appropriate for the return value of ObjectSpace.memsize_of.

• Currently, the method returns 0 for Symbol. However, creating one non-inline symbol uses one slot in mrb_state::symtbl (which may also uses a slot in mrb_state::symhash). Also, if the symbol name is not ro_data, allocate a buffer for it.
• Currently, the method returns RVALUE size for shared String. In the case of shared Strings, I think shared buffer size (including size of mrb_shared_string) should be accounted for in one of the objects that share the same buffer. However, I don't think it's obvious which objects should be accounted for. Information may need to be added to RString to identify the object, even if it is clear. I don't think it's desirable to increase runtime costs and so on just for the feature.
• Currently, the method retuerns 0 for Fixnum. However, it may be using a slot in literal pool.

I made a separated gem for memsize_of.

My intention was to provide it in a different repository than mruby/mruby (e.g. RoryO/mruby-os-memsize).

I personally feel that return value definition of the method is unclear. The meaning of the value may change due to changes in internal implementation because it strongly depends on the internal implementation.

Still, the methods may have some uses depending on the use case, but I'm a little wondering if it is necessary to officially provide such a seemingly unclear feature (also, I'm a little worried that the maintenance cost seems to be high against the frequency of use).

I agree with separating gems when maintaining in mruby/mruby because the method is required require 'objspace' in CRuby.

[matz]

Considering the fact memsize_of is that highly depends on the mruby internal, it should be maintained mainly by the core developer (i.e. me).

[shuujii]

• The result seems to be incorrect.

Example (64-bit no-boxing):

Hash Size	Expected Result	Actual Result
1	184	72
8	360	184
16	600	312
24	1176	1464
32	1800	2616

• IV table size should also be included.

• The result may not fit into mrb_int.

[RoryO]

Thank you for this example! Do you mean that a Hash object should also include mrb_obj_iv_tbl_memsize, like this

--- a/mrbgems/mruby-os-memsize/src/memsize.c
+++ b/mrbgems/mruby-os-memsize/src/memsize.c
@@ -74,6 +74,7 @@ os_memsize_of_object(mrb_state* mrb, mrb_value obj)
     }
     case MRB_TT_HASH: {
       size += mrb_objspace_page_slot_size() +
+              mrb_obj_iv_tbl_memsize(mrb, obj) +
               mrb_os_memsize_of_hash_table(obj);
       break;
     }

Do you mean the numeric limit of mrb_int? That has a minimum limit of INT32_MAX. I suppose it's possible for one object consuming 2gb of memory.

[shuujii]

Do you mean that a Hash object should also include mrb_obj_iv_tbl_memsize, like this

Since IV table is part of a Hash object, I think it's natural that its memory size be included in the result of mrb_os_memsize_of_hash_table().

However, in CRuby, for example, the result of rb_ary_memsize() seems not to include memory size of IV table, so I think that it will look like the code you presented if follow CRuby.

Do you mean the numeric limit of mrb_int?

Yes. I have no opinion on the behavior when it exceeds MRB_INT_MAX, but I'd like to know if it is not taken into consideration or if it is considered and fixed.