Optimize memory used by the publish calls

[jreidinger]

Problem

Agame and also yast consume quite bit of memory, so lets try to optimize it.

Solution

Add some profiling code and based on it, apply some optimization.

Optimization results ( testing on Leap15.4 so with ruby 2.5 ) just totals:

Original result:

Total allocated: 33846260 bytes (365250 objects)
Total retained:  5387014 bytes (47980 objects)

after first optimization:

Total allocated: 32843452 bytes (364422 objects)
Total retained:  5383110 bytes (47979 objects)

after second optimization:

Total allocated: 31132062 bytes (351286 objects)
Total retained:  4337427 bytes (41452 objects)

So result is roughly saved 2650 KiB of allocated and in persistent it is 1021 KiB on testing sample that just require yast and imports some yast modules.

[mvidner]

Reproduced the failure with:

rm -rf build
mkdir build
cd build
cmake ..
make -j`nproc`
../tests/yast_spec.rb

[mvidner]

BTW I've tried enabling frozen string literals to see what memory effect it has, but fast_gettext does not like it, starting here

$ ruby --enable=frozen-string-literal -r ../tests/test_helper.rb ../profiling/yast_import.rb  |& o
/usr/lib64/ruby/gems/2.5.0/gems/fast_gettext-1.6.0/lib/fast_gettext/vendor/mofile.rb:52:in `force_encoding': can't modify frozen String (FrozenError)
        from /usr/lib64/ruby/gems/2.5.0/gems/fast_gettext-1.6.0/lib/fast_gettext/vendor/mofile.rb:52:in `<class:MOFile>'
...

https://github.com/grosser/fast_gettext/blob/a4628d516cfd511bef3acc0762d453934f5a9acd/lib/fast_gettext/vendor/mofile.rb#L52

[mvidner]

First, frozen string literals are possible if I apply this diff

--- /usr/lib64/ruby/gems/2.5.0/gems/fast_gettext-1.6.0/lib/fast_gettext/vendor/mofile.rb.orig   2023-04-03 14:03:16.711532621 +0200
+++ /usr/lib64/ruby/gems/2.5.0/gems/fast_gettext-1.6.0/lib/fast_gettext/vendor/mofile.rb        2023-04-03 14:06:38.496909306 +0200
@@ -46,11 +46,14 @@
         :trans_sysdep_tab_offset
       end
 
-      MAGIC_BIG_ENDIAN    = "\x95\x04\x12\xde"
-      MAGIC_LITTLE_ENDIAN = "\xde\x12\x04\x95"
+      mbe = "\x95\x04\x12\xde"
+      mle = "\xde\x12\x04\x95"
       if "".respond_to?(:force_encoding)
-        MAGIC_BIG_ENDIAN.force_encoding("ASCII-8BIT")
-        MAGIC_LITTLE_ENDIAN.force_encoding("ASCII-8BIT")
+        MAGIC_BIG_ENDIAN = mbe.dup.force_encoding("ASCII-8BIT")
+       MAGIC_LITTLE_ENDIAN = mle.dup.force_encoding("ASCII-8BIT")
+      else
+        MAGIC_BIG_ENDIAN = mbe
+       MAGIC_LITTLE_ENDIAN = mle
       end
 
       def self.open(arg = nil, output_charset = nil)

Then, as I was documenting it, I looked at your Exportable optimization and realized that it can be improved 😄

(I've added the under_scores manually)

before frozen path
Total allocated: 31_308_339 bytes (353285 objects)
Total retained: 4_718_083 bytes (46366 objects)

with frozen path: note that it does not affect the retained values
Total allocated: 31_289_098 bytes (352803 objects)
Total retained: 4_718_083 bytes (46366 objects)

with global string freeze (would need the above fix in fast_gettext)
Total allocated: 30_608_658 bytes (335792 objects)
Total retained: 4_685_883 bytes (45561 objects)

pairs instead of name+type hashes, without global string freeze
Total allocated: 31_116_547 bytes (352806 objects)
Total retained: 4_545_259 bytes (46366 objects)

pairs instead of name+type hashes, with global string freeze
Total allocated: 30_436_107 bytes (335795 objects)
Total retained: 4_513_059 bytes (45561 objects)

I really wonder how this performs on a real workload

[mvidner]

more improvements

[mvidner]

@jreidinger later, I think it is worth looking at the retained foo part of the memory_profiler report, but for a real workload, not just for an initial import like your test

[yast-bot]

✔️ Public Jenkins job #70 successfully finished
✔️ Created OBS submit request #1077500

[yast-bot]

✔️ Internal Jenkins job #58 successfully finished
✔️ Created IBS submit request #304735