Bug: Memory Leak

[FireLemons]

Impacted User Types

Tech Leads

Environment

production

Current Behavior

The rise after the restart probably comes from people using the app at 10PM. But the memory usage keeps climbing well after users go to sleep.

How to access the QA site

Login Details:
Link to QA site

Test Environment Login Emails:

• volunteer1@example.com view site as a volunteer
• supervisor1@example.com view site as a supervisor
• casa_admin1@example.com view site as an admin
• all_casa_admin1@example.com view site as an all casa admin
  • go to /all_casa_admins/sign_in

password for all users: 12345678

Questions? Join Slack!

We highly recommend that you join us in slack #casa channel to ask questions quickly. And discord for office hours (currently Tuesday 5-7pm Pacific), stakeholder news, and upcoming new issues.

[FireLemons]

2026-02-12T00:17:11.290187+00:00 app[web.1]: [Scout] [02/12/26 00:17:11 +0000 web.1 (2)] INFO : [00:15] Delivering 2 Metrics for 0 requests and 0 Slow Transaction Traces and 0 Job Traces, Recorded from 1 processes.
2026-02-12T00:17:11.295284+00:00 app[web.1]: [Scout] [02/12/26 00:17:11 +0000 web.1 (2)] INFO : Posting payload to "https://checkin.scoutapp.com"
2026-02-12T00:17:22.481095+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:17:22.482971+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:17:44.411869+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:17:44.413846+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:18:06.892520+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:18:06.897222+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:18:11.288600+00:00 app[web.1]: [Scout] [02/12/26 00:18:11 +0000 web.1 (2)] INFO : [00:16] Delivering 2 Metrics for 0 requests and 0 Slow Transaction Traces and 0 Job Traces, Recorded from 1 processes.
2026-02-12T00:18:11.293860+00:00 app[web.1]: [Scout] [02/12/26 00:18:11 +0000 web.1 (2)] INFO : Posting payload to "https://checkin.scoutapp.com"
2026-02-12T00:18:28.578427+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:18:28.579983+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:18:50.571769+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:18:50.573306+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:19:11.288389+00:00 app[web.1]: [Scout] [02/12/26 00:19:11 +0000 web.1 (2)] INFO : [00:17] Delivering 2 Metrics for 0 requests and 0 Slow Transaction Traces and 0 Job Traces, Recorded from 1 processes.
2026-02-12T00:19:11.294307+00:00 app[web.1]: [Scout] [02/12/26 00:19:11 +0000 web.1 (2)] INFO : Posting payload to "https://checkin.scoutapp.com"
2026-02-12T00:19:13.039740+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:19:13.041964+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:19:34.377111+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:19:34.379445+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:19:56.612199+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:19:56.614504+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:20:11.288791+00:00 app[web.1]: [Scout] [02/12/26 00:20:11 +0000 web.1 (2)] INFO : [00:18] Delivering 2 Metrics for 0 requests and 0 Slow Transaction Traces and 0 Job Traces, Recorded from 1 processes.
2026-02-12T00:20:11.291608+00:00 app[web.1]: [Scout] [02/12/26 00:20:11 +0000 web.1 (2)] INFO : Posting payload to "https://checkin.scoutapp.com"
2026-02-12T00:20:19.058237+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:20:19.059792+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:20:40.501531+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:20:40.503653+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2026-02-12T00:21:02.479711+00:00 heroku[web.1]: Process running mem=558M(108.3%)
2026-02-12T00:21:02.481582+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

It might be scout because it seems like the only background process but the memory was already high before the logs started so inconclusive really.

[FireLemons]

We have a bot in discord that takes snapshots of the app's memory stats. Here is a breakdown of all the fields in the hash it prints:

time: Total time spent performing garbage collection in milliseconds
marking_time: Time spent in the "marking" phase of garbage collection in milliseconds. Marking is the process of traversing live objects and marking the ones in use.
sweeping_time: Time spent in the "sweeping" phase of garbage collection in milliseconds. Sweeping is the process of freeing the memory used by objects that haven't been marked during the marking phase.

minor_gc_count: the count of a limited but fast garbage collection
major_gc_count: the count of a slower more complete garbage collection
count: minor_gc_count + major_gc_count

"Heap" here refers to a region of memory reserved for the application not to be confused with the tree based data structure of the same name. It keeps records to all live ruby objects.

A heap is divided into pages.

heap_allocated_pages: the number of pages in use
heap_allocatable_pages: the number of pages not in use
heap_sorted_length: the size of the heap in pages or heap_allocatable_pages + heap_allocated_pages

Each heap page contains many slots. One slot holds one ruby object.

heap_live_slots: the number of occupied slots
heap_free_slots: the number of unoccupied slots
heap_available_slots: the total number of slots across the entire heap or heap_free_slots + heap_live_slots
heap_marked_slots: the number of slots marked after the most recent marking phase
heap_eden_pages: The number of pages that contain at least one live object
heap_tomb_pages: The number of pages that contain no live objects. These may potentially be returned to the OS.

heap_final_slots: the number of slots with finalizers attached to them. Finalizers are procs that trigger when an object is freed.

These next field names are somewhat misnomers. They're not counts of resources but rather cumulative counts of actions performed on the type of resource.
total_allocated_pages: The total count of page allocations
total_freed_pages: The total count of page frees
total_allocated_objects: The total count of object allocations
total_freed_objects: The total count of object frees

malloc_increase_bytes: A fluctuating value. Every time the process wants to allocate more memory this value goes up to minimize the amount of memory allocations needed to allocate memory.
malloc_increase_bytes_limit: The limit for malloc_increase_bytes. When malloc_increase_bytes pushes against this limit, a major garbage collection is triggered and malloc_increase_bytes is reset to a low value.

old_objects: The number of objects marked as old. Old objects survive multiple garbage collection passes.
old_objects_limit: When old objects passes this limit, a major garbage collection is triggered.

oldmalloc_increase_bytes: The same as malloc_increase_bytes but limited to objects that have survived a few garbage collection passes.
oldmalloc_increase_bytes_limit: The same as malloc_increase_bytes_limit but limited to objects that have survived a few garbage collection passes

compact_count: The number of times a compact was run. Compacting is defragmenting the heap. It's not beneficial for most processes and turned off by default.

read_barrier_faults: Read barriers are used while the garbage collector is moving objects while compacting to prevent the reading of an old reference to an object.
total_moved_objects: The cumulative number of object moves by compaction since the process started. Again this is not the count of a resource but rather an action.

Some context for these next 2: The remembered set is a list of old objects which have a reference to a new object. The write-barrier is simply a interface between the Ruby runtime and an object, so that we can track references to and from the object when they’re created. C-extensions can create new references to objects without going through the write-barrier, so objects which have been touched by C-extensions are called “write-barrier unprotected”*. The garbage collector must therefore keep track of these objects so that it doesn't delete a new object that is referenced by an old object by accident.

remembered_wb_unprotected_objects: The count of write barrier unprotected objects that are currently in the remembered set.
remembered_wb_unprotected_objects_limit: When remembered_wb_unprotected_objects pushes against this limit, a major garbage collection is performed.

Also worth noting:

The contents of the hash are implementation-specific and may change in the future without notice.

Sources:

• https://www.speedshop.co/blog/a-guide-to-gc-stat/
• https://docs.ruby-lang.org/en/3.4/GC.html

[github-actions]

This issue has been inactive for 242 hours (10.08 days) and will be unassigned after 118 more hours (4.92 days). If you have questions, please

If you are still working on this, comment here to tell the bot to give you more time

[FireLemons]

dsadadadsadsadsad

[compwron]

dsadadadsadsadsad

?