Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
tree: 68150e7587
Fetching contributors…

Cannot retrieve contributors at this time

7692 lines (6622 sloc) 237.771 kb
/*
* sgen-gc.c: Simple generational GC.
*
* Author:
* Paolo Molaro (lupus@ximian.com)
*
* Copyright 2005-2010 Novell, Inc (http://www.novell.com)
*
* Thread start/stop adapted from Boehm's GC:
* Copyright (c) 1994 by Xerox Corporation. All rights reserved.
* Copyright (c) 1996 by Silicon Graphics. All rights reserved.
* Copyright (c) 1998 by Fergus Henderson. All rights reserved.
* Copyright (c) 2000-2004 by Hewlett-Packard Company. All rights reserved.
*
* THIS MATERIAL IS PROVIDED AS IS, WITH ABSOLUTELY NO WARRANTY EXPRESSED
* OR IMPLIED. ANY USE IS AT YOUR OWN RISK.
*
* Permission is hereby granted to use or copy this program
* for any purpose, provided the above notices are retained on all copies.
* Permission to modify the code and to distribute modified code is granted,
* provided the above notices are retained, and a notice that the code was
* modified is included with the above copyright notice.
*
*
* Copyright 2001-2003 Ximian, Inc
* Copyright 2003-2010 Novell, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining
* a copy of this software and associated documentation files (the
* "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish,
* distribute, sublicense, and/or sell copies of the Software, and to
* permit persons to whom the Software is furnished to do so, subject to
* the following conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
* LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
* OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*
*
* Important: allocation provides always zeroed memory, having to do
* a memset after allocation is deadly for performance.
* Memory usage at startup is currently as follows:
* 64 KB pinned space
* 64 KB internal space
* size of nursery
* We should provide a small memory config with half the sizes
*
* We currently try to make as few mono assumptions as possible:
* 1) 2-word header with no GC pointers in it (first vtable, second to store the
* forwarding ptr)
* 2) gc descriptor is the second word in the vtable (first word in the class)
* 3) 8 byte alignment is the minimum and enough (not true for special structures (SIMD), FIXME)
* 4) there is a function to get an object's size and the number of
* elements in an array.
* 5) we know the special way bounds are allocated for complex arrays
* 6) we know about proxies and how to treat them when domains are unloaded
*
* Always try to keep stack usage to a minimum: no recursive behaviour
* and no large stack allocs.
*
* General description.
* Objects are initially allocated in a nursery using a fast bump-pointer technique.
* When the nursery is full we start a nursery collection: this is performed with a
* copying GC.
* When the old generation is full we start a copying GC of the old generation as well:
* this will be changed to mark&sweep with copying when fragmentation becomes to severe
* in the future. Maybe we'll even do both during the same collection like IMMIX.
*
* The things that complicate this description are:
* *) pinned objects: we can't move them so we need to keep track of them
* *) no precise info of the thread stacks and registers: we need to be able to
* quickly find the objects that may be referenced conservatively and pin them
* (this makes the first issues more important)
* *) large objects are too expensive to be dealt with using copying GC: we handle them
* with mark/sweep during major collections
* *) some objects need to not move even if they are small (interned strings, Type handles):
* we use mark/sweep for them, too: they are not allocated in the nursery, but inside
* PinnedChunks regions
*/
/*
* TODO:
*) we could have a function pointer in MonoClass to implement
customized write barriers for value types
*) investigate the stuff needed to advance a thread to a GC-safe
point (single-stepping, read from unmapped memory etc) and implement it.
This would enable us to inline allocations and write barriers, for example,
or at least parts of them, like the write barrier checks.
We may need this also for handling precise info on stacks, even simple things
as having uninitialized data on the stack and having to wait for the prolog
to zero it. Not an issue for the last frame that we scan conservatively.
We could always not trust the value in the slots anyway.
*) modify the jit to save info about references in stack locations:
this can be done just for locals as a start, so that at least
part of the stack is handled precisely.
*) test/fix endianess issues
*) Implement a card table as the write barrier instead of remembered
sets? Card tables are not easy to implement with our current
memory layout. We have several different kinds of major heap
objects: Small objects in regular blocks, small objects in pinned
chunks and LOS objects. If we just have a pointer we have no way
to tell which kind of object it points into, therefore we cannot
know where its card table is. The least we have to do to make
this happen is to get rid of write barriers for indirect stores.
(See next item)
*) Get rid of write barriers for indirect stores. We can do this by
telling the GC to wbarrier-register an object once we do an ldloca
or ldelema on it, and to unregister it once it's not used anymore
(it can only travel downwards on the stack). The problem with
unregistering is that it needs to happen eventually no matter
what, even if exceptions are thrown, the thread aborts, etc.
Rodrigo suggested that we could do only the registering part and
let the collector find out (pessimistically) when it's safe to
unregister, namely when the stack pointer of the thread that
registered the object is higher than it was when the registering
happened. This might make for a good first implementation to get
some data on performance.
*) Some sort of blacklist support? Blacklists is a concept from the
Boehm GC: if during a conservative scan we find pointers to an
area which we might use as heap, we mark that area as unusable, so
pointer retention by random pinning pointers is reduced.
*) experiment with max small object size (very small right now - 2kb,
because it's tied to the max freelist size)
*) add an option to mmap the whole heap in one chunk: it makes for many
simplifications in the checks (put the nursery at the top and just use a single
check for inclusion/exclusion): the issue this has is that on 32 bit systems it's
not flexible (too much of the address space may be used by default or we can't
increase the heap as needed) and we'd need a race-free mechanism to return memory
back to the system (mprotect(PROT_NONE) will still keep the memory allocated if it
was written to, munmap is needed, but the following mmap may not find the same segment
free...)
*) memzero the major fragments after restarting the world and optionally a smaller
chunk at a time
*) investigate having fragment zeroing threads
*) separate locks for finalization and other minor stuff to reduce
lock contention
*) try a different copying order to improve memory locality
*) a thread abort after a store but before the write barrier will
prevent the write barrier from executing
*) specialized dynamically generated markers/copiers
*) Dynamically adjust TLAB size to the number of threads. If we have
too many threads that do allocation, we might need smaller TLABs,
and we might get better performance with larger TLABs if we only
have a handful of threads. We could sum up the space left in all
assigned TLABs and if that's more than some percentage of the
nursery size, reduce the TLAB size.
*) Explore placing unreachable objects on unused nursery memory.
Instead of memset'ng a region to zero, place an int[] covering it.
A good place to start is add_nursery_frag. The tricky thing here is
placing those objects atomically outside of a collection.
*/
#include "config.h"
#ifdef HAVE_SGEN_GC
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <semaphore.h>
#include <signal.h>
#include <errno.h>
#include <assert.h>
#ifdef __MACH__
#undef _XOPEN_SOURCE
#endif
#include <pthread.h>
#ifdef __MACH__
#define _XOPEN_SOURCE
#endif
#include "metadata/sgen-gc.h"
#include "metadata/metadata-internals.h"
#include "metadata/class-internals.h"
#include "metadata/gc-internal.h"
#include "metadata/object-internals.h"
#include "metadata/threads.h"
#include "metadata/sgen-cardtable.h"
#include "metadata/sgen-protocol.h"
#include "metadata/sgen-archdep.h"
#include "metadata/sgen-bridge.h"
#include "metadata/mono-gc.h"
#include "metadata/method-builder.h"
#include "metadata/profiler-private.h"
#include "metadata/monitor.h"
#include "metadata/threadpool-internals.h"
#include "metadata/mempool-internals.h"
#include "metadata/marshal.h"
#include "metadata/runtime.h"
#include "utils/mono-mmap.h"
#include "utils/mono-time.h"
#include "utils/mono-semaphore.h"
#include "utils/mono-counters.h"
#include "utils/mono-proclib.h"
#include <mono/utils/memcheck.h>
#if defined(__MACH__)
#include "utils/mach-support.h"
#endif
#define OPDEF(a,b,c,d,e,f,g,h,i,j) \
a = i,
enum {
#include "mono/cil/opcode.def"
CEE_LAST
};
#undef OPDEF
#undef pthread_create
#undef pthread_join
#undef pthread_detach
/*
* ######################################################################
* ######## Types and constants used by the GC.
* ######################################################################
*/
/* 0 means not initialized, 1 is initialized, -1 means in progress */
static gint32 gc_initialized = 0;
/* If set, do a minor collection before every X allocation */
static guint32 collect_before_allocs = 0;
/* If set, do a heap consistency check before each minor collection */
static gboolean consistency_check_at_minor_collection = FALSE;
/* If set, check that there are no references to the domain left at domain unload */
static gboolean xdomain_checks = FALSE;
/* If not null, dump the heap after each collection into this file */
static FILE *heap_dump_file = NULL;
/* If set, mark stacks conservatively, even if precise marking is possible */
static gboolean conservative_stack_mark = FALSE;
/* If set, do a plausibility check on the scan_starts before and after
each collection */
static gboolean do_scan_starts_check = FALSE;
static gboolean disable_minor_collections = FALSE;
static gboolean disable_major_collections = FALSE;
#ifdef HEAVY_STATISTICS
static long long stat_objects_alloced = 0;
static long long stat_bytes_alloced = 0;
long long stat_objects_alloced_degraded = 0;
long long stat_bytes_alloced_degraded = 0;
static long long stat_bytes_alloced_los = 0;
long long stat_copy_object_called_nursery = 0;
long long stat_objects_copied_nursery = 0;
long long stat_copy_object_called_major = 0;
long long stat_objects_copied_major = 0;
long long stat_scan_object_called_nursery = 0;
long long stat_scan_object_called_major = 0;
long long stat_nursery_copy_object_failed_from_space = 0;
long long stat_nursery_copy_object_failed_forwarded = 0;
long long stat_nursery_copy_object_failed_pinned = 0;
static long long stat_store_remsets = 0;
static long long stat_store_remsets_unique = 0;
static long long stat_saved_remsets_1 = 0;
static long long stat_saved_remsets_2 = 0;
static long long stat_local_remsets_processed = 0;
static long long stat_global_remsets_added = 0;
static long long stat_global_remsets_readded = 0;
static long long stat_global_remsets_processed = 0;
static long long stat_global_remsets_discarded = 0;
static long long stat_wasted_fragments_used = 0;
static long long stat_wasted_fragments_bytes = 0;
static int stat_wbarrier_set_field = 0;
static int stat_wbarrier_set_arrayref = 0;
static int stat_wbarrier_arrayref_copy = 0;
static int stat_wbarrier_generic_store = 0;
static int stat_wbarrier_generic_store_remset = 0;
static int stat_wbarrier_set_root = 0;
static int stat_wbarrier_value_copy = 0;
static int stat_wbarrier_object_copy = 0;
#endif
static long long stat_pinned_objects = 0;
static long long time_minor_pre_collection_fragment_clear = 0;
static long long time_minor_pinning = 0;
static long long time_minor_scan_remsets = 0;
static long long time_minor_scan_card_table = 0;
static long long time_minor_scan_pinned = 0;
static long long time_minor_scan_registered_roots = 0;
static long long time_minor_scan_thread_data = 0;
static long long time_minor_finish_gray_stack = 0;
static long long time_minor_fragment_creation = 0;
static long long time_major_pre_collection_fragment_clear = 0;
static long long time_major_pinning = 0;
static long long time_major_scan_pinned = 0;
static long long time_major_scan_registered_roots = 0;
static long long time_major_scan_thread_data = 0;
static long long time_major_scan_alloc_pinned = 0;
static long long time_major_scan_finalized = 0;
static long long time_major_scan_big_objects = 0;
static long long time_major_finish_gray_stack = 0;
static long long time_major_free_bigobjs = 0;
static long long time_major_los_sweep = 0;
static long long time_major_sweep = 0;
static long long time_major_fragment_creation = 0;
#define DEBUG(level,a) do {if (G_UNLIKELY ((level) <= SGEN_MAX_DEBUG_LEVEL && (level) <= gc_debug_level)) a;} while (0)
int gc_debug_level = 0;
FILE* gc_debug_file;
/*
void
mono_gc_flush_info (void)
{
fflush (gc_debug_file);
}
*/
/*
* Define this to allow the user to change the nursery size by
* specifying its value in the MONO_GC_PARAMS environmental
* variable. See mono_gc_base_init for details.
*/
#define USER_CONFIG 1
#define TV_DECLARE SGEN_TV_DECLARE
#define TV_GETTIME SGEN_TV_GETTIME
#define TV_ELAPSED SGEN_TV_ELAPSED
#define TV_ELAPSED_MS SGEN_TV_ELAPSED_MS
#define ALIGN_TO(val,align) ((((guint64)val) + ((align) - 1)) & ~((align) - 1))
/* The method used to clear the nursery */
/* Clearing at nursery collections is the safest, but has bad interactions with caches.
* Clearing at TLAB creation is much faster, but more complex and it might expose hard
* to find bugs.
*/
typedef enum {
CLEAR_AT_GC,
CLEAR_AT_TLAB_CREATION
} NurseryClearPolicy;
static NurseryClearPolicy nursery_clear_policy = CLEAR_AT_TLAB_CREATION;
/*
* The young generation is divided into fragments. This is because
* we can hand one fragments to a thread for lock-less fast alloc and
* because the young generation ends up fragmented anyway by pinned objects.
* Once a collection is done, a list of fragments is created. When doing
* thread local alloc we use smallish nurseries so we allow new threads to
* allocate memory from gen0 without triggering a collection. Threads that
* are found to allocate lots of memory are given bigger fragments. This
* should make the finalizer thread use little nursery memory after a while.
* We should start assigning threads very small fragments: if there are many
* threads the nursery will be full of reserved space that the threads may not
* use at all, slowing down allocation speed.
* Thread local allocation is done from areas of memory Hotspot calls Thread Local
* Allocation Buffers (TLABs).
*/
typedef struct _Fragment Fragment;
struct _Fragment {
Fragment *next;
char *fragment_start;
char *fragment_limit; /* the current soft limit for allocation */
char *fragment_end;
};
/* the runtime can register areas of memory as roots: we keep two lists of roots,
* a pinned root set for conservatively scanned roots and a normal one for
* precisely scanned roots (currently implemented as a single list).
*/
typedef struct _RootRecord RootRecord;
struct _RootRecord {
RootRecord *next;
char *start_root;
char *end_root;
mword root_desc;
};
/*
* We're never actually using the first element. It's always set to
* NULL to simplify the elimination of consecutive duplicate
* entries.
*/
#define STORE_REMSET_BUFFER_SIZE 1023
typedef struct _GenericStoreRememberedSet GenericStoreRememberedSet;
struct _GenericStoreRememberedSet {
GenericStoreRememberedSet *next;
/* We need one entry less because the first entry of store
remset buffers is always a dummy and we don't copy it. */
gpointer data [STORE_REMSET_BUFFER_SIZE - 1];
};
/* we have 4 possible values in the low 2 bits */
enum {
REMSET_LOCATION, /* just a pointer to the exact location */
REMSET_RANGE, /* range of pointer fields */
REMSET_OBJECT, /* mark all the object for scanning */
REMSET_VTYPE, /* a valuetype array described by a gc descriptor, a count and a size */
REMSET_TYPE_MASK = 0x3
};
#ifdef HAVE_KW_THREAD
static __thread RememberedSet *remembered_set MONO_TLS_FAST;
#endif
static pthread_key_t remembered_set_key;
static RememberedSet *global_remset;
static RememberedSet *freed_thread_remsets;
static GenericStoreRememberedSet *generic_store_remsets = NULL;
/*A two slots cache for recently inserted remsets */
static gpointer global_remset_cache [2];
/* FIXME: later choose a size that takes into account the RememberedSet struct
* and doesn't waste any alloc paddin space.
*/
#define DEFAULT_REMSET_SIZE 1024
static RememberedSet* alloc_remset (int size, gpointer id, gboolean global);
#define object_is_forwarded SGEN_OBJECT_IS_FORWARDED
#define object_is_pinned SGEN_OBJECT_IS_PINNED
#define pin_object SGEN_PIN_OBJECT
#define unpin_object SGEN_UNPIN_OBJECT
#define ptr_in_nursery(p) (SGEN_PTR_IN_NURSERY ((p), DEFAULT_NURSERY_BITS, nursery_start, nursery_end))
#define LOAD_VTABLE SGEN_LOAD_VTABLE
static const char*
safe_name (void* obj)
{
MonoVTable *vt = (MonoVTable*)LOAD_VTABLE (obj);
return vt->klass->name;
}
#define safe_object_get_size mono_sgen_safe_object_get_size
const char*
mono_sgen_safe_name (void* obj)
{
return safe_name (obj);
}
/*
* ######################################################################
* ######## Global data.
* ######################################################################
*/
static LOCK_DECLARE (gc_mutex);
static int gc_disabled = 0;
static int num_minor_gcs = 0;
static int num_major_gcs = 0;
static gboolean use_cardtable;
#ifdef USER_CONFIG
/* good sizes are 512KB-1MB: larger ones increase a lot memzeroing time */
#define DEFAULT_NURSERY_SIZE (default_nursery_size)
static int default_nursery_size = (1 << 22);
#ifdef SGEN_ALIGN_NURSERY
/* The number of trailing 0 bits in DEFAULT_NURSERY_SIZE */
#define DEFAULT_NURSERY_BITS (default_nursery_bits)
static int default_nursery_bits = 22;
#endif
#else
#define DEFAULT_NURSERY_SIZE (4*1024*1024)
#ifdef SGEN_ALIGN_NURSERY
#define DEFAULT_NURSERY_BITS 22
#endif
#endif
#ifndef SGEN_ALIGN_NURSERY
#define DEFAULT_NURSERY_BITS -1
#endif
#define MIN_MINOR_COLLECTION_ALLOWANCE (DEFAULT_NURSERY_SIZE * 4)
#define SCAN_START_SIZE SGEN_SCAN_START_SIZE
/* the minimum size of a fragment that we consider useful for allocation */
#define FRAGMENT_MIN_SIZE (512)
static mword pagesize = 4096;
static mword nursery_size;
static int degraded_mode = 0;
static mword total_alloc = 0;
/* use this to tune when to do a major/minor collection */
static mword memory_pressure = 0;
static mword minor_collection_allowance;
static int minor_collection_sections_alloced = 0;
static GCMemSection *nursery_section = NULL;
static mword lowest_heap_address = ~(mword)0;
static mword highest_heap_address = 0;
static LOCK_DECLARE (interruption_mutex);
static LOCK_DECLARE (global_remset_mutex);
static LOCK_DECLARE (pin_queue_mutex);
#define LOCK_GLOBAL_REMSET pthread_mutex_lock (&global_remset_mutex)
#define UNLOCK_GLOBAL_REMSET pthread_mutex_unlock (&global_remset_mutex)
#define LOCK_PIN_QUEUE pthread_mutex_lock (&pin_queue_mutex)
#define UNLOCK_PIN_QUEUE pthread_mutex_unlock (&pin_queue_mutex)
typedef struct _FinalizeEntry FinalizeEntry;
struct _FinalizeEntry {
FinalizeEntry *next;
void *object;
};
typedef struct _FinalizeEntryHashTable FinalizeEntryHashTable;
struct _FinalizeEntryHashTable {
FinalizeEntry **table;
mword size;
int num_registered;
};
typedef struct _DisappearingLink DisappearingLink;
struct _DisappearingLink {
DisappearingLink *next;
void **link;
};
typedef struct _DisappearingLinkHashTable DisappearingLinkHashTable;
struct _DisappearingLinkHashTable {
DisappearingLink **table;
mword size;
int num_links;
};
typedef struct _EphemeronLinkNode EphemeronLinkNode;
struct _EphemeronLinkNode {
EphemeronLinkNode *next;
char *array;
};
typedef struct {
void *key;
void *value;
} Ephemeron;
int current_collection_generation = -1;
/*
* The link pointer is hidden by negating each bit. We use the lowest
* bit of the link (before negation) to store whether it needs
* resurrection tracking.
*/
#define HIDE_POINTER(p,t) ((gpointer)(~((gulong)(p)|((t)?1:0))))
#define REVEAL_POINTER(p) ((gpointer)((~(gulong)(p))&~3L))
#define DISLINK_OBJECT(d) (REVEAL_POINTER (*(d)->link))
#define DISLINK_TRACK(d) ((~(gulong)(*(d)->link)) & 1)
/* objects that are ready to be finalized */
static FinalizeEntry *fin_ready_list = NULL;
static FinalizeEntry *critical_fin_list = NULL;
static EphemeronLinkNode *ephemeron_list;
static int num_ready_finalizers = 0;
static int no_finalize = 0;
enum {
ROOT_TYPE_NORMAL = 0, /* "normal" roots */
ROOT_TYPE_PINNED = 1, /* roots without a GC descriptor */
ROOT_TYPE_WBARRIER = 2, /* roots with a write barrier */
ROOT_TYPE_NUM
};
/* registered roots: the key to the hash is the root start address */
/*
* Different kinds of roots are kept separate to speed up pin_from_roots () for example.
*/
static RootRecord **roots_hash [ROOT_TYPE_NUM] = { NULL, NULL };
static int roots_hash_size [ROOT_TYPE_NUM] = { 0, 0, 0 };
static mword roots_size = 0; /* amount of memory in the root set */
static int num_roots_entries [ROOT_TYPE_NUM] = { 0, 0, 0 };
#define GC_ROOT_NUM 32
typedef struct {
int count;
void *objects [GC_ROOT_NUM];
int root_types [GC_ROOT_NUM];
uintptr_t extra_info [GC_ROOT_NUM];
} GCRootReport;
static void
notify_gc_roots (GCRootReport *report)
{
if (!report->count)
return;
mono_profiler_gc_roots (report->count, report->objects, report->root_types, report->extra_info);
report->count = 0;
}
static void
add_profile_gc_root (GCRootReport *report, void *object, int rtype, uintptr_t extra_info)
{
if (report->count == GC_ROOT_NUM)
notify_gc_roots (report);
report->objects [report->count] = object;
report->root_types [report->count] = rtype;
report->extra_info [report->count++] = (uintptr_t)((MonoVTable*)LOAD_VTABLE (object))->klass;
}
/*
* The current allocation cursors
* We allocate objects in the nursery.
* The nursery is the area between nursery_start and nursery_end.
* Allocation is done from a Thread Local Allocation Buffer (TLAB). TLABs are allocated
* from nursery fragments.
* tlab_next is the pointer to the space inside the TLAB where the next object will
* be allocated.
* tlab_temp_end is the pointer to the end of the temporary space reserved for
* the allocation: it allows us to set the scan starts at reasonable intervals.
* tlab_real_end points to the end of the TLAB.
* nursery_frag_real_end points to the end of the currently used nursery fragment.
* nursery_first_pinned_start points to the start of the first pinned object in the nursery
* nursery_last_pinned_end points to the end of the last pinned object in the nursery
* At the next allocation, the area of the nursery where objects can be present is
* between MIN(nursery_first_pinned_start, first_fragment_start) and
* MAX(nursery_last_pinned_end, nursery_frag_real_end)
*/
static char *nursery_start = NULL;
static char *nursery_next = NULL;
static char *nursery_frag_real_end = NULL;
static char *nursery_end = NULL;
static char *nursery_last_pinned_end = NULL;
#ifdef HAVE_KW_THREAD
#define TLAB_ACCESS_INIT
#define TLAB_START tlab_start
#define TLAB_NEXT tlab_next
#define TLAB_TEMP_END tlab_temp_end
#define TLAB_REAL_END tlab_real_end
#define REMEMBERED_SET remembered_set
#define STORE_REMSET_BUFFER store_remset_buffer
#define STORE_REMSET_BUFFER_INDEX store_remset_buffer_index
#define IN_CRITICAL_REGION thread_info->in_critical_region
#else
static pthread_key_t thread_info_key;
#define TLAB_ACCESS_INIT SgenThreadInfo *__thread_info__ = pthread_getspecific (thread_info_key)
#define TLAB_START (__thread_info__->tlab_start)
#define TLAB_NEXT (__thread_info__->tlab_next)
#define TLAB_TEMP_END (__thread_info__->tlab_temp_end)
#define TLAB_REAL_END (__thread_info__->tlab_real_end)
#define REMEMBERED_SET (__thread_info__->remset)
#define STORE_REMSET_BUFFER (__thread_info__->store_remset_buffer)
#define STORE_REMSET_BUFFER_INDEX (__thread_info__->store_remset_buffer_index)
#define IN_CRITICAL_REGION (__thread_info__->in_critical_region)
#endif
#ifndef DISABLE_CRITICAL_REGION
/* we use the memory barrier only to prevent compiler reordering (a memory constraint may be enough) */
#define ENTER_CRITICAL_REGION do {IN_CRITICAL_REGION = 1;mono_memory_barrier ();} while (0)
#define EXIT_CRITICAL_REGION do {IN_CRITICAL_REGION = 0;mono_memory_barrier ();} while (0)
#endif
/*
* FIXME: What is faster, a TLS variable pointing to a structure, or separate TLS
* variables for next+temp_end ?
*/
#ifdef HAVE_KW_THREAD
static __thread SgenThreadInfo *thread_info;
static __thread char *tlab_start;
static __thread char *tlab_next;
static __thread char *tlab_temp_end;
static __thread char *tlab_real_end;
static __thread gpointer *store_remset_buffer;
static __thread long store_remset_buffer_index;
/* Used by the managed allocator/wbarrier */
static __thread char **tlab_next_addr;
static __thread char *stack_end;
static __thread long *store_remset_buffer_index_addr;
#endif
/* The size of a TLAB */
/* The bigger the value, the less often we have to go to the slow path to allocate a new
* one, but the more space is wasted by threads not allocating much memory.
* FIXME: Tune this.
* FIXME: Make this self-tuning for each thread.
*/
static guint32 tlab_size = (1024 * 4);
/*How much space is tolerable to be wasted from the current fragment when allocating a new TLAB*/
#define MAX_NURSERY_TLAB_WASTE 512
/* fragments that are free and ready to be used for allocation */
static Fragment *nursery_fragments = NULL;
/* freeelist of fragment structures */
static Fragment *fragment_freelist = NULL;
#define MAX_SMALL_OBJ_SIZE SGEN_MAX_SMALL_OBJ_SIZE
/* Functions supplied by the runtime to be called by the GC */
static MonoGCCallbacks gc_callbacks;
#define ALLOC_ALIGN SGEN_ALLOC_ALIGN
#define ALLOC_ALIGN_BITS SGEN_ALLOC_ALIGN_BITS
#define ALIGN_UP SGEN_ALIGN_UP
#define MOVED_OBJECTS_NUM 64
static void *moved_objects [MOVED_OBJECTS_NUM];
static int moved_objects_idx = 0;
/* Vtable of the objects used to fill out nursery fragments before a collection */
static MonoVTable *array_fill_vtable;
#ifdef SGEN_DEBUG_INTERNAL_ALLOC
pthread_t main_gc_thread = NULL;
#endif
/*
* ######################################################################
* ######## Heap size accounting
* ######################################################################
*/
/*heap limits*/
static mword max_heap_size = ((mword)0)- ((mword)1);
static mword allocated_heap;
/*Object was pinned during the current collection*/
static mword objects_pinned;
void
mono_sgen_release_space (mword size, int space)
{
allocated_heap -= size;
}
static size_t
available_free_space (void)
{
return max_heap_size - MIN (allocated_heap, max_heap_size);
}
gboolean
mono_sgen_try_alloc_space (mword size, int space)
{
if (available_free_space () < size)
return FALSE;
allocated_heap += size;
return TRUE;
}
static void
init_heap_size_limits (glong max_heap)
{
if (max_heap == 0)
return;
if (max_heap < nursery_size * 4) {
fprintf (stderr, "max-heap-size must be at least 4 times larger than nursery size.\n");
exit (1);
}
max_heap_size = max_heap - nursery_size;
}
/*
* ######################################################################
* ######## Macros and function declarations.
* ######################################################################
*/
inline static void*
align_pointer (void *ptr)
{
mword p = (mword)ptr;
p += sizeof (gpointer) - 1;
p &= ~ (sizeof (gpointer) - 1);
return (void*)p;
}
typedef SgenGrayQueue GrayQueue;
typedef void (*CopyOrMarkObjectFunc) (void**, GrayQueue*);
typedef char* (*ScanObjectFunc) (char*, GrayQueue*);
/* forward declarations */
static int stop_world (int generation);
static int restart_world (int generation);
static void scan_thread_data (void *start_nursery, void *end_nursery, gboolean precise, GrayQueue *queue);
static void scan_from_global_remsets (void *start_nursery, void *end_nursery, GrayQueue *queue);
static void scan_from_remsets (void *start_nursery, void *end_nursery, GrayQueue *queue);
static void scan_from_registered_roots (CopyOrMarkObjectFunc copy_func, char *addr_start, char *addr_end, int root_type, GrayQueue *queue);
static void scan_finalizer_entries (CopyOrMarkObjectFunc copy_func, FinalizeEntry *list, GrayQueue *queue);
static void report_finalizer_roots (void);
static void report_registered_roots (void);
static void find_pinning_ref_from_thread (char *obj, size_t size);
static void update_current_thread_stack (void *start);
static void finalize_in_range (CopyOrMarkObjectFunc copy_func, char *start, char *end, int generation, GrayQueue *queue);
static void process_fin_stage_entries (void);
static void null_link_in_range (CopyOrMarkObjectFunc copy_func, char *start, char *end, int generation, gboolean before_finalization, GrayQueue *queue);
static void null_links_for_domain (MonoDomain *domain, int generation);
static void process_dislink_stage_entries (void);
static gboolean alloc_fragment_for_size (size_t size);
static int alloc_fragment_for_size_range (size_t desired_size, size_t minimum_size);
static void clear_nursery_fragments (char *next);
static void pin_from_roots (void *start_nursery, void *end_nursery, GrayQueue *queue);
static int pin_objects_from_addresses (GCMemSection *section, void **start, void **end, void *start_nursery, void *end_nursery, GrayQueue *queue);
static void optimize_pin_queue (int start_slot);
static void clear_remsets (void);
static void clear_tlabs (void);
static void sort_addresses (void **array, int size);
static gboolean drain_gray_stack (GrayQueue *queue, int max_objs);
static void finish_gray_stack (char *start_addr, char *end_addr, int generation, GrayQueue *queue);
static gboolean need_major_collection (mword space_needed);
static void major_collection (const char *reason);
static void mono_gc_register_disappearing_link (MonoObject *obj, void **link, gboolean track, gboolean in_gc);
static gboolean mono_gc_is_critical_method (MonoMethod *method);
void describe_ptr (char *ptr);
void check_object (char *start);
static void check_consistency (void);
static void check_major_refs (void);
static void check_scan_starts (void);
static void check_for_xdomain_refs (void);
static void dump_heap (const char *type, int num, const char *reason);
void mono_gc_scan_for_specific_ref (MonoObject *key, gboolean precise);
static void init_stats (void);
static int mark_ephemerons_in_range (CopyOrMarkObjectFunc copy_func, char *start, char *end, GrayQueue *queue);
static void clear_unreachable_ephemerons (CopyOrMarkObjectFunc copy_func, char *start, char *end, GrayQueue *queue);
static void null_ephemerons_for_domain (MonoDomain *domain);
SgenMajorCollector major_collector;
#include "sgen-pinning.c"
#include "sgen-pinning-stats.c"
#include "sgen-gray.c"
#include "sgen-workers.c"
#include "sgen-cardtable.c"
/* Root bitmap descriptors are simpler: the lower three bits describe the type
* and we either have 30/62 bitmap bits or nibble-based run-length,
* or a complex descriptor, or a user defined marker function.
*/
enum {
ROOT_DESC_CONSERVATIVE, /* 0, so matches NULL value */
ROOT_DESC_BITMAP,
ROOT_DESC_RUN_LEN,
ROOT_DESC_COMPLEX,
ROOT_DESC_USER,
ROOT_DESC_TYPE_MASK = 0x7,
ROOT_DESC_TYPE_SHIFT = 3,
};
#define MAKE_ROOT_DESC(type,val) ((type) | ((val) << ROOT_DESC_TYPE_SHIFT))
#define MAX_USER_DESCRIPTORS 16
static gsize* complex_descriptors = NULL;
static int complex_descriptors_size = 0;
static int complex_descriptors_next = 0;
static MonoGCRootMarkFunc user_descriptors [MAX_USER_DESCRIPTORS];
static int user_descriptors_next = 0;
static int
alloc_complex_descriptor (gsize *bitmap, int numbits)
{
int nwords, res, i;
numbits = ALIGN_TO (numbits, GC_BITS_PER_WORD);
nwords = numbits / GC_BITS_PER_WORD + 1;
LOCK_GC;
res = complex_descriptors_next;
/* linear search, so we don't have duplicates with domain load/unload
* this should not be performance critical or we'd have bigger issues
* (the number and size of complex descriptors should be small).
*/
for (i = 0; i < complex_descriptors_next; ) {
if (complex_descriptors [i] == nwords) {
int j, found = TRUE;
for (j = 0; j < nwords - 1; ++j) {
if (complex_descriptors [i + 1 + j] != bitmap [j]) {
found = FALSE;
break;
}
}
if (found) {
UNLOCK_GC;
return i;
}
}
i += complex_descriptors [i];
}
if (complex_descriptors_next + nwords > complex_descriptors_size) {
int new_size = complex_descriptors_size * 2 + nwords;
complex_descriptors = g_realloc (complex_descriptors, new_size * sizeof (gsize));
complex_descriptors_size = new_size;
}
DEBUG (6, fprintf (gc_debug_file, "Complex descriptor %d, size: %d (total desc memory: %d)\n", res, nwords, complex_descriptors_size));
complex_descriptors_next += nwords;
complex_descriptors [res] = nwords;
for (i = 0; i < nwords - 1; ++i) {
complex_descriptors [res + 1 + i] = bitmap [i];
DEBUG (6, fprintf (gc_debug_file, "\tvalue: %p\n", (void*)complex_descriptors [res + 1 + i]));
}
UNLOCK_GC;
return res;
}
gsize*
mono_sgen_get_complex_descriptor (mword desc)
{
return complex_descriptors + (desc >> LOW_TYPE_BITS);
}
/*
* Descriptor builders.
*/
void*
mono_gc_make_descr_for_string (gsize *bitmap, int numbits)
{
return (void*) DESC_TYPE_RUN_LENGTH;
}
void*
mono_gc_make_descr_for_object (gsize *bitmap, int numbits, size_t obj_size)
{
int first_set = -1, num_set = 0, last_set = -1, i;
mword desc = 0;
size_t stored_size = obj_size;
for (i = 0; i < numbits; ++i) {
if (bitmap [i / GC_BITS_PER_WORD] & ((gsize)1 << (i % GC_BITS_PER_WORD))) {
if (first_set < 0)
first_set = i;
last_set = i;
num_set++;
}
}
/*
* We don't encode the size of types that don't contain
* references because they might not be aligned, i.e. the
* bottom two bits might be set, which would clash with the
* bits we need to encode the descriptor type. Since we don't
* use the encoded size to skip objects, other than for
* processing remsets, in which case only the positions of
* references are relevant, this is not a problem.
*/
if (first_set < 0)
return (void*)DESC_TYPE_RUN_LENGTH;
g_assert (!(stored_size & 0x3));
if (stored_size <= MAX_SMALL_OBJ_SIZE) {
/* check run-length encoding first: one byte offset, one byte number of pointers
* on 64 bit archs, we can have 3 runs, just one on 32.
* It may be better to use nibbles.
*/
if (first_set < 0) {
desc = DESC_TYPE_RUN_LENGTH | (stored_size << 1);
DEBUG (6, fprintf (gc_debug_file, "Ptrfree descriptor %p, size: %zd\n", (void*)desc, stored_size));
return (void*) desc;
} else if (first_set < 256 && num_set < 256 && (first_set + num_set == last_set + 1)) {
desc = DESC_TYPE_RUN_LENGTH | (stored_size << 1) | (first_set << 16) | (num_set << 24);
DEBUG (6, fprintf (gc_debug_file, "Runlen descriptor %p, size: %zd, first set: %d, num set: %d\n", (void*)desc, stored_size, first_set, num_set));
return (void*) desc;
}
}
/* we know the 2-word header is ptr-free */
if (last_set < LARGE_BITMAP_SIZE + OBJECT_HEADER_WORDS) {
desc = DESC_TYPE_LARGE_BITMAP | ((*bitmap >> OBJECT_HEADER_WORDS) << LOW_TYPE_BITS);
DEBUG (6, fprintf (gc_debug_file, "Largebitmap descriptor %p, size: %zd, last set: %d\n", (void*)desc, stored_size, last_set));
return (void*) desc;
}
/* it's a complex object ... */
desc = DESC_TYPE_COMPLEX | (alloc_complex_descriptor (bitmap, last_set + 1) << LOW_TYPE_BITS);
return (void*) desc;
}
/* If the array holds references, numbits == 1 and the first bit is set in elem_bitmap */
void*
mono_gc_make_descr_for_array (int vector, gsize *elem_bitmap, int numbits, size_t elem_size)
{
int first_set = -1, num_set = 0, last_set = -1, i;
mword desc = vector? DESC_TYPE_VECTOR: DESC_TYPE_ARRAY;
for (i = 0; i < numbits; ++i) {
if (elem_bitmap [i / GC_BITS_PER_WORD] & ((gsize)1 << (i % GC_BITS_PER_WORD))) {
if (first_set < 0)
first_set = i;
last_set = i;
num_set++;
}
}
/* See comment at the definition of DESC_TYPE_RUN_LENGTH. */
if (first_set < 0)
return (void*)DESC_TYPE_RUN_LENGTH;
if (elem_size <= MAX_ELEMENT_SIZE) {
desc |= elem_size << VECTOR_ELSIZE_SHIFT;
if (!num_set) {
return (void*)(desc | VECTOR_SUBTYPE_PTRFREE);
}
/* Note: we also handle structs with just ref fields */
if (num_set * sizeof (gpointer) == elem_size) {
return (void*)(desc | VECTOR_SUBTYPE_REFS | ((gssize)(-1) << 16));
}
/* FIXME: try run-len first */
/* Note: we can't skip the object header here, because it's not present */
if (last_set <= SMALL_BITMAP_SIZE) {
return (void*)(desc | VECTOR_SUBTYPE_BITMAP | (*elem_bitmap << 16));
}
}
/* it's am array of complex structs ... */
desc = DESC_TYPE_COMPLEX_ARR;
desc |= alloc_complex_descriptor (elem_bitmap, last_set + 1) << LOW_TYPE_BITS;
return (void*) desc;
}
/* Return the bitmap encoded by a descriptor */
gsize*
mono_gc_get_bitmap_for_descr (void *descr, int *numbits)
{
mword d = (mword)descr;
gsize *bitmap;
switch (d & 0x7) {
case DESC_TYPE_RUN_LENGTH: {
int first_set = (d >> 16) & 0xff;
int num_set = (d >> 24) & 0xff;
int i;
bitmap = g_new0 (gsize, (first_set + num_set + 7) / 8);
for (i = first_set; i < first_set + num_set; ++i)
bitmap [i / GC_BITS_PER_WORD] |= ((gsize)1 << (i % GC_BITS_PER_WORD));
*numbits = first_set + num_set;
return bitmap;
}
default:
g_assert_not_reached ();
}
}
static gboolean
is_xdomain_ref_allowed (gpointer *ptr, char *obj, MonoDomain *domain)
{
MonoObject *o = (MonoObject*)(obj);
MonoObject *ref = (MonoObject*)*(ptr);
int offset = (char*)(ptr) - (char*)o;
if (o->vtable->klass == mono_defaults.thread_class && offset == G_STRUCT_OFFSET (MonoThread, internal_thread))
return TRUE;
if (o->vtable->klass == mono_defaults.internal_thread_class && offset == G_STRUCT_OFFSET (MonoInternalThread, current_appcontext))
return TRUE;
if (mono_class_has_parent (o->vtable->klass, mono_defaults.real_proxy_class) &&
offset == G_STRUCT_OFFSET (MonoRealProxy, unwrapped_server))
return TRUE;
/* Thread.cached_culture_info */
if (!strcmp (ref->vtable->klass->name_space, "System.Globalization") &&
!strcmp (ref->vtable->klass->name, "CultureInfo") &&
!strcmp(o->vtable->klass->name_space, "System") &&
!strcmp(o->vtable->klass->name, "Object[]"))
return TRUE;
/*
* at System.IO.MemoryStream.InternalConstructor (byte[],int,int,bool,bool) [0x0004d] in /home/schani/Work/novell/trunk/mcs/class/corlib/System.IO/MemoryStream.cs:121
* at System.IO.MemoryStream..ctor (byte[]) [0x00017] in /home/schani/Work/novell/trunk/mcs/class/corlib/System.IO/MemoryStream.cs:81
* at (wrapper remoting-invoke-with-check) System.IO.MemoryStream..ctor (byte[]) <IL 0x00020, 0xffffffff>
* at System.Runtime.Remoting.Messaging.CADMethodCallMessage.GetArguments () [0x0000d] in /home/schani/Work/novell/trunk/mcs/class/corlib/System.Runtime.Remoting.Messaging/CADMessages.cs:327
* at System.Runtime.Remoting.Messaging.MethodCall..ctor (System.Runtime.Remoting.Messaging.CADMethodCallMessage) [0x00017] in /home/schani/Work/novell/trunk/mcs/class/corlib/System.Runtime.Remoting.Messaging/MethodCall.cs:87
* at System.AppDomain.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage,byte[]&,System.Runtime.Remoting.Messaging.CADMethodReturnMessage&) [0x00018] in /home/schani/Work/novell/trunk/mcs/class/corlib/System/AppDomain.cs:1213
* at (wrapper remoting-invoke-with-check) System.AppDomain.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage,byte[]&,System.Runtime.Remoting.Messaging.CADMethodReturnMessage&) <IL 0x0003d, 0xffffffff>
* at System.Runtime.Remoting.Channels.CrossAppDomainSink.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage) [0x00008] in /home/schani/Work/novell/trunk/mcs/class/corlib/System.Runtime.Remoting.Channels/CrossAppDomainChannel.cs:198
* at (wrapper runtime-invoke) object.runtime_invoke_CrossAppDomainSink/ProcessMessageRes_object_object (object,intptr,intptr,intptr) <IL 0x0004c, 0xffffffff>
*/
if (!strcmp (ref->vtable->klass->name_space, "System") &&
!strcmp (ref->vtable->klass->name, "Byte[]") &&
!strcmp (o->vtable->klass->name_space, "System.IO") &&
!strcmp (o->vtable->klass->name, "MemoryStream"))
return TRUE;
/* append_job() in threadpool.c */
if (!strcmp (ref->vtable->klass->name_space, "System.Runtime.Remoting.Messaging") &&
!strcmp (ref->vtable->klass->name, "AsyncResult") &&
!strcmp (o->vtable->klass->name_space, "System") &&
!strcmp (o->vtable->klass->name, "Object[]") &&
mono_thread_pool_is_queue_array ((MonoArray*) o))
return TRUE;
return FALSE;
}
static void
check_reference_for_xdomain (gpointer *ptr, char *obj, MonoDomain *domain)
{
MonoObject *o = (MonoObject*)(obj);
MonoObject *ref = (MonoObject*)*(ptr);
int offset = (char*)(ptr) - (char*)o;
MonoClass *class;
MonoClassField *field;
char *str;
if (!ref || ref->vtable->domain == domain)
return;
if (is_xdomain_ref_allowed (ptr, obj, domain))
return;
field = NULL;
for (class = o->vtable->klass; class; class = class->parent) {
int i;
for (i = 0; i < class->field.count; ++i) {
if (class->fields[i].offset == offset) {
field = &class->fields[i];
break;
}
}
if (field)
break;
}
if (ref->vtable->klass == mono_defaults.string_class)
str = mono_string_to_utf8 ((MonoString*)ref);
else
str = NULL;
g_print ("xdomain reference in %p (%s.%s) at offset %d (%s) to %p (%s.%s) (%s) - pointed to by:\n",
o, o->vtable->klass->name_space, o->vtable->klass->name,
offset, field ? field->name : "",
ref, ref->vtable->klass->name_space, ref->vtable->klass->name, str ? str : "");
mono_gc_scan_for_specific_ref (o, TRUE);
if (str)
g_free (str);
}
#undef HANDLE_PTR
#define HANDLE_PTR(ptr,obj) check_reference_for_xdomain ((ptr), (obj), domain)
static void
scan_object_for_xdomain_refs (char *start, mword size, void *data)
{
MonoDomain *domain = ((MonoObject*)start)->vtable->domain;
#include "sgen-scan-object.h"
}
static gboolean scan_object_for_specific_ref_precise = TRUE;
#undef HANDLE_PTR
#define HANDLE_PTR(ptr,obj) do { \
if ((MonoObject*)*(ptr) == key) { \
g_print ("found ref to %p in object %p (%s) at offset %td\n", \
key, (obj), safe_name ((obj)), ((char*)(ptr) - (char*)(obj))); \
} \
} while (0)
static void
scan_object_for_specific_ref (char *start, MonoObject *key)
{
char *forwarded;
if ((forwarded = SGEN_OBJECT_IS_FORWARDED (start)))
start = forwarded;
if (scan_object_for_specific_ref_precise) {
#include "sgen-scan-object.h"
} else {
mword *words = (mword*)start;
size_t size = safe_object_get_size ((MonoObject*)start);
int i;
for (i = 0; i < size / sizeof (mword); ++i) {
if (words [i] == (mword)key) {
g_print ("found possible ref to %p in object %p (%s) at offset %td\n",
key, start, safe_name (start), i * sizeof (mword));
}
}
}
}
void
mono_sgen_scan_area_with_callback (char *start, char *end, IterateObjectCallbackFunc callback, void *data, gboolean allow_flags)
{
while (start < end) {
size_t size;
char *obj;
if (!*(void**)start) {
start += sizeof (void*); /* should be ALLOC_ALIGN, really */
continue;
}
if (allow_flags) {
if (!(obj = SGEN_OBJECT_IS_FORWARDED (start)))
obj = start;
} else {
obj = start;
}
size = ALIGN_UP (safe_object_get_size ((MonoObject*)obj));
callback (obj, size, data);
start += size;
}
}
static void
scan_object_for_specific_ref_callback (char *obj, size_t size, MonoObject *key)
{
scan_object_for_specific_ref (obj, key);
}
static void
check_root_obj_specific_ref (RootRecord *root, MonoObject *key, MonoObject *obj)
{
if (key != obj)
return;
g_print ("found ref to %p in root record %p\n", key, root);
}
static MonoObject *check_key = NULL;
static RootRecord *check_root = NULL;
static void
check_root_obj_specific_ref_from_marker (void **obj)
{
check_root_obj_specific_ref (check_root, check_key, *obj);
}
static void
scan_roots_for_specific_ref (MonoObject *key, int root_type)
{
int i;
RootRecord *root;
check_key = key;
for (i = 0; i < roots_hash_size [root_type]; ++i) {
for (root = roots_hash [root_type][i]; root; root = root->next) {
void **start_root = (void**)root->start_root;
mword desc = root->root_desc;
check_root = root;
switch (desc & ROOT_DESC_TYPE_MASK) {
case ROOT_DESC_BITMAP:
desc >>= ROOT_DESC_TYPE_SHIFT;
while (desc) {
if (desc & 1)
check_root_obj_specific_ref (root, key, *start_root);
desc >>= 1;
start_root++;
}
return;
case ROOT_DESC_COMPLEX: {
gsize *bitmap_data = complex_descriptors + (desc >> ROOT_DESC_TYPE_SHIFT);
int bwords = (*bitmap_data) - 1;
void **start_run = start_root;
bitmap_data++;
while (bwords-- > 0) {
gsize bmap = *bitmap_data++;
void **objptr = start_run;
while (bmap) {
if (bmap & 1)
check_root_obj_specific_ref (root, key, *objptr);
bmap >>= 1;
++objptr;
}
start_run += GC_BITS_PER_WORD;
}
break;
}
case ROOT_DESC_USER: {
MonoGCRootMarkFunc marker = user_descriptors [desc >> ROOT_DESC_TYPE_SHIFT];
marker (start_root, check_root_obj_specific_ref_from_marker);
break;
}
case ROOT_DESC_RUN_LEN:
g_assert_not_reached ();
default:
g_assert_not_reached ();
}
}
}
check_key = NULL;
check_root = NULL;
}
void
mono_gc_scan_for_specific_ref (MonoObject *key, gboolean precise)
{
RootRecord *root;
int i;
scan_object_for_specific_ref_precise = precise;
mono_sgen_scan_area_with_callback (nursery_section->data, nursery_section->end_data,
(IterateObjectCallbackFunc)scan_object_for_specific_ref_callback, key, TRUE);
major_collector.iterate_objects (TRUE, TRUE, (IterateObjectCallbackFunc)scan_object_for_specific_ref_callback, key);
mono_sgen_los_iterate_objects ((IterateObjectCallbackFunc)scan_object_for_specific_ref_callback, key);
scan_roots_for_specific_ref (key, ROOT_TYPE_NORMAL);
scan_roots_for_specific_ref (key, ROOT_TYPE_WBARRIER);
for (i = 0; i < roots_hash_size [ROOT_TYPE_PINNED]; ++i) {
for (root = roots_hash [ROOT_TYPE_PINNED][i]; root; root = root->next) {
void **ptr = (void**)root->start_root;
while (ptr < (void**)root->end_root) {
check_root_obj_specific_ref (root, *ptr, key);
++ptr;
}
}
}
}
static void
clear_current_nursery_fragment (char *next)
{
if (nursery_clear_policy == CLEAR_AT_TLAB_CREATION) {
g_assert (next <= nursery_frag_real_end);
DEBUG (4, fprintf (gc_debug_file, "Clear nursery frag %p-%p\n", next, nursery_frag_real_end));
memset (next, 0, nursery_frag_real_end - next);
}
}
/* Clear all remaining nursery fragments */
static void
clear_nursery_fragments (char *next)
{
Fragment *frag;
if (nursery_clear_policy == CLEAR_AT_TLAB_CREATION) {
clear_current_nursery_fragment (next);
for (frag = nursery_fragments; frag; frag = frag->next) {
DEBUG (4, fprintf (gc_debug_file, "Clear nursery frag %p-%p\n", frag->fragment_start, frag->fragment_end));
memset (frag->fragment_start, 0, frag->fragment_end - frag->fragment_start);
}
}
}
static gboolean
need_remove_object_for_domain (char *start, MonoDomain *domain)
{
if (mono_object_domain (start) == domain) {
DEBUG (4, fprintf (gc_debug_file, "Need to cleanup object %p\n", start));
binary_protocol_cleanup (start, (gpointer)LOAD_VTABLE (start), safe_object_get_size ((MonoObject*)start));
return TRUE;
}
return FALSE;
}
static void
process_object_for_domain_clearing (char *start, MonoDomain *domain)
{
GCVTable *vt = (GCVTable*)LOAD_VTABLE (start);
if (vt->klass == mono_defaults.internal_thread_class)
g_assert (mono_object_domain (start) == mono_get_root_domain ());
/* The object could be a proxy for an object in the domain
we're deleting. */
if (mono_class_has_parent (vt->klass, mono_defaults.real_proxy_class)) {
MonoObject *server = ((MonoRealProxy*)start)->unwrapped_server;
/* The server could already have been zeroed out, so
we need to check for that, too. */
if (server && (!LOAD_VTABLE (server) || mono_object_domain (server) == domain)) {
DEBUG (4, fprintf (gc_debug_file, "Cleaning up remote pointer in %p to object %p\n",
start, server));
((MonoRealProxy*)start)->unwrapped_server = NULL;
}
}
}
static MonoDomain *check_domain = NULL;
static void
check_obj_not_in_domain (void **o)
{
g_assert (((MonoObject*)(*o))->vtable->domain != check_domain);
}
static void
scan_for_registered_roots_in_domain (MonoDomain *domain, int root_type)
{
int i;
RootRecord *root;
check_domain = domain;
for (i = 0; i < roots_hash_size [root_type]; ++i) {
for (root = roots_hash [root_type][i]; root; root = root->next) {
void **start_root = (void**)root->start_root;
mword desc = root->root_desc;
/* The MonoDomain struct is allowed to hold
references to objects in its own domain. */
if (start_root == (void**)domain)
continue;
switch (desc & ROOT_DESC_TYPE_MASK) {
case ROOT_DESC_BITMAP:
desc >>= ROOT_DESC_TYPE_SHIFT;
while (desc) {
if ((desc & 1) && *start_root)
check_obj_not_in_domain (*start_root);
desc >>= 1;
start_root++;
}
break;
case ROOT_DESC_COMPLEX: {
gsize *bitmap_data = complex_descriptors + (desc >> ROOT_DESC_TYPE_SHIFT);
int bwords = (*bitmap_data) - 1;
void **start_run = start_root;
bitmap_data++;
while (bwords-- > 0) {
gsize bmap = *bitmap_data++;
void **objptr = start_run;
while (bmap) {
if ((bmap & 1) && *objptr)
check_obj_not_in_domain (*objptr);
bmap >>= 1;
++objptr;
}
start_run += GC_BITS_PER_WORD;
}
break;
}
case ROOT_DESC_USER: {
MonoGCRootMarkFunc marker = user_descriptors [desc >> ROOT_DESC_TYPE_SHIFT];
marker (start_root, check_obj_not_in_domain);
break;
}
case ROOT_DESC_RUN_LEN:
g_assert_not_reached ();
default:
g_assert_not_reached ();
}
}
}
check_domain = NULL;
}
static void
check_for_xdomain_refs (void)
{
LOSObject *bigobj;
mono_sgen_scan_area_with_callback (nursery_section->data, nursery_section->end_data,
(IterateObjectCallbackFunc)scan_object_for_xdomain_refs, NULL, FALSE);
major_collector.iterate_objects (TRUE, TRUE, (IterateObjectCallbackFunc)scan_object_for_xdomain_refs, NULL);
for (bigobj = los_object_list; bigobj; bigobj = bigobj->next)
scan_object_for_xdomain_refs (bigobj->data, bigobj->size, NULL);
}
static gboolean
clear_domain_process_object (char *obj, MonoDomain *domain)
{
gboolean remove;
process_object_for_domain_clearing (obj, domain);
remove = need_remove_object_for_domain (obj, domain);
if (remove && ((MonoObject*)obj)->synchronisation) {
void **dislink = mono_monitor_get_object_monitor_weak_link ((MonoObject*)obj);
if (dislink)
mono_gc_register_disappearing_link (NULL, dislink, FALSE, TRUE);
}
return remove;
}
static void
clear_domain_process_minor_object_callback (char *obj, size_t size, MonoDomain *domain)
{
if (clear_domain_process_object (obj, domain))
memset (obj, 0, size);
}
static void
clear_domain_process_major_object_callback (char *obj, size_t size, MonoDomain *domain)
{
clear_domain_process_object (obj, domain);
}
static void
clear_domain_free_major_non_pinned_object_callback (char *obj, size_t size, MonoDomain *domain)
{
if (need_remove_object_for_domain (obj, domain))
major_collector.free_non_pinned_object (obj, size);
}
static void
clear_domain_free_major_pinned_object_callback (char *obj, size_t size, MonoDomain *domain)
{
if (need_remove_object_for_domain (obj, domain))
major_collector.free_pinned_object (obj, size);
}
/*
* When appdomains are unloaded we can easily remove objects that have finalizers,
* but all the others could still be present in random places on the heap.
* We need a sweep to get rid of them even though it's going to be costly
* with big heaps.
* The reason we need to remove them is because we access the vtable and class
* structures to know the object size and the reference bitmap: once the domain is
* unloaded the point to random memory.
*/
void
mono_gc_clear_domain (MonoDomain * domain)
{
LOSObject *bigobj, *prev;
int i;
LOCK_GC;
process_fin_stage_entries ();
process_dislink_stage_entries ();
clear_nursery_fragments (nursery_next);
if (xdomain_checks && domain != mono_get_root_domain ()) {
scan_for_registered_roots_in_domain (domain, ROOT_TYPE_NORMAL);
scan_for_registered_roots_in_domain (domain, ROOT_TYPE_WBARRIER);
check_for_xdomain_refs ();
}
mono_sgen_scan_area_with_callback (nursery_section->data, nursery_section->end_data,
(IterateObjectCallbackFunc)clear_domain_process_minor_object_callback, domain, FALSE);
/*Ephemerons and dislinks must be processed before LOS since they might end up pointing
to memory returned to the OS.*/
null_ephemerons_for_domain (domain);
for (i = GENERATION_NURSERY; i < GENERATION_MAX; ++i)
null_links_for_domain (domain, i);
/* We need two passes over major and large objects because
freeing such objects might give their memory back to the OS
(in the case of large objects) or obliterate its vtable
(pinned objects with major-copying or pinned and non-pinned
objects with major-mark&sweep), but we might need to
dereference a pointer from an object to another object if
the first object is a proxy. */
major_collector.iterate_objects (TRUE, TRUE, (IterateObjectCallbackFunc)clear_domain_process_major_object_callback, domain);
for (bigobj = los_object_list; bigobj; bigobj = bigobj->next)
clear_domain_process_object (bigobj->data, domain);
prev = NULL;
for (bigobj = los_object_list; bigobj;) {
if (need_remove_object_for_domain (bigobj->data, domain)) {
LOSObject *to_free = bigobj;
if (prev)
prev->next = bigobj->next;
else
los_object_list = bigobj->next;
bigobj = bigobj->next;
DEBUG (4, fprintf (gc_debug_file, "Freeing large object %p\n",
bigobj->data));
mono_sgen_los_free_object (to_free);
continue;
}
prev = bigobj;
bigobj = bigobj->next;
}
major_collector.iterate_objects (TRUE, FALSE, (IterateObjectCallbackFunc)clear_domain_free_major_non_pinned_object_callback, domain);
major_collector.iterate_objects (FALSE, TRUE, (IterateObjectCallbackFunc)clear_domain_free_major_pinned_object_callback, domain);
UNLOCK_GC;
}
static void
global_remset_cache_clear (void)
{
memset (global_remset_cache, 0, sizeof (global_remset_cache));
}
/*
* Tries to check if a given remset location was already added to the global remset.
* It can
*
* A 2 entry, LRU cache of recently saw location remsets.
*
* It's hand-coded instead of done using loops to reduce the number of memory references on cache hit.
*
* Returns TRUE is the element was added..
*/
static gboolean
global_remset_location_was_not_added (gpointer ptr)
{
gpointer first = global_remset_cache [0], second;
if (first == ptr) {
HEAVY_STAT (++stat_global_remsets_discarded);
return FALSE;
}
second = global_remset_cache [1];
if (second == ptr) {
/*Move the second to the front*/
global_remset_cache [0] = second;
global_remset_cache [1] = first;
HEAVY_STAT (++stat_global_remsets_discarded);
return FALSE;
}
global_remset_cache [0] = second;
global_remset_cache [1] = ptr;
return TRUE;
}
/*
* mono_sgen_add_to_global_remset:
*
* The global remset contains locations which point into newspace after
* a minor collection. This can happen if the objects they point to are pinned.
*
* LOCKING: If called from a parallel collector, the global remset
* lock must be held. For serial collectors that is not necessary.
*/
void
mono_sgen_add_to_global_remset (gpointer ptr)
{
RememberedSet *rs;
gboolean lock = major_collector.is_parallel;
if (use_cardtable) {
sgen_card_table_mark_address ((mword)ptr);
return;
}
g_assert (!ptr_in_nursery (ptr) && ptr_in_nursery (*(gpointer*)ptr));
if (lock)
LOCK_GLOBAL_REMSET;
if (!global_remset_location_was_not_added (ptr))
goto done;
DEBUG (8, fprintf (gc_debug_file, "Adding global remset for %p\n", ptr));
binary_protocol_global_remset (ptr, *(gpointer*)ptr, (gpointer)LOAD_VTABLE (*(gpointer*)ptr));
HEAVY_STAT (++stat_global_remsets_added);
/*
* FIXME: If an object remains pinned, we need to add it at every minor collection.
* To avoid uncontrolled growth of the global remset, only add each pointer once.
*/
if (global_remset->store_next + 3 < global_remset->end_set) {
*(global_remset->store_next++) = (mword)ptr;
goto done;
}
rs = alloc_remset (global_remset->end_set - global_remset->data, NULL, TRUE);
rs->next = global_remset;
global_remset = rs;
*(global_remset->store_next++) = (mword)ptr;
{
int global_rs_size = 0;
for (rs = global_remset; rs; rs = rs->next) {
global_rs_size += rs->store_next - rs->data;
}
DEBUG (4, fprintf (gc_debug_file, "Global remset now has size %d\n", global_rs_size));
}
done:
if (lock)
UNLOCK_GLOBAL_REMSET;
}
/*
* drain_gray_stack:
*
* Scan objects in the gray stack until the stack is empty. This should be called
* frequently after each object is copied, to achieve better locality and cache
* usage.
*/
static gboolean
drain_gray_stack (GrayQueue *queue, int max_objs)
{
char *obj;
if (current_collection_generation == GENERATION_NURSERY) {
for (;;) {
GRAY_OBJECT_DEQUEUE (queue, obj);
if (!obj)
return TRUE;
DEBUG (9, fprintf (gc_debug_file, "Precise gray object scan %p (%s)\n", obj, safe_name (obj)));
major_collector.minor_scan_object (obj, queue);
}
} else {
int i;
if (major_collector.is_parallel && queue == &workers_distribute_gray_queue)
return TRUE;
do {
for (i = 0; i != max_objs; ++i) {
GRAY_OBJECT_DEQUEUE (queue, obj);
if (!obj)
return TRUE;
DEBUG (9, fprintf (gc_debug_file, "Precise gray object scan %p (%s)\n", obj, safe_name (obj)));
major_collector.major_scan_object (obj, queue);
}
} while (max_objs < 0);
return FALSE;
}
}
/*
* Addresses from start to end are already sorted. This function finds
* the object header for each address and pins the object. The
* addresses must be inside the passed section. The (start of the)
* address array is overwritten with the addresses of the actually
* pinned objects. Return the number of pinned objects.
*/
static int
pin_objects_from_addresses (GCMemSection *section, void **start, void **end, void *start_nursery, void *end_nursery, GrayQueue *queue)
{
void *last = NULL;
int count = 0;
void *search_start;
void *last_obj = NULL;
size_t last_obj_size = 0;
void *addr;
int idx;
void **definitely_pinned = start;
Fragment *frag;
/*
* The code below starts the search from an entry in scan_starts, which might point into a nursery
* fragment containing random data. Clearing the nursery fragments takes a lot of time, and searching
* though them too, so lay arrays at each location inside a fragment where a search can start:
* - scan_locations[i]
* - start_nursery
* - the start of each fragment (the last_obj + last_obj case)
* The third encompasses the first two, since scan_locations [i] can't point inside a nursery fragment.
*/
for (frag = nursery_fragments; frag; frag = frag->next) {
MonoArray *o;
g_assert (frag->fragment_end - frag->fragment_start >= sizeof (MonoArray));
o = (MonoArray*)frag->fragment_start;
memset (o, 0, sizeof (MonoArray));
g_assert (array_fill_vtable);
o->obj.vtable = array_fill_vtable;
/* Mark this as not a real object */
o->obj.synchronisation = GINT_TO_POINTER (-1);
o->max_length = (frag->fragment_end - frag->fragment_start) - sizeof (MonoArray);
g_assert (frag->fragment_start + safe_object_get_size ((MonoObject*)o) == frag->fragment_end);
}
while (start < end) {
addr = *start;
/* the range check should be reduntant */
if (addr != last && addr >= start_nursery && addr < end_nursery) {
DEBUG (5, fprintf (gc_debug_file, "Considering pinning addr %p\n", addr));
/* multiple pointers to the same object */
if (addr >= last_obj && (char*)addr < (char*)last_obj + last_obj_size) {
start++;
continue;
}
idx = ((char*)addr - (char*)section->data) / SCAN_START_SIZE;
g_assert (idx < section->num_scan_start);
search_start = (void*)section->scan_starts [idx];
if (!search_start || search_start > addr) {
while (idx) {
--idx;
search_start = section->scan_starts [idx];
if (search_start && search_start <= addr)
break;
}
if (!search_start || search_start > addr)
search_start = start_nursery;
}
if (search_start < last_obj)
search_start = (char*)last_obj + last_obj_size;
/* now addr should be in an object a short distance from search_start
* Note that search_start must point to zeroed mem or point to an object.
*/
do {
if (!*(void**)search_start) {
/* Consistency check */
/*
for (frag = nursery_fragments; frag; frag = frag->next) {
if (search_start >= frag->fragment_start && search_start < frag->fragment_end)
g_assert_not_reached ();
}
*/
search_start = (void*)ALIGN_UP ((mword)search_start + sizeof (gpointer));
continue;
}
last_obj = search_start;
last_obj_size = ALIGN_UP (safe_object_get_size ((MonoObject*)search_start));
if (((MonoObject*)last_obj)->synchronisation == GINT_TO_POINTER (-1)) {
/* Marks the beginning of a nursery fragment, skip */
} else {
DEBUG (8, fprintf (gc_debug_file, "Pinned try match %p (%s), size %zd\n", last_obj, safe_name (last_obj), last_obj_size));
if (addr >= search_start && (char*)addr < (char*)last_obj + last_obj_size) {
DEBUG (4, fprintf (gc_debug_file, "Pinned object %p, vtable %p (%s), count %d\n", search_start, *(void**)search_start, safe_name (search_start), count));
binary_protocol_pin (search_start, (gpointer)LOAD_VTABLE (search_start), safe_object_get_size (search_start));
pin_object (search_start);
GRAY_OBJECT_ENQUEUE (queue, search_start);
if (heap_dump_file)
mono_sgen_pin_stats_register_object (search_start, last_obj_size);
definitely_pinned [count] = search_start;
count++;
break;
}
}
/* skip to the next object */
search_start = (void*)((char*)search_start + last_obj_size);
} while (search_start <= addr);
/* we either pinned the correct object or we ignored the addr because
* it points to unused zeroed memory.
*/
last = addr;
}
start++;
}
//printf ("effective pinned: %d (at the end: %d)\n", count, (char*)end_nursery - (char*)last);
if (mono_profiler_get_events () & MONO_PROFILE_GC_ROOTS) {
GCRootReport report;
report.count = 0;
for (idx = 0; idx < count; ++idx)
add_profile_gc_root (&report, definitely_pinned [idx], MONO_PROFILE_GC_ROOT_PINNING, 0);
notify_gc_roots (&report);
}
stat_pinned_objects += count;
return count;
}
void
mono_sgen_pin_objects_in_section (GCMemSection *section, GrayQueue *queue)
{
int num_entries = section->pin_queue_num_entries;
if (num_entries) {
void **start = section->pin_queue_start;
int reduced_to;
reduced_to = pin_objects_from_addresses (section, start, start + num_entries,
section->data, section->next_data, queue);
section->pin_queue_num_entries = reduced_to;
if (!reduced_to)
section->pin_queue_start = NULL;
}
}
void
mono_sgen_pin_object (void *object, GrayQueue *queue)
{
if (major_collector.is_parallel) {
LOCK_PIN_QUEUE;
/*object arrives pinned*/
pin_stage_ptr (object);
++objects_pinned ;
UNLOCK_PIN_QUEUE;
} else {
SGEN_PIN_OBJECT (object);
pin_stage_ptr (object);
++objects_pinned;
}
GRAY_OBJECT_ENQUEUE (queue, object);
}
/* Sort the addresses in array in increasing order.
* Done using a by-the book heap sort. Which has decent and stable performance, is pretty cache efficient.
*/
static void
sort_addresses (void **array, int size)
{
int i;
void *tmp;
for (i = 1; i < size; ++i) {
int child = i;
while (child > 0) {
int parent = (child - 1) / 2;
if (array [parent] >= array [child])
break;
tmp = array [parent];
array [parent] = array [child];
array [child] = tmp;
child = parent;
}
}
for (i = size - 1; i > 0; --i) {
int end, root;
tmp = array [i];
array [i] = array [0];
array [0] = tmp;
end = i - 1;
root = 0;
while (root * 2 + 1 <= end) {
int child = root * 2 + 1;
if (child < end && array [child] < array [child + 1])
++child;
if (array [root] >= array [child])
break;
tmp = array [root];
array [root] = array [child];
array [child] = tmp;
root = child;
}
}
}
static G_GNUC_UNUSED void
print_nursery_gaps (void* start_nursery, void *end_nursery)
{
int i;
gpointer first = start_nursery;
gpointer next;
for (i = 0; i < next_pin_slot; ++i) {
next = pin_queue [i];
fprintf (gc_debug_file, "Nursery range: %p-%p, size: %td\n", first, next, (char*)next-(char*)first);
first = next;
}
next = end_nursery;
fprintf (gc_debug_file, "Nursery range: %p-%p, size: %td\n", first, next, (char*)next-(char*)first);
}
/* reduce the info in the pin queue, removing duplicate pointers and sorting them */
static void
optimize_pin_queue (int start_slot)
{
void **start, **cur, **end;
/* sort and uniq pin_queue: we just sort and we let the rest discard multiple values */
/* it may be better to keep ranges of pinned memory instead of individually pinning objects */
DEBUG (5, fprintf (gc_debug_file, "Sorting pin queue, size: %d\n", next_pin_slot));
if ((next_pin_slot - start_slot) > 1)
sort_addresses (pin_queue + start_slot, next_pin_slot - start_slot);
start = cur = pin_queue + start_slot;
end = pin_queue + next_pin_slot;
while (cur < end) {
*start = *cur++;
while (*start == *cur && cur < end)
cur++;
start++;
};
next_pin_slot = start - pin_queue;
DEBUG (5, fprintf (gc_debug_file, "Pin queue reduced to size: %d\n", next_pin_slot));
//DEBUG (6, print_nursery_gaps (start_nursery, end_nursery));
}
/*
* Scan the memory between start and end and queue values which could be pointers
* to the area between start_nursery and end_nursery for later consideration.
* Typically used for thread stacks.
*/
static void
conservatively_pin_objects_from (void **start, void **end, void *start_nursery, void *end_nursery, int pin_type)
{
int count = 0;
while (start < end) {
if (*start >= start_nursery && *start < end_nursery) {
/*
* *start can point to the middle of an object
* note: should we handle pointing at the end of an object?
* pinning in C# code disallows pointing at the end of an object
* but there is some small chance that an optimizing C compiler
* may keep the only reference to an object by pointing
* at the end of it. We ignore this small chance for now.
* Pointers to the end of an object are indistinguishable
* from pointers to the start of the next object in memory
* so if we allow that we'd need to pin two objects...
* We queue the pointer in an array, the
* array will then be sorted and uniqued. This way
* we can coalesce several pinning pointers and it should
* be faster since we'd do a memory scan with increasing
* addresses. Note: we can align the address to the allocation
* alignment, so the unique process is more effective.
*/
mword addr = (mword)*start;
addr &= ~(ALLOC_ALIGN - 1);
if (addr >= (mword)start_nursery && addr < (mword)end_nursery)
pin_stage_ptr ((void*)addr);
if (heap_dump_file)
pin_stats_register_address ((char*)addr, pin_type);
DEBUG (6, if (count) fprintf (gc_debug_file, "Pinning address %p from %p\n", (void*)addr, start));
count++;
}
start++;
}
DEBUG (7, if (count) fprintf (gc_debug_file, "found %d potential pinned heap pointers\n", count));
}
/*
* Debugging function: find in the conservative roots where @obj is being pinned.
*/
static G_GNUC_UNUSED void
find_pinning_reference (char *obj, size_t size)
{
RootRecord *root;
int i;
char *endobj = obj + size;
for (i = 0; i < roots_hash_size [0]; ++i) {
for (root = roots_hash [0][i]; root; root = root->next) {
/* if desc is non-null it has precise info */
if (!root->root_desc) {
char ** start = (char**)root->start_root;
while (start < (char**)root->end_root) {
if (*start >= obj && *start < endobj) {
DEBUG (0, fprintf (gc_debug_file, "Object %p referenced in pinned roots %p-%p (at %p in record %p)\n", obj, root->start_root, root->end_root, start, root));
}
start++;
}
}
}
}
find_pinning_ref_from_thread (obj, size);
}
/*
* The first thing we do in a collection is to identify pinned objects.
* This function considers all the areas of memory that need to be
* conservatively scanned.
*/
static void
pin_from_roots (void *start_nursery, void *end_nursery, GrayQueue *queue)
{
RootRecord *root;
int i;
DEBUG (2, fprintf (gc_debug_file, "Scanning pinned roots (%d bytes, %d/%d entries)\n", (int)roots_size, num_roots_entries [ROOT_TYPE_NORMAL], num_roots_entries [ROOT_TYPE_PINNED]));
/* objects pinned from the API are inside these roots */
for (i = 0; i < roots_hash_size [ROOT_TYPE_PINNED]; ++i) {
for (root = roots_hash [ROOT_TYPE_PINNED][i]; root; root = root->next) {
DEBUG (6, fprintf (gc_debug_file, "Pinned roots %p-%p\n", root->start_root, root->end_root));
conservatively_pin_objects_from ((void**)root->start_root, (void**)root->end_root, start_nursery, end_nursery, PIN_TYPE_OTHER);
}
}
/* now deal with the thread stacks
* in the future we should be able to conservatively scan only:
* *) the cpu registers
* *) the unmanaged stack frames
* *) the _last_ managed stack frame
* *) pointers slots in managed frames
*/
scan_thread_data (start_nursery, end_nursery, FALSE, queue);
evacuate_pin_staging_area ();
}
typedef struct {
CopyOrMarkObjectFunc func;
GrayQueue *queue;
} UserCopyOrMarkData;
static pthread_key_t user_copy_or_mark_key;
static void
init_user_copy_or_mark_key (void)
{
pthread_key_create (&user_copy_or_mark_key, NULL);
}
static void
set_user_copy_or_mark_data (UserCopyOrMarkData *data)
{
static pthread_once_t init_control = PTHREAD_ONCE_INIT;
pthread_once (&init_control, init_user_copy_or_mark_key);
pthread_setspecific (user_copy_or_mark_key, data);
}
static void
single_arg_user_copy_or_mark (void **obj)
{
UserCopyOrMarkData *data = pthread_getspecific (user_copy_or_mark_key);
data->func (obj, data->queue);
}
/*
* The memory area from start_root to end_root contains pointers to objects.
* Their position is precisely described by @desc (this means that the pointer
* can be either NULL or the pointer to the start of an object).
* This functions copies them to to_space updates them.
*
* This function is not thread-safe!
*/
static void
precisely_scan_objects_from (CopyOrMarkObjectFunc copy_func, void** start_root, void** end_root, char* n_start, char *n_end, mword desc, GrayQueue *queue)
{
switch (desc & ROOT_DESC_TYPE_MASK) {
case ROOT_DESC_BITMAP:
desc >>= ROOT_DESC_TYPE_SHIFT;
while (desc) {
if ((desc & 1) && *start_root) {
copy_func (start_root, queue);
DEBUG (9, fprintf (gc_debug_file, "Overwrote root at %p with %p\n", start_root, *start_root));
drain_gray_stack (queue, -1);
}
desc >>= 1;
start_root++;
}
return;
case ROOT_DESC_COMPLEX: {
gsize *bitmap_data = complex_descriptors + (desc >> ROOT_DESC_TYPE_SHIFT);
int bwords = (*bitmap_data) - 1;
void **start_run = start_root;
bitmap_data++;
while (bwords-- > 0) {
gsize bmap = *bitmap_data++;
void **objptr = start_run;
while (bmap) {
if ((bmap & 1) && *objptr) {
copy_func (objptr, queue);
DEBUG (9, fprintf (gc_debug_file, "Overwrote root at %p with %p\n", objptr, *objptr));
drain_gray_stack (queue, -1);
}
bmap >>= 1;
++objptr;
}
start_run += GC_BITS_PER_WORD;
}
break;
}
case ROOT_DESC_USER: {
UserCopyOrMarkData data = { copy_func, queue };
MonoGCRootMarkFunc marker = user_descriptors [desc >> ROOT_DESC_TYPE_SHIFT];
set_user_copy_or_mark_data (&data);
marker (start_root, single_arg_user_copy_or_mark);
set_user_copy_or_mark_data (NULL);
break;
}
case ROOT_DESC_RUN_LEN:
g_assert_not_reached ();
default:
g_assert_not_reached ();
}
}
static void
reset_heap_boundaries (void)
{
lowest_heap_address = ~(mword)0;
highest_heap_address = 0;
}
void
mono_sgen_update_heap_boundaries (mword low, mword high)
{
mword old;
do {
old = lowest_heap_address;
if (low >= old)
break;
} while (SGEN_CAS_PTR ((gpointer*)&lowest_heap_address, (gpointer)low, (gpointer)old) != (gpointer)old);
do {
old = highest_heap_address;
if (high <= old)
break;
} while (SGEN_CAS_PTR ((gpointer*)&highest_heap_address, (gpointer)high, (gpointer)old) != (gpointer)old);
}
static Fragment*
alloc_fragment (void)
{
Fragment *frag = fragment_freelist;
if (frag) {
fragment_freelist = frag->next;
frag->next = NULL;
return frag;
}
frag = mono_sgen_alloc_internal (INTERNAL_MEM_FRAGMENT);
frag->next = NULL;
return frag;
}
static void
add_fragment (char *start, char *end)
{
Fragment *fragment;
fragment = alloc_fragment ();
fragment->fragment_start = start;
fragment->fragment_limit = start;
fragment->fragment_end = end;
fragment->next = nursery_fragments;
nursery_fragments = fragment;
}
/* size must be a power of 2 */
void*
mono_sgen_alloc_os_memory_aligned (mword size, mword alignment, gboolean activate)
{
/* Allocate twice the memory to be able to put the block on an aligned address */
char *mem = mono_sgen_alloc_os_memory (size + alignment, activate);
char *aligned;
g_assert (mem);
aligned = (char*)((mword)(mem + (alignment - 1)) & ~(alignment - 1));
g_assert (aligned >= mem && aligned + size <= mem + size + alignment && !((mword)aligned & (alignment - 1)));
if (aligned > mem)
mono_sgen_free_os_memory (mem, aligned - mem);
if (aligned + size < mem + size + alignment)
mono_sgen_free_os_memory (aligned + size, (mem + size + alignment) - (aligned + size));
return aligned;
}
/*
* Allocate and setup the data structures needed to be able to allocate objects
* in the nursery. The nursery is stored in nursery_section.
*/
static void
alloc_nursery (void)
{
GCMemSection *section;
char *data;
int scan_starts;
int alloc_size;
if (nursery_section)
return;
DEBUG (2, fprintf (gc_debug_file, "Allocating nursery size: %lu\n", (unsigned long)nursery_size));
/* later we will alloc a larger area for the nursery but only activate
* what we need. The rest will be used as expansion if we have too many pinned
* objects in the existing nursery.
*/
/* FIXME: handle OOM */
section = mono_sgen_alloc_internal (INTERNAL_MEM_SECTION);
g_assert (nursery_size == DEFAULT_NURSERY_SIZE);
alloc_size = nursery_size;
#ifdef SGEN_ALIGN_NURSERY
data = major_collector.alloc_heap (alloc_size, alloc_size, DEFAULT_NURSERY_BITS);
#else
data = major_collector.alloc_heap (alloc_size, 0, DEFAULT_NURSERY_BITS);
#endif
nursery_start = data;
nursery_end = nursery_start + nursery_size;
mono_sgen_update_heap_boundaries ((mword)nursery_start, (mword)nursery_end);
DEBUG (4, fprintf (gc_debug_file, "Expanding nursery size (%p-%p): %lu, total: %lu\n", data, data + alloc_size, (unsigned long)nursery_size, (unsigned long)total_alloc));
section->data = section->next_data = data;
section->size = alloc_size;
section->end_data = nursery_end;
scan_starts = (alloc_size + SCAN_START_SIZE - 1) / SCAN_START_SIZE;
section->scan_starts = mono_sgen_alloc_internal_dynamic (sizeof (char*) * scan_starts, INTERNAL_MEM_SCAN_STARTS);
section->num_scan_start = scan_starts;
section->block.role = MEMORY_ROLE_GEN0;
section->block.next = NULL;
nursery_section = section;
/* Setup the single first large fragment */
add_fragment (nursery_start, nursery_end);
}
void*
mono_gc_get_nursery (int *shift_bits, size_t *size)
{
*size = nursery_size;
#ifdef SGEN_ALIGN_NURSERY
*shift_bits = DEFAULT_NURSERY_BITS;
#else
*shift_bits = -1;
#endif
return nursery_start;
}
gboolean
mono_gc_precise_stack_mark_enabled (void)
{
return !conservative_stack_mark;
}
FILE *
mono_gc_get_logfile (void)
{
return mono_sgen_get_logfile ();
}
static void
report_finalizer_roots_list (FinalizeEntry *list)
{
GCRootReport report;
FinalizeEntry *fin;
report.count = 0;
for (fin = list; fin; fin = fin->next) {
if (!fin->object)
continue;
add_profile_gc_root (&report, fin->object, MONO_PROFILE_GC_ROOT_FINALIZER, 0);
}
notify_gc_roots (&report);
}
static void
report_finalizer_roots (void)
{
report_finalizer_roots_list (fin_ready_list);
report_finalizer_roots_list (critical_fin_list);
}
static GCRootReport *root_report;
static void
single_arg_report_root (void **obj)
{
if (*obj)
add_profile_gc_root (root_report, *obj, MONO_PROFILE_GC_ROOT_OTHER, 0);
}
static void
precisely_report_roots_from (GCRootReport *report, void** start_root, void** end_root, mword desc)
{
switch (desc & ROOT_DESC_TYPE_MASK) {
case ROOT_DESC_BITMAP:
desc >>= ROOT_DESC_TYPE_SHIFT;
while (desc) {
if ((desc & 1) && *start_root) {
add_profile_gc_root (report, *start_root, MONO_PROFILE_GC_ROOT_OTHER, 0);
}
desc >>= 1;
start_root++;
}
return;
case ROOT_DESC_COMPLEX: {
gsize *bitmap_data = complex_descriptors + (desc >> ROOT_DESC_TYPE_SHIFT);
int bwords = (*bitmap_data) - 1;
void **start_run = start_root;
bitmap_data++;
while (bwords-- > 0) {
gsize bmap = *bitmap_data++;
void **objptr = start_run;
while (bmap) {
if ((bmap & 1) && *objptr) {
add_profile_gc_root (report, *objptr, MONO_PROFILE_GC_ROOT_OTHER, 0);
}
bmap >>= 1;
++objptr;
}
start_run += GC_BITS_PER_WORD;
}
break;
}
case ROOT_DESC_USER: {
MonoGCRootMarkFunc marker = user_descriptors [desc >> ROOT_DESC_TYPE_SHIFT];
root_report = report;
marker (start_root, single_arg_report_root);
break;
}
case ROOT_DESC_RUN_LEN:
g_assert_not_reached ();
default:
g_assert_not_reached ();
}
}
static void
report_registered_roots_by_type (int root_type)
{
GCRootReport report;
int i;
RootRecord *root;
report.count = 0;
for (i = 0; i < roots_hash_size [root_type]; ++i) {
for (root = roots_hash [root_type][i]; root; root = root->next) {
DEBUG (6, fprintf (gc_debug_file, "Precise root scan %p-%p (desc: %p)\n", root->start_root, root->end_root, (void*)root->root_desc));
precisely_report_roots_from (&report, (void**)root->start_root, (void**)root->end_root, root->root_desc);
}
}
notify_gc_roots (&report);
}
static void
report_registered_roots (void)
{
report_registered_roots_by_type (ROOT_TYPE_NORMAL);
report_registered_roots_by_type (ROOT_TYPE_WBARRIER);
}
static void
scan_finalizer_entries (CopyOrMarkObjectFunc copy_func, FinalizeEntry *list, GrayQueue *queue)
{
FinalizeEntry *fin;
for (fin = list; fin; fin = fin->next) {
if (!fin->object)
continue;
DEBUG (5, fprintf (gc_debug_file, "Scan of fin ready object: %p (%s)\n", fin->object, safe_name (fin->object)));
copy_func (&fin->object, queue);
}
}
static mword fragment_total = 0;
/*
* We found a fragment of free memory in the nursery: memzero it and if
* it is big enough, add it to the list of fragments that can be used for
* allocation.
*/
static void
add_nursery_frag (size_t frag_size, char* frag_start, char* frag_end)
{
DEBUG (4, fprintf (gc_debug_file, "Found empty fragment: %p-%p, size: %zd\n", frag_start, frag_end, frag_size));
binary_protocol_empty (frag_start, frag_size);
/* Not worth dealing with smaller fragments: need to tune */
if (frag_size >= FRAGMENT_MIN_SIZE) {
/* memsetting just the first chunk start is bound to provide better cache locality */
if (nursery_clear_policy == CLEAR_AT_GC)
memset (frag_start, 0, frag_size);
add_fragment (frag_start, frag_end);
fragment_total += frag_size;
} else {
/* Clear unused fragments, pinning depends on this */
/*TODO place an int[] here instead of the memset if size justify it*/
memset (frag_start, 0, frag_size);
}
}
static const char*
generation_name (int generation)
{
switch (generation) {
case GENERATION_NURSERY: return "nursery";
case GENERATION_OLD: return "old";
default: g_assert_not_reached ();
}
}
static MonoObject **finalized_array = NULL;
static int finalized_array_capacity = 0;
static int finalized_array_entries = 0;
static void
bridge_register_finalized_object (MonoObject *object)
{
if (!finalized_array)
return;
if (finalized_array_entries >= finalized_array_capacity) {
MonoObject **new_array;
g_assert (finalized_array_entries == finalized_array_capacity);
finalized_array_capacity *= 2;
new_array = mono_sgen_alloc_internal_dynamic (sizeof (MonoObject*) * finalized_array_capacity, INTERNAL_MEM_BRIDGE_DATA);
memcpy (new_array, finalized_array, sizeof (MonoObject*) * finalized_array_entries);
mono_sgen_free_internal_dynamic (finalized_array, sizeof (MonoObject*) * finalized_array_entries, INTERNAL_MEM_BRIDGE_DATA);
finalized_array = new_array;
}
finalized_array [finalized_array_entries++] = object;
}
static void
finish_gray_stack (char *start_addr, char *end_addr, int generation, GrayQueue *queue)
{
TV_DECLARE (atv);
TV_DECLARE (btv);
int fin_ready;
int ephemeron_rounds = 0;
int num_loops;
CopyOrMarkObjectFunc copy_func = current_collection_generation == GENERATION_NURSERY ? major_collector.copy_object : major_collector.copy_or_mark_object;
/*
* We copied all the reachable objects. Now it's the time to copy
* the objects that were not referenced by the roots, but by the copied objects.
* we built a stack of objects pointed to by gray_start: they are
* additional roots and we may add more items as we go.
* We loop until gray_start == gray_objects which means no more objects have
* been added. Note this is iterative: no recursion is involved.
* We need to walk the LO list as well in search of marked big objects
* (use a flag since this is needed only on major collections). We need to loop
* here as well, so keep a counter of marked LO (increasing it in copy_object).
* To achieve better cache locality and cache usage, we drain the gray stack
* frequently, after each object is copied, and just finish the work here.
*/
drain_gray_stack (queue, -1);
TV_GETTIME (atv);
DEBUG (2, fprintf (gc_debug_file, "%s generation done\n", generation_name (generation)));
/*
We must clear weak links that don't track resurrection before processing object ready for
finalization so they can be cleared before that.
*/
null_link_in_range (copy_func, start_addr, end_addr, generation, TRUE, queue);
if (generation == GENERATION_OLD)
null_link_in_range (copy_func, start_addr, end_addr, GENERATION_NURSERY, TRUE, queue);
if (finalized_array == NULL && mono_sgen_need_bridge_processing ()) {
finalized_array_capacity = 32;
finalized_array = mono_sgen_alloc_internal_dynamic (sizeof (MonoObject*) * finalized_array_capacity, INTERNAL_MEM_BRIDGE_DATA);
}
finalized_array_entries = 0;
/* walk the finalization queue and move also the objects that need to be
* finalized: use the finalized objects as new roots so the objects they depend
* on are also not reclaimed. As with the roots above, only objects in the nursery
* are marked/copied.
* We need a loop here, since objects ready for finalizers may reference other objects
* that are fin-ready. Speedup with a flag?
*/
num_loops = 0;
do {
/*
* Walk the ephemeron tables marking all values with reachable keys. This must be completely done
* before processing finalizable objects to avoid finalizing reachable values.
*
* It must be done inside the finalizaters loop since objects must not be removed from CWT tables
* while they are been finalized.
*/
int done_with_ephemerons = 0;
do {
done_with_ephemerons = mark_ephemerons_in_range (copy_func, start_addr, end_addr, queue);
drain_gray_stack (queue, -1);
++ephemeron_rounds;
} while (!done_with_ephemerons);
fin_ready = num_ready_finalizers;
finalize_in_range (copy_func, start_addr, end_addr, generation, queue);
if (generation == GENERATION_OLD)
finalize_in_range (copy_func, nursery_start, nursery_end, GENERATION_NURSERY, queue);
if (fin_ready != num_ready_finalizers) {
++num_loops;
if (finalized_array != NULL)
mono_sgen_bridge_processing (finalized_array_entries, finalized_array);
}
/* drain the new stack that might have been created */
DEBUG (6, fprintf (gc_debug_file, "Precise scan of gray area post fin\n"));
drain_gray_stack (queue, -1);
} while (fin_ready != num_ready_finalizers);
if (mono_sgen_need_bridge_processing ())
g_assert (num_loops <= 1);
/*
* Clear ephemeron pairs with unreachable keys.
* We pass the copy func so we can figure out if an array was promoted or not.
*/
clear_unreachable_ephemerons (copy_func, start_addr, end_addr, queue);
TV_GETTIME (btv);
DEBUG (2, fprintf (gc_debug_file, "Finalize queue handling scan for %s generation: %d usecs %d ephemeron roundss\n", generation_name (generation), TV_ELAPSED (atv, btv), ephemeron_rounds));
/*
* handle disappearing links
* Note we do this after checking the finalization queue because if an object
* survives (at least long enough to be finalized) we don't clear the link.
* This also deals with a possible issue with the monitor reclamation: with the Boehm
* GC a finalized object my lose the monitor because it is cleared before the finalizer is
* called.
*/
g_assert (gray_object_queue_is_empty (queue));
for (;;) {
null_link_in_range (copy_func, start_addr, end_addr, generation, FALSE, queue);
if (generation == GENERATION_OLD)
null_link_in_range (copy_func, start_addr, end_addr, GENERATION_NURSERY, FALSE, queue);
if (gray_object_queue_is_empty (queue))
break;
drain_gray_stack (queue, -1);
}
g_assert (gray_object_queue_is_empty (queue));
}
void
mono_sgen_check_section_scan_starts (GCMemSection *section)
{
int i;
for (i = 0; i < section->num_scan_start; ++i) {
if (section->scan_starts [i]) {
guint size = safe_object_get_size ((MonoObject*) section->scan_starts [i]);
g_assert (size >= sizeof (MonoObject) && size <= MAX_SMALL_OBJ_SIZE);
}
}
}
static void
check_scan_starts (void)
{
if (!do_scan_starts_check)
return;
mono_sgen_check_section_scan_starts (nursery_section);
major_collector.check_scan_starts ();
}
static int last_num_pinned = 0;
static void
build_nursery_fragments (void **start, int num_entries)
{
char *frag_start, *frag_end;
size_t frag_size;
int i;
while (nursery_fragments) {
Fragment *next = nursery_fragments->next;
nursery_fragments->next = fragment_freelist;
fragment_freelist = nursery_fragments;
nursery_fragments = next;
}
frag_start = nursery_start;
fragment_total = 0;
/* clear scan starts */
memset (nursery_section->scan_starts, 0, nursery_section->num_scan_start * sizeof (gpointer));
for (i = 0; i < num_entries; ++i) {
frag_end = start [i];
/* remove the pin bit from pinned objects */
unpin_object (frag_end);
nursery_section->scan_starts [((char*)frag_end - (char*)nursery_section->data)/SCAN_START_SIZE] = frag_end;
frag_size = frag_end - frag_start;
if (frag_size)
add_nursery_frag (frag_size, frag_start, frag_end);
frag_size = ALIGN_UP (safe_object_get_size ((MonoObject*)start [i]));
frag_start = (char*)start [i] + frag_size;
}
nursery_last_pinned_end = frag_start;
frag_end = nursery_end;
frag_size = frag_end - frag_start;
if (frag_size)
add_nursery_frag (frag_size, frag_start, frag_end);
if (!nursery_fragments) {
DEBUG (1, fprintf (gc_debug_file, "Nursery fully pinned (%d)\n", num_entries));
for (i = 0; i < num_entries; ++i) {
DEBUG (3, fprintf (gc_debug_file, "Bastard pinning obj %p (%s), size: %d\n", start [i], safe_name (start [i]), safe_object_get_size (start [i])));
}
degraded_mode = 1;
}
nursery_next = nursery_frag_real_end = NULL;
/* Clear TLABs for all threads */
clear_tlabs ();
}
static void
scan_from_registered_roots (CopyOrMarkObjectFunc copy_func, char *addr_start, char *addr_end, int root_type, GrayQueue *queue)
{
int i;
RootRecord *root;
for (i = 0; i < roots_hash_size [root_type]; ++i) {
for (root = roots_hash [root_type][i]; root; root = root->next) {
DEBUG (6, fprintf (gc_debug_file, "Precise root scan %p-%p (desc: %p)\n", root->start_root, root->end_root, (void*)root->root_desc));
precisely_scan_objects_from (copy_func, (void**)root->start_root, (void**)root->end_root, addr_start, addr_end, root->root_desc, queue);
}
}
}
void
mono_sgen_dump_occupied (char *start, char *end, char *section_start)
{
fprintf (heap_dump_file, "<occupied offset=\"%td\" size=\"%td\"/>\n", start - section_start, end - start);
}
void
mono_sgen_dump_section (GCMemSection *section, const char *type)
{
char *start = section->data;
char *end = section->data + section->size;
char *occ_start = NULL;
GCVTable *vt;
char *old_start = NULL; /* just for debugging */
fprintf (heap_dump_file, "<section type=\"%s\" size=\"%lu\">\n", type, (unsigned long)section->size);
while (start < end) {
guint size;
MonoClass *class;
if (!*(void**)start) {
if (occ_start) {
mono_sgen_dump_occupied (occ_start, start, section->data);
occ_start = NULL;
}
start += sizeof (void*); /* should be ALLOC_ALIGN, really */
continue;
}
g_assert (start < section->next_data);
if (!occ_start)
occ_start = start;
vt = (GCVTable*)LOAD_VTABLE (start);
class = vt->klass;
size = ALIGN_UP (safe_object_get_size ((MonoObject*) start));
/*
fprintf (heap_dump_file, "<object offset=\"%d\" class=\"%s.%s\" size=\"%d\"/>\n",
start - section->data,
vt->klass->name_space, vt->klass->name,
size);
*/
old_start = start;
start += size;
}
if (occ_start)
mono_sgen_dump_occupied (occ_start, start, section->data);
fprintf (heap_dump_file, "</section>\n");
}
static void
dump_object (MonoObject *obj, gboolean dump_location)
{
static char class_name [1024];
MonoClass *class = mono_object_class (obj);
int i, j;
/*
* Python's XML parser is too stupid to parse angle brackets
* in strings, so we just ignore them;
*/
i = j = 0;
while (class->name [i] && j < sizeof (class_name) - 1) {
if (!strchr ("<>\"", class->name [i]))
class_name [j++] = class->name [i];
++i;
}
g_assert (j < sizeof (class_name));
class_name [j] = 0;
fprintf (heap_dump_file, "<object class=\"%s.%s\" size=\"%d\"",
class->name_space, class_name,
safe_object_get_size (obj));
if (dump_location) {
const char *location;
if (ptr_in_nursery (obj))
location = "nursery";
else if (safe_object_get_size (obj) <= MAX_SMALL_OBJ_SIZE)
location = "major";
else
location = "LOS";
fprintf (heap_dump_file, " location=\"%s\"", location);
}
fprintf (heap_dump_file, "/>\n");
}
static void
dump_heap (const char *type, int num, const char *reason)
{
ObjectList *list;
LOSObject *bigobj;
fprintf (heap_dump_file, "<collection type=\"%s\" num=\"%d\"", type, num);
if (reason)
fprintf (heap_dump_file, " reason=\"%s\"", reason);
fprintf (heap_dump_file, ">\n");
fprintf (heap_dump_file, "<other-mem-usage type=\"mempools\" size=\"%ld\"/>\n", mono_mempool_get_bytes_allocated ());
mono_sgen_dump_internal_mem_usage (heap_dump_file);
fprintf (heap_dump_file, "<pinned type=\"stack\" bytes=\"%zu\"/>\n", pinned_byte_counts [PIN_TYPE_STACK]);
/* fprintf (heap_dump_file, "<pinned type=\"static-data\" bytes=\"%d\"/>\n", pinned_byte_counts [PIN_TYPE_STATIC_DATA]); */
fprintf (heap_dump_file, "<pinned type=\"other\" bytes=\"%zu\"/>\n", pinned_byte_counts [PIN_TYPE_OTHER]);
fprintf (heap_dump_file, "<pinned-objects>\n");
for (list = pinned_objects; list; list = list->next)
dump_object (list->obj, TRUE);
fprintf (heap_dump_file, "</pinned-objects>\n");
mono_sgen_dump_section (nursery_section, "nursery");
major_collector.dump_heap (heap_dump_file);
fprintf (heap_dump_file, "<los>\n");
for (bigobj = los_object_list; bigobj; bigobj = bigobj->next)
dump_object ((MonoObject*)bigobj->data, FALSE);
fprintf (heap_dump_file, "</los>\n");
fprintf (heap_dump_file, "</collection>\n");
}
void
mono_sgen_register_moved_object (void *obj, void *destination)
{
g_assert (mono_profiler_events & MONO_PROFILE_GC_MOVES);
/* FIXME: handle this for parallel collector */
g_assert (!major_collector.is_parallel);
if (moved_objects_idx == MOVED_OBJECTS_NUM) {
mono_profiler_gc_moves (moved_objects, moved_objects_idx);
moved_objects_idx = 0;
}
moved_objects [moved_objects_idx++] = obj;
moved_objects [moved_objects_idx++] = destination;
}
static void
init_stats (void)
{
static gboolean inited = FALSE;
if (inited)
return;
mono_counters_register ("Minor fragment clear", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_pre_collection_fragment_clear);
mono_counters_register ("Minor pinning", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_pinning);
mono_counters_register ("Minor scan remsets", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_scan_remsets);
mono_counters_register ("Minor scan cardtables", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_scan_card_table);
mono_counters_register ("Minor scan pinned", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_scan_pinned);
mono_counters_register ("Minor scan registered roots", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_scan_registered_roots);
mono_counters_register ("Minor scan thread data", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_scan_thread_data);
mono_counters_register ("Minor finish gray stack", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_finish_gray_stack);
mono_counters_register ("Minor fragment creation", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_minor_fragment_creation);
mono_counters_register ("Major fragment clear", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_pre_collection_fragment_clear);
mono_counters_register ("Major pinning", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_pinning);
mono_counters_register ("Major scan pinned", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_scan_pinned);
mono_counters_register ("Major scan registered roots", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_scan_registered_roots);
mono_counters_register ("Major scan thread data", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_scan_thread_data);
mono_counters_register ("Major scan alloc_pinned", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_scan_alloc_pinned);
mono_counters_register ("Major scan finalized", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_scan_finalized);
mono_counters_register ("Major scan big objects", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_scan_big_objects);
mono_counters_register ("Major finish gray stack", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_finish_gray_stack);
mono_counters_register ("Major free big objects", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_free_bigobjs);
mono_counters_register ("Major LOS sweep", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_los_sweep);
mono_counters_register ("Major sweep", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_sweep);
mono_counters_register ("Major fragment creation", MONO_COUNTER_GC | MONO_COUNTER_LONG, &time_major_fragment_creation);
mono_counters_register ("Number of pinned objects", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_pinned_objects);
#ifdef HEAVY_STATISTICS
mono_counters_register ("WBarrier set field", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_set_field);
mono_counters_register ("WBarrier set arrayref", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_set_arrayref);
mono_counters_register ("WBarrier arrayref copy", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_arrayref_copy);
mono_counters_register ("WBarrier generic store called", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_generic_store);
mono_counters_register ("WBarrier generic store stored", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_generic_store_remset);
mono_counters_register ("WBarrier set root", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_set_root);
mono_counters_register ("WBarrier value copy", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_value_copy);
mono_counters_register ("WBarrier object copy", MONO_COUNTER_GC | MONO_COUNTER_INT, &stat_wbarrier_object_copy);
mono_counters_register ("# objects allocated", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_objects_alloced);
mono_counters_register ("bytes allocated", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_bytes_alloced);
mono_counters_register ("# objects allocated degraded", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_objects_alloced_degraded);
mono_counters_register ("bytes allocated degraded", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_bytes_alloced_degraded);
mono_counters_register ("bytes allocated in LOS", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_bytes_alloced_los);
mono_counters_register ("# copy_object() called (nursery)", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_copy_object_called_nursery);
mono_counters_register ("# objects copied (nursery)", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_objects_copied_nursery);
mono_counters_register ("# copy_object() called (major)", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_copy_object_called_major);
mono_counters_register ("# objects copied (major)", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_objects_copied_major);
mono_counters_register ("# scan_object() called (nursery)", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_scan_object_called_nursery);
mono_counters_register ("# scan_object() called (major)", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_scan_object_called_major);
mono_counters_register ("# nursery copy_object() failed from space", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_nursery_copy_object_failed_from_space);
mono_counters_register ("# nursery copy_object() failed forwarded", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_nursery_copy_object_failed_forwarded);
mono_counters_register ("# nursery copy_object() failed pinned", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_nursery_copy_object_failed_pinned);
mono_counters_register ("# wasted fragments used", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_wasted_fragments_used);
mono_counters_register ("bytes in wasted fragments", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_wasted_fragments_bytes);
mono_counters_register ("Store remsets", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_store_remsets);
mono_counters_register ("Unique store remsets", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_store_remsets_unique);
mono_counters_register ("Saved remsets 1", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_saved_remsets_1);
mono_counters_register ("Saved remsets 2", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_saved_remsets_2);
mono_counters_register ("Non-global remsets processed", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_local_remsets_processed);
mono_counters_register ("Global remsets added", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_global_remsets_added);
mono_counters_register ("Global remsets re-added", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_global_remsets_readded);
mono_counters_register ("Global remsets processed", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_global_remsets_processed);
mono_counters_register ("Global remsets discarded", MONO_COUNTER_GC | MONO_COUNTER_LONG, &stat_global_remsets_discarded);
#endif
inited = TRUE;
}
static gboolean need_calculate_minor_collection_allowance;
static int last_collection_old_num_major_sections;
static mword last_collection_los_memory_usage = 0;
static mword last_collection_old_los_memory_usage;
static mword last_collection_los_memory_alloced;
static void
reset_minor_collection_allowance (void)
{
need_calculate_minor_collection_allowance = TRUE;
}
static void
try_calculate_minor_collection_allowance (gboolean overwrite)
{
int num_major_sections, num_major_sections_saved, save_target, allowance_target;
mword los_memory_saved;
if (overwrite)
g_assert (need_calculate_minor_collection_allowance);
if (!need_calculate_minor_collection_allowance)
return;
if (!*major_collector.have_swept) {
if (overwrite)
minor_collection_allowance = MIN_MINOR_COLLECTION_ALLOWANCE;
return;
}
num_major_sections = major_collector.get_num_major_sections ();
num_major_sections_saved = MAX (last_collection_old_num_major_sections - num_major_sections, 0);
los_memory_saved = MAX (last_collection_old_los_memory_usage - last_collection_los_memory_usage, 1);
save_target = ((num_major_sections * major_collector.section_size) + los_memory_saved) / 2;
/*
* We aim to allow the allocation of as many sections as is
* necessary to reclaim save_target sections in the next
* collection. We assume the collection pattern won't change.
* In the last cycle, we had num_major_sections_saved for
* minor_collection_sections_alloced. Assuming things won't
* change, this must be the same ratio as save_target for
* allowance_target, i.e.
*
* num_major_sections_saved save_target
* --------------------------------- == ----------------
* minor_collection_sections_alloced allowance_target
*
* hence:
*/
allowance_target = (mword)((double)save_target * (double)(minor_collection_sections_alloced * major_collector.section_size + last_collection_los_memory_alloced) / (double)(num_major_sections_saved * major_collector.section_size + los_memory_saved));
minor_collection_allowance = MAX (MIN (allowance_target, num_major_sections * major_collector.section_size + los_memory_usage), MIN_MINOR_COLLECTION_ALLOWANCE);
if (major_collector.have_computed_minor_collection_allowance)
major_collector.have_computed_minor_collection_allowance ();
need_calculate_minor_collection_allowance = FALSE;
}
static gboolean
need_major_collection (mword space_needed)
{
mword los_alloced = los_memory_usage - MIN (last_collection_los_memory_usage, los_memory_usage);
return (space_needed > available_free_space ()) ||
minor_collection_sections_alloced * major_collector.section_size + los_alloced > minor_collection_allowance;
}
gboolean
mono_sgen_need_major_collection (mword space_needed)
{
return need_major_collection (space_needed);
}
static GrayQueue*
job_gray_queue (WorkerData *worker_data)
{
return worker_data ? &worker_data->private_gray_queue : WORKERS_DISTRIBUTE_GRAY_QUEUE;
}
typedef struct
{
char *heap_start;
char *heap_end;
} ScanFromRemsetsJobData;
static void
job_scan_from_remsets (WorkerData *worker_data, void *job_data_untyped)
{
ScanFromRemsetsJobData *job_data = job_data_untyped;
scan_from_remsets (job_data->heap_start, job_data->heap_end, job_gray_queue (worker_data));
}
typedef struct
{
CopyOrMarkObjectFunc func;
char *heap_start;
char *heap_end;
int root_type;
} ScanFromRegisteredRootsJobData;
static void
job_scan_from_registered_roots (WorkerData *worker_data, void *job_data_untyped)
{
ScanFromRegisteredRootsJobData *job_data = job_data_untyped;
scan_from_registered_roots (job_data->func,
job_data->heap_start, job_data->heap_end,
job_data->root_type,
job_gray_queue (worker_data));
}
typedef struct
{
char *heap_start;
char *heap_end;
} ScanThreadDataJobData;
static void
job_scan_thread_data (WorkerData *worker_data, void *job_data_untyped)
{
ScanThreadDataJobData *job_data = job_data_untyped;
scan_thread_data (job_data->heap_start, job_data->heap_end, TRUE,
job_gray_queue (worker_data));
}
/*
* Collect objects in the nursery. Returns whether to trigger a major
* collection.
*/
static gboolean
collect_nursery (size_t requested_size)
{
gboolean needs_major;
size_t max_garbage_amount;
char *orig_nursery_next;
ScanFromRemsetsJobData sfrjd;
ScanFromRegisteredRootsJobData scrrjd_normal, scrrjd_wbarrier;
ScanThreadDataJobData stdjd;
TV_DECLARE (all_atv);
TV_DECLARE (all_btv);
TV_DECLARE (atv);
TV_DECLARE (btv);
if (disable_minor_collections)
return TRUE;
mono_perfcounters->gc_collections0++;
current_collection_generation = GENERATION_NURSERY;
binary_protocol_collection (GENERATION_NURSERY);
check_scan_starts ();
degraded_mode = 0;
objects_pinned = 0;
orig_nursery_next = nursery_next;
nursery_next = MAX (nursery_next, nursery_last_pinned_end);
/* FIXME: optimize later to use the higher address where an object can be present */
nursery_next = MAX (nursery_next, nursery_end);
DEBUG (1, fprintf (gc_debug_file, "Start nursery collection %d %p-%p, size: %d\n", num_minor_gcs, nursery_start, nursery_next, (int)(nursery_next - nursery_start)));
max_garbage_amount = nursery_next - nursery_start;
g_assert (nursery_section->size >= max_garbage_amount);
/* world must be stopped already */
TV_GETTIME (all_atv);
atv = all_atv;
/* Pinning no longer depends on clearing all nursery fragments */
clear_current_nursery_fragment (orig_nursery_next);
TV_GETTIME (btv);
time_minor_pre_collection_fragment_clear += TV_ELAPSED_MS (atv, btv);
if (xdomain_checks)
check_for_xdomain_refs ();
nursery_section->next_data = nursery_next;
major_collector.start_nursery_collection ();
try_calculate_minor_collection_allowance (FALSE);
gray_object_queue_init (&gray_queue);
workers_init_distribute_gray_queue ();
num_minor_gcs++;
mono_stats.minor_gc_count ++;
global_remset_cache_clear ();
process_fin_stage_entries ();
process_dislink_stage_entries ();
/* pin from pinned handles */
init_pinning ();
mono_profiler_gc_event (MONO_GC_EVENT_MARK_START, 0);
pin_from_roots (nursery_start, nursery_next, WORKERS_DISTRIBUTE_GRAY_QUEUE);
/* identify pinned objects */
optimize_pin_queue (0);
next_pin_slot = pin_objects_from_addresses (nursery_section, pin_queue, pin_queue + next_pin_slot, nursery_start, nursery_next, WORKERS_DISTRIBUTE_GRAY_QUEUE);
nursery_section->pin_queue_start = pin_queue;
nursery_section->pin_queue_num_entries = next_pin_slot;
TV_GETTIME (atv);
time_minor_pinning += TV_ELAPSED_MS (btv, atv);
DEBUG (2, fprintf (gc_debug_file, "Finding pinned pointers: %d in %d usecs\n", next_pin_slot, TV_ELAPSED (btv, atv)));
DEBUG (4, fprintf (gc_debug_file, "Start scan with %d pinned objects\n", next_pin_slot));
if (consistency_check_at_minor_collection)
check_consistency ();
workers_start_all_workers ();
/*
* Walk all the roots and copy the young objects to the old
* generation, starting from to_space.
*
* The global remsets must be processed before the workers start
* marking because they might add global remsets.
*/
scan_from_global_remsets (nursery_start, nursery_next, WORKERS_DISTRIBUTE_GRAY_QUEUE);
workers_start_marking ();
sfrjd.heap_start = nursery_start;
sfrjd.heap_end = nursery_next;
workers_enqueue_job (job_scan_from_remsets, &sfrjd);
/* we don't have complete write barrier yet, so we scan all the old generation sections */
TV_GETTIME (btv);
time_minor_scan_remsets += TV_ELAPSED_MS (atv, btv);
DEBUG (2, fprintf (gc_debug_file, "Old generation scan: %d usecs\n", TV_ELAPSED (atv, btv)));
if (use_cardtable) {
atv = btv;
card_tables_collect_stats (TRUE);
scan_from_card_tables (nursery_start, nursery_next, WORKERS_DISTRIBUTE_GRAY_QUEUE);
TV_GETTIME (btv);
time_minor_scan_card_table += TV_ELAPSED_MS (atv, btv);
}
if (!major_collector.is_parallel)
drain_gray_stack (&gray_queue, -1);
if (mono_profiler_get_events () & MONO_PROFILE_GC_ROOTS)
report_registered_roots ();
if (mono_profiler_get_events () & MONO_PROFILE_GC_ROOTS)
report_finalizer_roots ();
TV_GETTIME (atv);
time_minor_scan_pinned += TV_ELAPSED_MS (btv, atv);
/* registered roots, this includes static fields */
scrrjd_normal.func = major_collector.copy_object;
scrrjd_normal.heap_start = nursery_start;
scrrjd_normal.heap_end = nursery_next;
scrrjd_normal.root_type = ROOT_TYPE_NORMAL;
workers_enqueue_job (job_scan_from_registered_roots, &scrrjd_normal);
scrrjd_wbarrier.func = major_collector.copy_object;
scrrjd_wbarrier.heap_start = nursery_start;
scrrjd_wbarrier.heap_end = nursery_next;
scrrjd_wbarrier.root_type = ROOT_TYPE_WBARRIER;
workers_enqueue_job (job_scan_from_registered_roots, &scrrjd_wbarrier);
TV_GETTIME (btv);
time_minor_scan_registered_roots += TV_ELAPSED_MS (atv, btv);
/* thread data */
stdjd.heap_start = nursery_start;
stdjd.heap_end = nursery_next;
workers_enqueue_job (job_scan_thread_data, &stdjd);
TV_GETTIME (atv);
time_minor_scan_thread_data += TV_ELAPSED_MS (btv, atv);
btv = atv;
if (major_collector.is_parallel) {
while (!gray_object_queue_is_empty (WORKERS_DISTRIBUTE_GRAY_QUEUE)) {
workers_distribute_gray_queue_sections ();
usleep (1000);
}
}
workers_join ();
if (major_collector.is_parallel)
g_assert (gray_object_queue_is_empty (&gray_queue));
finish_gray_stack (nursery_start, nursery_next, GENERATION_NURSERY, &gray_queue);
TV_GETTIME (atv);
time_minor_finish_gray_stack += TV_ELAPSED_MS (btv, atv);
mono_profiler_gc_event (MONO_GC_EVENT_MARK_END, 0);
/*
* The (single-threaded) finalization code might have done
* some copying/marking so we can only reset the GC thread's
* worker data here instead of earlier when we joined the
* workers.
*/
if (major_collector.reset_worker_data)
major_collector.reset_worker_data (workers_gc_thread_data.major_collector_data);
if (objects_pinned) {
evacuate_pin_staging_area ();
optimize_pin_queue (0);
nursery_section->pin_queue_start = pin_queue;
nursery_section->pin_queue_num_entries = next_pin_slot;
}
/* walk the pin_queue, build up the fragment list of free memory, unmark
* pinned objects as we go, memzero() the empty fragments so they are ready for the
* next allocations.
*/
mono_profiler_gc_event (MONO_GC_EVENT_RECLAIM_START, 0);
build_nursery_fragments (pin_queue, next_pin_slot);
mono_profiler_gc_event (MONO_GC_EVENT_RECLAIM_END, 0);
TV_GETTIME (btv);
time_minor_fragment_creation += TV_ELAPSED_MS (atv, btv);
DEBUG (2, fprintf (gc_debug_file, "Fragment creation: %d usecs, %lu bytes available\n", TV_ELAPSED (atv, btv), (unsigned long)fragment_total));
if (consistency_check_at_minor_collection)
check_major_refs ();
major_collector.finish_nursery_collection ();
TV_GETTIME (all_btv);
mono_stats.minor_gc_time_usecs += TV_ELAPSED (all_atv, all_btv);
if (heap_dump_file)
dump_heap ("minor", num_minor_gcs - 1, NULL);
/* prepare the pin queue for the next collection */
last_num_pinned = next_pin_slot;
next_pin_slot = 0;
if (fin_ready_list || critical_fin_list) {
DEBUG (4, fprintf (gc_debug_file, "Finalizer-thread wakeup: ready %d\n", num_ready_finalizers));
mono_gc_finalize_notify ();
}
pin_stats_reset ();
g_assert (gray_object_queue_is_empty (&gray_queue));
if (use_cardtable)
card_tables_collect_stats (FALSE);
check_scan_starts ();
binary_protocol_flush_buffers (FALSE);
/*objects are late pinned because of lack of memory, so a major is a good call*/
needs_major = need_major_collection (0) || objects_pinned;
current_collection_generation = -1;
objects_pinned = 0;
return needs_major;
}
typedef struct
{
FinalizeEntry *list;
} ScanFinalizerEntriesJobData;
static void
job_scan_finalizer_entries (WorkerData *worker_data, void *job_data_untyped)
{
ScanFinalizerEntriesJobData *job_data = job_data_untyped;
scan_finalizer_entries (major_collector.copy_or_mark_object,
job_data->list,
job_gray_queue (worker_data));
}
static void
major_do_collection (const char *reason)
{
LOSObject *bigobj, *prevbo;
TV_DECLARE (all_atv);
TV_DECLARE (all_btv);
TV_DECLARE (atv);
TV_DECLARE (btv);
/* FIXME: only use these values for the precise scan
* note that to_space pointers should be excluded anyway...
*/
char *heap_start = NULL;
char *heap_end = (char*)-1;
int old_next_pin_slot;
ScanFromRegisteredRootsJobData scrrjd_normal, scrrjd_wbarrier;
ScanThreadDataJobData stdjd;
ScanFinalizerEntriesJobData sfejd_fin_ready, sfejd_critical_fin;
mono_perfcounters->gc_collections1++;
last_collection_old_num_major_sections = major_collector.get_num_major_sections ();
/*
* A domain could have been freed, resulting in
* los_memory_usage being less than last_collection_los_memory_usage.
*/
last_collection_los_memory_alloced = los_memory_usage - MIN (last_collection_los_memory_usage, los_memory_usage);
last_collection_old_los_memory_usage = los_memory_usage;
objects_pinned = 0;
//count_ref_nonref_objs ();
//consistency_check ();
binary_protocol_collection (GENERATION_OLD);
check_scan_starts ();
gray_object_queue_init (&gray_queue);
workers_init_distribute_gray_queue ();
degraded_mode = 0;
DEBUG (1, fprintf (gc_debug_file, "Start major collection %d\n", num_major_gcs));
num_major_gcs++;
mono_stats.major_gc_count ++;
/* world must be stopped already */
TV_GETTIME (all_atv);
atv = all_atv;
/* Pinning depends on this */
clear_nursery_fragments (nursery_next);
TV_GETTIME (btv);
time_major_pre_collection_fragment_clear += TV_ELAPSED_MS (atv, btv);
nursery_section->next_data = nursery_end;
/* we should also coalesce scanning from sections close to each other
* and deal with pointers outside of the sections later.
*/
if (major_collector.start_major_collection)
major_collector.start_major_collection ();
*major_collector.have_swept = FALSE;
reset_minor_collection_allowance ();
if (xdomain_checks)
check_for_xdomain_refs ();
/* The remsets are not useful for a major collection */
clear_remsets ();
global_remset_cache_clear ();
if (use_cardtable)
card_table_clear ();
process_fin_stage_entries ();
process_dislink_stage_entries ();
TV_GETTIME (atv);
init_pinning ();
DEBUG (6, fprintf (gc_debug_file, "Collecting pinned addresses\n"));
pin_from_roots ((void*)lowest_heap_address, (void*)highest_heap_address, WORKERS_DISTRIBUTE_GRAY_QUEUE);
optimize_pin_queue (0);
/*
* pin_queue now contains all candidate pointers, sorted and
* uniqued. We must do two passes now to figure out which
* objects are pinned.
*
* The first is to find within the pin_queue the area for each
* section. This requires that the pin_queue be sorted. We
* also process the LOS objects and pinned chunks here.
*
* The second, destructive, pass is to reduce the section
* areas to pointers to the actually pinned objects.
*/
DEBUG (6, fprintf (gc_debug_file, "Pinning from sections\n"));
/* first pass for the sections */
mono_sgen_find_section_pin_queue_start_end (nursery_section);
major_collector.find_pin_queue_start_ends (WORKERS_DISTRIBUTE_GRAY_QUEUE);
/* identify possible pointers to the insize of large objects */
DEBUG (6, fprintf (gc_debug_file, "Pinning from large objects\n"));
for (bigobj = los_object_list; bigobj; bigobj = bigobj->next) {
int dummy;
if (mono_sgen_find_optimized_pin_queue_area (bigobj->data, (char*)bigobj->data + bigobj->size, &dummy)) {
pin_object (bigobj->data);
/* FIXME: only enqueue if object has references */
GRAY_OBJECT_ENQUEUE (WORKERS_DISTRIBUTE_GRAY_QUEUE, bigobj->data);
if (heap_dump_file)
mono_sgen_pin_stats_register_object ((char*) bigobj->data, safe_object_get_size ((MonoObject*) bigobj->data));
DEBUG (6, fprintf (gc_debug_file, "Marked large object %p (%s) size: %lu from roots\n", bigobj->data, safe_name (bigobj->data), (unsigned long)bigobj->size));
}
}
/* second pass for the sections */
mono_sgen_pin_objects_in_section (nursery_section, WORKERS_DISTRIBUTE_GRAY_QUEUE);
major_collector.pin_objects (WORKERS_DISTRIBUTE_GRAY_QUEUE);
old_next_pin_slot = next_pin_slot;
TV_GETTIME (btv);
time_major_pinning += TV_ELAPSED_MS (atv, btv);
DEBUG (2, fprintf (gc_debug_file, "Finding pinned pointers: %d in %d usecs\n", next_pin_slot, TV_ELAPSED (atv, btv)));
DEBUG (4, fprintf (gc_debug_file, "Start scan with %d pinned objects\n", next_pin_slot));
major_collector.init_to_space ();
#ifdef SGEN_DEBUG_INTERNAL_ALLOC
main_gc_thread = pthread_self ();
#endif
workers_start_all_workers ();
workers_start_marking ();
if (mono_profiler_get_events () & MONO_PROFILE_GC_ROOTS)
report_registered_roots ();
TV_GETTIME (atv);
time_major_scan_pinned += TV_ELAPSED_MS (btv, atv);
/* registered roots, this includes static fields */
scrrjd_normal.func = major_collector.copy_or_mark_object;
scrrjd_normal.heap_start = heap_start;
scrrjd_normal.heap_end = heap_end;
scrrjd_normal.root_type = ROOT_TYPE_NORMAL;
workers_enqueue_job (job_scan_from_registered_roots, &scrrjd_normal);
scrrjd_wbarrier.func = major_collector.copy_or_mark_object;
scrrjd_wbarrier.heap_start = heap_start;
scrrjd_wbarrier.heap_end = heap_end;
scrrjd_wbarrier.root_type = ROOT_TYPE_WBARRIER;
workers_enqueue_job (job_scan_from_registered_roots, &scrrjd_wbarrier);
TV_GETTIME (btv);
time_major_scan_registered_roots += TV_ELAPSED_MS (atv, btv);
/* Threads */
stdjd.heap_start = heap_start;
stdjd.heap_end = heap_end;
workers_enqueue_job (job_scan_thread_data, &stdjd);
TV_GETTIME (atv);
time_major_scan_thread_data += TV_ELAPSED_MS (btv, atv);
TV_GETTIME (btv);
time_major_scan_alloc_pinned += TV_ELAPSED_MS (atv, btv);
if (mono_profiler_get_events () & MONO_PROFILE_GC_ROOTS)
report_finalizer_roots ();
/* scan the list of objects ready for finalization */
sfejd_fin_ready.list = fin_ready_list;
workers_enqueue_job (job_scan_finalizer_entries, &sfejd_fin_ready);
sfejd_critical_fin.list = critical_fin_list;
workers_enqueue_job (job_scan_finalizer_entries, &sfejd_critical_fin);
TV_GETTIME (atv);
time_major_scan_finalized += TV_ELAPSED_MS (btv, atv);
DEBUG (2, fprintf (gc_debug_file, "Root scan: %d usecs\n", TV_ELAPSED (btv, atv)));
TV_GETTIME (btv);
time_major_scan_big_objects += TV_ELAPSED_MS (atv, btv);
if (major_collector.is_parallel) {
while (!gray_object_queue_is_empty (WORKERS_DISTRIBUTE_GRAY_QUEUE)) {