Permalink
Browse files

Merge branch 'rickard-sverker/supercarrier/OTP-11149' into maint

* rickard-sverker/supercarrier/OTP-11149: (29 commits)
  erts: Add test case for erts_mmap
  erts: Add mutex to init_atoms in erts_mmap.c
  erts: Fix lock violation for init_atoms in erl_mmap.c
  erts: Fix misc minor bugs in supercarrier initialization
  erts: Add erts_mmap stats
  erts: Add erts_bld_tupleX macros
  erts: Rename erts_bld_atom_uint_2tup_list to *_uword_*
  erts: Fix bug in lookup_free_seg
  erts: Fix race bug in erts_munmap
  erts: Add HARD_DBG_MSEG
  erts: Refactor rbt_insert in erl_mmap
  erts: erts_mmap improved free seg desc management
  erts: Add documentation for +MMsc* system flags
  erts: Allow page aligned erts_munmap()
  erts: Sort tree in super aligned sizes (SA_SZ_ADDR_ORDER)
  erts: Fix ASSERT bug and void* arithmetics
  erts: Add mmap argument to erts_debug:get_internal_state
  erts: Improve erts_mmap out of free descriptor management
  erts: Cleanup erl_mmap
  erts: Add __func__ to ERTS_ASSERT macro
  ...
  • Loading branch information...
sverker committed Oct 15, 2013
2 parents bbe80d2 + 5db2c05 commit 356c1e9312a640ac211b6d5d4110372742acfcde
Showing with 3,428 additions and 847 deletions.
  1. +59 −0 erts/doc/src/erts_alloc.xml
  2. +1 −0 erts/emulator/Makefile.in
  3. +2 −2 erts/emulator/beam/dist.c
  4. +63 −7 erts/emulator/beam/erl_alloc.c
  5. +10 −10 erts/emulator/beam/erl_alloc_util.c
  6. +3 −3 erts/emulator/beam/erl_alloc_util.h
  7. +0 −3 erts/emulator/beam/erl_ao_firstfit_alloc.c
  8. +0 −3 erts/emulator/beam/erl_bestfit_alloc.c
  9. +4 −1 erts/emulator/beam/erl_bif_info.c
  10. +3 −3 erts/emulator/beam/erl_binary.h
  11. +2 −2 erts/emulator/beam/erl_db.c
  12. +1 −1 erts/emulator/beam/erl_db_util.h
  13. +4 −1 erts/emulator/beam/erl_lock_check.c
  14. +5 −5 erts/emulator/beam/erl_node_container_utils.h
  15. +1 −1 erts/emulator/beam/erl_process.c
  16. +2 −2 erts/emulator/beam/erl_process.h
  17. +19 −19 erts/emulator/beam/erl_trace.c
  18. +3 −0 erts/emulator/beam/erl_unicode.c
  19. +7 −3 erts/emulator/beam/erl_utils.h
  20. +7 −7 erts/emulator/beam/erl_vm.h
  21. +2 −2 erts/emulator/beam/external.h
  22. +3 −3 erts/emulator/beam/global.h
  23. +1 −1 erts/emulator/beam/io.c
  24. +12 −11 erts/emulator/beam/sys.h
  25. +2 −2 erts/emulator/beam/utils.c
  26. +1 −1 erts/emulator/drivers/common/inet_drv.c
  27. +2,832 −0 erts/emulator/sys/common/erl_mmap.c
  28. +134 −0 erts/emulator/sys/common/erl_mmap.h
  29. +148 −728 erts/emulator/sys/common/erl_mseg.c
  30. +19 −17 erts/emulator/sys/common/erl_mseg.h
  31. +5 −5 erts/emulator/sys/unix/sys.c
  32. +1 −1 erts/emulator/sys/win32/sys.c
  33. +66 −3 erts/emulator/test/alloc_SUITE.erl
  34. +4 −0 erts/etc/common/erlexec.c
  35. BIN erts/preloaded/ebin/erlang.beam
  36. +2 −0 erts/preloaded/src/erlang.erl
@@ -271,6 +271,65 @@
memory segment cache is not reused if its size exceeds the
requested size with more than relative max cache bad fit
percent of the requested size. Default value is 20.</item>
<tag><marker id="MMscmgc"><c><![CDATA[+MMscmgc <amount>]]></c></marker></tag>
<item>
Set <seealso marker="#MMscs">super carrier</seealso> max guaranteed
no of carriers. This parameter defaults to <c>65536</c>. This
parameter determines an amount of pre-allocated structures that is
needed in order to keep track of different areas in the super carrier.
When the system runs out of such structures it may crash due to an
out of memory condition.
</item>
<tag><marker id="MMsco"><c><![CDATA[+MMsco true|false]]></c></marker></tag>
<item>
Set <seealso marker="#MMscs">super carrier</seealso> only flag. This
flag defaults to <c>true</c>. When a super carrier is used and this
flag is <c>true</c>, the system will crash when a carrier request
cannot be satisfied by the super carrier. When the flag is <c>false</c>
the system will try to create requested carrier by other means.
<br/><br/>
<em>NOTE</em>: Setting this flag to <c>false</c> may not be supported
on all systems. This flag will in that case be ignored.
<br/><br/>
<em>NOTE</em>: The super carrier cannot be enabled nor
disabled on halfword heap systems. This flag will be
ignored on halfword heap systems.
</item>
<tag><marker id="MMscrpm"><c><![CDATA[+MMscrpm true|false]]></c></marker></tag>
<item>
Set <seealso marker="#MMscs">super carrier</seealso> reserve physical
memory flag. This flag defaults to <c>true</c>. When this flag is
<c>true</c>, physical memory will be reserved for the whole super
carrier at once when it is created. The reservation will after that
be left unchanged. When this flag is set to <c>false</c> only virtual
address space will be reserved for the super carrier upon creation.
The system will attempt to reserve physical memory upon carrier
creations in the super carrier, and attempt to unreserve physical
memory upon carrier destructions in the super carrier.
<br/><br/>
<em>NOTE</em>: What reservation of physical memory actually means
highly depends on the operating system, and how it is configured. For
example, different memory overcommit settings on Linux drastically
change the behaviour. Also note, setting this flag to <c>false</c>
may not be supported on all systems. This flag will in that case
be ignored.
<br/><br/>
<em>NOTE</em>: The super carrier cannot be enabled nor
disabled on halfword heap systems. This flag will be
ignored on halfword heap systems.
</item>
<tag><marker id="MMscs"><c><![CDATA[+MMscs <size in MB>]]></c></marker></tag>
<item>
Set super carrier size (in MB). The super carrier size defaults to
zero; i.e, the super carrier is by default disabled. The super
carrier is a large continuous area in the virtual address space.
The system will always try to create new carriers in the super
carrier.
<br/><br/>
<em>NOTE</em>: The super carrier cannot be enabled nor
disabled on halfword heap systems. This flag will be
ignored on halfword heap systems.
</item>
<tag><marker id="MMmcs"><c><![CDATA[+MMmcs <amount>]]></c></marker></tag>
<item>
Max cached segments. The maximum number of memory segments
@@ -808,6 +808,7 @@ OS_OBJS += $(OBJDIR)/erl_poll.o \
endif
OS_OBJS += $(OBJDIR)/erl_mseg.o \
$(OBJDIR)/erl_mmap.o \
$(OBJDIR)/erl_$(ERLANG_OSTYPE)_sys_ddll.o \
$(OBJDIR)/erl_mtrace_sys_wrap.o \
$(OBJDIR)/erl_sys_common_misc.o
@@ -353,7 +353,7 @@ static void doit_link_net_exits_sub(ErtsLink *sublnk, void *vlnecp)
static void doit_link_net_exits(ErtsLink *lnk, void *vnecp)
{
LinkNetExitsContext lnec = {(NetExitsContext *) vnecp, lnk};
ASSERT(lnk->type == LINK_PID)
ASSERT(lnk->type == LINK_PID);
erts_sweep_links(ERTS_LINK_ROOT(lnk), &doit_link_net_exits_sub, (void *) &lnec);
#ifdef DEBUG
ERTS_LINK_ROOT(lnk) = NULL;
@@ -369,7 +369,7 @@ static void doit_node_link_net_exits(ErtsLink *lnk, void *vnecp)
Process *rp;
ErtsLink *rlnk;
Uint i,n;
ASSERT(lnk->type == LINK_NODE)
ASSERT(lnk->type == LINK_NODE);
if (is_internal_pid(lnk->pid)) {
ErtsProcLocks rp_locks = ERTS_PROC_LOCK_LINK;
rp = erts_pid2proc(NULL, 0, lnk->pid, rp_locks);
@@ -718,6 +718,7 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop)
init.mseg.nos = erts_no_schedulers;
erts_mseg_init(&init.mseg);
#endif
erts_alcu_init(&init.alloc_util);
erts_afalc_init();
erts_bfalc_init();
@@ -1174,6 +1175,25 @@ get_kb_value(char *param_end, char** argv, int* ip)
return ((Uint) tmp)*1024;
}
static UWord
get_mb_value(char *param_end, char** argv, int* ip)
{
SWord tmp;
UWord max = ((~((UWord) 0))/(1024*1024)) + 1;
char *rest;
char *param = argv[*ip]+1;
char *value = get_value(param_end, argv, ip);
errno = 0;
tmp = (SWord) ErtsStrToSint(value, &rest, 10);
if (errno != 0 || rest == value || tmp < 0 || max < ((UWord) tmp))
bad_value(param, param_end, value);
if (max == (UWord) tmp)
return ~((UWord) 0);
else
return ((UWord) tmp)*1024*1024;
}
#if 0
static Uint
get_byte_value(char *param_end, char** argv, int* ip)
@@ -1448,6 +1468,30 @@ handle_args(int *argc, char **argv, erts_alc_hndl_args_init_t *init)
#endif
get_amount_value(argv[i]+6, argv, &i);
}
else if (has_prefix("scs", argv[i]+3)) {
#if HAVE_ERTS_MSEG
init->mseg.mmap.scs =
#endif
get_mb_value(argv[i]+6, argv, &i);
}
else if (has_prefix("sco", argv[i]+3)) {
#if HAVE_ERTS_MSEG
init->mseg.mmap.sco =
#endif
get_bool_value(argv[i]+6, argv, &i);
}
else if (has_prefix("scrpm", argv[i]+3)) {
#if HAVE_ERTS_MSEG
init->mseg.mmap.scrpm =
#endif
get_bool_value(argv[i]+8, argv, &i);
}
else if (has_prefix("scmgc", argv[i]+3)) {
#if HAVE_ERTS_MSEG
init->mseg.mmap.scmgc =
#endif
get_amount_value(argv[i]+8, argv, &i);
}
else {
bad_param(param, param+2);
}
@@ -2668,6 +2712,7 @@ erts_allocator_info(int to, void *arg)
#if HAVE_ERTS_MSEG
{
struct erts_mmap_info_struct emis;
#ifdef ERTS_SMP
int max = (int) erts_no_schedulers;
#else
@@ -2678,6 +2723,8 @@ erts_allocator_info(int to, void *arg)
erts_print(to, arg, "=allocator:mseg_alloc[%d]\n", i);
erts_mseg_info(i, &to, arg, 0, NULL, NULL);
}
erts_print(to, arg, "=allocator:mseg_alloc.erts_mmap\n");
erts_mmap_info(&to, arg, NULL, NULL, &emis);
}
#endif
@@ -2904,6 +2951,7 @@ reply_alloc_info(void *vair)
Uint sz, *szp;
ErlOffHeap *ohp = NULL;
ErlHeapFragment *bp = NULL;
struct erts_mmap_info_struct emis;
int i;
Eterm (*info_func)(Allctr_t *,
int,
@@ -3016,15 +3064,23 @@ reply_alloc_info(void *vair)
? NIL
: erts_mseg_info(0, NULL, NULL, hpp != NULL,
hpp, szp));
ainfo = erts_bld_tuple(hpp, szp, 3,
alloc_atom,
make_small(0),
ainfo);
ainfo = erts_bld_tuple3(hpp, szp,
alloc_atom,
make_small(0),
ainfo);
ai_list = erts_bld_cons(hpp, szp,
ainfo, ai_list);
ainfo = (air->only_sz ? NIL : erts_mmap_info(NULL, NULL, hpp, szp, &emis));
ainfo = erts_bld_tuple3(hpp, szp,
alloc_atom,
erts_bld_atom(hpp,szp,"erts_mmap"),
ainfo);
#else
ainfo = erts_bld_tuple(hpp, szp, 2, alloc_atom,
am_false);
ainfo = erts_bld_tuple2(hpp, szp, alloc_atom,
am_false);
#endif
break;
break;
default:
alloc_atom = erts_bld_atom(hpp, szp,
(char *) ERTS_ALC_A2AD(ai));
@@ -87,9 +87,6 @@ static int initialized = 0;
#define SYS_ALLOC_CARRIER_CEILING(X) \
SYS_ALLOC_CARRIER_FLOOR((X) + INV_SYS_ALLOC_CARRIER_MASK)
#undef ASSERT
#define ASSERT ASSERT_EXPR
#if 0
/* Can be useful for debugging */
#define MBC_REALLOC_ALWAYS_MOVES
@@ -759,8 +756,9 @@ static ERTS_INLINE void *
alcu_mseg_alloc(Allctr_t *allctr, Uint *size_p, Uint flags)
{
void *res;
res = erts_mseg_alloc_opt(allctr->alloc_no, size_p, flags, &allctr->mseg_opt);
UWord size = (UWord) *size_p;
res = erts_mseg_alloc_opt(allctr->alloc_no, &size, flags, &allctr->mseg_opt);
*size_p = (Uint) size;
INC_CC(allctr->calls.mseg_alloc);
return res;
}
@@ -769,17 +767,18 @@ static ERTS_INLINE void *
alcu_mseg_realloc(Allctr_t *allctr, void *seg, Uint old_size, Uint *new_size_p)
{
void *res;
res = erts_mseg_realloc_opt(allctr->alloc_no, seg, old_size, new_size_p,
UWord new_size = (UWord) *new_size_p;
res = erts_mseg_realloc_opt(allctr->alloc_no, seg, (UWord) old_size, &new_size,
ERTS_MSEG_FLG_NONE, &allctr->mseg_opt);
*new_size_p = (Uint) new_size;
INC_CC(allctr->calls.mseg_realloc);
return res;
}
static ERTS_INLINE void
alcu_mseg_dealloc(Allctr_t *allctr, void *seg, Uint size, Uint flags)
{
erts_mseg_dealloc_opt(allctr->alloc_no, seg, size, flags, &allctr->mseg_opt);
erts_mseg_dealloc_opt(allctr->alloc_no, seg, (UWord) size, flags, &allctr->mseg_opt);
INC_CC(allctr->calls.mseg_dealloc);
}
@@ -3226,10 +3225,12 @@ static void CHECK_1BLK_CARRIER(Allctr_t* A, int SBC, int MSEGED, Carrier_t* C,
ASSERT(IS_MBC_BLK((B)));
ASSERT(IS_MB_CARRIER((C)));
ASSERT(FBLK_TO_MBC(B) == (C));
if ((MSEGED)) {
ASSERT_ERTS_SACRR_UNIT_SIZE_MULTIPLE((CSZ));
}
}
if ((MSEGED)) {
ASSERT(IS_MSEG_CARRIER((C)));
ASSERT_ERTS_SACRR_UNIT_SIZE_MULTIPLE((CSZ));
}
else {
ASSERT(IS_SYS_ALLOC_CARRIER((C)));
@@ -3601,7 +3602,6 @@ destroy_carrier(Allctr_t *allctr, Block_t *blk, Carrier_t **busy_pcrr_pp)
#if HAVE_ERTS_MSEG
if (IS_MSEG_CARRIER(crr)) {
ASSERT(crr_sz % ERTS_SACRR_UNIT_SZ == 0);
STAT_MSEG_SBC_FREE(allctr, crr_sz, blk_sz);
}
else
@@ -254,9 +254,9 @@ erts_aint32_t erts_alcu_fix_alloc_shrink(Allctr_t *, erts_aint32_t);
# define MBC_ABLK_SZ_MASK (~FLG_MASK)
#endif
#define MBC_ABLK_SZ(B) (ASSERT_EXPR(!is_sbc_blk(B)), (B)->bhdr & MBC_ABLK_SZ_MASK)
#define MBC_FBLK_SZ(B) (ASSERT_EXPR(!is_sbc_blk(B)), (B)->bhdr & MBC_FBLK_SZ_MASK)
#define SBC_BLK_SZ(B) (ASSERT_EXPR(is_sbc_blk(B)), (B)->bhdr & SBC_BLK_SZ_MASK)
#define MBC_ABLK_SZ(B) (ASSERT(!is_sbc_blk(B)), (B)->bhdr & MBC_ABLK_SZ_MASK)
#define MBC_FBLK_SZ(B) (ASSERT(!is_sbc_blk(B)), (B)->bhdr & MBC_FBLK_SZ_MASK)
#define SBC_BLK_SZ(B) (ASSERT(is_sbc_blk(B)), (B)->bhdr & SBC_BLK_SZ_MASK)
#define CARRIER_SZ(C) \
((C)->chdr & CARRIER_SZ_MASK)
@@ -85,9 +85,6 @@
#define SET_RED(N) (((AOFF_RBTree_t *) (N))->flags |= RED_FLG)
#define SET_BLACK(N) (((AOFF_RBTree_t *) (N))->flags &= ~RED_FLG)
#undef ASSERT
#define ASSERT ASSERT_EXPR
#if 1
#define RBT_ASSERT ASSERT
#else
@@ -75,9 +75,6 @@
#define BF_BLK_SZ(B) MBC_FBLK_SZ(&(B)->hdr)
#undef ASSERT
#define ASSERT ASSERT_EXPR
#if 1
#define RBT_ASSERT ASSERT
#else
@@ -2091,7 +2091,7 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1)
BIF_RET(res);
} else if (BIF_ARG_1 == am_sequential_tracer) {
val = erts_get_system_seq_tracer();
ASSERT(is_internal_pid(val) || is_internal_port(val) || val==am_false)
ASSERT(is_internal_pid(val) || is_internal_port(val) || val==am_false);
hp = HAlloc(BIF_P, 3);
res = TUPLE2(hp, am_sequential_tracer, val);
BIF_RET(res);
@@ -3289,6 +3289,9 @@ BIF_RETTYPE erts_debug_get_internal_state_1(BIF_ALIST_1)
erts_smp_thr_progress_unblock();
BIF_RET(res);
}
else if (ERTS_IS_ATOM_STR("mmap", BIF_ARG_1)) {
BIF_RET(erts_mmap_debug_info(BIF_P));
}
}
else if (is_tuple(BIF_ARG_1)) {
Eterm* tp = tuple_val(BIF_ARG_1);
@@ -1,7 +1,7 @@
/*
* %CopyrightBegin%
*
* Copyright Ericsson AB 2000-2011. All Rights Reserved.
* Copyright Ericsson AB 2000-2013. All Rights Reserved.
*
* The contents of this file are subject to the Erlang Public License,
* Version 1.1, (the "License"); you may not use this file except in
@@ -153,7 +153,7 @@ do { \
#define binary_bytes(Bin) \
(*binary_val(Bin) == HEADER_PROC_BIN ? \
((ProcBin *) binary_val(Bin))->bytes : \
(ASSERT_EXPR(thing_subtag(*binary_val(Bin)) == HEAP_BINARY_SUBTAG), \
(ASSERT(thing_subtag(*binary_val(Bin)) == HEAP_BINARY_SUBTAG), \
(byte *)(&(((ErlHeapBin *) binary_val(Bin))->data))))
void erts_init_binary(void);
@@ -183,7 +183,7 @@ BIF_RETTYPE erts_binary_part(Process *p, Eterm binary, Eterm epos, Eterm elen);
#endif
#define ERTS_CHK_BIN_ALIGNMENT(B) \
do { ASSERT(!(B) || (((UWord) &((Binary *)(B))->orig_bytes[0]) & ERTS_BIN_ALIGNMENT_MASK) == ((UWord) 0)) } while(0)
do { ASSERT(!(B) || (((UWord) &((Binary *)(B))->orig_bytes[0]) & ERTS_BIN_ALIGNMENT_MASK) == ((UWord) 0)); } while(0)
ERTS_GLB_INLINE byte* erts_get_aligned_binary_bytes(Eterm bin, byte** base_ptr);
ERTS_GLB_INLINE void erts_free_aligned_binary_bytes(byte* buf);
@@ -2236,7 +2236,7 @@ static BIF_RETTYPE ets_select_trap_1(BIF_ALIST_1)
CHECK_TABLES();
tptr = tuple_val(a1);
ASSERT(arityval(*tptr) >= 1)
ASSERT(arityval(*tptr) >= 1);
if ((tb = db_get_table(p, tptr[1], DB_READ, kind)) == NULL) {
BIF_ERROR(p, BADARG);
@@ -2403,7 +2403,7 @@ static BIF_RETTYPE ets_select_count_1(BIF_ALIST_1)
CHECK_TABLES();
tptr = tuple_val(a1);
ASSERT(arityval(*tptr) >= 1)
ASSERT(arityval(*tptr) >= 1);
if ((tb = db_get_table(p, tptr[1], DB_READ, kind)) == NULL) {
BIF_ERROR(p, BADARG);
}
@@ -457,7 +457,7 @@ int erts_db_is_compiled_ms(Eterm term);
&& ERTS_MAGIC_BIN_DESTRUCTOR((BP)) == erts_db_match_prog_destructor)
#define Binary2MatchProg(BP) \
(ASSERT_EXPR(IsMatchProgBinary((BP))), \
(ASSERT(IsMatchProgBinary((BP))), \
((MatchProg *) ERTS_MAGIC_BIN_DATA((BP))))
/*
** Debugging
Oops, something went wrong.

0 comments on commit 356c1e9

Please sign in to comment.