New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

L4Linux: memory usage is limited #414

Closed
skalk opened this Issue Oct 15, 2012 · 10 comments

Comments

Projects
None yet
5 participants
@skalk
Member

skalk commented Oct 15, 2012

This issue was brought up on the Linux mailing-list:

http://sourceforge.net/mailarchive/message.php?msg_id=29953101

When L4Linux gets more memory (dependent on the platform, e.g. x86_32 single core > 512MB) internal errors are triggered, potentially due to internal memory corruption.

@iloskutov

This comment has been minimized.

Show comment
Hide comment
@iloskutov

iloskutov Feb 13, 2013

Contributor

I try to investigate this issue and found that it occurred when l4linux tries to allocate Dataspace like this:

enum { DS_SIZE=513*1024*1024 };
Dataspace_capability cap = Genode::env()->ram_session()->alloc(DS_SIZE);
unsigned char *data = static_cast<unsigned char*>(Genode::env()->rm_session()->attach(cap));

When size of Dataspace is more that 512M in function Ram_session_component::alloc called function Ram_session_component::_clear_ds, which caused a page fault when tried to clear allocated memory. I don't understand, how _clear_ds used phys_addr for clear? It is correct?

memset((void *)ds->phys_addr(), 0, ds->size());

I tried comment this line, but the same error occurred when I read or write to the memory after 512M.

I try to allocate the large dataspace in L4re. It seems it works. Part of code:

enum {  DS_SIZE = 800*1024*1024, };
...
int err =  L4Re::Env::env()->mem_alloc()->alloc(DS_SIZE, *_ds, 0);
char *_addr = 0;
err =  L4Re::Env::env()->rm()->attach(&_addr, (*_ds)->size(), L4Re::Rm::Search_addr, *_ds);
memset(_addr, 0, DS_SIZE);

I have tested it on foc_x86_32 and foc_x86_64. It seems the issue is in Genode.

Contributor

iloskutov commented Feb 13, 2013

I try to investigate this issue and found that it occurred when l4linux tries to allocate Dataspace like this:

enum { DS_SIZE=513*1024*1024 };
Dataspace_capability cap = Genode::env()->ram_session()->alloc(DS_SIZE);
unsigned char *data = static_cast<unsigned char*>(Genode::env()->rm_session()->attach(cap));

When size of Dataspace is more that 512M in function Ram_session_component::alloc called function Ram_session_component::_clear_ds, which caused a page fault when tried to clear allocated memory. I don't understand, how _clear_ds used phys_addr for clear? It is correct?

memset((void *)ds->phys_addr(), 0, ds->size());

I tried comment this line, but the same error occurred when I read or write to the memory after 512M.

I try to allocate the large dataspace in L4re. It seems it works. Part of code:

enum {  DS_SIZE = 800*1024*1024, };
...
int err =  L4Re::Env::env()->mem_alloc()->alloc(DS_SIZE, *_ds, 0);
char *_addr = 0;
err =  L4Re::Env::env()->rm()->attach(&_addr, (*_ds)->size(), L4Re::Rm::Search_addr, *_ds);
memset(_addr, 0, DS_SIZE);

I have tested it on foc_x86_32 and foc_x86_64. It seems the issue is in Genode.

@ghost ghost assigned cproc Feb 19, 2013

@cproc

This comment has been minimized.

Show comment
Hide comment
@cproc

cproc Feb 20, 2013

Member

I made the following changes to the l4linux run script for testing:

diff --git a/ports-foc/run/l4linux.run b/ports-foc/run/l4linux.run
index e757651..a822cb3 100644
--- a/ports-foc/run/l4linux.run
+++ b/ports-foc/run/l4linux.run
@@ -88,11 +88,11 @@ append_if [have_spec lan9118] config {

 append config {
        <start name="l4linux">
-               <resource name="RAM" quantum="128M"/>
+               <resource name="RAM" quantum="2G"/>
 }

 append_if [have_spec x86] config {
-               <config args="mem=64M console=ttyS0 l4x_rd=initrd.gz l4x_cpus=2 l4x_cpus_map=0,1"/>
+               <config args="mem=512M console=ttyS0 l4x_rd=initrd.gz l4x_cpus=2 l4x_cpus_map=0,1"/>
 }

 append_if [have_spec arm] config {
@@ -131,9 +131,9 @@ build_boot_image  [join $boot_modules " "]
 #
 # Qemu
 #
-append qemu_args " -m 128 -nographic "
-append qemu_args " -serial file:kdb.log "
+append qemu_args " -m 768 -nographic "
 append qemu_args " -serial mon:stdio "
+append qemu_args " -serial file:kdb.log "
 append_if [have_spec     x86] qemu_args " -smp 2,cores=2 "
 append_if [have_spec     x86] qemu_args " -net nic,model=e1000 -net user "
 append_if [have_spec lan9118] qemu_args " -net nic,model=lan9118 -net user "

With these changes, when running the l4linux test built with -O0 on foc_x86_32, the bug check statement

BUG_ON((unsigned long)high_memory       > VMALLOC_START);

in file l4linux/arch/l4/mm/arch-x86/init_32.c triggers..

high_memory is basically "main memory start address + main memory size".

In file l4linux/arch/l4/kernel/main.c, the main memory start address gets set from the result of a l4re_rm_reserve_area() call with a given start address of 0 and the L4RE_RM_SEARCH_ADDR flag.

With the tested configuration, the resulting main memory start address is 0x20000000 and the main memory size is 0x20000000, too.

A bit later, the VMALLOC_START (l4x_vmalloc_memory_start) address gets set from the result of another l4re_rm_reserve_area() call, now with a given start address of (main memory start address + main memory size) = 0x40000000 and the L4RE_RM_SEARCH_ADDR flag.

Because the virtual memory area at 0x40000000 is already reserved for Genode thread contexts, an alternative range gets searched and as result, VMALLOC_START becomes 0x8000000, which is below the main memory area.

According to the documentation for the L4Re::Rm::reserve_area() function, the search for an alternative area should start at the given start address, but in the current Genode-specific implementation the search starts at address 0.

Member

cproc commented Feb 20, 2013

I made the following changes to the l4linux run script for testing:

diff --git a/ports-foc/run/l4linux.run b/ports-foc/run/l4linux.run
index e757651..a822cb3 100644
--- a/ports-foc/run/l4linux.run
+++ b/ports-foc/run/l4linux.run
@@ -88,11 +88,11 @@ append_if [have_spec lan9118] config {

 append config {
        <start name="l4linux">
-               <resource name="RAM" quantum="128M"/>
+               <resource name="RAM" quantum="2G"/>
 }

 append_if [have_spec x86] config {
-               <config args="mem=64M console=ttyS0 l4x_rd=initrd.gz l4x_cpus=2 l4x_cpus_map=0,1"/>
+               <config args="mem=512M console=ttyS0 l4x_rd=initrd.gz l4x_cpus=2 l4x_cpus_map=0,1"/>
 }

 append_if [have_spec arm] config {
@@ -131,9 +131,9 @@ build_boot_image  [join $boot_modules " "]
 #
 # Qemu
 #
-append qemu_args " -m 128 -nographic "
-append qemu_args " -serial file:kdb.log "
+append qemu_args " -m 768 -nographic "
 append qemu_args " -serial mon:stdio "
+append qemu_args " -serial file:kdb.log "
 append_if [have_spec     x86] qemu_args " -smp 2,cores=2 "
 append_if [have_spec     x86] qemu_args " -net nic,model=e1000 -net user "
 append_if [have_spec lan9118] qemu_args " -net nic,model=lan9118 -net user "

With these changes, when running the l4linux test built with -O0 on foc_x86_32, the bug check statement

BUG_ON((unsigned long)high_memory       > VMALLOC_START);

in file l4linux/arch/l4/mm/arch-x86/init_32.c triggers..

high_memory is basically "main memory start address + main memory size".

In file l4linux/arch/l4/kernel/main.c, the main memory start address gets set from the result of a l4re_rm_reserve_area() call with a given start address of 0 and the L4RE_RM_SEARCH_ADDR flag.

With the tested configuration, the resulting main memory start address is 0x20000000 and the main memory size is 0x20000000, too.

A bit later, the VMALLOC_START (l4x_vmalloc_memory_start) address gets set from the result of another l4re_rm_reserve_area() call, now with a given start address of (main memory start address + main memory size) = 0x40000000 and the L4RE_RM_SEARCH_ADDR flag.

Because the virtual memory area at 0x40000000 is already reserved for Genode thread contexts, an alternative range gets searched and as result, VMALLOC_START becomes 0x8000000, which is below the main memory area.

According to the documentation for the L4Re::Rm::reserve_area() function, the search for an alternative area should start at the given start address, but in the current Genode-specific implementation the search starts at address 0.

@cproc

This comment has been minimized.

Show comment
Hide comment
@cproc

cproc Feb 21, 2013

Member

@iloskutov: The page fault in Ram_session_component::_clear_ds() can happen if more than 1G of physical memory are available. Currently, the physical memory is mapped 1:1 to core virtual memory in base-foc (that's why ds->phys_addr() can be used) and if more than 1G of physical memory is available, the Ram_session_component allocates memory from core's own thread context area at 0x40000000 and _clear_ds() then clears the entrypoint thread's stack. I believe @ssumpf has found a solution for this problem recently.

Member

cproc commented Feb 21, 2013

@iloskutov: The page fault in Ram_session_component::_clear_ds() can happen if more than 1G of physical memory are available. Currently, the physical memory is mapped 1:1 to core virtual memory in base-foc (that's why ds->phys_addr() can be used) and if more than 1G of physical memory is available, the Ram_session_component allocates memory from core's own thread context area at 0x40000000 and _clear_ds() then clears the entrypoint thread's stack. I believe @ssumpf has found a solution for this problem recently.

@cproc

This comment has been minimized.

Show comment
Hide comment
@cproc

cproc Feb 21, 2013

Member

@iloskutov: 4d9d83fd52ebc57bf5b202043a72f66cf221dc01 (from issue #660) should fix the _clear_ds() problem.

Member

cproc commented Feb 21, 2013

@iloskutov: 4d9d83fd52ebc57bf5b202043a72f66cf221dc01 (from issue #660) should fix the _clear_ds() problem.

@cproc

This comment has been minimized.

Show comment
Hide comment
@cproc

cproc Mar 11, 2013

Member

07a460dfc8df147494812b7aeb9ef65a53269a71 should hopefully fix the remaining stability problems with larger memory amounts (tested by running the 'find' program with 952MB reported to l4linux on the Pandaboard and 1655MB on the Arndale board).

The changes made to the l4linux contrib code are a bit hard to spot from looking at the commit, because it's a modification of a patch file. The actual changes can be seen in https://github.com/cproc/genode/blob/edd767e1af0e546e68a2649120d323d435eff973/ports-foc/contrib/l4linux/patches-issue414/l4linux_issue414.diff.

Member

cproc commented Mar 11, 2013

07a460dfc8df147494812b7aeb9ef65a53269a71 should hopefully fix the remaining stability problems with larger memory amounts (tested by running the 'find' program with 952MB reported to l4linux on the Pandaboard and 1655MB on the Arndale board).

The changes made to the l4linux contrib code are a bit hard to spot from looking at the commit, because it's a modification of a patch file. The actual changes can be seen in https://github.com/cproc/genode/blob/edd767e1af0e546e68a2649120d323d435eff973/ports-foc/contrib/l4linux/patches-issue414/l4linux_issue414.diff.

cproc added a commit to cproc/genode that referenced this issue Mar 13, 2013

L4Linux stability improvements
- search for alternative virtual address regions upwards, starting from
  the given start address, in the 'l4re_rm_attach()' and
  'Region_manager::reserve_range()' functions

- don't treat memory locations above 0x80000000 in l4linux's virtual
  address space as device memory

- align the start address of the vmalloc area according to the assumption
  in 'devicemaps_init()'

Fixes genodelabs#414.
@iloskutov

This comment has been minimized.

Show comment
Hide comment
@iloskutov

iloskutov Mar 13, 2013

Contributor

I've tested this patch. It works for me. Thanks.

Contributor

iloskutov commented Mar 13, 2013

I've tested this patch. It works for me. Thanks.

@yefe

This comment has been minimized.

Show comment
Hide comment
@yefe

yefe Mar 14, 2013

Question about code:
In function
Region* Region_manager::reserve_range(Genode::size_t size, int align,
Genode::addr_t start)
There is a call to attach():
env()->rm_session()->attach(rmc->dataspace());
If it call causes no exceptions we use a native alignment=log2(size) and ignore align parameter in function reserve_range.
For my tests it may cause a problem with 512MB (region allocated by alloc_aligned has different alignment).

yefe commented Mar 14, 2013

Question about code:
In function
Region* Region_manager::reserve_range(Genode::size_t size, int align,
Genode::addr_t start)
There is a call to attach():
env()->rm_session()->attach(rmc->dataspace());
If it call causes no exceptions we use a native alignment=log2(size) and ignore align parameter in function reserve_range.
For my tests it may cause a problem with 512MB (region allocated by alloc_aligned has different alignment).

@cproc

This comment has been minimized.

Show comment
Hide comment
@cproc

cproc Mar 14, 2013

Member

@yefe: memory addresses in Ram_session_component::alloc() refer to core's address space, whereas memory addresses in Rm_session_component::attach() refer to l4linux's address space. There should be no problem if the alignment of those addresses differs.

The align function argument of Region_manager::reserve_range() is only used when searching for alternative ranges. That's according to the documentation of the l4re_rm_reserve_area() function (see http://os.inf.tu-dresden.de/L4Re/doc/classL4Re_1_1Rm.html#a5c934ae58a023f8cbe839a96bf046c2c), which is the primary caller of the Region_manager::reserve_range() function.

Just to be sure: have you applied the 'L4Linux stability improvements' patch (latest version is eea26f3) and run 'make prepare' afterwards in the 'ports-foc' repository?

Member

cproc commented Mar 14, 2013

@yefe: memory addresses in Ram_session_component::alloc() refer to core's address space, whereas memory addresses in Rm_session_component::attach() refer to l4linux's address space. There should be no problem if the alignment of those addresses differs.

The align function argument of Region_manager::reserve_range() is only used when searching for alternative ranges. That's according to the documentation of the l4re_rm_reserve_area() function (see http://os.inf.tu-dresden.de/L4Re/doc/classL4Re_1_1Rm.html#a5c934ae58a023f8cbe839a96bf046c2c), which is the primary caller of the Region_manager::reserve_range() function.

Just to be sure: have you applied the 'L4Linux stability improvements' patch (latest version is eea26f3) and run 'make prepare' afterwards in the 'ports-foc' repository?

@yefe

This comment has been minimized.

Show comment
Hide comment
@yefe

yefe Mar 14, 2013

No, i havent. Thanks for hint.

yefe commented Mar 14, 2013

No, i havent. Thanks for hint.

@nfeske

This comment has been minimized.

Show comment
Hide comment
@nfeske

nfeske Mar 14, 2013

Member

@cproc is the patch eea26f3 ready to be merged into staging?

Member

nfeske commented Mar 14, 2013

@cproc is the patch eea26f3 ready to be merged into staging?

iloskutov added a commit to Ksys-labs/genode that referenced this issue Mar 20, 2013

L4Linux stability improvements
- search for alternative virtual address regions upwards, starting from
  the given start address, in the 'l4re_rm_attach()' and
  'Region_manager::reserve_range()' functions

- don't treat memory locations above 0x80000000 in l4linux's virtual
  address space as device memory

- align the start address of the vmalloc area according to the assumption
  in 'devicemaps_init()'

Fixes genodelabs#414.

@cproc cproc closed this in b38fee2 Mar 21, 2013

cproc added a commit to cproc/genode that referenced this issue May 12, 2014

L4Linux stability improvements
- search for alternative virtual address regions upwards, starting from
  the given start address, in the 'l4re_rm_attach()' and
  'Region_manager::reserve_range()' functions

- don't treat memory locations above 0x80000000 in l4linux's virtual
  address space as device memory

- align the start address of the vmalloc area according to the assumption
  in 'devicemaps_init()'

Fixes genodelabs#414.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment