Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

maxmem_kb leak during migration #955

Closed
amarao opened this Issue Dec 22, 2012 · 9 comments

Comments

Projects
None yet
7 participants
Contributor

amarao commented Dec 22, 2012

XCP 1.6:

If vm (pv) migrated on localhost, every migration raise maxmem_kb limit for domain.

Steps to reproduce:
create and start vm (f.e. test).
get an maxmem_kb value for domain from xc.domain_getinfo()
for a in seq 1 1000;do xe vm-migration vm=test host=hostname;done
recheck maxmem_kb from xc.domain_getinfo()

Expected: same value
Actual: very large value, raising up after every new migration.

Contributor

mcclurmc commented Jan 3, 2013

I believe that this is a known issue, though it was only recently discovered by the @jeromemaloberti on the Xapi team. He's on holiday now, but I'll assign this issue to him so that he can confirm whether this is the same issue he found.

@ghost ghost assigned jeromemaloberti Jan 3, 2013

Contributor

jeromemaloberti commented Jan 3, 2013

It is the same the same issue. Some additional memory is allocated for the migration, but it is never reclaimed. The solution is not known for now.

Contributor

amarao commented Jan 3, 2013

03.01.2013 19:07, jeromemaloberti пишет:

May be just restore original value for domain after successfull
migration? That change for new domain caused by xapi or by xenguest ?

It is the same the same issue. Some additional memory is allocated for
the migration, but it is never reclaimed. The solution is not known
for now.


Reply to this email directly or view it on GitHub
xen-org#955 (comment).

Contributor

jeromemaloberti commented Jan 3, 2013

The memory is requested by xapi through squeezed, but then transfered to xen.
Well, if you have an idea how to fix it, I will be very happy to test it when I return from holiday :).
However, I don't know what xen is actually doing with the additional memory. If it is just used during the migration, that's fine, but if xen keeps using it after, it will be a problem. Any suggestion ?

Contributor

amarao commented Jan 3, 2013

It's not the memory, actually, but maxmem value (just an internal Xen
limit for domain). That leak do not cause any direct damage (like 'out
of memory' condition for host), but just raise a theoretical memory
limit for domain, so domain can (if want) can ask for more, more and
more memory beyond static-max limit of xapi for that VM.

03.01.2013 19:18, jeromemaloberti пишет:

The memory is requested by xapi through squeezed, but then transfered
to xen.
Well, if you have an idea how to fix it, I will be very happy to test
it when I return from holiday :).
However, I don't know what xen is actually doing with the additional
memory. If it is just used during the migration, that's fine, but if
xen keeps using it after, it will be a problem. Any suggestion ?


Reply to this email directly or view it on GitHub
xen-org#955 (comment).

Collaborator

djs55 commented Jan 3, 2013

I think we're just missing the subtraction on the receiver (in xenopsd).
It's probably easiest to fix this by "printf debugging" rather than
studying the code too carefully.

On 3 Jan 2013, at 15:41, George Shuklin notifications@github.com wrote:

It's not the memory, actually, but maxmem value (just an internal Xen
limit for domain). That leak do not cause any direct damage (like 'out
of memory' condition for host), but just raise a theoretical memory
limit for domain, so domain can (if want) can ask for more, more and
more memory beyond static-max limit of xapi for that VM.

03.01.2013 19:18, jeromemaloberti пишет:

The memory is requested by xapi through squeezed, but then transfered
to xen.
Well, if you have an idea how to fix it, I will be very happy to test
it when I return from holiday :).
However, I don't know what xen is actually doing with the additional
memory. If it is just used during the migration, that's fine, but if
xen keeps using it after, it will be a problem. Any suggestion ?


Reply to this email directly or view it on GitHub
xen-org#955 (comment).


Reply to this email directly or view it on
GitHubhttps://github.com/xen-org/xen-api/issues/955#issuecomment-11847909.

Contributor

andyhhp commented Jul 1, 2013

The root of the problem here appears to be confusion over Xen's memory handling.

A VM is made up of static-max pages and delta, where delta is the extra overhead for Xen.

This delta is dependent on the version of Xen, the hardware available, command line options at boot, VM configuration, runtime tweaks to the VM and even down to the VM physmap fragmentation.

The delta for the VM on its source host bares no specific relation to the delta for the same VM on the destination host (although in general it should approximate to linear function of static-max).

The reason VMs get bigger across migrate is that the xenops finds VM(static-max + delta) on the source and creates a new domain on the destination of the same size. This is completely wrong, and defeats the purpose it seems to have been implemented for. When the new domain is static-max + old delta, Xen must find a new slightly larger delta from elsewhere in memory.

The target domain must be created as static-max, to be the same as source domain. Xen will find its delta from other free memory.

If the toolstack wishes to approximate this delta, it should look at the ratio for HVM domains on the target host as a better guess, but there is no way to reserve this memory for Xen; Attempting to do so by making a bigger domain is actively counter-productive.

Collaborator

johnelse commented Sep 6, 2013

Potential fix: #1486

@ghost ghost assigned johnelse Oct 15, 2013

Collaborator

simonjbeaumont commented Mar 9, 2015

Looks like there was a pull request to address this issue. I'll close this now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment