update: Return memory to host on memory update. #793

jcvenegas · 2018-09-27T15:43:15Z

If memory update will reduce memory we want to get memory
back to the host and restrict the guest to use more memory.

$ docker exec -ti kata-1 free -h
              total        used        free      shared  buff/cache   available
Mem:           5.9G         23M        5.8G        8.2M         18M        5.8G
Swap:            0B          0B          0B
$ docker update --memory 2G kata-1
kata-1
$ docker exec -ti kata-1 free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G         56M        1.8G        8.2M         18M        1.8G
Swap:            0B          0B          0B

Fixes: #790

Signed-off-by: Jose Carlos Venegas Munoz jose.carlos.venegas.munoz@intel.com

jcvenegas · 2018-09-27T16:22:45Z

Create container with 6GB and stress the memory with 75%

+ docker run -tdi --name kata-1 -m 6G --rm --runtime kata-runtime stress-ng bash
55d15db690fc059f66edc9f286add9856e9eaf29f033b5a0c0ea3f4feb462d4d
+ docker exec -ti kata-1 free -h
              total        used        free      shared  buff/cache   available
Mem:           5.9G         24M        5.8G        8.2M         18M        5.8G
Swap:            0B          0B          0B
# qemu memory consumption (smem)
  PID User     Command                         Swap      USS      PSS      RSS
28465 root     /opt/kata/bin/qemu-system-x        0   286.4M   286.4M   286.4M
+ docker exec -ti kata-1 stress-ng --vm 1 --vm-bytes 75% --vm-method all --verify -t 20s -v
stress-ng: debug: [12] 1 processor online, 1 processor configured
stress-ng: debug: [12] main: can't set oom_score_adj
stress-ng: info:  [12] dispatching hogs: 1 vm
stress-ng: debug: [12] cache allocate: default cache size: 16384K
stress-ng: debug: [12] starting stressors
stress-ng: debug: [12] 1 stressor spawned
stress-ng: debug: [16] stress-ng-vm: can't set oom_score_adj
stress-ng: debug: [16] stress-ng-vm: started [16] (instance 0)
stress-ng: debug: [16] stress-ng-vm using method 'all'
stress-ng: debug: [16] stress-ng-vm: exited [16] (instance 0)
stress-ng: debug: [12] process [16] terminated
stress-ng: info:  [12] successful run completed in 20.16s
# qemu memory consumption (smem)
  PID User     Command                         Swap      USS      PSS      RSS
28465 root     /opt/kata/bin/qemu-system-x        0     4.6G     4.6G     4.6G

update memory to reduce it.

With balloon:

+ docker update --memory 1G kata-1
kata-1
+ sleep 5s
+ docker exec -ti kata-1 free -h
              total        used        free      shared  buff/cache   available
Mem:           898M         40M        839M        8.2M         18M        795M
Swap:            0B          0B          0B
# qemu memory consumption (smem)
  PID User     Command                         Swap      USS      PSS      RSS
28465 root     /opt/kata/bin/qemu-system-x        0   310.8M   310.8M   310.8M

without balloon:

+ docker update --memory 1G kata-1
kata-1
+ sleep 5s
+ docker exec -ti kata-1 free -h
              total        used        free      shared  buff/cache   available
Mem:           5.9G         24M        5.8G        8.2M         18M        5.8G
Swap:            0B          0B          0B
# qemu memory consumption (smem)
  PID User     Command                         Swap      USS      PSS      RSS 
30038 root     /opt/kata/bin/qemu-system-x        0     4.6G     4.6G     4.6G

jcvenegas · 2018-09-27T17:50:23Z

So I am getting an strange behavior when hotplug memory just by adding -balloon virtio. When I hotplug memory the kernel total memory is not near what I expect. It is increased but not as I would like.

[jcvenega@jcvenega-nuc runtime]$ docker run -dti -m 256M --runtime kata-runtime --name kata-1 stress-ng
ec02de8650364902004e3a73047ea8219d077d8fe0b321f2c3c8440263fad725
[jcvenega@jcvenega-nuc runtime]$ docker exec -ti kata-1  free -h
              total        used        free      shared  buff/cache   available
Mem:           234M         20M        201M        1.7M         12M        204M
Swap:            0B          0B          0B
[jcvenega@jcvenega-nuc runtime]$ docker update --memory 4G --memory-swap 4G kata-1
kata-1
[jcvenega@jcvenega-nuc runtime]$ docker exec -ti kata-1  free -h
              total        used        free      shared  buff/cache   available
Mem:           746M         81M        652M        1.7M         12M        647M
Swap:            0B          0B          0B

jcvenegas · 2018-09-27T18:56:01Z

update: I just tried in a VM with the kata kernel and same qemu binary and worked I'll debug in the kata guest OS.

jcvenegas · 2018-09-27T22:06:33Z

found the issue, will send the fix

jodh-intel · 2018-09-28T09:04:13Z

nice.

lgtm

jcvenegas · 2018-10-04T15:54:58Z

depends on kata-containers/govmm#55

jcvenegas · 2018-10-04T15:57:00Z

@jcvenegas @sboeuf I think this is ready to a new review. I think this is almost the final version, just waiting to merge kata-containers/govmm#55 and I will create a PR to test this on tests repo.

sboeuf · 2018-10-04T17:57:03Z

@jcvenegas Ok I'll take a look later today!

jcvenegas · 2018-10-04T21:19:16Z

/test

jcvenegas · 2018-10-04T23:09:22Z

/test

codecov · 2018-10-04T23:48:08Z

Codecov Report

Merging #793 into master will increase coverage by 1.70%.
The diff coverage is 65.78%.

@@            Coverage Diff             @@
##           master     #793      +/-   ##
==========================================
+ Coverage   51.67%   53.38%   +1.70%     
==========================================
  Files         107      110       +3     
  Lines       14615    18759    +4144     
==========================================
+ Hits         7552    10014    +2462     
- Misses       6152     7592    +1440     
- Partials      911     1153     +242

sboeuf · 2018-10-05T17:02:19Z

virtcontainers/container.go

+		addMemDevice := &memoryDevice{
+			sizeMB: 0,
+		}
+		data, err := c.sandbox.hypervisor.hotplugAddDevice(addMemDevice, memoryDev)


Why are we trying to hotplug more memory here? We're in the case where oldMemMB == newMemMB, and as mentioned by the debug log the current memory will not be modified.

sboeuf

Discussed offline with @jcvenegas. This needs some changes to prevent from introducing the balloon knowledge at the container.go level.

sboeuf · 2018-10-05T17:06:57Z

virtcontainers/qemu.go

-		return 0, fmt.Errorf("Unable to hotplug %d MiB memory, the SB has %d MiB and the maximum amount is %d MiB",
-			memDev.sizeMB, currentMemory, q.config.MemorySize)
+	if memDev.sizeMB == 0 {
+		// handle case when not memory added (because not needed)


// handle the case where no memory is added (because not needed)

Well, actually I would remove the comment since there's nothing to do in this case.
Just print a log and return.

sboeuf · 2018-10-05T17:09:13Z

virtcontainers/qemu.go

+				"hotplug": "memory",
+			},
+		).Debug("Not needed to hotplug, updating balloon")
+		return 0, q.updateMemoryBalloon(currentMemory)


Why do we need to update the balloon, this case should be a simple no-op, right?

jcvenegas · 2018-10-11T01:40:05Z

/test

jcvenegas · 2018-10-11T02:44:26Z

/test

jcvenegas · 2018-10-11T14:17:36Z

/test

jcvenegas · 2018-10-11T15:14:28Z

@sboeuf take a look, I think now is more explicit the the balloon usage. @linzichang @clarecch please take a look this PR is planning to move part of the logic on how we manage memory. The last to commits do that.

cedriccchen · 2018-10-12T10:07:02Z

Hi @jcvenegas , most of this PR looks nice, but two doubts：

Why should we move hotplug logic from container to sandbox? I think c.sandbox.XXX is the same as s.XXX.
I am glad to see the thought of virtcontainers: hotplug memory with kata-runtime update command. #624 (comment) is almost realized, as @linzichang and me are too busy to handle the Support elastic memory hotplug #788 issue. Just a question, do we need vm_reserved_memory memtions in virtcontainers: hotplug memory with kata-runtime update command. #624 (comment) to keep the vm running normally when sandbox memory is all used by container? With vm_reserved_memory , we can still hotplug memory from hostos as long as hostos has enough free memory , and the guestos will not panic even if all sandbox memory is used by container.

cedriccchen · 2018-10-15T03:22:30Z

virtcontainers/sandbox.go

+
+	// Memory is not updated if memory limit not set
+	if newResources.MemMB != 0 {
+		c.config.Resources.MemMB = newResources.MemMB


If just a part of memory is hotplugged successfully, the memory limit cgroup of container should not be set to newResources.MemMB, which is expected to be hotplugged.

@clarecch I need to test this,
from my perspective we still need to update the cgroup

the container should update its cgroup

The sandbox manage the memory from the guest.

I expect that cgroup limit does not depends on the guest physical memory, so we can set the cgroup with more memory than the memory that is available in the VM.

An example that is valid is when:

have a sandbox/pod/VM with all (almost ) the memory of the host added.
create sandbox, create container -memory all the host memory

Then we add another container to the Sandbox/pod/VM without any memory cgroup.

After we update the cgroup of the new container to increase memory.
a. The sandbox wont increase the memory but the container will share the memory with other containers and the cgroup will be limited.

Thanks for explanation.

raravena80 · 2019-03-25T21:51:03Z

@jcvenegas ping, any updates?

raravena80 · 2019-04-18T00:30:50Z

@jcvenegas ping, any updates? Thx.

grahamwhaley · 2019-05-01T14:05:49Z

Branch is now conflicted (again). @jcvenegas - can you give us an update on your plan with this at least - are you actively pursuing and still hoping to land?

raravena80 · 2019-05-22T21:50:16Z

@jcvenegas any updates? Thx!

caoruidong · 2019-06-17T10:02:47Z

ping @jcvenegas

raravena80 · 2019-06-27T03:44:01Z

@jcvenegas ping.

raravena80 · 2019-07-31T18:13:42Z

ping @jcvenegas

caoruidong · 2019-08-29T03:32:32Z

ping @jcvenegas Shall we close it?

jcvenegas · 2019-09-12T21:54:34Z

/test

Request to return memory back using balloon. Fixes: kata-containers#790 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

Now that we are in qemu 4.x it is not need to handle it in an special way for balloon. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

jcvenegas · 2019-09-13T16:33:19Z

/test

jcvenegas · 2019-09-13T20:29:08Z

I'll try to get this PR merged this year, @bergwolf @egernst please take a look would be nice to have it merged for 1.9

jcvenegas · 2019-10-02T17:50:43Z

@devimc @egernst @bergwolf ping place take a look

devimc

thanks @jcvenegas

devimc · 2019-10-02T19:47:18Z

virtcontainers/qemu.go

@@ -1128,6 +1134,9 @@ func (q *qemu) hotplugVFIODevice(device *config.VFIODev, op operation) (err erro
 	devID := device.ID

 	if op == addDevice {
+
+		// When HasVFIODevice is set balloon size is set to maximal memory


and where is HasVFIODevice set ?

raravena80 · 2019-11-15T19:45:25Z

@jcvenegas any update on this PR?

jcvenegas · 2019-11-19T18:10:47Z

@raravena80 thanks for ping on this I was waiting for feedback from @egernst and @bergwolf if still we want to enable this, the PR is functional but this PR does not have a lot of priority today.

raravena80 · 2019-12-19T22:24:19Z

@jcvenegas any updates?

Your weekly Kata herder.

jodh-intel · 2020-01-20T10:32:08Z

Hmm, this is a pain - the only failing CI is the nemu one. But we no longer support that so we could land this. But GitHub won't let us. Thoughts @jcvenegas?

@chavafg - btw, as nemu is dead, do we still need the nemu CI jobs?

grahamwhaley · 2020-01-20T10:34:32Z

a re-submit as a new PR might be the easiest thing to do.
Good thought on the CI - if we have not done it already.

chavafg · 2020-01-20T14:58:47Z

@jodh-intel we already removed the nemu CI for master branch.
We still have it running for stable branches (1.8.x)... Although not sure if we still support that branch. Maybe we can now remove it definitely.

The nemu register above was from an old execution, we can ignore it, or as @grahamwhaley comments, the PR can be re-pushed.

jcvenegas · 2020-01-28T16:24:54Z

@jodh-intel its being a long time of this PR I think make sense to ask to @kata-containers/architecture-committee if we want it or should I close it.

raravena80 · 2020-04-04T19:52:01Z

@chavafg @jcvenegas any updates on this PR? Thx

jcvenegas added wip feature New functionality labels Sep 27, 2018

jcvenegas force-pushed the balloon-update-remove branch 5 times, most recently from 61a8e9c to d25cee5 Compare October 4, 2018 15:52

jcvenegas force-pushed the balloon-update-remove branch from d25cee5 to 157e713 Compare October 4, 2018 19:17

jcvenegas force-pushed the balloon-update-remove branch from 418bfc4 to 31fcea0 Compare October 5, 2018 15:46

sboeuf reviewed Oct 5, 2018

View reviewed changes

sboeuf suggested changes Oct 5, 2018

View reviewed changes

jcvenegas force-pushed the balloon-update-remove branch from 759640e to 80b21f9 Compare October 11, 2018 02:44

cedriccchen reviewed Oct 15, 2018

View reviewed changes

jcvenegas force-pushed the balloon-update-remove branch from f01caba to a006319 Compare September 12, 2019 21:47

jcvenegas force-pushed the balloon-update-remove branch 3 times, most recently from ab8ac47 to fc0f210 Compare September 13, 2019 16:07

jcvenegas added 2 commits September 13, 2019 11:15

update: Memory use balloon to reduce memory on host.

ba00888

Request to return memory back using balloon. Fixes: kata-containers#790 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

vfio: remove vfio counter for balloon.

131d5bc

Now that we are in qemu 4.x it is not need to handle it in an special way for balloon. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

jcvenegas force-pushed the balloon-update-remove branch from fc0f210 to 131d5bc Compare September 13, 2019 16:26

devimc reviewed Oct 2, 2019

View reviewed changes

teawater mentioned this pull request Feb 6, 2020

how-to: Add how-to for Kata Containers with virtio-mem kata-containers/documentation#592

Merged

jcvenegas closed this May 27, 2020

update: Return memory to host on memory update. #793

update: Return memory to host on memory update. #793

Conversation

jcvenegas commented Sep 27, 2018

jcvenegas commented Sep 27, 2018 • edited

update memory to reduce it.

jcvenegas commented Sep 27, 2018

jcvenegas commented Sep 27, 2018

jcvenegas commented Sep 27, 2018

jodh-intel commented Sep 28, 2018 • edited by amshinde

jcvenegas commented Oct 4, 2018

jcvenegas commented Oct 4, 2018

sboeuf commented Oct 4, 2018

jcvenegas commented Oct 4, 2018

jcvenegas commented Oct 4, 2018

codecov bot commented Oct 4, 2018 • edited

Codecov Report

sboeuf Oct 5, 2018

Choose a reason for hiding this comment

sboeuf left a comment

Choose a reason for hiding this comment

sboeuf Oct 5, 2018

Choose a reason for hiding this comment

sboeuf Oct 5, 2018

Choose a reason for hiding this comment

sboeuf Oct 5, 2018

Choose a reason for hiding this comment

jcvenegas commented Oct 11, 2018

jcvenegas commented Oct 11, 2018

jcvenegas commented Oct 11, 2018

jcvenegas commented Oct 11, 2018

cedriccchen commented Oct 12, 2018

cedriccchen Oct 15, 2018

Choose a reason for hiding this comment

jcvenegas Oct 15, 2018 • edited

Choose a reason for hiding this comment

cedriccchen Oct 16, 2018

Choose a reason for hiding this comment

raravena80 commented Mar 25, 2019

raravena80 commented Apr 18, 2019

grahamwhaley commented May 1, 2019

raravena80 commented May 22, 2019

caoruidong commented Jun 17, 2019

raravena80 commented Jun 27, 2019

raravena80 commented Jul 31, 2019

caoruidong commented Aug 29, 2019

jcvenegas commented Sep 12, 2019

jcvenegas commented Sep 13, 2019

jcvenegas commented Sep 13, 2019

jcvenegas commented Oct 2, 2019

devimc left a comment

Choose a reason for hiding this comment

devimc Oct 2, 2019

Choose a reason for hiding this comment

raravena80 commented Nov 15, 2019 • edited

jcvenegas commented Nov 19, 2019

raravena80 commented Dec 19, 2019

jodh-intel commented Jan 20, 2020

grahamwhaley commented Jan 20, 2020

chavafg commented Jan 20, 2020

jcvenegas commented Jan 28, 2020

raravena80 commented Apr 4, 2020

jcvenegas commented Sep 27, 2018 •

edited

jodh-intel commented Sep 28, 2018 •

edited by amshinde

codecov bot commented Oct 4, 2018 •

edited

jcvenegas Oct 15, 2018 •

edited

raravena80 commented Nov 15, 2019 •

edited