Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VMware 4.11.2.0 system VMs memory consumption grows overtime until heavy swapping occurs #3039

Closed
rohityadavcloud opened this issue Nov 20, 2018 · 11 comments · Fixed by #3126
Closed

VMware 4.11.2.0 system VMs memory consumption grows overtime until heavy swapping occurs #3039

rohityadavcloud opened this issue Nov 20, 2018 · 11 comments · Fixed by #3126
Milestone
4.13.0.0

Comments

@rohityadavcloud
Copy link
Member

rohityadavcloud commented Nov 20, 2018 •
edited by PaulAngus
Loading

Memory Consumption on vSphere
vRouter are starting to swap with low memory available.
Increasing memory only causes a longer time period before heavy swapping occurs.

@resmo shared a related issue on dev list: systemd/systemd#8015

ISSUE TYPE
  • Bug Report
COMPONENT NAME
VR
CLOUDSTACK VERSION
ACS 4.11.x
OS / ENVIRONMENT

Seen only in VMware environments

STEPS TO REPRODUCE
- Deploy VR in vmware
- SSH into it from management server
- See memory growth over time
EXPECTED RESULTS
No additional memory consumption.
@rohityadavcloud rohityadavcloud added this to the 4.11.3.0 milestone Nov 20, 2018
@PaulAngus PaulAngus changed the title Memory consumption grows in VR due to SSH (vmware) VMware 4.11.2.0 system VMs memory consumption grows overtime until heavy swapping occurs Nov 26, 2018
@PaulAngus PaulAngus modified the milestones: 4.11.3.0, 4.11.2.1 Nov 26, 2018
@onitake
Copy link
Contributor

onitake commented Dec 3, 2018

Hi @rhtyd @PaulAngus - thanks for taking on this issue.

If you have any suggestions on more tests we could run, please share.

@onitake
Copy link
Contributor

onitake commented Dec 10, 2018

Is the fix going to be something like the one mentioned here https://lists.apache.org/thread.html/cb36b5efe8327c080919086688707650a3ff036c884a2dd87dd6e66d@%3Cdev.cloudstack.apache.org%3E , or is this an unrelated issue?

@PaulAngus
Copy link
Member

Hi Gregor,
Yes, it is the same issue.
I'm currently uploading the latest test tempates to:
http://packages.shapeblue.com/systemvmtemplate/public-testing/41121rc0/

Latest have the date of the 10th Dec on them.

They include

  • adding cache back pressure
  • setting a target minimum amount of free memory (reduces caching)
  • reducing the number of startup Apache HTTPD threads/processes
  • tighter control over the starting of guest daemons for hypervisors
  • fix for misconfigured vmware tools conf file
  • explicit ordering of cloudstack start processes
  • and proper termination of startup services

The memory usage is much less..

I don't think that hammering with ssh connections is a useful test. instead i create a network with a couple of VMs and then constantly add and remove firewall rules and port-forwarding rules to them. one every couple of seconds to ensure that they are separate connections.

I have left this running for 3 days with no swapping on a 256MB VR.

@onitake
Copy link
Contributor

onitake commented Dec 11, 2018

We were able to drive memory usage through the roof with SSH connections, because each session would leak memory. But we also had success with lots of network reconfiguration requests, so I guess both are valid test cases.

@onitake
Copy link
Contributor

onitake commented Jan 7, 2019

@PaulAngus Is it also possible to get just the templates in lieu of full 4.11.2.1 RPMs?
If I try to access your package server through a web browser, I just get a blank page.

This was referenced Jan 11, 2019
Merged
Closed
@rohityadavcloud rohityadavcloud modified the milestones: 4.11.2.1, 4.11.3.0 Mar 12, 2019
@andrijapanicsb
Copy link
Contributor

Any progress on this one - anyone else to share experience or a proper fix ?

@PaulAngus
Copy link
Member

This is addressed by #3127

@andrijapanicsb
Copy link
Contributor

Pardon my blind eyes ... ... ...

@rohityadavcloud
Copy link
Member Author

Interesting notes from systemd/systemd#8015 (comment)

On Debian I fix the memory consumption commenting the following line on /etc/pam.d/common-session:

#session optional pam_systemd.so

But, with that, the ssh sessions doesn't terminate correctly. The fix is already done:

cp /usr/share/doc/openssh-client/examples/ssh-session-cleanup.service /etc/systemd/system/
systemctl  enable ssh-session-cleanup.service
Other alternative solution for Debian, add to the end of the file /etc/pam.d/systemd-user :

@include null

It doesn't include any file, because null doesn't exists.

@andrijapanicsb
Copy link
Contributor

^^^ included in #3126 and #3127

@rohityadavcloud
Copy link
Member Author

Carry on here: #3126

@rohityadavcloud rohityadavcloud modified the milestones: 4.11.3.0, 4.13.0.0 May 27, 2019
rohityadavcloud pushed a commit that referenced this issue Jun 26, 2019
@rohityadavcloud
systemvm: improve SystemVM startup and memory usage (#3126)
0331999 
In order to reduce memory footprint and improve boot speed/predictability.
The following changes have been made:

- add vm.min_free_kbytes to sysctl
- periodically clear disk cache (depending on memory size)
- only start guest services specific to hypervisor
- use systemvm code to determine hypervisor type (not systemd)
- start cloud service at end of post init rather than through systemd
- reduce initial threads started for httpd
- fix vmtools config file

Fixes #3039

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
4.13.0.0
Development

Successfully merging a pull request may close this issue.

Improve System VM startup and memory usage shapeblue/cloudstack
4 participants
@rohityadavcloud @onitake @PaulAngus @andrijapanicsb