Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gitlab v15.0 RC1 memory leak #1106

Closed
danielpclark opened this issue Jun 2, 2018 · 8 comments

Comments

Projects
None yet
2 participants
@danielpclark
Copy link

commented Jun 2, 2018

Using this commit turnkeylinux-apps/gitlab@08bcacf I found only two issues.

  • First the auto-generated password for logging in doesn't show up
  • Second there is a memory leak in this particular setup

To get around the first issue I SSH'd in and changed to the /home/git/gitlab directory and ran bundle exec rails console production to access the first user.

user = User.first
user.username
# => 'gitlab-admin'

user.password = "new_password"
user.save

The real issue I'm seeing though is the memory leak. Without using the web interface the memory will very slowly increase over time. Using the Gitlab site web interface rapidly increases used memory.

I tried just switching to latest versions in building through the TKLDev build but that ran into Rails issues and Gitaly connectivity issues so clearly an upgrade will require more involved work.

@danielpclark

This comment has been minimized.

Copy link
Author

commented Jun 2, 2018

Alright, slightly more detail now. It takes a long time to consume more memory if you don't browse the site. My guess is the leak is about 5Mbs per hour with no use. I'll be looking further into it with performance tools so I'll let you know later.

@JedMeister

This comment has been minimized.

Copy link
Member

commented Jun 3, 2018

Hi @danielpclark - thanks for your feedback and input - it's really appreciated! 👍

I'm curious what you mean by:

the auto-generated password for logging in doesn't show up

The autogenerated password isn't intended to be used, you should be setting a password on firstboot via the inithook. As you can see there, it should be setting the user password (albeit for user 1 rather than gitlab-admin although by my understanding they should be one and the same?) So I'm not really clear what you mean there?! Could you please clarify? Having said that, I haven't actually done any pre-release "real world" testing, so perhaps my code review overlooked something that doesn't work as intended.

Re the memory leak, that doesn't sound good. If you have any further info on that, I'd certainly love to hear.

@danielpclark

This comment has been minimized.

Copy link
Author

commented Jun 4, 2018

What I mean by

the auto-generated password for logging in doesn't show up

Is that I remember from the v14 series stable release install that a random password was generated for use in the Rails application and was displayed at boot time either before the root login or after. That made installation and usage a breeze.

This isn't there this time. This time, at no point was I ever asked or shown a password for the first time logging in with Gitlab's web interface. At least I don't believe I was asked… I wrote down each of the passwords I gave during installation and none of those would work.

Of course I didn't know what the username was because of that info not being display at boot so I only tried the passwords for usernames admin and root on the Gitlab web login. And that could have been why they didn't work. admin was what I expected the username to be and I read somewhere during the install (I believe) that the password should be left blank, but the web login wouldn't allow for that.

Re the memory leak, that doesn't sound good. If you have any further info on that, I'd certainly love to hear.

I don't have the time to look into this right away as I've got several deadlines this week. I will be looking into this further afterwards.

@JedMeister

This comment has been minimized.

Copy link
Member

commented Jun 4, 2018

Is that I remember from the v14 series stable release install that a random password was generated for use in the Rails application and was displayed at boot time either before the root login or after. That made installation and usage a breeze.

Hmm, I'm not sure. I've just looked back through the code, and unless it was some unintended "feature" you were leveraging, I'm not sure how that happened/worked. From what I can see, other than updates to GitLab itself, our firstboot process is essentially unchanged, at least within v14.x (and v15.0).

This isn't there this time. This time, at no point was I ever asked or shown a password for the first time logging in with Gitlab's web interface. At least I don't believe I was asked… I wrote down each of the passwords I gave during installation and none of those would work.

Of course I didn't know what the username was because of that info not being display at boot so I only tried the passwords for usernames admin and root on the Gitlab web login. And that could have been why they didn't work. admin was what I expected the username to be and I read somewhere during the install (I believe) that the password should be left blank, but the web login wouldn't allow for that.

FWIW, on firstboot, via the local console you should be presented with a series of questions (these are what we call the inithooks). Two of those should be "GitLab 'admin' email address" and "GitLab 'admin' password". You should then use that email (as username) and password combo to log into the webUI.

Although as I say, I haven't actually tested the v15.0 appliance yet (and we're still a little while away from publishing it, so it's considered "release candidate" code at best). So perhaps despite appearing to be OK, there is a bug that wasn't obvious during my code review.

Out of interest, are you just building an ISO in TKLDev then installing to a VM? Or are you creating and installing it a different way? (E.g. via buildtasks)

I don't have the time to look into this right away as I've got several deadlines this week. I will be looking into this further afterwards.

Fantastic. I look forward to hearing what you find.

@JedMeister JedMeister added this to the 15.0 milestone Jun 4, 2018

@danielpclark

This comment has been minimized.

Copy link
Author

commented Jun 4, 2018

Out of interest, are you just building an ISO in TKLDev then installing to a VM? Or are you creating and installing it a different way? (E.g. via buildtasks)

I built the ISO from TKLDev v15.0 RC1 from the image provided at https://www.turnkeylinux.org/blog/v15.0rc1-core-and-tkldev . I recommend providing a link to the release candidate on the main downloads page as I initially downloaded the wrong one and that didn't work before I realized I had to use the link provided from the blog post.

@JedMeister

This comment has been minimized.

Copy link
Member

commented Jun 4, 2018

I recommend providing a link to the release candidate on the main downloads page as I initially downloaded the wrong one and that didn't work before I realized I had to use the link provided from the blog post.

Yes, great point, thanks. I'll do that right now! 👍

Done. As suggested, both the Core and TKLDev appliance pages now have download links for the v15.0rc1 builds.

@danielpclark

This comment has been minimized.

Copy link
Author

commented Jul 2, 2018

Alright, I provisioned more memory for my VM and gave it several days of no interaction to see what the story with the memory management is. It takes three days for the memory usage to plateau and stay level which is a great sign. This indicates that there isn't a memory leak.

Also Gitlab's documentation on recommended RAM for provisioning is wrong so they need to fix it. For just one person it recommended 2Gb of RAM, but the actual minimum should be 4Gb if you like to live dangerously.

They do have a bit of an issue where the background workers don't clean up the memory usage after running. But they've hard coded in a solution to completely kill the background worker system at certain hard limits. So it's a ragtag solution, but it works. Here's my VM memory snapshots for three days.

gitlab-memory

@JedMeister

This comment has been minimized.

Copy link
Member

commented Jul 2, 2018

Thanks for the update @danielpclark Very much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.