New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Micro-optimizations to Plugin data to reduce minimum memory for Jenkins #3654

Merged
merged 10 commits into from Oct 6, 2018

Conversation

4 participants
@svanoort
Member

svanoort commented Sep 27, 2018

Applying part of findings from a small investigation into how to reduce overall Jenkins memory footprint -- too small to really merit a JIRA.

WIP until CI tests pass

  • Created utility to reduce memory waste of HashMaps, especially for small maps
  • Use map sizing utility to reduce wasted HashMap oversizing for plugin dependency maps
  • Intern a whole bunch of duplicated Plugin strings (name, short-name, version, etc)
  • Use a canonical empty String array rather than creating empty ones just for casting (and use this where previously we were using static fields throughout core).
  • Should be covered by existing tests

Proposed changelog entries

  • Minor improvements to reduce minimum memory footprint for Jenkins, especially around Update Center & plugin metadata

Submitter checklist

  • Changelog entry appropriate for the audience affected by the change (users or developer, depending on the change). Examples
    * Use the Internal: prefix if the change has no user-visible impact (API, test frameworks, etc.)

Desired reviewers

@oleg-nenashev - may be of a bit of interest as a basic platform enhancement that can be applied in other places to make Jenkins more friendly for lightweight cloud deployment.

@svanoort

This comment has been minimized.

Show comment
Hide comment
@svanoort

svanoort Sep 27, 2018

Member

Other opportunities to reduce memory footprint:

Starting from a baseline of about 107 MB of actual, strongly-held heap used after doing a bunch of tuning and running a build or two:

  • Something in Jenkins or plugins is kicking off the H2 database, besides the Pipeline Maven Plugin -- which is a great lightweight embedded database, but carries an extra 15 MB of memory footprint which I can't find a real justification for us to have (especially in headless master use-cases)
  • Jelly: whole bunch of duplicated strings
  • Stapler: org.kohsuke.stapler.bind.BoundObjectTable$Table - 10 MB, including a lot of org.kohsuke.stapler.jelly.CustomTagLibrary and org.kohsuke.stapler.jelly.CustomJellyContext objects which eat several hundred kB to 1 MB of retained object memory
Member

svanoort commented Sep 27, 2018

Other opportunities to reduce memory footprint:

Starting from a baseline of about 107 MB of actual, strongly-held heap used after doing a bunch of tuning and running a build or two:

  • Something in Jenkins or plugins is kicking off the H2 database, besides the Pipeline Maven Plugin -- which is a great lightweight embedded database, but carries an extra 15 MB of memory footprint which I can't find a real justification for us to have (especially in headless master use-cases)
  • Jelly: whole bunch of duplicated strings
  • Stapler: org.kohsuke.stapler.bind.BoundObjectTable$Table - 10 MB, including a lot of org.kohsuke.stapler.jelly.CustomTagLibrary and org.kohsuke.stapler.jelly.CustomJellyContext objects which eat several hundred kB to 1 MB of retained object memory
@oleg-nenashev

Nice! Test failures are real, so some fixes are needed. It would be also great to split empty arrays and string interning to separate pull requests. Empty arrays could be merged quickly, but String interning needs a really careful review.

@svanoort svanoort closed this Sep 27, 2018

@svanoort svanoort reopened this Sep 27, 2018

Addressed via fixes

@svanoort

This comment has been minimized.

Show comment
Hide comment
@svanoort

svanoort Sep 28, 2018

Member
  • Comparing heap dumps shows a flat drop in memory retained for the UpdateSite object: 4.3 MB retained before, 2.9 MB retained after
  • Even more interesting if you look at memory use (1) latest build off this PR (2) The dip is restarting using core 1.121.3 LTS (without this PR applied).

screen shot 2018-09-28 at 2 14 13 pm

A little info on test environment: this is my scalability test environment that I'll be showing off at Jenkins World France, running a suite of Pipelines simulating various user loads (waiting until all the builds in the Multibranch project complete, then running 5x System.gc() and waiting a few minutes).

I have already done a fair bit of tuning to reduce memory use in that environment (primarily manipulating Java system settings for GC and threads, etc).

But yeah, oddly enough the before-and-after (technically, after and then before in the time sequence of the graph) shows something like a 25% reduction in memory footprint (potentially there may be other factors at play there).

Member

svanoort commented Sep 28, 2018

  • Comparing heap dumps shows a flat drop in memory retained for the UpdateSite object: 4.3 MB retained before, 2.9 MB retained after
  • Even more interesting if you look at memory use (1) latest build off this PR (2) The dip is restarting using core 1.121.3 LTS (without this PR applied).

screen shot 2018-09-28 at 2 14 13 pm

A little info on test environment: this is my scalability test environment that I'll be showing off at Jenkins World France, running a suite of Pipelines simulating various user loads (waiting until all the builds in the Multibranch project complete, then running 5x System.gc() and waiting a few minutes).

I have already done a fair bit of tuning to reduce memory use in that environment (primarily manipulating Java system settings for GC and threads, etc).

But yeah, oddly enough the before-and-after (technically, after and then before in the time sequence of the graph) shows something like a 25% reduction in memory footprint (potentially there may be other factors at play there).

@svanoort svanoort requested review from daniel-beck and oleg-nenashev Sep 28, 2018

@oleg-nenashev

I see no issues with the specific String interning here. Even if something goes wrong, we will just have a full list of plugin IDs and versions on memory. Should be ok

@daniel-beck

This comment has been minimized.

Show comment
Hide comment
@daniel-beck

daniel-beck Oct 5, 2018

Member

Can't help but wonder whether this is worth the additional code. @jenkinsci/code-reviewers perhaps?

Member

daniel-beck commented Oct 5, 2018

Can't help but wonder whether this is worth the additional code. @jenkinsci/code-reviewers perhaps?

@svanoort

This comment has been minimized.

Show comment
Hide comment
@svanoort

svanoort Oct 5, 2018

Member

@daniel-beck I mean, I did provide a benchmark and the code changes are short and simple. While the before-and-after isn't a perfect apples-to-apples comparison because there are potentially other core changes impacting it (though it's unlikely) it does suggest a fairly large overall impact: 125 vs. ~190 MB of heap actively used (look at the blue line), and ~215 vs ~260 MB of heap committed (memory claimed from the OS).

With less-tuned GC settings the committed heap will be much larger relative to the used heap.

I expect the overall impact to become much more important over time as we move towards less mega-master Jenkins installs and offload more of the work of running Pipelines from the Jenkins master itself.

In a containerized world with high-density systems, every MB of memory claimed is another container we can't run without adding another server to the cluster.

Member

svanoort commented Oct 5, 2018

@daniel-beck I mean, I did provide a benchmark and the code changes are short and simple. While the before-and-after isn't a perfect apples-to-apples comparison because there are potentially other core changes impacting it (though it's unlikely) it does suggest a fairly large overall impact: 125 vs. ~190 MB of heap actively used (look at the blue line), and ~215 vs ~260 MB of heap committed (memory claimed from the OS).

With less-tuned GC settings the committed heap will be much larger relative to the used heap.

I expect the overall impact to become much more important over time as we move towards less mega-master Jenkins installs and offload more of the work of running Pipelines from the Jenkins master itself.

In a containerized world with high-density systems, every MB of memory claimed is another container we can't run without adding another server to the cluster.

@oleg-nenashev oleg-nenashev merged commit e0ed9a8 into jenkinsci:master Oct 6, 2018

1 check passed

continuous-integration/jenkins/pr-merge This commit looks good
Details

@svanoort svanoort deleted the svanoort:memory-micro-optimizations branch Oct 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment