Add a performance pipeline to build.go.cd

[zabil]

Set up continuous performance runs on build.go.cd before releasing it as experimental.

[zabil]

Here's our current problem.

• We run our performance tests too close to a release. This includes setting up our environment, Go servers and agents.
• The setup process is not neatly scripted (unlike our release process), software dependencies are managed manually.
• Tests runs require manual triggering and analysis.
• Our test suite which runs on "ab" does not generate easy to read report making it hard to debug issues.
• We've not set benchmarks with respect response times, cpu or memory usage. Our decision is based on if we've regressed on performance w.r.t to a previous release.
• We only have stress tests.
• We only cover get requests on a few pages.
• We do not capture periodic jvm or machine stats during the test.
• We do not have anything in place to monitor our servers or agents while the tests run.
• We do not have or turn on a lightweight profiling for diagnosis.
• Our scripts mix performance environment setup and test scenarios making it difficult to identify where to make changes.

Here's initial thought on what we'll be doing.

• Make spinning up our performance instances easier.
• Identify benchmarks for response times, memory and cpu usage.
• Track performance runs to commits by integrating it with our release pipeline.
• Better reports. Also make it easy to compare reports from multiple runs.
• Additional test suites (load, stress and soak)
• Tests in a language that's Turing complete.
• Integrate some level of profiling and monitoring.

[zabil]

Oh and our performance tests are not open source.

[rajiesh]

As initial steps towards making performance pipeline working, we need to understand the system and usage patterns of GO users. Here is the snapshot of the build.go.cd environment
Would like to get similar information from GO users about their environment so that it can be used to model the performance test scenarios
Please add if you think more info can help

=======================
System Info
=======================

8 Cores
model name    : Intel Core i7 9xx (Nehalem Class Core i7)
cpu MHz        : 2260.998
cache size    : 4096 KB

=================
Config Statistics
=================

Number of pipelines: 17
Number of agents: 42
Number of environments: 3
Number of unique materials: 20
Number of schedulable materials: 20


=================
JVM ARGS
=================

Instaled Java: java-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_64

  -Xms1024m
  -Xmx8g
  -XX:PermSize=128m
  -XX:MaxPermSize=400m


==========================
Usage Statistics
==========================

Top Hit Pages:(measured over a period of 1 month from access_log file)
----------------------------------------------------------------------

Page                                                  Hits
----------------------------------------------------------
Pipeline Dashboard                                    625
Stage History                                         400
Job Status                                            125
Config XML                                            68
Environments                                          61
Pipeline Edit                                         50

Users : 90% of users of build.go.cd are admins, but it can vary depending on organizations

[henriquegemignani]

I'd love to contribute. I've checked the output of /api/support, but it's not in the same format you posted.
Oh, do I post the information here or is there a better place?

[zabil]

@henriquegemignani it need not be in the format mentioned above.
We've scraped this together from api/support apache logs and system info.

These are what we are looking for but a subset is fine too.

[greenmoss]

I hear the call!

Here's a reduced set of output from /api/support. The full output is at https://gist.github.com/greenmoss/a7f08600af10326b5117cc91ff9c4957

=================
Config Statistics
=================

Valid Config.
Number of pipelines: 492
Number of agents: 56
Number of environments: 13
Number of unique materials: 900
Number of schedulable materials: 897
Security:


=================
Go Server Version
=================

16.2.0(3024-0946fc7a9a54555f437fda398fafde0fbc7985bd)




==============
OS information
==============

Linux, amd64, 3.13.0-83-generic, 8, 4.86


===================
Runtime information
===================

Name - 1535@gocd-server
Uptime - 194439641 [About 54 hours, 0 minutes, 39 seconds]
Spec Name - Java Virtual Machine Specification
Spec Vendor - Oracle Corporation
Spec Version - 1.7

Input Arguments
---------------
  -> -Xms240000m
  -> -Xmx240000m
  -> -XX:PermSize=128m
  -> -XX:MaxPermSize=256m
  -> -Dgo.plugin.build.status.go-server=http://localhost:8080
  -> -Dgo.config.repo.gc.periodic=Y
  -> -Duser.language=en
  -> -Djruby.rack.request.size.threshold.bytes=30000000
  -> -Duser.country=US
  -> -Dcruise.config.dir=/etc/go
  -> -Dcruise.config.file=/etc/go/cruise-config.xml
  -> -Dcruise.server.port=8080
  -> -Dcruise.server.ssl.port=8081
  -> -Djava.security.egd=file:/dev/./urandom

25-30% of our users are admins. If we could restrict environments to groups, this number would go down a lot.

[rajiesh]

This is the load test result on 16.12.0-4305 version of GoCD - https://build.go.cd/go/tab/build/detail/performance-benchmark/115/benchmark/1/performance_monitor

Load test configuration:

Number of pipelines : 750
Number of agents: 75
Number of unique materials: 750
Git Materials: 625
TFS Materials: 75

Hardware Configuration:
Openstack VM
CPI:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                3

Memory: 8g
Swap not enabled

Server Configuration:
Heap min: 4g
Heap max: 6g


SCM Checkins: 1 checkin into each unique material every 10secs 
Config Updates: 1 config update every 30 secs
Load duration: 20 mins
Hits per second on server: 4 hits/sec

Same setup can be used to run a soak test. To run a soak test for 12 hours please update the parameters here to values given below,

NO_OF_AGENTS: 35
NO_OF_PIPELINES: 350
GIT_COMMIT_INTERVAL: 10
NUMBER_OF_COMMITS: 4320
CONFIG_SAVE_INTERVAL: 30
NUMBER_OF_CONFIG_SAVES: 1440
NO_OF_THREAD_GROUPS: 1
LOAD_TEST_DURATION:43200
SERVER_MAX_MEM: 4g
SERVER_MEM: 2g

[lenucksi]

That sounds like a good thing and good progress on it.
Might I ask which complexity and interaction between the pipelines the the performance test data has?

[rajiesh]

@lenucksi the pipeline dependency is something like shown in this image. Every 10 pipelines in the test data has upstreams and downstreams as shown here.

[arvindsv]

This has been done. Is continuously being improved, mostly by @rajiesh, but I don't think this needs an issue open at this point.