Repository navigation
[JENKINS-65308] Jenkins.trimLabels gets increasingly slower as number of nodes and labels increase #11987
Description
We've been trying to track down some issues we've been seeing around Queue lock
contention on one of our Jenkins clusters. The lock contention manifests in
both UI instability/slowness and failures with REST API calls to add, update or
remove nodes. We use the Swarm plugin on the primary and swarm-client (version
3.24) on the agents to connect to the primary. The REST API failures aren't due
to exceptions from Jenkins, but to the API calls exceeding the configured
proxy_read_timeout (180s) for the nginx instance we have in front of Jenkins.
That manifests in the swarm-client process on the agents receiving a 504 from
nginx since Jenkins didn't respond in time.
Thread dumps gathered during periods of instability show that hundreds of
threads are waiting for the Queue lock to be able to add, update or remove
a node.
"Handling POST /plugin/swarm/createSlave from 10.224.1.234 : Jetty (winstone)-1218487" #1218487 prio=5 os_prio=0 tid=0x00007f4732c7f800 nid=0x2f96 waiting on condition [0x00007f3c6b5f2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f3f117ca288> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at hudson.model.Queue._withLock(Queue.java:1441)
[snip..]
The vast majority of the time, the thread holding the Queue lock during each
thread dump is performing operations within the Jenkins.trimLabels method as
part of adding, updating or removing a node.
"Handling POST /plugin/swarm/createSlave from 10.240.81.215 : Jetty (winstone)-1218362" #1218362 prio=5 os_prio=0 tid=0x00007f47265fd000 nid=0x2e11 runnable [0x00007f3c85ceb000]
java.lang.Thread.State: RUNNABLE
at hudson.util.QuotedStringTokenizer.hasMoreTokens(QuotedStringTokenizer.java:184)
at hudson.model.Label.parse(Label.java:585)
at hudson.model.Node.getAssignedLabels(Node.java:303)
at hudson.model.Label.matches(Label.java:196)
at hudson.model.Label.getNodes(Label.java:233)
at hudson.model.Label.isEmpty(Label.java:430)
at jenkins.model.Jenkins.trimLabels(Jenkins.java:2201)
at jenkins.model.Nodes$4.call(Nodes.java:214)
at jenkins.model.Nodes$4.call(Nodes.java:210)
at hudson.model.Queue._withLock(Queue.java:1443)
at hudson.model.Queue.withLock(Queue.java:1304)
at jenkins.model.Nodes.updateNode(Nodes.java:210)
at jenkins.model.Jenkins.updateNode(Jenkins.java:2176)
at hudson.model.Node.save(Node.java:139)
at hudson.model.Node.setTemporaryOfflineCause(Node.java:274)
at hudson.model.Computer.setNode(Computer.java:820)
at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:895)
at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:137)
at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:43)
at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:223)
at hudson.model.Queue._withLock(Queue.java:1384)
at hudson.model.Queue.withLock(Queue.java:1261)
at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:206)
at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1632)
at jenkins.model.Nodes$2.run(Nodes.java:151)
at hudson.model.Queue._withLock(Queue.java:1384)
at hudson.model.Queue.withLock(Queue.java:1261)
at jenkins.model.Nodes.addNode(Nodes.java:147)
at jenkins.model.Jenkins.addNode(Jenkins.java:2155)
at hudson.plugins.swarm.PluginImpl.doCreateSlave(PluginImpl.java:224)
I've attached a couple of archives created using the collectPerformanceData
script that contain the relevant thread dumps.
During the aforementioned periods of instability there are between 1500-1600
unique labels and 400-500 workers, as gathered from the script console using
Jenkins.instance.labels.size() and Jenkins.instance.nodes.size().
I'm able to replicate the increasing slowness using Groovy scripts that mirror
what our worker creation steps look like. I've attached both scripts.
create-workers.groovy creates the workers, remove-workers.groovy removes
them. To make it match our swarm-client workflow we create SwarmSlave agents
in the script but that detail probably doesn't matter for reproduction
purposes.
Creating and then removing workers with a single label is fast, as you'd
expect. Here's some snipped output for creation (full output attached as create-workers-single-label.log):
...
uniqueLabels: 395 nodes: 393 swarm-test-392: 63ms
uniqueLabels: 396 nodes: 394 swarm-test-393: 65ms
uniqueLabels: 397 nodes: 395 swarm-test-394: 87ms
uniqueLabels: 398 nodes: 396 swarm-test-395: 62ms
uniqueLabels: 399 nodes: 397 swarm-test-396: 62ms
uniqueLabels: 400 nodes: 398 swarm-test-397: 62ms
uniqueLabels: 401 nodes: 399 swarm-test-398: 63ms
uniqueLabels: 402 nodes: 400 swarm-test-399: 63ms
uniqueLabels: 403 nodes: 401 swarm-test-400: 64ms
Total time to create 400 workers: 9183ms
And then the same for removal (full output attached as remove-workers-single-label.log):
...
uniqueLabels: 10 nodes: 8 swarm-test-91: 0ms
uniqueLabels: 9 nodes: 7 swarm-test-92: 0ms
uniqueLabels: 8 nodes: 6 swarm-test-93: 1ms
uniqueLabels: 7 nodes: 5 swarm-test-94: 0ms
uniqueLabels: 6 nodes: 4 swarm-test-95: 1ms
uniqueLabels: 5 nodes: 3 swarm-test-96: 0ms
uniqueLabels: 4 nodes: 2 swarm-test-97: 1ms
uniqueLabels: 3 nodes: 1 swarm-test-98: 0ms
uniqueLabels: 1 nodes: 0 swarm-test-99: 1ms
Total time to remove 401 workers: 8675ms
But once you start adding more labels, thing start slowing down drastically.
Here's some snipped output for creation (full output attached as create-workers-multiple-labels.log):
...
uniqueLabels: 809 nodes: 393 swarm-test-392: 1875ms
uniqueLabels: 811 nodes: 394 swarm-test-393: 1875ms
uniqueLabels: 813 nodes: 395 swarm-test-394: 1883ms
uniqueLabels: 815 nodes: 396 swarm-test-395: 1888ms
uniqueLabels: 817 nodes: 397 swarm-test-396: 1901ms
uniqueLabels: 819 nodes: 398 swarm-test-397: 1913ms
uniqueLabels: 821 nodes: 399 swarm-test-398: 1915ms
uniqueLabels: 823 nodes: 400 swarm-test-399: 1927ms
uniqueLabels: 825 nodes: 401 swarm-test-400: 1939ms
Total time to create 400 workers: 261866ms
And then the same for removal (full output attached as remove-workers-multiple-labels.log):
...
uniqueLabels: 39 nodes: 8 swarm-test-91: 3ms
uniqueLabels: 37 nodes: 7 swarm-test-92: 2ms
uniqueLabels: 35 nodes: 6 swarm-test-93: 2ms
uniqueLabels: 33 nodes: 5 swarm-test-94: 1ms
uniqueLabels: 31 nodes: 4 swarm-test-95: 1ms
uniqueLabels: 29 nodes: 3 swarm-test-96: 0ms
uniqueLabels: 27 nodes: 2 swarm-test-97: 0ms
uniqueLabels: 25 nodes: 1 swarm-test-98: 1ms
uniqueLabels: 1 nodes: 0 swarm-test-99: 0ms
Total time to remove 401 workers: 258555ms
Increasing (roughly doubling it in this case) the number of unique labels makes
the same process that originally took about 9s for each operation take about
4 minutes and 20 seconds for each operation.
Is there some way to make Jenkins.trimLabels less expensive even in the
face of thousands of labels and hundreds of workers? To my eye it looks like
the current code path has several nested loops (outer loop over every label,
inner loop over every worker, inner loop over every parsed token from the label
tokenizer, inner loop over every char in the raw label str) which are what
contribute to the increase in execution time as the inputs get larger.
Originally reported by 
jonahbull, imported from: Jenkins.trimLabels gets increasingly slower as number of nodes and labels increase
- assignee: raihaan
- status: Closed
- priority: Major
- component(s): core
- label(s): 2.277.4-rejected
- resolution: Fixed
- resolved: 2021-04-19T09:23:26+00:00
- votes: 3
- watchers: 9
- imported: 2025-11-24
Raw content of original issue
We've been trying to track down some issues we've been seeing around Queue lock contention on one of our Jenkins clusters. The lock contention manifests in both UI instability/slowness and failures with REST API calls to add, update or remove nodes. We use the Swarm plugin on the primary and swarm-client (version 3.24) on the agents to connect to the primary. The REST API failures aren't due to exceptions from Jenkins, but to the API calls exceeding the configured proxy_read_timeout (180s) for the nginx instance we have in front of Jenkins. That manifests in the swarm-client process on the agents receiving a 504 from nginx since Jenkins didn't respond in time.
Thread dumps gathered during periods of instability show that hundreds of threads are waiting for the Queue lock to be able to add, update or remove a node.
"Handling POST /plugin/swarm/createSlave from 10.224.1.234 : Jetty (winstone)-1218487" #1218487 prio=5 os_prio=0 tid=0x00007f4732c7f800 nid=0x2f96 waiting on condition [0x00007f3c6b5f2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00007f3f117ca288> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1441) [snip..]The vast majority of the time, the thread holding the Queue lock during each thread dump is performing operations within the Jenkins.trimLabels method as part of adding, updating or removing a node.
"Handling POST /plugin/swarm/createSlave from 10.240.81.215 : Jetty (winstone)-1218362" #1218362 prio=5 os_prio=0 tid=0x00007f47265fd000 nid=0x2e11 runnable [0x00007f3c85ceb000] java.lang.Thread.State: RUNNABLE at hudson.util.QuotedStringTokenizer.hasMoreTokens(QuotedStringTokenizer.java:184) at hudson.model.Label.parse(Label.java:585) at hudson.model.Node.getAssignedLabels(Node.java:303) at hudson.model.Label.matches(Label.java:196) at hudson.model.Label.getNodes(Label.java:233) at hudson.model.Label.isEmpty(Label.java:430) at jenkins.model.Jenkins.trimLabels(Jenkins.java:2201) at jenkins.model.Nodes$4.call(Nodes.java:214) at jenkins.model.Nodes$4.call(Nodes.java:210) at hudson.model.Queue._withLock(Queue.java:1443) at hudson.model.Queue.withLock(Queue.java:1304) at jenkins.model.Nodes.updateNode(Nodes.java:210) at jenkins.model.Jenkins.updateNode(Jenkins.java:2176) at hudson.model.Node.save(Node.java:139) at hudson.model.Node.setTemporaryOfflineCause(Node.java:274) at hudson.model.Computer.setNode(Computer.java:820) at hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:895) at hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:137) at hudson.model.AbstractCIBase.access$000(AbstractCIBase.java:43) at hudson.model.AbstractCIBase$2.run(AbstractCIBase.java:223) at hudson.model.Queue._withLock(Queue.java:1384) at hudson.model.Queue.withLock(Queue.java:1261) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:206) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1632) at jenkins.model.Nodes$2.run(Nodes.java:151) at hudson.model.Queue._withLock(Queue.java:1384) at hudson.model.Queue.withLock(Queue.java:1261) at jenkins.model.Nodes.addNode(Nodes.java:147) at jenkins.model.Jenkins.addNode(Jenkins.java:2155) at hudson.plugins.swarm.PluginImpl.doCreateSlave(PluginImpl.java:224)I've attached a couple of archives created using the collectPerformanceData script that contain the relevant thread dumps.
During the aforementioned periods of instability there are between 1500-1600 unique labels and 400-500 workers, as gathered from the script console using Jenkins.instance.labels.size() and Jenkins.instance.nodes.size().
I'm able to replicate the increasing slowness using Groovy scripts that mirror what our worker creation steps look like. I've attached both scripts. create-workers.groovy creates the workers, remove-workers.groovy removes them. To make it match our swarm-client workflow we create SwarmSlave agents in the script but that detail probably doesn't matter for reproduction purposes.
Creating and then removing workers with a single label is fast, as you'd expect. Here's some snipped output for creation (full output attached as create-workers-single-label.log):
... uniqueLabels: 395 nodes: 393 swarm-test-392: 63ms uniqueLabels: 396 nodes: 394 swarm-test-393: 65ms uniqueLabels: 397 nodes: 395 swarm-test-394: 87ms uniqueLabels: 398 nodes: 396 swarm-test-395: 62ms uniqueLabels: 399 nodes: 397 swarm-test-396: 62ms uniqueLabels: 400 nodes: 398 swarm-test-397: 62ms uniqueLabels: 401 nodes: 399 swarm-test-398: 63ms uniqueLabels: 402 nodes: 400 swarm-test-399: 63ms uniqueLabels: 403 nodes: 401 swarm-test-400: 64ms Total time to create 400 workers: 9183msAnd then the same for removal (full output attached as remove-workers-single-label.log):
... uniqueLabels: 10 nodes: 8 swarm-test-91: 0ms uniqueLabels: 9 nodes: 7 swarm-test-92: 0ms uniqueLabels: 8 nodes: 6 swarm-test-93: 1ms uniqueLabels: 7 nodes: 5 swarm-test-94: 0ms uniqueLabels: 6 nodes: 4 swarm-test-95: 1ms uniqueLabels: 5 nodes: 3 swarm-test-96: 0ms uniqueLabels: 4 nodes: 2 swarm-test-97: 1ms uniqueLabels: 3 nodes: 1 swarm-test-98: 0ms uniqueLabels: 1 nodes: 0 swarm-test-99: 1ms Total time to remove 401 workers: 8675msBut once you start adding more labels, thing start slowing down drastically. Here's some snipped output for creation (full output attached as create-workers-multiple-labels.log):
... uniqueLabels: 809 nodes: 393 swarm-test-392: 1875ms uniqueLabels: 811 nodes: 394 swarm-test-393: 1875ms uniqueLabels: 813 nodes: 395 swarm-test-394: 1883ms uniqueLabels: 815 nodes: 396 swarm-test-395: 1888ms uniqueLabels: 817 nodes: 397 swarm-test-396: 1901ms uniqueLabels: 819 nodes: 398 swarm-test-397: 1913ms uniqueLabels: 821 nodes: 399 swarm-test-398: 1915ms uniqueLabels: 823 nodes: 400 swarm-test-399: 1927ms uniqueLabels: 825 nodes: 401 swarm-test-400: 1939ms Total time to create 400 workers: 261866msAnd then the same for removal (full output attached as remove-workers-multiple-labels.log):
... uniqueLabels: 39 nodes: 8 swarm-test-91: 3ms uniqueLabels: 37 nodes: 7 swarm-test-92: 2ms uniqueLabels: 35 nodes: 6 swarm-test-93: 2ms uniqueLabels: 33 nodes: 5 swarm-test-94: 1ms uniqueLabels: 31 nodes: 4 swarm-test-95: 1ms uniqueLabels: 29 nodes: 3 swarm-test-96: 0ms uniqueLabels: 27 nodes: 2 swarm-test-97: 0ms uniqueLabels: 25 nodes: 1 swarm-test-98: 1ms uniqueLabels: 1 nodes: 0 swarm-test-99: 0ms Total time to remove 401 workers: 258555msIncreasing (roughly doubling it in this case) the number of unique labels makes the same process that originally took about 9s for each operation take about 4 minutes and 20 seconds for each operation.
Is there some way to make Jenkins.trimLabels less expensive even in the face of thousands of labels and hundreds of workers? To my eye it looks like the current code path has several nested loops (outer loop over every label, inner loop over every worker, inner loop over every parsed token from the label tokenizer, inner loop over every char in the raw label str) which are what contribute to the increase in execution time as the inputs get larger.
environment
agents jdk information:<br/>
swarm-client-3.24.jar<br/>
openjdk version "1.8.0_282"<br/>
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~16.04-b08)<br/>
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)<br/>
primary node jdk information:<br/>
openjdk version "1.8.0_282"<br/>
OpenJDK Runtime Environment (build 1.8.0_282-8u282-b08-0ubuntu1~18.04-b08)<br/>
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)<br/>
Jenkins: 2.276<br/>
OS: Linux - 5.4.0-1036-gcp<br/>
---<br/>
ace-editor:1.1<br/>
pipeline-rest-api:2.19<br/>
pipeline-input-step:2.12<br/>
dtkit-api:3.0.0<br/>
build-name-setter:2.1.0<br/>
blueocean-pipeline-editor:1.24.4<br/>
extended-read-permission:3.2<br/>
swarm:3.24<br/>
pipeline-utility-steps:2.6.1<br/>
ldap:2.3<br/>
jaxb:2.3.0<br/>
blueocean-display-url:2.4.0<br/>
blueocean-pipeline-api-impl:1.24.4<br/>
handlebars:1.1.1<br/>
trilead-api:1.0.13<br/>
bootstrap4-api:4.6.0-1<br/>
conditional-buildstep:1.4.1<br/>
scm-api:2.6.4<br/>
powershell:1.4<br/>
pipeline-stage-view:2.19<br/>
okhttp-api:3.14.9<br/>
command-launcher:1.5<br/>
aws-java-sdk:1.11.930<br/>
blueocean-bitbucket-pipeline:1.24.4<br/>
s3:0.11.6<br/>
blueocean-rest:1.24.4<br/>
credentials-binding:1.24<br/>
pipeline-graph-analysis:1.10<br/>
lockable-resources:2.10<br/>
postbuildscript:2.11.0<br/>
github-branch-source:2.10.0<br/>
build-user-vars-plugin:1.6<br/>
cloudbees-disk-usage-simple:0.10<br/>
blueocean-git-pipeline:1.24.4<br/>
node-iterator-api:1.5.0<br/>
junit-attachments:1.6<br/>
momentjs:1.1.1<br/>
plain-credentials:1.7<br/>
workflow-support:3.8<br/>
workflow-aggregator:2.6<br/>
credentials:2.3.14<br/>
workflow-cps-global-lib:2.17<br/>
email-ext:2.81<br/>
authorize-project:1.3.0<br/>
docker-commons:1.17<br/>
gradle:1.36<br/>
cobertura:1.16<br/>
blueocean-commons:1.24.4<br/>
antisamy-markup-formatter:2.1<br/>
favorite:2.3.3<br/>
embeddable-build-status:2.0.3<br/>
jjwt-api:0.11.2-7.a257d5ff5a6b<br/>
jsch:0.1.55.2<br/>
mailer:1.32.1<br/>
pipeline-stage-tags-metadata:1.8.1<br/>
external-monitor-job:1.7<br/>
ssh-slaves:1.31.5<br/>
built-on-column:1.1<br/>
jenkins-multijob-plugin:1.36<br/>
popper-api:1.16.1-1<br/>
token-macro:2.15<br/>
blueocean-github-pipeline:1.24.4<br/>
ws-cleanup:0.38<br/>
workflow-basic-steps:2.23<br/>
rebuild:1.31<br/>
pipeline-model-definition:1.8.1<br/>
junit:1.48<br/>
ghprb:1.42.1<br/>
job-dsl:1.77<br/>
jenkins-design-language:1.24.4<br/>
pipeline-milestone-step:1.3.2<br/>
blueocean-rest-impl:1.24.4<br/>
github-api:1.122<br/>
workflow-multibranch:2.22<br/>
pipeline-model-api:1.8.1<br/>
allure-jenkins-plugin:2.29.0<br/>
docker-workflow:1.25<br/>
display-url-api:2.3.4<br/>
git-server:1.9<br/>
durable-task:1.35<br/>
pipeline-build-step:2.13<br/>
pam-auth:1.6<br/>
vsphere-cloud:2.25<br/>
google-storage-plugin:1.5.3<br/>
slack:2.45<br/>
google-oauth-plugin:1.0.3<br/>
authentication-tokens:1.4<br/>
snakeyaml-api:1.27.0<br/>
resource-disposer:0.14<br/>
workflow-job:2.40<br/>
apache-httpcomponents-client-4-api:4.5.13-1.0<br/>
jdk-tool:1.4<br/>
ssh-credentials:1.18.1<br/>
blueocean-core-js:1.24.4<br/>
blueocean-config:1.24.4<br/>
github-scm-trait-notification-context:1.1<br/>
workflow-scm-step:2.11<br/>
yaml-axis:0.3.0<br/>
build-timeout:1.20<br/>
basic-branch-build-strategies:1.3.2<br/>
variant:1.4<br/>
pipeline-githubnotify-step:1.0.5<br/>
envinject-api:1.7<br/>
matrix-combinations-parameter:1.3.1<br/>
plugin-util-api:1.6.1<br/>
checks-api:1.3.0<br/>
htmlpublisher:1.25<br/>
generic-webhook-trigger:1.72<br/>
blueocean-pipeline-scm-api:1.24.4<br/>
cloudbees-folder:6.15<br/>
blueocean-autofavorite:1.2.4<br/>
blueocean-i18n:1.24.4<br/>
echarts-api:4.9.0-3<br/>
structs:1.22<br/>
metrics:4.0.2.7<br/>
cloudbees-bitbucket-branch-source:2.9.7<br/>
blueocean-web:1.24.4<br/>
github:1.32.0<br/>
windows-slaves:1.7<br/>
maven-plugin:3.9<br/>
jackson2-api:2.12.1<br/>
git:4.5.2<br/>
envinject:2.4.0<br/>
handy-uri-templates-2-api:2.1.8-1.0<br/>
matrix-project:1.18<br/>
ssh-agent:1.20<br/>
timestamper:1.11.8<br/>
disable-github-multibranch-status:1.2<br/>
bouncycastle-api:2.18<br/>
blueocean:1.24.4<br/>
run-condition:1.5<br/>
blueocean-events:1.24.4<br/>
jquery3-api:3.5.1-2<br/>
pipeline-stage-step:2.5<br/>
font-awesome-api:5.15.2-1<br/>
blueocean-dashboard:1.24.4<br/>
git-client:3.6.0<br/>
leastload:3.0.0<br/>
javadoc:1.6<br/>
oauth-credentials:0.4<br/>
workflow-api:2.41<br/>
sse-gateway:1.24<br/>
blueocean-jwt:1.24.4<br/>
script-security:1.76<br/>
github-oauth:0.33<br/>
branch-api:2.6.3<br/>
custom-build-properties:1.9.1<br/>
ant:1.11<br/>
workflow-durable-task-step:2.37<br/>
workflow-step-api:2.23<br/>
hashicorp-vault-plugin:3.7.0<br/>
ansicolor:0.5.3<br/>
pipeline-github:2.7<br/>
parameterized-trigger:2.40<br/>
code-coverage-api:1.3.1<br/>
copyartifact:1.46<br/>
jquery:1.12.4-1<br/>
mask-passwords:3.0<br/>
blueocean-personalization:1.24.4<br/>
prometheus:2.0.9<br/>
google-metadata-plugin:0.3.1<br/>
pipeline-model-extensions:1.8.1<br/>
xunit:3.0.0<br/>
workflow-cps:2.87<br/>
pubsub-light:1.13<br/>
matrix-auth:2.6.5
Activity
jenkins-infra-bot commentedon Apr 7, 2021
Remote links associated with this issue:
jenkins-infra-bot commentedon Apr 12, 2021
Raw content of original comment:
Really appreciate the quick response to this issue! After the linked PR was
merged I built jenkins.war from master locally and built a jdk11 Docker
image based on the that WAR file using the Makefile from the Jenkins docker
repo. Unfortunately in my tests the behavior I described looks worse and not
better after these changes.
Here's some snipped output for creation using latest master:
And then the same for removal:
In contrast, here's the snipped output for creation using the 2.276 docker
image:
And then the same for removal:
This is all using the groovy scripts attached to this issue. I'll dig some more
tomorrow and see if I can provide some further data.
jenkins-infra-bot commentedon Apr 12, 2021
raihaan:
Raw content of original comment:
Hey Jonah thanks for the feedback, I'll probably write another PR which you can hopefully test.
In my limited testing, this improved things but it doesn't seem to have helped you.
jenkins-infra-bot commentedon Apr 12, 2021
raihaan:
Raw content of original comment:
Hey jonahbull if you get the chance could you try out the newer PR.
jenkins-infra-bot commentedon Apr 13, 2021
Raw content of original comment:
Hey Raihaan, thanks again for the quick work on these PRs! I tested the newer PR this morning and got not-very-improved results, which didn't make any sense to me. I slowly realized that my timing/reporting code in the worker creation loop was probably what was dominating the time now since it has to get the number of labels and nodes to print each iteration. I commented the timing code in the worker creation loop, leaving just the measurement of the total time to create/remove all the workers. That seems to have done the trick as far as reporting accurate results. So the first PR was actually a massive improvement from our baseline on 2.276 and the second PR is even better still!
2.276:
PR-5402 (first PR):
PR-5412 (second PR):
Personally I find the implementation of the second PR clearer so I'd love to see that one land as well. I'm not sure if there are any concerns over whether the second PR changes the contract of the trimLabels method significantly, hopefully folks who know that area of the code better will chime in on that aspect.
jenkins-infra-bot commentedon Apr 13, 2021
raihaan:
Raw content of original comment:
Hey Jonah, thanks a ton for testing. I'll be getting the newer PR merged as well but since this improves your situation. You can use the official 2.288 that contains the first implementation in the meantime and hopefully next week the second implementation gets merged.
jenkins-infra-bot commentedon Apr 15, 2021
Raw content of original comment:
raihaan Could we consider this improvement for back porting to LTS release aswell?
jenkins-infra-bot commentedon Apr 15, 2021
raihaan:
Raw content of original comment:
I can label it as such but, there is no guarantees it will make it to LTS
jenkins-infra-bot commentedon Apr 15, 2021
Raw content of original comment:
Cool... Well atleast it will be considered now...
jenkins-infra-bot commentedon Apr 19, 2021
raihaan:
Raw content of original comment:
Marking this resolved as the first fix technically rectifies the issue. And the second fix will be going in for 2.289
jenkins-infra-bot commentedon Apr 21, 2021
Raw content of original comment:
It is unclear what should be backported into 2.277.x as there are two related PRs, and one isn't even out a full day yet.
jenkins-infra-bot commentedon Apr 23, 2021
Raw content of original comment:
I just thought I'd drop by and say thank you from myself and Jonah for the awesome work on getting this issue resolved soo quickly.
We deployed the latest `2.289` version to production yesterday, and have seen dramatic improvements on the performance of creating new agents. The 99th percentile for the `POST /plugin/swarm/createSlave` call is now a mere 2.2 seconds!!!
jenkins-infra-bot commentedon Apr 23, 2021
raihaan:
Raw content of original comment:
Hey Gavin, thanks to both you and Jonah that this was able to be resolved so quickly. The initial investigation done was superb which allowed me to focus on the issue. Its collaboration like this that helps issues get resolved quickly.
jenkins-infra-bot commentedon Apr 26, 2021
Raw content of original comment:
Hi bmunoz
I just noticed that this issue has been labelled as `2.277.4-rejected`.
Does that mean it's not going to be part of the next LTS release?
Could you elaborate on the reasoning behind that? From our testing and production usage thus far, this change has a massive benefit on overall Jenkins performance, especially when runnings hundreds of connected agents that are being dynamically created...
Thanks
jenkins-infra-bot commentedon Apr 26, 2021
raihaan:
Raw content of original comment:
Hey Gavin, it was rejected because the change has not been in weekly releases for long and it is not a regression. Thus it was decided that it be left out because this change poses a risk of a regression. So yes it will not be in 2.277.4, it will likely be in the LTS after that whose baseline is likely to be 2.289.
jenkins-infra-bot commentedon Apr 27, 2021
Raw content of original comment:
Hey Gavin. What Raihaan said is correct. Here the discussion
jenkins-infra-bot commentedon Apr 28, 2021
Raw content of original comment:
Thanks for the detail both