Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the size of the `agent-plugins.zip` (#3583) (#121) #3584

Merged
merged 2 commits into from Jun 19, 2017

Conversation

Projects
None yet
3 participants
@ketan
Copy link
Member

commented Jun 13, 2017

No description provided.

@zabil zabil added the in progress label Jun 13, 2017

@ketan ketan changed the title Reduce the size of the `agent-plugins.zip`. (#3583) Reduce the size of the `agent-plugins.zip`. (#3583) (#121) Jun 13, 2017

@ketan ketan changed the title Reduce the size of the `agent-plugins.zip`. (#3583) (#121) Reduce the size of the `agent-plugins.zip` (#3583) (#121) Jun 13, 2017

@ketan ketan force-pushed the ketan:improve/plugin-listener branch from 015e36f to 30615dc Jun 14, 2017

Simplify some of the logic around creating plugin zip for agent
* Remove `PluginsZipUpdater` and move that functionality to `PluginsZip`
* Remove redundant `PluginsFolderChangeListener`
* The `PluginsZip` will now listen to plugin changes and will create a
  plugin zip consisting of plugins of type scm, task and
  package-material. To ensure consistency, the plugins are ordered by
  whether they are bundled, followed by whether their plugin id.
* Simplify how the plugin zip checksum is computed, it's now the
  checksum of all the relevant plugin jars concatenated with each other.
@ketan

This comment has been minimized.

Copy link
Member Author

commented Jun 14, 2017

Verified. Dropped a bunch of plugins and the only ones that come out are task, scm and package material plugins.

$ ls external
docker-elastic-agents-0.6.1.jar     github-pr-poller-1.3.0-RC2.jar
docker-task-assembly-0.1.27.jar     go-maven-poller.jar
ecs-elastic-agent-plugin-0.0.3.jar  gocd-ldap-authorization-plugin-0.0.1.jar
gerrit-cs-status-1.2.jar            gocd-nuget-poller-plugin-v1.0.3.jar

$ http://localhost:8153/go/admin/agent-plugins.zip ; unzip -l agent-plugins.zip; rm agent-plugins.zip
--2017-06-14 15:50:14--  http://localhost:8153/go/admin/agent-plugins.zip
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8153... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘agent-plugins.zip’

agent-plugins.zip                                [ <=>                                                                                         ]   7.02M  --.-KB/s    in 0.01s

2017-06-14 15:50:14 (571 MB/s) - ‘agent-plugins.zip’ saved [7357180]

Archive:  agent-plugins.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
   621954  06-14-17 15:49   external/docker-task-assembly-0.1.27.jar
  6016714  06-14-17 15:49   external/github-pr-poller-1.3.0-RC2.jar
   787806  06-14-17 15:49   external/gocd-nuget-poller-plugin-v1.0.3.jar
 --------                   -------
  7426474                   3 files
@varshavaradarajan

This comment has been minimized.

Copy link
Contributor

commented Jun 19, 2017

I can't verify the part where a new plugin is added and agent exits and downloads the plugins zip all over again. These are the steps I followed -

  1. ./gradlew clean serverGenericZip agentGenericZip
  2. mkdir -p plugins/external in server working directory and put in some plugins
  3. start the server and agent
  4. This only downloads the task and material plugins to the agent.
  5. Add a new task plugin in <server_work_dir>/plugins/external

Nothing changes on the agent side after adding the new task plugin - on master or on this PR.
@ketan said he shall reproduce this. Otherwise, LGTM.

@ketan

This comment has been minimized.

Copy link
Member Author

commented Jun 19, 2017

@varshavaradarajan — I just tried the following steps, between the 2 sets of bash statements, I removed the docker-task-assembly-0.1.27.jar plugin jar, and the agent does restart on its own (logs at the end)

GO_SERVER_SYSTEM_PROPERTIES=-DpluginLocationMonitor.sleepTimeInSecs=5 ./server.sh
$ http://localhost:8153/go/admin/latest-agent.status -I ; wget http://localhost:8153/go/admin/agent-plugins.zip ; unzip -l agent-plugins.zip; rm agent-plugins.zip
HTTP/1.1 200 OK
Date: Mon, 19 Jun 2017 05:39:24 GMT
Set-Cookie: JSESSIONID=13kwz7imm82bb178rzmhcycrhf;Path=/go;Expires=Mon, 03-Jul-2017 05:39:24 GMT;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Agent-Content-MD5: rbPRP/N2zLujPLoI3Vi+jw==
Agent-Launcher-Content-MD5: CI7KVwA5HFghStmlfzonQQ==
Agent-Plugins-Content-MD5: 237430418a6f12285e4b7ea8141d4623
TFS-SDK-Content-MD5: /rEhlcK4KRNfF7MBxt/IiQ==
Cruise-Server-Ssl-Port: 8154
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: chrome=1
Content-Length: 0

--2017-06-19 11:09:24--  http://localhost:8153/go/admin/agent-plugins.zip
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8153... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘agent-plugins.zip’

agent-plugins.zip                                [ <=>                                                                                         ]   7.02M  --.-KB/s    in 0.01s

2017-06-19 11:09:24 (709 MB/s) - ‘agent-plugins.zip’ saved [7357180]

Archive:  agent-plugins.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
   621954  06-19-17 11:08   external/docker-task-assembly-0.1.27.jar
  6016714  06-19-17 11:08   external/github-pr-poller-1.3.0-RC2.jar
   787806  06-19-17 11:08   external/gocd-nuget-poller-plugin-v1.0.3.jar
 --------                   -------
  7426474                   3 files
$ curl http://localhost:8153/go/admin/latest-agent.status -I ; wget http://localhost:8153/go/admin/agent-plugins.zip ; unzip -l agent-plugins.zip; rm agent-plugins.zip
HTTP/1.1 200 OK
Date: Mon, 19 Jun 2017 05:39:27 GMT
Set-Cookie: JSESSIONID=4y3cq4npnn3qldt9zahjgrd4;Path=/go;Expires=Mon, 03-Jul-2017 05:39:27 GMT;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Agent-Content-MD5: rbPRP/N2zLujPLoI3Vi+jw==
Agent-Launcher-Content-MD5: CI7KVwA5HFghStmlfzonQQ==
Agent-Plugins-Content-MD5: 63c5b306c49dc0c7915249f7741f7de4
TFS-SDK-Content-MD5: /rEhlcK4KRNfF7MBxt/IiQ==
Cruise-Server-Ssl-Port: 8154
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: chrome=1
Content-Length: 0

--2017-06-19 11:09:27--  http://localhost:8153/go/admin/agent-plugins.zip
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8153... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘agent-plugins.zip’

agent-plugins.zip                                [ <=>                                                                                         ]   6.48M  --.-KB/s    in 0.01s

2017-06-19 11:09:27 (583 MB/s) - ‘agent-plugins.zip’ saved [6796743]

Archive:  agent-plugins.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
  6016714  06-19-17 11:09   external/github-pr-poller-1.3.0-RC2.jar
   787806  06-19-17 11:09   external/gocd-nuget-poller-plugin-v1.0.3.jar
 --------                   -------
  6804520                   2 files

Agent logs:

2017-06-19 11:12:25,997 [loopThread] INFO  go.agent.service.SslInfrastructureService:92 - [Agent Registration] Successfully registered agent.
2017-06-19 11:12:25,997 [loopThread] INFO  thoughtworks.go.agent.AgentHTTPClientController:105 - About to get cookie from the server.
2017-06-19 11:12:26,063 [loopThread] INFO  thoughtworks.go.agent.AgentHTTPClientController:108 - Got cookie: 79918e7d-39e0-44a0-b311-c5f7bcb12611





2017-06-19 11:12:45,659 [loopThread] FATAL go.agent.service.AgentUpgradeService:46 - [Agent Upgrade] Agent needs to upgrade plugins. Currently has md5 [237430418a6f12285e4b7ea8141d4623] but server version has md5 [63c5b306c49dc0c7915249f7741f7de4]. Exiting.
2017-06-19 11:12:45,660 [Thread-6 ] INFO  springframework.context.support.ClassPathXmlApplicationContext:1032 - Closing org.springframework.context.support.ClassPathXmlApplicationContext@446cdf90: startup date [Mon Jun 19 11:12:23 IST 2017]; root of context hierarchy
2017-06-19 11:12:45,660 [Thread-6 ] INFO  beans.factory.support.DefaultListableBeanFactory:434 - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@78e03bb5: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,defaultPluginJarChangeListener,defaultPluginJarLocationMonitor,defaultPluginRegistry,goPluginDescriptorBuilder,pluginWriter,pluginValidator,goFileSystem,defaultPluginManager,felixGoPluginOSGiFramework,defaultGoPluginOSGiManifestCreator,pluginRequestProcessorRegistry,pluginsZip,zipUtil,systemEnvironment,propertyConfigurer,buildLoopServer,urlService,httpService,artifactsManipulator,sslInfrastructureService,agentUpgradeService,agentRegistry,subprocessLogger,agentPluginsInitializer,packageRepositoryExtension,scmExtension,taskExtension,httpClient,httpClientBuilder,webSocketClientBuilder,webSocketClientHandler,webSocketSessionHandler,agentControllerFactory,agentController,agentControllerLooper,agentControllerPinger,agentControllerInstructionExecutor,loopThread,pingThread,instructionExecuteThread,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy
2017-06-19 11:12:45,661 [Thread-6 ] INFO  springframework.scheduling.timer.TimerFactoryBean:165 - Cancelling Timer
2017-06-19 11:12:45,661 [Thread-6 ] INFO  springframework.scheduling.timer.TimerFactoryBean:165 - Cancelling Timer
2017-06-19 11:12:45,662 [Thread-6 ] INFO  springframework.scheduling.timer.TimerFactoryBean:165 - Cancelling Timer

==> logs/go-agent-bootstrapper.log <==
2017-06-19 11:12:50,700 [main     ] INFO  go.agent.bootstrapper.AgentBootstrapper:86 - Launcher returned with code 0(0x0)
2017-06-19 11:12:50,731 [main     ] INFO  go.agent.bootstrapper.AgentBootstrapper:131 - Destroying launcher creator
2017-06-19 11:12:51,758 [main     ] INFO  go.agent.bootstrapper.DefaultAgentLauncherCreatorImpl:89 - Attempt No: 1 to cleanup launcher temp files
2017-06-19 11:12:51,760 [main     ] INFO  go.agent.bootstrapper.AgentBootstrapper:114 - Waiting for 10000 ms before re-launch....
2017-06-19 11:13:01,775 [main     ] INFO  agent.common.util.JarUtil:68 - Attempting to load Go-Agent-Launcher-Class from feec6b7e-2dba-47a8-8440-36c1952e2959agent-launcher.jar File:
2017-06-19 11:13:01,776 [main     ] INFO  agent.common.util.JarUtil:77 - manifestLibDirKey: Go-Agent-Launcher-Lib-Dir: libs
2017-06-19 11:13:01,797 [main     ] INFO  agent.common.util.JarUtil:83 - manifestClassKey: Go-Agent-Launcher-Class: com.thoughtworks.go.agent.launcher.AgentLauncherImpl
2017-06-19 11:13:01,799 [main     ] INFO  agent.common.util.ParentClassAccessFilteringClassloader:43 - Loading com.thoughtworks.cruise.agent.common.launcher.AgentLauncher using com.simontuffs.onejar.JarClassLoader
2017-06-19 11:13:01,814 [main     ] INFO  go.agent.bootstrapper.AgentBootstrapper:82 - Attempting create and start launcher...
2017-06-19 11:13:01,836 [main     ] INFO  agent.common.util.ParentClassAccessFilteringClassloader:43 - Loading com.thoughtworks.cruise.agent.common.launcher.AgentLaunchDescriptor using com.simontuffs.onejar.JarClassLoader

==> logs/go-agent-launcher.log <==
2017-06-19 11:13:01,836 [TouchLoopThread-8] INFO  go.agent.launcher.Lockfile:102 - Using lock file: /Users/ketanpadegaonkar/projects/gocd/gocd/installers/target/distributions/zip/go-agent-17.6.0/.agent-bootstrapper.running
2017-06-19 11:13:02,030 [main     ] INFO  agent.common.util.JarUtil:68 - Attempting to load Go-Agent-Bootstrap-Class from agent.jar File:
2017-06-19 11:13:02,030 [main     ] INFO  agent.common.util.JarUtil:83 - manifestClassKey: Go-Agent-Bootstrap-Class: com.thoughtworks.go.agent.AgentProcessParentImpl
2017-06-19 11:13:02,032 [main     ] INFO  thoughtworks.go.agent.AgentProcessParentImpl:52 - Agent is version: 17.6.0-5112
2017-06-19 11:13:02,144 [main     ] INFO  go.agent.launcher.ServerBinaryDownloader:108 - Downloading agent-plugins.zip
2017-06-19 11:13:02,169 [main     ] INFO  go.agent.launcher.ServerBinaryDownloader:115 - Got server response
2017-06-19 11:13:02,235 [main     ] INFO  go.agent.launcher.ServerBinaryDownloader:123 - Piped the stream to admin/agent-plugins.zip
2017-06-19 11:13:02,241 [main     ] INFO  thoughtworks.go.util.PerfTimer:53 - Performance: Downloading new admin/agent-plugins.zip with md5 signature: 63c5b306c49dc0c7915249f7741f7de4 took 97ms
2017-06-19 11:13:02,322 [main     ] INFO  thoughtworks.go.agent.AgentProcessParentImpl:70 - Launching Agent with command: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/bin/java -Dcruise.console.publish.interval=10 -Xms128m -Xmx256m -Djava.io.tmpdir=/var/folders/fg/nyq9dgfx7ndfd0ng_qvd1p140000gn/T/ -Djava.security.egd=file:/dev/./urandom -Dagent.plugins.md5=63c5b306c49dc0c7915249f7741f7de4 -Dagent.binary.md5=rbPRP/N2zLujPLoI3Vi+jw== -Dagent.launcher.md5=CI7KVwA5HFghStmlfzonQQ== -Dagent.tfs.md5=/rEhlcK4KRNfF7MBxt/IiQ== -jar agent.jar -serverUrl https://127.0.0.1:8154/go -sslVerificationMode NONE

==> logs/go-agent-stderr.log <==
Using log4j configuration from config/agent-log4j.properties

==> logs/go-agent.log <==
2017-06-19 11:13:03,213 [main     ] INFO  springframework.context.support.ClassPathXmlApplicationContext:503 - Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@446cdf90: startup date [Mon Jun 19 11:13:03 IST 2017]; root of context hierarchy
2017-06-19 11:13:03,234 [main     ] INFO  beans.factory.xml.XmlBeanDefinitionReader:315 - Loading XML bean definitions from class path resource [applicationContext.xml]
2017-06-19 11:13:03,320 [main     ] INFO  beans.factory.xml.XmlBeanDefinitionReader:315 - Loading XML bean definitions from class path resource [applicationContext-plugin-infra.xml]
2017-06-19 11:13:03,444 [main     ] INFO  beans.factory.config.PropertyPlaceholderConfigurer:177 - Loading properties file from URL [onejar:main/agent-17.6.0-5112-classes.jar/agent.properties]
2017-06-19 11:13:03,448 [main     ] INFO  beans.factory.annotation.AutowiredAnnotationBeanPostProcessor:139 - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-06-19 11:13:03,456 [main     ] INFO  beans.factory.support.DefaultListableBeanFactory:577 - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@78e03bb5: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,defaultPluginJarChangeListener,defaultPluginJarLocationMonitor,defaultPluginRegistry,goPluginDescriptorBuilder,pluginWriter,pluginValidator,goFileSystem,defaultPluginManager,felixGoPluginOSGiFramework,defaultGoPluginOSGiManifestCreator,pluginRequestProcessorRegistry,pluginsZip,zipUtil,systemEnvironment,propertyConfigurer,buildLoopServer,urlService,httpService,artifactsManipulator,sslInfrastructureService,agentUpgradeService,agentRegistry,subprocessLogger,agentPluginsInitializer,packageRepositoryExtension,scmExtension,taskExtension,httpClient,httpClientBuilder,webSocketClientBuilder,webSocketClientHandler,webSocketSessionHandler,agentControllerFactory,agentController,agentControllerLooper,agentControllerPinger,agentControllerInstructionExecutor,loopThread,pingThread,instructionExecuteThread,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy

==> logs/go-agent-stderr.log <==
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

==> logs/go-agent.log <==
2017-06-19 11:13:03,845 [main     ] INFO  thoughtworks.go.agent.AgentControllerFactory:102 - Connecting to server using HTTP(S)
2017-06-19 11:13:03,877 [main     ] INFO  springframework.scheduling.timer.TimerFactoryBean:97 - Initializing Timer
2017-06-19 11:13:03,879 [main     ] INFO  springframework.scheduling.timer.TimerFactoryBean:97 - Initializing Timer
2017-06-19 11:13:03,880 [main     ] INFO  springframework.scheduling.timer.TimerFactoryBean:97 - Initializing Timer
2017-06-19 11:13:05,047 [loopThread] INFO  thoughtworks.go.agent.AgentHTTPClientController:105 - About to get cookie from the server.
2017-06-19 11:13:05,073 [loopThread] INFO  thoughtworks.go.agent.AgentHTTPClientController:108 - Got cookie: 25219759-1302-441c-b6b0-113c5fccfac2

@varshavaradarajan varshavaradarajan merged commit 8821f76 into gocd:master Jun 19, 2017

8 checks passed

build-linux-PR/build-non-server
Details
build-linux-PR/build-server
Details
build-windows-PR/build-non-server
Details
build-windows-PR/build-server
Details
installers-PR/dist
Details
license/cla Contributor License Agreement is signed.
Details
plugins-PR/build
Details
trigger/do-nothing
Details

@varshavaradarajan varshavaradarajan added this to the Release 17.6 milestone Jun 19, 2017

@ketan ketan deleted the ketan:improve/plugin-listener branch Jun 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.