[INFRA-1809] Private update center improvement

[LinuxSuRen]

INFRA-1809

[jenkinsadmin]

Build failed; the context from the latest run is:

Expand to view

remote: Enumerating objects
remote: Counting objects
remote: Compressing objects
Receiving objects
Resolving deltas
Updating references
Fetching without tags
Merging remotes/origin/master commit 78099017a8e368e5ca72957d2a4d57863e898f3b into PR head commit 32a30039181f7591db927a94390de64e96445343
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to ubuntu-jenkinsinfra-highmem027b30
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
		at sun.reflect.GeneratedMethodAccessor379.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
		at com.sun.proxy.$Proxy254.execute(Unknown Source)
		at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:122)
		at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1094)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1187)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
		at hudson.security.ACL.impersonate(ACL.java:290)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
hudson.plugins.git.GitException: Failed to merge AnyObjectId[78099017a8e368e5ca72957d2a4d57863e898f3b]
	at org.jenkinsci.plugins.gitclient.JGitAPIImpl$6.execute(JGitAPIImpl.java:1560)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Powered by the Comment Logger

[daniel-beck]

Features not exercised in Jenkins project infra are unlikely to be maintained well.

Note that this will not be merged, as it has content specific to your environment, rather than just generalizations (which could be). It seems you should rather maintain a fork with your specific data, rather than seek merging into upstream.

Wouldn't https://github.com/yandex-qatools/juseppe be better for your use case?

[LinuxSuRen]

I think there is a misunderstanding. I just find I commit some private files. That's no need. I will remove them.
So I mean this PR will don't contain any specific environment information.

[oleg-nenashev]

Looks good to me. Some documentation for README would be helpful tho

[LinuxSuRen]

Okay, just miss this.

[rasmusjelsgaard]

We have the same use case where I work as @LinuxSuRen has described as the motivation for this PR.
It seems it has been waiting for some time. Anything I could do to help to speed up things? Or is there anything preventing this from being merged?

[LinuxSuRen]

ping @daniel-beck How do you think about this PR? Thanks for your time.

[jenkinsadmin]

Build failed; the context from the latest run is:

Expand to view

	at org.apache.maven.artifact.manager.DefaultWagonManager.getArtifact(DefaultWagonManager.java:379)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:167)
	... 15 more
Downloading org.jenkins-ci.plugins:token-macro:hpi:1.9
java.io.IOException: Failed to read manifest from org.jenkins-ci.plugins:token-macro:1.9:null:hpi
	at org.jvnet.hudson.update_center.MavenArtifactSource.getManifest(MavenArtifactSource.java:40)
	at org.jvnet.hudson.update_center.MavenArtifact.getManifest(MavenArtifact.java:172)
	at org.jvnet.hudson.update_center.MavenArtifact.getManifestAttributes(MavenArtifact.java:178)
	at org.jvnet.hudson.update_center.HPI.getRequiredJenkinsVersion(HPI.java:85)
	at org.jvnet.hudson.update_center.VersionCappedMavenRepository.listHudsonPlugins(VersionCappedMavenRepository.java:72)
	at org.jvnet.hudson.update_center.Main.buildPlugins(Main.java:475)
	at org.jvnet.hudson.update_center.Main.buildUpdateCenterJson(Main.java:350)
	at org.jvnet.hudson.update_center.Main.run(Main.java:285)
	at org.jvnet.hudson.update_center.Main.run(Main.java:203)
	at org.jvnet.hudson.update_center.Main.main(Main.java:193)
Caused by: java.io.IOException: Failed to resolve artifact org.jenkins-ci.plugins:token-macro:1.9:null:hpi
	at org.jvnet.hudson.update_center.MavenArtifact.resolve(MavenArtifact.java:97)
	at org.jvnet.hudson.update_center.MavenArtifactSource.getZipFileEntry(MavenArtifactSource.java:46)
	at org.jvnet.hudson.update_center.MavenArtifactSource.getManifest(MavenArtifactSource.java:37)
	... 9 more
Caused by: org.apache.maven.artifact.resolver.ArtifactNotFoundException: Unable to download the artifact from any repository

Try downloading the file manually from the project website.

Then, install it using the command: 
    mvn install:install-file -DgroupId=org.jenkins-ci.plugins -DartifactId=token-macro -Dversion=1.9 -Dpackaging=hpi -Dfile=/path/to/file

Alternatively, if you host your own repository you can depCannot contact ubuntu-jenkinsinfrad381f0: java.lang.InterruptedException
Resuming build at Wed Mar 27 09:43:50 GMT 2019 after Jenkins restart
Waiting to resume part of Infra » update-center2 » PR-245 #23: Waiting for next available executor on ‘ubuntu-jenkinsinfrad381f0’
Ready to run at Wed Mar 27 09:43:52 GMT 2019
Cannot contact ubuntu-jenkinsinfrad381f0: java.lang.InterruptedException
Resuming build at Wed Mar 27 13:40:54 GMT 2019 after Jenkins restart
Waiting to resume part of Infra » update-center2 » PR-245 #23: Finished waiting
Ready to run at Wed Mar 27 13:40:57 GMT 2019
Cannot contact ubuntu-jenkinsinfrad381f0: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on ubuntu-jenkinsinfrad381f0 failed. The channel is closing down or has closed down
Aborted by olblakCould not connect to ubuntu-jenkinsinfrad381f0 to send interrupt signal to process

[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

Finished: ABORTED

Powered by the Comment Logger

[LinuxSuRen]

Anyone could help me to review this? Thanks.

[daniel-beck]

I recommend you maintain a fork of this repository if you want to use it on your own infrastructure.

It's not a goal of this tool to be generally useful, and if we have to cut corners, or even overhaul the tool, to optimize it for use in the Jenkins project infrastructure, we will.

I'll poke Tyler to see whether he disagrees, but I'm not particularly inclined to add features (and the resulting risk of getting bug reports and RFEs I have no intention in working on) that would not be exercised in Jenkins infra.

[LinuxSuRen]

@daniel-beck I can understand what you're thinking. And I agree with you that this org (jenkins-infra) is setting up for Jenkins infrastructure. I can maintain this feature. But I believe that there're lots of people have the same demands as me. Anyway, thanks for your viewpoint.

[daniel-beck]

@LinuxSuRen Thanks for your understanding. I've talked with others in the past and they mentioned their problems in keeping their installations up to date with this repo, and their specific needs (like ones implemented here), so I understand the use case. It's just something we don't really have the capacity to support. As I wrote I will check in with @rtyler before closing this PR in case he offers a different perspective.

A real fork (not like GitHub forks, more like Hudson/Jenkins) of this repo for end user update sites, or perhaps an entirely different tool like https://github.com/jenkinsci/juseppe, seems like a much easier way to handle this use.

[rtyler]

@daniel-beck I definitely understand your concerns and share many of them myself. If I look at pipeline-library however, there has been a fair bit of work done to make that library work outside of the Jenkins project's own CI infrastructure.

In that case, the iteration on the code for uses not strictly part of the original intent for pipeline-library have proven to be overall beneficial to the usage on ci.jenkins.io.

I think if @LinuxSuRen is volunteering to contribute and improve the code in this repository, that is an overall long term benefit to the update center process running for the Jenkins project, even if it may seem unrelated in the short term.

Fortunately, we know @LinuxSuRen so I'm not worried about this being a "drive-by contribution." 😸

[jenkinsadmin]

Build failed; the context from the latest run is:

Expand to view

	at org.jvnet.hudson.update_center.Main.buildPlugins(Main.java:475)
	at org.jvnet.hudson.update_center.Main.buildUpdateCenterJson(Main.java:350)
	at org.jvnet.hudson.update_center.Main.run(Main.java:285)
	at org.jvnet.hudson.update_center.Main.run(Main.java:203)
	at org.jvnet.hudson.update_center.Main.main(Main.java:193)
Caused by: java.io.IOException: Failed to resolve artifact org.jenkins-ci.ui:momentjs:1.0:null:hpi
	at org.jvnet.hudson.update_center.MavenArtifact.resolve(MavenArtifact.java:97)
	at org.jvnet.hudson.update_center.MavenArtifactSource.getZipFileEntry(MavenArtifactSource.java:46)
	at org.jvnet.hudson.update_center.MavenArtifactSource.getManifest(MavenArtifactSource.java:37)
	... 9 more
Caused by: org.apache.maven.artifact.resolver.ArtifactNotFoundException: Unable to download the artifact from any repository

Try downloading the file manually from the project website.

Then, install it using the command: 
    mvn install:install-file -DgroupId=org.jenkins-ci.ui -DartifactId=momentjs -Dversion=1.0 -Dpackaging=hpi -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there: 
    mvn deploy:deploy-file -DgroupId=org.jenkins-ci.ui -DartifactId=momentjs -Dversion=1.0 -Dpackaging=hpi -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]


  org.jenkins-ci.ui:momentjs:hpi:1.0

from the specified remote repositories:
  public (http://repo.jenkins-ci.org/public/)

	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:179)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:82)
	at org.jvnet.hudson.update_center.MavenRepositoryImpl.resolve(MavenRepositoryImpl.java:245)
	at org.jvnet.hudson.update_center.MavenRepository.resolve(MavenRepository.java:83)
	at org.jvnet.hudson.update_center.MavenArtifact.resolve(MavenArtifact.java:71)
	... 11 more
Caused by: org.apache.maven.wagon.ResourceDoesNotExistException: Unable to download the artifact from any repository
	at org.apache.maven.artifact.manager.DefaultWagonManager.getArtifact(DefaultWagonManager.java:379)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:167)
	... 15 more
Downloading org.jenkins-ci.plugins:mongodb:hpi:1.3
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: missing workspace /mnt/agent-workspace/workspace/Infra_update-center2_PR-245 on ubuntu-jenkinsinfra-highmemdaea60
Finished: FAILURE

Powered by the Comment Logger

[Siegfriedk]

We have the same issue: For security reasons and building jenkins images on the fly, we are very dependend on it and need an internal mirror.

The code itself is not much and it looks very maintainable. What would make it possible to integrate it in this repo? Any official commitment?

I have found a few ways of how people are simulating what this tool is doing. Not sure whats more beneficial for jenkins-infra: Some hacky way which is not controlable or an official one :).

I will clone it and work with LinuxSuRen branch. @LinuxSuRen tx for it

[LinuxSuRen]

I really hope this official one could also support a private environment. At least, this feature would not break any infra I think.

And Thanks @Siegfriedk and @rasmusjelsgaard to using this PR.

[reenberg]

Awesome, I was actually in the process of implementing something like this.

However I was also thinking of implementing a limit on how many plugins and war files to keep around. It seems a bit silly to offer every version of every plugin inside a closed network.

[Siegfriedk]

Awesome, I was actually in the process of implementing something like this.

However I was also thinking of implementing a limit on how many plugins and war files to keep around. It seems a bit silly to offer every version of every plugin inside a closed network.

I thought about this as well but hadn't time to look into it. I would probably add a filter like "last 5 versions" or "younger then 5 month"

For now, with the cache, it works quite well and when you look into the setup / content of all plugins there are not tooo many which are huge.

For example, a quick check on one of our older jenkins mirror setups:
483M selenium 497M docker-build-step 524M azure-vm-agents 572M dependency-check-jenkins-plugin 579M ontrack 597M doktor 609M openshift-sync 623M cucumber-living-documentation 628M findbugs 659M pmd 681M yet-another-docker-plugin 798M kubernetes 810M xframium 882M aws-codepipeline 888M DotCi 950M crossbrowsertesting 1.2G artifactory 1.2G aws-device-farm 1.2G aws-java-sdk 1.3G whitesource 2.0G maven-plugin 7.5G sauce-ondemand

[reenberg]

I thought about this as well but hadn't time to look into it. I would probably add a filter like "last 5 versions" or "younger then 5 month"
Just my idea.

For now, with the cache, it works quite well and when you look into the setup / content of all plugins there are not tooo many which are huge.

Size was one thing, but also sync time was a concern for me. Lastly i was thinking about reducing the load on the jenkins-ci infrastructure.
Note that I haven't completely looked into how much network traffic it generates when you already have synched it once, but I have noticed that it takes quite some time.

[reenberg]

For what it is worth, I have tried giving it a stab.

I have forked master and rebased the changes made by @LinuxSuRen. However they had some issues. For example, it was only the plugins that had their url changed in the produced update-center.json file, the .war artifacts was still using the original hard coded url.

Since the whole concept of a cache server didn't make sense to me, as the -download option will produce the needed artifacts for you, and since you can now change the URL, I decided to more or less remove all the additions that was made, and then I renamed the parameter for setting the url, to something that I found more natural.

In order to clean up some code, I decided to implement a static Options class, that contains all the cli parameters. This made it possible to read those parameters from the HPI and HudsonWar classes, which made everything way simpler and cleaner to implement.

I totally understand if this is way to much change to accept back upstream.

Besides that I decided to implement a MavenRepository class called TruncatedMavenRepository (terrible name, i know), which add two new options, such that you can limit the number of .war's and .hpi's pr plugins. So if you want to keep the last 5 Jenkins versions and 3 plugin versions, that is now possible. This reduced my download folder from about 100GiB to ~19GiB (as far as I remember).

It still doesn't clean up your local .m2/repositories/ folder though. This is a option I would also like to add at some point. Without this, you need to manually wipe that folder once in a while to still get any space savings. But hey, at least i'm not storring everything anymore.

The above is implemented in my feature/private-update-center branch.

I also got a bit fed up with not having propper output. When syncing from scratch, you will have quite a big period of time with nothing happening on the screen, as it is downloading large ,war files. I added a Progress Bar as a TransferListener which the wagon class can take and utilize, in order to make this a bit more user friendly, I also implemented logging using log4j 2 instead of those System.out / System.err which makes it way better when running unattended for log collection, as it now also has actual debug output.

This needs some more attention, as most of the original System.out was turned into INFO logs, but I think at least some of them should get a second opinion. Also I have just added a very crude log4j2.xml which contains class names, etc. This takes up quite a bit of screen real estate, and is ugly to look at. This needs to be simplified before it is really useful.

The progress bar and logging is implemented ontop of the other branch in my feature/better-logging branch.

Note that I may still rebase those branches at will when I find bugs or find stuff that I forgot, so don't rely on them 100%.

Comments are more than welcome.

[LinuxSuRen]

@reenberg Thanks for sharing your ideas here. There're actually some issues with this PR. But I will continue to fix them.

[silent-night-no-trace]

please view https://www.oschina.net/news/111266/jenkins-plugin-mirror to solve plugins download slowly

[LinuxSuRen]

please view https://www.oschina.net/news/111266/jenkins-plugin-mirror to solve plugins download slowly

There's no relationship between this repo and mirror. Please visit here to get more help. And thanks for your feedback.

[daniel-beck]

FWIW this is still on my backlog to review and test in depth; unfortunately I have not yet found the time to do that.

[daniel-beck]

Looks OK overall. Sorry it took me so long. I think this should be mergeable after the findings below are addressed.

[daniel-beck]

Requesting that previous feedback is addressed.

[LinuxSuRen]

Requesting that previous feedback is addressed.

I see it. Sorry, I'll do it as soon as possible.

[daniel-beck]

@LinuxSuRen No worries, just making sure this doesn't get stalled any longer 👍

[LinuxSuRen]

@LinuxSuRen No worries, just making sure this doesn't get stalled any longer 👍

Sure. Totally understand. And thanks for the reviewing work.

[LinuxSuRen]

@daniel-beck I've finished my changes.

[daniel-beck]

@LinuxSuRen Please try #365. It is a big overhaul of the update center generator internally and with new parameters, but after the latest changes, I think everything you need is included:

• --whitelist-file is a new parameter for whitelists that are similar to yours. You can specify specific versions, or * for all
• The system property or environment variable DOWNLOADS_ROOT_URL will be used for download URLs
• --downloads-directory (previously -download) would write the plugins to a directory tree.

While not exactly the same as what this PR does (especially around "cache"), I think this should take care of your needs. Please let me know how this goes.

[daniel-beck]

@LinuxSuRen I just merged #265 which changes a lot about the update center. For interest to you with this PR:

• A whitelist feature exists that seems more flexible than this implementation.
• The whitelist can also basically differentiate between "-cache" and "-cacheAll" from this PR when running with what used to be the -download argument.
• An environment variable or system property named DOWNLOADS_ROOT_URL will be used for the download base URL, see the new HPI.java.

I think this PR may no longer be needed. Please look at the new version and tell me what you think.

[daniel-beck]

As a matter of housekeeping, I'm closing this pull request. This isn't a rejection of the proposed changes; just acknowledging that they're based on top of a much older update-center2 and not usable in their current state (plus they might be obsolete).

I'd still like to know whether the implemented changes I mention in #245 (comment) / #245 (comment) address your use case, please let me know. If not, I'm open to finding a solution together.

[LinuxSuRen]

Good to hear more improvements from here. I'll let you know if there're more ideas from me.