Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add mirror option to speed the download #252

Merged
merged 3 commits into from
Apr 2, 2020
Merged

Add mirror option to speed the download #252

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
58c14f2
Add mirror option to speed the download
LinuxSuRen Mar 8, 2020
0095566
Remove one dependency
LinuxSuRen Mar 8, 2020
749fb31
Update bootstrap/src/main/java/io/jenkins/jenkinsfile/runner/bootstra…
LinuxSuRen Mar 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 13 additions & 2 deletions bootstrap/src/main/java/io/jenkins/jenkinsfile/runner/bootstrap/Bootstrap.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ public class Bootstrap {
@Option(name = "-p", aliases = { "--plugins" }, usage = "plugins required to run pipeline. Either a plugins.txt file or a /plugins installation directory. Defaults to plugins.txt.")
public File pluginsDir;

@Option(name = "-m", aliases = {"--mirror"}, usage = "mirror site of Jenkins, defaults to http://updates.jenkins.io/download. Get the mirror list from http://mirrors.jenkins-ci.org/status.html.")
public String mirror;

/**
* Checked out copy of the working space.
*/
Expand Down Expand Up @@ -261,7 +264,7 @@ private File getJenkinsWar() throws IOException {
File war = new File(cache, String.format("war/%s/jenkins-war-%s.war", version, version));
if (!war.exists()) {
war.getParentFile().mkdirs();
final URL url = new URL(String.format("http://updates.jenkins.io/download/war/%s/jenkins.war", version));
final URL url = new URL(getMirrorURL(String.format("http://updates.jenkins.io/download/war/%s/jenkins.war", version)));
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why we use http instead of https here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our current mirror infrastructure works for HTTP only. For HTTPs you get no mirrors. AFAIK @olblak is exploring replacement options

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#258 as a follow-up

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As @oleg-nenashev mention we don't have a very good support of https at the moment and I am working on different initiatives
There is an open Pull Request for https support of http://archives.jenkins-ci.org
I also deployed get.jenkins.io which is not as update as mirror.jenkins.io but instead of using mirrorbrain we switch to mirrorbits so we support https on our mirror infrastructure and it also provides more information like this

Basically the challenges that I am facing now are:

  • htacess support, we rely a lot on htacess so I have to switch from nginx to apache or get rid of htacess. I am more considering switch to apache
  • Updating artifacts, either we have push or a pull approach
    mirror.azure.jenkins.io has a pull approach and update its data every 5minutes
  • https://get.jenkins.io and the futur https://archives.jenkins.io are configured with a push approach but currently it's taking too long, around 15min for each run

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regarding update-center will probably follow once our mirror infra fully support https

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if you specify https here, you'll just end up on a HTTP mirror at the moment

System.out.printf("Downloading jenkins %s...\n", version);
FileUtils.copyURLToFile(url, war);
}
Expand All @@ -275,14 +278,22 @@ private void installPlugin(String shortname, String version) throws IOException
File plugin = new File(cache, String.format("plugins/%s/%s-%s.hpi", shortname, shortname, version));
if (!plugin.exists() || ("latest".equals(version) && plugin.lastModified() < CACHE_EXPIRE) ) {
plugin.getParentFile().mkdirs();
final URL url = new URL(String.format("http://updates.jenkins.io/download/plugins/%s/%s/%s.hpi", shortname, version, shortname));
final URL url = new URL(getMirrorURL(String.format("http://updates.jenkins.io/download/plugins/%s/%s/%s.hpi", shortname, version, shortname)));
System.out.printf("Downloading jenkins plugin %s (%s)...\n", shortname, version);
FileUtils.copyURLToFile(url, plugin);
}

Files.createSymbolicLink(install.toPath(), plugin.toPath());
}

private String getMirrorURL(String url) {
if (this.mirror == null || "".equals(this.mirror.trim())) {
return url;
}

return url.replace("http://updates.jenkins.io/download", this.mirror);
}

public int run() throws Throwable {
String appClassName = "io.jenkins.jenkinsfile.runner.App";
if (hasClass(appClassName)) {
Expand Down