Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

New S3-plugin features for S3 artifacts management #2

Closed
wants to merge 13 commits into from

7 participants

Brice Figureau Richard Dallaway Nicolas De loof Matt Jonas Rabbe David Beer domi
Brice Figureau

Hi,

Here is a pull request to add to the S3 plugin several features to allow S3 to become a more integrated repository for artifacts built by jenkins. The idea is for S3 to become a substitute of the master artifact storage area.

This pull request notably adds the following features:

  • allow to upload from the slave (this is essential if like me you are
    using EC2 slaves with a master outside EC2, to prevent uploading from slave to master then from master to S3).

  • S3 uploaded artifacts fingerprinting (to allow finding what builds produced what artifacts)

  • S3 Copy artifact build step, based on the "Copy Artifact" plugin, it allows to download an artifact from a build using the S3 publisher plugin back into the workspace. This also produces fingerprints to link jobs together.

  • automatically delete artifacts in S3 from deleted jobs/builds.

  • Direct S3 download links of the artifacts in the job page or build home page.

This code is in production for about one week on my work jenkins installation with great success.

Please review this changeset, I'd be happy to modify it so that it can be merged upstream.
Many thanks!

Brice

and others added some commits February 17, 2012
Brice Figureau s3 plugin works fine with jenkins 1.424
so just say so.

Signed-off-by: Brice Figureau <brice@daysofwonder.com>
731214d
Brice Figureau Upload from the slave instead of relaying through the master
It's very useful when running a master outside of AWS EC2 but
with slaves on EC2.

There is no possibility yet to allow the previous way of
working.

Signed-off-by: Brice Figureau <brice@daysofwonder.com>
7d322c1
Brice Figureau Allow to upload fromm slave or master
This adds a checkbox in the Publish S3 configuration view of a job, to
either publish to S3 from the slave or the master.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
b5a5e82
Brice Figureau Record fingerprints of uploaded to S3 artifacts
This also adds an "S3 Artifact" links to the run side panel to have
a direct access to the uploaded artifacts.


Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
247510a
Brice Figureau Copy Artifacts from S3
This new extension allows to copy artifacts uploaded to S3 by a previous
run to the current build workspace, like the "Copy Artifacts" plugin
does.

It also fingerprint downloaded artifacts to be able to link builds
together.

This extension requires the Copy Artifact plugin to be installed to work
correctly.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
4c77473
Brice Figureau Refactor: moving the download/upload callable to their own types
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
32b3860
Brice Figureau Refactor: add some factories to Destination
This allows to create a Destination from a run and an S3Artifact.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
6b8b4d0
Brice Figureau Remove the Workspace build selector from the build selector list
This selector has no meaning for S3 artifacts copy.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
6754197
Brice Figureau Respect 'filter' when copying artifact back from S3
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
f29b453
Brice Figureau Delete old artifacts from S3 when the run is deleted
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
e0719aa
Brice Figureau Allow to download artifacts on the fly from S3
* removes the s3url field from FingerprintRecord
* adds an S3ArtifactAction download action that creates a
presigned S3 url for the given artifact

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
af9f79a
Brice Figureau Adds a build section that displays the last succesfull artifacts
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
7cbe589
Richard Dallaway
Collaborator
d6y commented on 4c77473 March 11, 2012

I've run into the following problem with the changes. When running mvn on this code I'm getting:

Tests in error: 
  /Users/richard/Developer/s3-plugin/target/classes/hudson/plugins/s3/S3CopyArtifact/index.jelly(org.jvnet.hudson.test.JellyTestSuiteBuilder$JellyCheck): Could not parse Jelly script : file:/Users/richard/Developer/s3-plugin/target/classes/hudson/plugins/s3/S3CopyArtifact/index.jelly

index.jelly is empty. This is with Maven 3.0.3.

Are there additional steps I need to make this build?

I must admit I didn't check the tests, since there are none included :)
Can you just remove this index.jelly file and try again?

I confirm that this empty index.jelly file shouldn't have been committed in the first place. Removing it allows the plugin to be built correctly.
I'm going to push an additional commit removing this empty file.

Brice Figureau Remove an empty unneeded index.jelly
This prevents unit test to run and this file isn't used.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
b1ddaa9
Richard Dallaway
Collaborator
d6y commented March 11, 2012

Thank you - I can now build and create the s3.hpi file. I'll test some more later in the week.

Richard Dallaway
Collaborator
d6y commented March 16, 2012

Disappointingly I've not been able to progress testing this week. I thought I'd be able to this weekend, but I'm going to be away and I'm not sure how much time I'm going to get to work on this. But I've not forgotten about this request! Just apologising for not moving faster on it.

Richard Dallaway
Collaborator
d6y commented March 25, 2012

I tried to drop the s3.hpi into my ~/.hudson/plugins directory under Jenkins 1.456 and the received the following errors during startup:

Mar 25, 2012 6:14:11 PM jenkins.InitReactorRunner$1 onTaskFailed 
SEVERE: Failed Loading plugin s3
java.io.IOException: Dependency copyartifact (1.21) doesn't exist
at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:446)
at hudson.PluginManager$2$1$1.run(PluginManager.java:305)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
at jenkins.model.Jenkins$6.runTask(Jenkins.java:840)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

and

Mar 25, 2012 6:14:14 PM jenkins.InitReactorRunner$1 onTaskFailed  
SEVERE: Failed Initializing plugin s3
java.lang.NullPointerException
at hudson.PluginManager$2$1$2.run(PluginManager.java:322)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
at jenkins.model.Jenkins$6.runTask(Jenkins.java:840)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

I did this in the hope of retaining my existing S3 plugin settings.

I then removed in ~/.hudson/plugin/s3* files and folders and restarted and uploaded the s3.hpi from the advanced tab of the plugin manager in Jenkins. Doing this produced the "Failed Initializing plugin s3" error above.

I removed my ~/.hudson directory and tried again, and I still get the above errors.

I rebuilt the s3 plugin against 1.456 and the same problem continued.

Can you suggest what I should do next?

Brice Figureau

The patch adds the S3 Copy Artifact feature that unfortunately requires to have the Copy Artifact plugin installed. This is I guess the cause of the first stack trace.
Can you try to install this plugin and try again?

We could theoretically remove this dependency, but we'll have to copy or re-implement code from this plugin.

Normally your previous S3 plugin settings should be compatible.

Richard Dallaway
Collaborator
d6y commented March 25, 2012

Thank you - I'll try that.

A thought: is there a way to add the copyartifact code as a library dependency in the POM? That is, use the code from the plugin without having to make the user install a separate plugin? I don't know if that's even possible.

Brice Figureau

It is already in the POM, as I've followed these instructions:
https://wiki.jenkins-ci.org/display/JENKINS/Dependencies+among+plugins

Richard Dallaway
Collaborator
d6y commented March 25, 2012

Thank you - I've installed the plugin, ran a project, and it produced files on S3.

One change is the output on S3 is in a structure based on the project name and build number. In my case it's called: jobs/oth/5/output-1.0-SNAPSHOT.jar

That's a big change from the current plugin which leaves files in the top-level of the bucket you specify. Is there a setting to allow the plugin to produce the older behaviour?

Brice Figureau

No, there is no option to restore the old behavior (it looks like I completely forgot this).
I'll add an option in a subsequent update, but if enabled, the user won't be able to use the S3 Copy Artifact part of the plugin (because we need to separate artifacts produced by different jobs).

Richard Dallaway
Collaborator
d6y commented March 25, 2012

Ok! Sounds good. I'll look out for that update.

Richard Dallaway
Collaborator

Is there any progress on this change set or should we close this one?

Nicolas De loof
Collaborator

incompatible change to bucket layout is a blocker imho. User expect plugin to be backward compatible and this will break the build chain. Should be an option, that S3CopyArtifact enables when used, but not the "classic" publisher (by default)

Brice Figureau

Grr, I missed your last two comments because of the new github notifications.
I do agree that the default should be compatible with previous versions.
I just need to find enough time and motivation to finally fix that :)

Matt

This would really be great to have merged in... Any chance we can motivate you to get it pulled in?

Brice Figureau

That's still on my list of things to do :)

Meanwhile you can build my patched version if you really need the functionality (I'm running it on production without any issues since I proposed this PR).

Jonas Rabbe

If you need motivation, @masterzen then have another vote. The current behavior of the S3 plugin is a huge blocker for us, since we need the hierarchy in our S3 bucket for our releases.

Brice Figureau

@jrabbe I'm motivated :)
I just need to find the time to bring back the backward compatibility.

Meanwhile you can clone the repository, build the plugin (mvn hpi:hpi) and install it manually in your jenkins instance.

We're using it in production to host our artifacts on s3 for more than 2 years without any adverse effect. It is the base of some very complex pipelines where artifacts are stored in s3 and automatically downloaded in the next stage of the pipeline, etc...

David Beer
Collaborator

Hi Guys,

I currently use the s3-plugin as is for deploing finished artifacts and delivering them afterwords via cloudfront or s3. We need to maintain the old capability, if anything it might be better to make the feature a separate plugin that depends on the s3-plugin. So that you have a customised s3-plugin for default artifact storage that does not interfere with standard s3-plugin upload and settings.

David

Brice Figureau

@dmbeer it's easy to restore the original behavior (it's still there in fact), it just lacks the UI.

Brice Figureau

Guys, I finally found the motivation to cleanup and finish this work, mostly clean up and restoring the original behavior of the plugin.

This happened in Pull Request #19.

Therefore I'm closing this one, as it has no object anymore.

Brice Figureau masterzen closed this November 02, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 13 unique commits by 2 authors.

Feb 17, 2012
Brice Figureau s3 plugin works fine with jenkins 1.424
so just say so.

Signed-off-by: Brice Figureau <brice@daysofwonder.com>
731214d
Brice Figureau Upload from the slave instead of relaying through the master
It's very useful when running a master outside of AWS EC2 but
with slaves on EC2.

There is no possibility yet to allow the previous way of
working.

Signed-off-by: Brice Figureau <brice@daysofwonder.com>
7d322c1
Feb 18, 2012
Brice Figureau Allow to upload fromm slave or master
This adds a checkbox in the Publish S3 configuration view of a job, to
either publish to S3 from the slave or the master.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
b5a5e82
Feb 22, 2012
Brice Figureau Record fingerprints of uploaded to S3 artifacts
This also adds an "S3 Artifact" links to the run side panel to have
a direct access to the uploaded artifacts.


Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
247510a
Feb 25, 2012
Brice Figureau Copy Artifacts from S3
This new extension allows to copy artifacts uploaded to S3 by a previous
run to the current build workspace, like the "Copy Artifacts" plugin
does.

It also fingerprint downloaded artifacts to be able to link builds
together.

This extension requires the Copy Artifact plugin to be installed to work
correctly.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
4c77473
Mar 10, 2012
Brice Figureau Refactor: moving the download/upload callable to their own types
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
32b3860
Brice Figureau Refactor: add some factories to Destination
This allows to create a Destination from a run and an S3Artifact.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
6b8b4d0
Brice Figureau Remove the Workspace build selector from the build selector list
This selector has no meaning for S3 artifacts copy.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
6754197
Brice Figureau Respect 'filter' when copying artifact back from S3
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
f29b453
Brice Figureau Delete old artifacts from S3 when the run is deleted
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
e0719aa
Brice Figureau Allow to download artifacts on the fly from S3
* removes the s3url field from FingerprintRecord
* adds an S3ArtifactAction download action that creates a
presigned S3 url for the given artifact

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
af9f79a
Brice Figureau Adds a build section that displays the last succesfull artifacts
Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
7cbe589
Mar 11, 2012
Brice Figureau Remove an empty unneeded index.jelly
This prevents unit test to run and this file isn't used.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
b1ddaa9
Something went wrong with that request. Please try again.