Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-30424] Move build.save to the migration thread #8

Merged
merged 5 commits into from Oct 23, 2015
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/main/java/org/jenkinsci/plugins/uniqueid/impl/Id.java
Expand Up @@ -71,7 +71,7 @@ public void onAttached(Run<?, ?> r) {
// NO-OP
}

/**
/**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could revert.

* Migrates the run away from using this Action.
*/
public void onLoad(Run<?, ?> r) {
Expand Down
Expand Up @@ -36,8 +36,8 @@ public class IdStoreMigratorV1ToV2 {

private static Logger LOGGER = Logger.getLogger(IdStoreMigratorV1ToV2.class.getName());

private static final String MARKER_FILE_NAME = "unique-id-migration.txt";
/* package */ static final String MARKER_FILE_NAME = "unique-id-migration.txt";

/**
* Migrates any IDs stored in Folder/Job/Run configuration
* @throws IOException
Expand Down Expand Up @@ -136,8 +136,16 @@ public void run() {
// touch something in the build just to force loading incase it gets more lazy in the future.
Object r = iterator.next();
if (r != null && r instanceof Run) {
((Run)r).getAllActions();
Run run = ((Run) r);
run.getAllActions();
try {
// Save the run here so its storage is updated (after being migrated in Id.onLoad(Run))
run.save();
} catch (IOException e) {
LOGGER.log(Level.WARNING, "Can not save build {0} on job {1}", new Object[] { run.getNumber(), job.getName() });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about including the IOException in the log?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not like to pollute the logs with stacktraces, since if this migration fails it will be addressed later by Id.onLoad on demand.
Moreover, all Level.WARNING logs in this plugin are not printing stacktraces, I don't want to break predefined style.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐜 This exception would be unexpected and logging it would probably be wise.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other loggers should log the message. If not please raise another jira

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

}
}

migratedBuilds++;
}
}
Expand Down
Expand Up @@ -4,10 +4,13 @@
import hudson.model.Action;
import hudson.model.Actionable;
import hudson.model.Run;
import jenkins.model.Jenkins;

import java.io.File;
import java.io.IOException;
import java.util.Iterator;
import java.util.List;
import java.util.logging.Level;

import org.kohsuke.accmod.Restricted;
import org.kohsuke.accmod.restrictions.NoExternalUse;
Expand All @@ -30,12 +33,26 @@ public void remove(Run run) throws IOException {
List<Id> ids = run.getActions(Id.class);
if (!ids.isEmpty()) {
actionList.removeAll(ids);
run.save();
saveIfNeeded(run);
}
}

@Override
public String get(Run thing) {
return Id.getId((Actionable) thing);
}

private void saveIfNeeded(Run run) throws IOException {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method name seems clear, but a javadoc will be nice.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's private so Javadocs are not really useful. There are some inline comments inside for developers.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not know this measure to add javadoc

Jenkins jenkins = Jenkins.getInstance();
if (jenkins == null) {
throw new IllegalStateException("Jenkins is null, so it is impossible to migrate the ID");
}
File marker = new File(jenkins.getRootDir(), IdStoreMigratorV1ToV2.MARKER_FILE_NAME);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probaly best as a static method on the migrator class

if (marker.exists()) {
// The background migration thread already finished (all builds have been already loaded at least once),
// so this is a belated load on a run that was not migrated in the main process (for some reason).
// Let's save it now.
run.save();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't r this have the same potential to blow up?

Prehaps scedule a timertask to persist with the id of the run?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. This call to save is only reached after the migration thread finished, so at that time all builds have been loaded at least once (then there is no concurrency risk with Run.onLoad())

}
}
}
Expand Up @@ -43,7 +43,7 @@ public class IdStoreMigratorV1ToV2Test {
public void testMigration() throws Exception {
Jenkins jenkins = jenkinsRule.jenkins;
assertThat("All jobs loaded correctly", jenkins.getAllItems(), hasSize(4));

Folder folderNoID = jenkins.getItem("folderNoID", jenkins, Folder.class);
Folder folderWithID = jenkins.getItem("folderWithID", jenkins, Folder.class);

Expand All @@ -54,7 +54,7 @@ public void testMigration() throws Exception {
assertThat(folderWithID, notNullValue());
assertThat(jobNoID, notNullValue());
assertThat(jobWithID, notNullValue());

checkID(folderNoID, null);
checkID(folderWithID, "YzUxN2JiZTYtNGVhZS00NDQxLTg5NT");

Expand All @@ -66,10 +66,24 @@ public void testMigration() throws Exception {
checkID(jobNoID.getBuildByNumber(2), null);

// build 1 had no id so its generated on the fly from the parent
checkID(jobWithID.getBuildByNumber(1), "ZGQxMDNhYzUtMTJlOC00YTc4LTgzOT_" + jobWithID.getBuildByNumber(1).getId());
checkID(jobWithID.getBuildByNumber(1), "ZGQxMDNhYzUtMTJlOC00YTc4LTgzOT_" + jobWithID.getBuildByNumber(1).getId());
checkID(jobWithID.getBuildByNumber(2), "NGQ0ODM2NjktZGM0OS00MjdkLWE3NT");
}


@Test
@Issue("JENKINS-30424")
@LocalData
public void testMigrationLongLoadingBuilds() throws Exception {
Jenkins jenkins = jenkinsRule.jenkins;
assertThat("All jobs loaded correctly", jenkins.getAllItems(), hasSize(2));

Folder folderWithID = jenkins.getItem("folderWithID", jenkins, Folder.class);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the folder needed for this test?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. But it's harmless and I saved some time by re-using the existing local data zip.

Job jobWithID = jenkins.getItem("jobWithID", (ItemGroup)folderWithID, Job.class);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jenkins.getItemByFullName("folderWithID/jobWithID", Job.class);

// Build #2 has a LongLoadingAction which simulates an action that blocks in the onLoad method
// This kind on delays causes issues on startup with version 2.1.0 (see issue linked to this test)
checkID(jobWithID.getBuildByNumber(2), "NGQ0ODM2NjktZGM0OS00MjdkLWE3NT");
}

@Test
@Issue("JENKINS-28883")
public void migrateUnsupportedType() throws Exception {
Expand Down
@@ -0,0 +1,31 @@
package org.jenkinsci.plugins.uniqueid.impl;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing copyright

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no class in this plugin with copyright header... what should I use?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Standard cloudbees MIT if you have done this on company time. I must have onitted the copyright on the earlier files 😞

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.


import hudson.model.Run;
import jenkins.model.RunAction2;

public class LongLoadingAction implements RunAction2 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this action neexed or can the onload in the job do the same?0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not strictly needed but it reproduces tho original issue in a wider range (not all Run implementations override onLoad, but the issue is potentially there if some RunAction2 takes some time to load).


public String getIconFileName() {
return null;
}

public String getDisplayName() {
return null;
}

public String getUrlName() {
return null;
}

public void onAttached(Run<?, ?> r) {
}

public void onLoad(Run<?, ?> r) {
try {
Thread.sleep(1000);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sleeping in tests isn't generally a good idea, ca. A wait/semaphore be used?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't think about it. It would be far more complex than this.
What's exactly wrong with Thread.sleep here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see how sleeping indefinitely (inside a loop) is a bad thing, but this seems fine to me, even if I don't know why it's there (mimic a load delay or something?).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's simulating a load delay (which is what causes the original issue actually).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, having looked at the test code I can now see why you're using the 1s delay. I can see now that this could (potentially) cause some flaky test behaviour, but I guess @amuniz is the one to decide if that's realistically possible or not. If it is possible it should be easy to fix by adding a flag on LongLoadingAction that gets toggled once the test is where it needs to be, telling LongLoadingAction.onLoad that it's ok to finish now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is what happens if the machine running the test is heavily loaded. If that causes a test failure when the code is good, that is a problem. If it just causes the test to pass even when the code regressed, that is OK, since this will not usually happen and we will find the failure soon.

} catch (InterruptedException e) {
e.printStackTrace();
}
}

}
@@ -0,0 +1,37 @@
package org.jenkinsci.plugins.uniqueid.impl;

import java.io.File;
import java.io.IOException;

import hudson.model.Build;

public class LongLoadingBuild extends Build<LongLoadingProject, LongLoadingBuild> {

private String check = null;

public LongLoadingBuild(LongLoadingProject project) throws IOException {
super(project);
}

public LongLoadingBuild(LongLoadingProject project, File buildDir) throws IOException {
super(project, buildDir);
}

@Override
public synchronized void save() throws IOException {
// Will produce a null pointer exception if called before onLoad finished.
check.toString();
super.save();
}

@Override
protected void onLoad() {
super.onLoad();
check = "Checked!";
}

@Override
public void run() {
}

}
@@ -0,0 +1,36 @@
package org.jenkinsci.plugins.uniqueid.impl;

import org.jvnet.hudson.test.TestExtension;
import hudson.model.ItemGroup;
import hudson.model.Messages;
import hudson.model.Project;
import hudson.model.TopLevelItem;
import hudson.model.TopLevelItemDescriptor;
import jenkins.model.Jenkins;

public class LongLoadingProject extends Project<LongLoadingProject, LongLoadingBuild> implements TopLevelItem {

public LongLoadingProject(ItemGroup parent, String name) {
super(parent, name);
}

@Override
protected Class<LongLoadingBuild> getBuildClass() {
return LongLoadingBuild.class;
}

@TestExtension
public static final class DescriptorImpl extends TopLevelItemDescriptor {
public String getDisplayName() {
return Messages.FreeStyleProject_DisplayName();
}

public LongLoadingProject newInstance(ItemGroup parent, String name) {
return new LongLoadingProject(parent,name);
}
}

public TopLevelItemDescriptor getDescriptor() {
return (DescriptorImpl) Jenkins.getInstance().getDescriptorOrDie(LongLoadingProject.class);
}
}
Binary file not shown.