Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block scanning APIs #10

Merged
merged 52 commits into from Aug 24, 2016
Merged
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
10e2eb4
For graph analysis add a FlowChunk object to use in analysis APIs
svanoort Jun 7, 2016
389543b
For flow scanning, add an in-memory representation of timed blocks
svanoort Jun 8, 2016
bd426b3
Add a parallel flow chunk for duration computation
svanoort Jun 8, 2016
2efcba4
Define interface for chunk storage
svanoort Jun 9, 2016
8ebbdc9
Save simple block visitor
svanoort Jun 10, 2016
f97a08f
Basic forkscanner block API
svanoort Jun 10, 2016
13ef687
Add status handling and memory storage engine to graph analysis
svanoort Jun 10, 2016
094cb36
Carve out advanced visitor methods
svanoort Jun 10, 2016
2116580
Advanced visitor for carving up flows (skeletal impl) and parallel ha…
svanoort Jun 10, 2016
698d0d3
Chunk and block storage types
svanoort Jul 24, 2016
b8ac909
Merge branch 'master' into block-scanning-APIs
svanoort Jul 24, 2016
64ec619
Customize SNAPSHOT version so we can distinguish this from master sna…
svanoort Jul 24, 2016
9efad20
Remove all the parts not needed for raw block scanning
svanoort Aug 5, 2016
1338b64
Formalize the block visitor API with parallels and branches yet again
svanoort Aug 5, 2016
6c4b8e3
Harden ChunkVisitor and start a FlowChunker
svanoort Aug 5, 2016
0237e96
Forkscanner now tracks state that shows what kind of nodes we have (a…
svanoort Aug 8, 2016
abdf296
When splitting chunks you need to know previous node for inclusive vs…
svanoort Aug 10, 2016
a955451
Add some peeking methods to ForkScanner and also a few small access/f…
svanoort Aug 10, 2016
6eb3705
Finish the internal SimpleChunkVisitor iteration logic in ForkScanner…
svanoort Aug 10, 2016
ac2609d
Remove the useless peek methods
svanoort Aug 10, 2016
e8e6203
Fix test failure
svanoort Aug 10, 2016
b5c2b9d
More small fixes
svanoort Aug 10, 2016
866e21d
Test visitor and tests for visitor iteration in ForkScanner
svanoort Aug 11, 2016
cd6e3dd
Soften requirements for MemoryFlowChunk
svanoort Aug 11, 2016
84656e5
Remove unused StandardSimpleChunkVisitor for now
svanoort Aug 11, 2016
55dc020
Add pause timing to MemoryFlowChunk
svanoort Aug 12, 2016
4af21f5
Fixes and test fixes for simple visitor parallels
svanoort Aug 12, 2016
1627155
Add standard visitor and convenience methods for forkscanner/simpleCh…
svanoort Aug 12, 2016
e1988cd
Cleanup for reviews
svanoort Aug 12, 2016
c9f2444
Merge branch 'generic-flow-analyzer' into block-scanning-APIs
svanoort Aug 12, 2016
a53b13e
Remove dangling file
svanoort Aug 12, 2016
aa18ee5
Oh yeah I meant to change the versioning
svanoort Aug 12, 2016
f22aba6
Better reset the pause time for StandardChunkVisitor
svanoort Aug 12, 2016
9880dc8
Rename finder and add the stagefinder that I removed earlier
svanoort Aug 12, 2016
700d009
Remove unused BlockVisitor from early prototypes
svanoort Aug 12, 2016
586979c
Fix minor oopsy
svanoort Aug 12, 2016
7bb581c
Remove StageChunkFinder because BlockScopedStages need some additiona…
svanoort Aug 15, 2016
7b9faca
Cleanup author annotation misformatting for javadoc
svanoort Aug 17, 2016
ae47cc2
Make all paragraph tags in javadocs not self-closing to placate javadocs
svanoort Aug 17, 2016
d21d67a
Placate more javadocs complaints
svanoort Aug 17, 2016
1678b7d
Fix more javadocs complaints and silence doclint nonsense
svanoort Aug 17, 2016
2030360
More exhaustive tests for SimpleVisitor and fix a pair of swapped arg…
svanoort Aug 19, 2016
6b6eba7
Address review comments from @oleg-nenashev
svanoort Aug 19, 2016
33e079e
Fix an off-by-one case in ForkScanner with parallels and add more ext…
svanoort Aug 21, 2016
56e42eb
Restrive access to some of ForkScanner internals
svanoort Aug 21, 2016
94e9169
Remove unused totalBranches field in ForkScanner
svanoort Aug 21, 2016
6cd4215
Address review comments from @oleg-nenashev
svanoort Aug 23, 2016
d332d8c
Re-enable linting just to see how much pain there is
svanoort Aug 23, 2016
9067f00
Javadocs update
svanoort Aug 24, 2016
8ed03ee
Fix more JavaDocs formatting due to DocLint being a jerk.
svanoort Aug 24, 2016
80b7ad4
Fix yet more doclint whining
svanoort Aug 24, 2016
cf4d5ac
OMG doclint really
svanoort Aug 24, 2016
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

Fix an off-by-one case in ForkScanner with parallels and add more ext…

…ensive test coverage
  • Loading branch information...
svanoort committed Aug 21, 2016
commit 33e079e1e9b161e58cfdab4d224c1ee036117e89
@@ -149,13 +149,12 @@ public boolean isWalkingFromFinish() {
/** Tracks state for parallel blocks, so we can ensure all are visited and know the branch starting point */
protected static class ParallelBlockStart {
protected BlockStartNode forkStart; // This is the node with child branches
protected int remainingBranches;
protected int totalBranches;
protected ArrayDeque<FlowNode> unvisited = new ArrayDeque<FlowNode>(); // Remaining branches of this that we have have not visited yet

protected ParallelBlockStart(BlockStartNode forkStart, int branchCount) {
this.forkStart = forkStart;
this.remainingBranches = branchCount;
this.totalBranches = branchCount;
}

/** Strictly for internal use in the least common ancestor problem */
@@ -248,7 +247,6 @@ public Fork(BlockStartNode forkNode) {
ParallelBlockStart start = new ParallelBlockStart();
start.totalBranches = f.following.size();
start.forkStart = f.forkStart;
start.remainingBranches = start.totalBranches;
start.unvisited = new ArrayDeque<FlowNode>();

// Add the nodes to the parallel starts here
@@ -378,7 +376,6 @@ protected void setHeads(@Nonnull Collection<FlowNode> heads) {
myCurrent = currentParallelStart.unvisited.pop();
myNext = myCurrent;
nextType = NodeType.PARALLEL_BRANCH_END;
currentParallelStart.remainingBranches--;
walkingFromFinish = false;
} else {
FlowNode f = heads.iterator().next();
@@ -432,7 +429,6 @@ protected FlowNode hitParallelEnd(BlockEndNode endNode, List<FlowNode> parents,
ParallelBlockStart parallelBlockStart = new ParallelBlockStart(start, branches.size());
output = branches.pop();
parallelBlockStart.totalBranches = parents.size();
parallelBlockStart.remainingBranches--;
parallelBlockStart.unvisited = branches;

if (currentParallelStart != null) {
@@ -452,7 +448,7 @@ protected FlowNode hitParallelStart() {
FlowNode output = null;

if (currentParallelStart != null) {
if ((currentParallelStart.remainingBranches--) <= 0) { // Strip off a completed branch
if (currentParallelStart.unvisited.isEmpty()) { // Strip off a completed branch
// We finished a nested set of parallel branches, visit the head and move up a level
output = currentParallelStartNode;

@@ -522,7 +518,6 @@ protected FlowNode next(@Nonnull FlowNode current, @Nonnull Collection<FlowNode>
if (currentParallelStart != null && currentParallelStart.unvisited.size() > 0) {
output = currentParallelStart.unvisited.pop();
nextType = NodeType.PARALLEL_BRANCH_END;
currentParallelStart.remainingBranches--;
}
if (output == null) {
nextType = null;
@@ -33,6 +33,7 @@
import org.jenkinsci.plugins.workflow.cps.steps.ParallelStep;
import org.jenkinsci.plugins.workflow.flow.FlowExecution;
import org.jenkinsci.plugins.workflow.graph.BlockStartNode;
import org.jenkinsci.plugins.workflow.graph.FlowGraphWalker;
import org.jenkinsci.plugins.workflow.graph.FlowNode;
import org.jenkinsci.plugins.workflow.job.WorkflowJob;
import org.jenkinsci.plugins.workflow.job.WorkflowRun;
@@ -48,6 +49,7 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.List;
@@ -201,7 +203,6 @@ public void testForkedScanner() throws Exception {

ForkScanner.ParallelBlockStart start = scanner.currentParallelStart;
Assert.assertEquals(2, start.totalBranches);
Assert.assertEquals(1, start.remainingBranches);
Assert.assertEquals(1, start.unvisited.size());
Assert.assertEquals(exec.getNode("4"), start.forkStart);

@@ -330,7 +331,6 @@ public void testLeastCommonAncestor() throws Exception {
ForkScanner.ParallelBlockStart start = starts.peek();
Assert.assertEquals(2, start.totalBranches);
Assert.assertEquals(2, start.unvisited.size());
Assert.assertEquals(2, start.remainingBranches);
Assert.assertEquals(exec.getNode("4"), start.forkStart);
Assert.assertArrayEquals(heads.toArray(), start.unvisited.toArray());

@@ -344,12 +344,10 @@ public void testLeastCommonAncestor() throws Exception {
ForkScanner.ParallelBlockStart inner = starts.getFirst();
ForkScanner.ParallelBlockStart outer = starts.getLast();

Assert.assertEquals(2, inner.remainingBranches);
Assert.assertEquals(2, inner.totalBranches);
Assert.assertEquals(2, inner.unvisited.size());
Assert.assertEquals(exec.getNode("12"), inner.forkStart);

Assert.assertEquals(2, outer.remainingBranches);
Assert.assertEquals(2, outer.totalBranches);
Assert.assertEquals(1, outer.unvisited.size());
Assert.assertEquals(exec.getNode("9"), outer.unvisited.peek());
@@ -424,4 +422,55 @@ public boolean apply(TestVisitor.CallEntry input) {
new TestVisitor.CallEntry(TestVisitor.CallType.PARALLEL_START, 4, 7).assertEquals(parallelCalls.get(5));

}

/** Checks for off-by one cases with multiple parallel */
@Test
public void testTripleParallel() throws Exception {
WorkflowJob job = r.jenkins.createProject(WorkflowJob.class, "TripleParallel");
job.setDefinition(new CpsFlowDefinition(
"stage 'test'\n"+ // Id 3, Id 2 before that has the FlowStartNode
"parallel 'unit':{\n" + // Id 4 starts parallel, Id 7 is the block start for the unit branch
" echo \"Unit testing...\"\n" + // Id 10
"},'integration':{\n" + // Id 11 is unit branch end, Id 8 is the branch start for integration branch
" echo \"Integration testing...\"\n" + // Id 12
"}, 'ui':{\n" + // Id 13 in integration branch end, Id 9 is branch start for UI branch
" echo \"UI testing...\"\n" + // Id 14
"}" // Node 15 is UI branch end node, Node 16 is Parallel End node, Node 17 is FlowWendNode
));
WorkflowRun b = r.assertBuildStatusSuccess(job.scheduleBuild2(0));

ForkScanner.setParallelStartPredicate(PARALLEL_START_PREDICATE);
FlowExecution exec = b.getExecution();
ForkScanner f = new ForkScanner();
f.setup(exec.getCurrentHeads());
TestVisitor visitor = new TestVisitor();
f.visitSimpleChunks(visitor, new BlockChunkFinder());

ArrayList<TestVisitor.CallEntry> parallels = Lists.newArrayList(Iterables.filter(visitor.calls,
Predicates.or(
predicateForCallEntryType(TestVisitor.CallType.PARALLEL_BRANCH_START),
predicateForCallEntryType(TestVisitor.CallType.PARALLEL_BRANCH_END))
)
);
Assert.assertEquals(6, parallels.size());

// Visiting from partially completed branches
// Verify we still get appropriate parallels callbacks for a branch end
// even if in-progress and no explicit end node
ArrayList<FlowNode> ends = new ArrayList<FlowNode>();
ends.add(exec.getNode("11"));
ends.add(exec.getNode("12"));
ends.add(exec.getNode("14"));
visitor = new TestVisitor();
f.setup(ends);
f.visitSimpleChunks(visitor, new BlockChunkFinder());
parallels = Lists.newArrayList(Iterables.filter(visitor.calls,
Predicates.or(
predicateForCallEntryType(TestVisitor.CallType.PARALLEL_BRANCH_START),
predicateForCallEntryType(TestVisitor.CallType.PARALLEL_BRANCH_END))
)
);
Assert.assertEquals(6, parallels.size());
Assert.assertEquals(17, visitor.calls.size());
}
}
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.