Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JEP-210] Log handling rewrite #15

Merged
merged 55 commits into from Oct 4, 2018

Conversation

@jglick
Copy link
Member

commented Sep 23, 2016

Downstream of jenkinsci/workflow-api-plugin#17. Subsumes #75.

JEP-210

  • basics
  • fix rendering of console output from Pipeline Steps
  • way to efficiently get all output from a branch
  • retain compatibility if only this plugin is updated and not workflow-job
  • optimize AnnotatedLogAction.getLogText
  • optimize AnnotatedLogAction.strip
@jglick jglick referenced this pull request Sep 23, 2016
6 of 11 tasks complete
@jglick

This comment has been minimized.

Copy link
Member Author

commented Sep 23, 2016

Not going to bother deploying upstream snapshots at this stage, so CI failures are expected.

jglick added 10 commits Sep 26, 2016
In practice it seems the type parameter of ConsoleNote must be Object…
… to be usable from LogAction.

(Core-defined notes do use Object, explicitly or implicitly via rawtypes.)
If we are using an old WorkflowRun with copyLogs, fall back to origin…
…al LogActionImpl.

Includes further encapsulation than that started in #10.
Also applying ConsoleLogFilter in new mode.
@svanoort

This comment has been minimized.

Copy link
Member

commented Aug 8, 2018

@jglick Does this fix the issue we discussed months ago where it requires reading the entire logfile to retrieve the fragment for a single step?

@jglick

This comment has been minimized.

Copy link
Member Author

commented Aug 8, 2018

reading the entire logfile to retrieve the fragment for a single step

That has since been refactored into jenkinsci/workflow-api-plugin#17 where it now lives in StreamLogStorage.stepLog. Since streaming a single file is relatively cheap for I/O, and the filtering is simple, that is just left as is to keep things simple. Of course if you are using, e.g., CloudWatch Logs there is a completely different implementation (in that case using server-side filtering).

Rendering speed is anyway not a priority for JEP-210, compared to optimizing writing during the build. (discussion)

* A marker for a node which had some log text.
*/
@Restricted(NoExternalUse.class) // for use from DefaultStepContext only
public class AnnotatedLogAction extends LogAction implements FlowNodeAction, PersistentAction {

This comment has been minimized.

Copy link
@jglick

jglick Aug 8, 2018

Author Member

This should be renamed.

@jglick

This comment has been minimized.

Copy link
Member Author

commented on 842ee5a Aug 9, 2018

jglick added 4 commits Aug 13, 2018
If during an upgrade we are resuming a build with a running step that…
… had used LogAction, it must continue to do so (now teeing to the overall log).

@dwnusbaum dwnusbaum self-requested a review Aug 15, 2018

public static @Nonnull TaskListener listenerFor(@Nonnull FlowNode node, @CheckForNull ConsoleLogFilter filter) throws IOException, InterruptedException {
FlowExecutionOwner owner = node.getExecution().getOwner();
if (LogActionImpl.isOld(owner) || node.getAction(LogActionImpl.class) != null) {
return LogActionImpl.stream(node, filter);

This comment has been minimized.

Copy link
@jglick

jglick Aug 22, 2018

Author Member

Probably the !FlowNode.active closing logic from LogActionImpl should be moved here—for some implementations it may be significant (though not for FileLogStorage). Also rather than PrintStream.close it should check for TaskListener & AutoCloseable. And the expected close behavior for both the overall and step logs should be documented in LogStorage.

This comment has been minimized.

Copy link
@jglick

jglick Aug 23, 2018

Author Member

(resolved)

jglick added 3 commits Aug 23, 2018
Centralizing support for closing step logs in DefaultStepContext, to …
…support LogStorageAction, and handling TaskListener & AutoCloseable.
@Override public void onNewHead(FlowNode newNode) {
if (!node.isActive()) {
node.getExecution().removeListener(this);
LOGGER.log(Level.FINE, "closing log for {0}", node.getDisplayFunctionName());

This comment has been minimized.

Copy link
@svanoort

svanoort Sep 12, 2018

Member

For a good time, try to disentangle what all the xxFunctionName/xxDisplayName function for a FlowNode do and which to use when.

And by a "good time" I mean "if you're a masochist."

@@ -97,6 +99,33 @@
}
}

private synchronized TaskListener getListener() throws IOException, InterruptedException {

This comment has been minimized.

Copy link
@svanoort

svanoort Sep 12, 2018

Member

Synchronizing is /probably/ the right move here but makes me nervous we're going to hit threading issues (as has happened with analogous changes in WorkflowRun with bottlenecks and deadlocks). Nothing I can point to explicitly wrong here, but something to keep a close eye on.

jglick added 2 commits Oct 3, 2018
If there is any output from Run.getEnvironment(TaskListener), and non…
…e is expected, send it to the overall build log not the step log.

This allows us to call context.get(EnvVars.class) even on a FlowNode which has not yet been attached to the graph.
If there is any output from Run.getEnvironment(TaskListener), and non…
…e is expected, send it to the overall build log not the step log.

This allows us to call context.get(EnvVars.class) even on a FlowNode which has not yet been attached to the graph.

(cherry picked from commit d5d1f46)

@jglick jglick merged commit 0dfa35c into jenkinsci:master Oct 4, 2018

2 checks passed

continuous-integration/jenkins/incrementals Deployed to Incrementals.
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details

@jglick jglick deleted the jglick:logs-JENKINS-38381 branch Oct 4, 2018

@@ -62,17 +67,14 @@
if (key == EnvVars.class) {
Run<?,?> run = get(Run.class);
EnvironmentAction a = run == null ? null : run.getAction(EnvironmentAction.class);
EnvVars customEnvironment = a != null ? a.getEnvironment() : run.getEnvironment(get(TaskListener.class));
EnvVars customEnvironment = a != null ? a.getEnvironment() : run.getEnvironment(getExecution().getOwner().getListener());

This comment has been minimized.

Copy link
@jglick

jglick Jan 18, 2019

Author Member

Amended in #62 FTR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.