Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job setting to suppress console logging #94

Merged

Conversation

Projects
None yet
3 participants
@dobbymoodge
Copy link

dobbymoodge commented Nov 9, 2016

This plugin is really noisy when you have loads of assets to archive,
so this adds a toggle to suppress all output to the build console.

Add job setting to suppress console logging
This plugin is really noisy when you have loads of assets to archive,
so this adds a toggle to suppress all output to the build console.
@@ -119,7 +127,9 @@ public static S3Profile getProfile(String profileName) {
}

private void log(final PrintStream logger, final String message) {
logger.println(StringUtils.defaultString(getDescriptor().getDisplayName()) + ' ' + message);
if(! suppressLogging) {

This comment has been minimized.

Copy link
@Jimilian

Jimilian Nov 10, 2016

Sorry, but I don't think that this is a right place for suppressing. Probably, you want to suppress only messages that's related to pushing files to s3 - not all messages (including warning and errors).
So, I can see two solutions:

  • Move this check to line 185/195
  • Introduce three different levels: errors (after such log you can see return statement or setResult(failed)), warning ('no files found', 'skip publishing', etc.) and info (this one you want to skip).

This comment has been minimized.

Copy link
@dobbymoodge

dobbymoodge Nov 10, 2016

Author

I've addressed this in my latest commit. Let me know if you like the implementation.

@Jimilian

This comment has been minimized.

Copy link

Jimilian commented Nov 10, 2016

Thanks for PR, please, check review

Change to loglevel-based filtering
Allow users to control the noisiness of the S3 plugin console output by
using the `java.util.logging.Level` field definitions.
@@ -66,6 +68,7 @@ public S3BucketPublisher(String profileName, List<Entry> entries, List<MetadataP
this.userMetadata = userMetadata;

this.dontWaitForConcurrentBuildCompletion = dontWaitForConcurrentBuildCompletion;
this.consoleLogLevel = Level.parse(consoleLogLevel);

This comment has been minimized.

Copy link
@Jimilian

Jimilian Nov 11, 2016

Unfortunately, parse is executing under synchronized. In highly loaded environment it's not very efficient (we run ~1.5k jobs in parallel). I will think how to do it better.

This comment has been minimized.

Copy link
@dobbymoodge

dobbymoodge Nov 11, 2016

Author

@Jimilian Please check my latest commit to see if that is a more acceptable solution.

This comment has been minimized.

Copy link
@Jimilian

Jimilian Nov 12, 2016

Yes, I was thinking about the same. Thanks you!

@Jimilian Jimilian merged commit 7e39a79 into jenkinsci:master Nov 12, 2016

1 check passed

Jenkins This pull request looks good
Details
@coltrey

This comment has been minimized.

Copy link

coltrey commented on 2efa667 Dec 30, 2016

Changing the constructor means that pipeline scripts that use this plugin in step() are now broken. Correct?

@dobbymoodge

This comment has been minimized.

Copy link
Author

dobbymoodge commented Jan 3, 2017

@coltrey I'm not familiary with how pipeline scripts work, but this is concerning. Do you have a reference you could link? I'm guessing this would be a significant disruption for plugin users, right?

@Jimilian

This comment has been minimized.

Copy link

Jimilian commented Jan 3, 2017

@coltrey, @dobbymoodge I'm not 100% sure as well, but IMHO everything should be ok. In case of step null should be provided to unknown parameters. This situation is addressed already. But anyway any tests for that is highly appreciated.

Unfortunately, I don't have time for testing that right now.

@coltrey

This comment has been minimized.

Copy link

coltrey commented Jan 3, 2017

@dobbymoodge I'm a lousy developer, but here's a an example and a stack trace, and I realize this comment might be on the wrong commit thread, but I think when the constructor was changed, it broke compatibility. Downgrading to 0.10.9 from 0.10.11 solved the issue (also I have more recently confirmed that 0.10.10 was also working). When I was troubleshooting, I pulled up the syntax generator and it looked like some things may've changed...:

This is the 0.10.11 syntax generator example, the generated values for consoleLogLEvel and pluginFailureResultConstraint are one of the things that made me question the constructor change:

step([$class: 'S3BucketPublisher',
   consoleLogLevel: <object of type java.util.logging.Level>,
   dontWaitForConcurrentBuildCompletion: false,
   entries: [[bucket: 'ourbucket',
        excludedFile: '',
        flatten: true,
        gzipFiles: false,
        keepForever: false,
        managedArtifacts: false,
        noUploadOnFailure: false,
        selectedRegion: 'us-east-1',
        showDirectlyInBrowser: false,
        sourceFile: '*.jar',
        storageClass: 'STANDARD',
        uploadFromSlave: false,
        useServerSideEncryption: false]],
   pluginFailureResultConstraint: <object of type hudson.model.Result>,
   profileName: 'jenkins_bucket',
   userMetadata: []
])

This works in 0.10.9, but not 0.10.11:

		step([$class: 'S3BucketPublisher',
			dontWaitForConcurrentBuildCompletion: false,
			entries: [[bucket: 'our_bucket/${JOB_NAME}_${BUILD_NUMBER}',
				excludedFile: '',
				flatten: true,
				gzipFiles: false,
				managedArtifacts: false,
				noUploadOnFailure: true,
				selectedRegion: 'us-east-1',
				sourceFile: '*.jar',
				storageClass: 'STANDARD',
				uploadFromSlave: true,
				useServerSideEncryption: false]],
			profileName: 'jenkins_bucket',
			userMetadata: []
		])

This is the stack trace we get:


java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:258)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.coerce(DescribableModel.java:372)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.buildArguments(DescribableModel.java:313)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:257)
	at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:194)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:181)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
	at WorkflowScript.run(WorkflowScript:70)
	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/opt/jenkins/home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:123)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
	at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.access$000(CollectionLiteralBlock.java:31)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock.eval(CollectionLiteralBlock.java:26)
	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at hudson.plugins.s3.S3BucketPublisher.parseLevel(S3BucketPublisher.java:76)
	at hudson.plugins.s3.S3BucketPublisher.<init>(S3BucketPublisher.java:71)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:258)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.coerce(DescribableModel.java:372)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.buildArguments(DescribableModel.java:313)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:257)
	at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:194)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:181)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
	... 37 more
Finished: FAILURE
@coltrey

This comment has been minimized.

Copy link

coltrey commented Jan 3, 2017

@dobbymoodge @Jimilian
Here's a minimal example for a pipeline script, just set BUCKET to something the "jenkins_bucket" s3 publisher profile is authorized to write.


node() {
    BUCKET="some_test_bucket/path/test/"
    sh 'touch foo'
    
    step([$class: 'S3BucketPublisher',
   dontWaitForConcurrentBuildCompletion: false,
   entries: [[bucket: $BUCKET,
        excludedFile: '',
        flatten: true,
        gzipFiles: false,
        keepForever: false,
        managedArtifacts: false,
        noUploadOnFailure: false,
        selectedRegion: 'us-east-1',
        showDirectlyInBrowser: false,
        sourceFile: 'foo',
        storageClass: 'STANDARD',
        uploadFromSlave: false,
        useServerSideEncryption: false]],
   profileName: 'jenkins_bucket',
   userMetadata: []
])

}
@dobbymoodge

This comment has been minimized.

Copy link
Author

dobbymoodge commented Jan 3, 2017

@coltrey This looks legit. Can you file an issue so I can reference it in the fix PR?

@coltrey

This comment has been minimized.

Copy link

coltrey commented Jan 3, 2017

@dobbymoodge JENKINS-40786

@Jimilian

This comment has been minimized.

Copy link

Jimilian commented Jan 3, 2017

@dobbymoodge it should be enough to create second constructor with old signature and set new variable to same value as in readResolve method to fix old pipeline scripts.
My assumption was correct. Pipeline script provides null to constructor, but I didn't fix that case in this commit: ab3e574
I fixed only old config files (xml deserialization doesn't use constructor - it loads fields directly and calls readResolve).

Caused by: java.lang.NullPointerException
	at hudson.plugins.s3.S3BucketPublisher.parseLevel(S3BucketPublisher.java:76)

So, it's better to introduce several not null checks...

@coltrey

This comment has been minimized.

Copy link

coltrey commented Jan 3, 2017

What about compatibility with the snippet generator? The 0.10.11 release will generate invalid snippets (ex: consoleLogLevel: <object of type java.util.logging.Level>")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.