Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job setting to suppress console logging #94

Merged

Conversation

dobbymoodge
Copy link

This plugin is really noisy when you have loads of assets to archive,
so this adds a toggle to suppress all output to the build console.

This plugin is really noisy when you have loads of assets to archive,
so this adds a toggle to suppress all output to the build console.
@@ -119,7 +127,9 @@ public static S3Profile getProfile(String profileName) {
}

private void log(final PrintStream logger, final String message) {
logger.println(StringUtils.defaultString(getDescriptor().getDisplayName()) + ' ' + message);
if(! suppressLogging) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but I don't think that this is a right place for suppressing. Probably, you want to suppress only messages that's related to pushing files to s3 - not all messages (including warning and errors).
So, I can see two solutions:

  • Move this check to line 185/195
  • Introduce three different levels: errors (after such log you can see return statement or setResult(failed)), warning ('no files found', 'skip publishing', etc.) and info (this one you want to skip).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've addressed this in my latest commit. Let me know if you like the implementation.

@Jimilian
Copy link

Thanks for PR, please, check review

Allow users to control the noisiness of the S3 plugin console output by
using the `java.util.logging.Level` field definitions.
@@ -66,6 +68,7 @@ public S3BucketPublisher(String profileName, List<Entry> entries, List<MetadataP
this.userMetadata = userMetadata;

this.dontWaitForConcurrentBuildCompletion = dontWaitForConcurrentBuildCompletion;
this.consoleLogLevel = Level.parse(consoleLogLevel);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, parse is executing under synchronized. In highly loaded environment it's not very efficient (we run ~1.5k jobs in parallel). I will think how to do it better.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jimilian Please check my latest commit to see if that is a more acceptable solution.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was thinking about the same. Thanks you!

@Jimilian Jimilian merged commit 7e39a79 into jenkinsci:master Nov 12, 2016
@dobbymoodge
Copy link
Author

@coltrey I'm not familiary with how pipeline scripts work, but this is concerning. Do you have a reference you could link? I'm guessing this would be a significant disruption for plugin users, right?

@Jimilian
Copy link

Jimilian commented Jan 3, 2017

@coltrey, @dobbymoodge I'm not 100% sure as well, but IMHO everything should be ok. In case of step null should be provided to unknown parameters. This situation is addressed already. But anyway any tests for that is highly appreciated.

Unfortunately, I don't have time for testing that right now.

@coltrey
Copy link

coltrey commented Jan 3, 2017

@dobbymoodge I'm a lousy developer, but here's a an example and a stack trace, and I realize this comment might be on the wrong commit thread, but I think when the constructor was changed, it broke compatibility. Downgrading to 0.10.9 from 0.10.11 solved the issue (also I have more recently confirmed that 0.10.10 was also working). When I was troubleshooting, I pulled up the syntax generator and it looked like some things may've changed...:

This is the 0.10.11 syntax generator example, the generated values for consoleLogLEvel and pluginFailureResultConstraint are one of the things that made me question the constructor change:

step([$class: 'S3BucketPublisher',
   consoleLogLevel: <object of type java.util.logging.Level>,
   dontWaitForConcurrentBuildCompletion: false,
   entries: [[bucket: 'ourbucket',
        excludedFile: '',
        flatten: true,
        gzipFiles: false,
        keepForever: false,
        managedArtifacts: false,
        noUploadOnFailure: false,
        selectedRegion: 'us-east-1',
        showDirectlyInBrowser: false,
        sourceFile: '*.jar',
        storageClass: 'STANDARD',
        uploadFromSlave: false,
        useServerSideEncryption: false]],
   pluginFailureResultConstraint: <object of type hudson.model.Result>,
   profileName: 'jenkins_bucket',
   userMetadata: []
])

This works in 0.10.9, but not 0.10.11:

		step([$class: 'S3BucketPublisher',
			dontWaitForConcurrentBuildCompletion: false,
			entries: [[bucket: 'our_bucket/${JOB_NAME}_${BUILD_NUMBER}',
				excludedFile: '',
				flatten: true,
				gzipFiles: false,
				managedArtifacts: false,
				noUploadOnFailure: true,
				selectedRegion: 'us-east-1',
				sourceFile: '*.jar',
				storageClass: 'STANDARD',
				uploadFromSlave: true,
				useServerSideEncryption: false]],
			profileName: 'jenkins_bucket',
			userMetadata: []
		])

This is the stack trace we get:


java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:258)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.coerce(DescribableModel.java:372)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.buildArguments(DescribableModel.java:313)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:257)
	at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:194)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:181)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
	at WorkflowScript.run(WorkflowScript:70)
	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/opt/jenkins/home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:123)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
	at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.access$000(CollectionLiteralBlock.java:31)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock.eval(CollectionLiteralBlock.java:26)
	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at hudson.plugins.s3.S3BucketPublisher.parseLevel(S3BucketPublisher.java:76)
	at hudson.plugins.s3.S3BucketPublisher.<init>(S3BucketPublisher.java:71)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:258)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.coerce(DescribableModel.java:372)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.buildArguments(DescribableModel.java:313)
	at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:257)
	at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:194)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:181)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
	... 37 more
Finished: FAILURE

@coltrey
Copy link

coltrey commented Jan 3, 2017

@dobbymoodge @Jimilian
Here's a minimal example for a pipeline script, just set BUCKET to something the "jenkins_bucket" s3 publisher profile is authorized to write.


node() {
    BUCKET="some_test_bucket/path/test/"
    sh 'touch foo'
    
    step([$class: 'S3BucketPublisher',
   dontWaitForConcurrentBuildCompletion: false,
   entries: [[bucket: $BUCKET,
        excludedFile: '',
        flatten: true,
        gzipFiles: false,
        keepForever: false,
        managedArtifacts: false,
        noUploadOnFailure: false,
        selectedRegion: 'us-east-1',
        showDirectlyInBrowser: false,
        sourceFile: 'foo',
        storageClass: 'STANDARD',
        uploadFromSlave: false,
        useServerSideEncryption: false]],
   profileName: 'jenkins_bucket',
   userMetadata: []
])

}

@dobbymoodge
Copy link
Author

@coltrey This looks legit. Can you file an issue so I can reference it in the fix PR?

@coltrey
Copy link

coltrey commented Jan 3, 2017

@dobbymoodge JENKINS-40786

@Jimilian
Copy link

Jimilian commented Jan 3, 2017

@dobbymoodge it should be enough to create second constructor with old signature and set new variable to same value as in readResolve method to fix old pipeline scripts.
My assumption was correct. Pipeline script provides null to constructor, but I didn't fix that case in this commit: ab3e574
I fixed only old config files (xml deserialization doesn't use constructor - it loads fields directly and calls readResolve).

Caused by: java.lang.NullPointerException
	at hudson.plugins.s3.S3BucketPublisher.parseLevel(S3BucketPublisher.java:76)

So, it's better to introduce several not null checks...

@coltrey
Copy link

coltrey commented Jan 3, 2017

What about compatibility with the snippet generator? The 0.10.11 release will generate invalid snippets (ex: consoleLogLevel: <object of type java.util.logging.Level>")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants