[BEAM-9242] No longer reporting Lulls as errors in the worker.

[pabloem]

These were misleading users to think their pipeline was having actual errors when it was in fact performing fine.

Furthermore, this reporting had been added to ensure users would be able to notice these logs. With the new Dataflow Job Details page, there is a Worker Logs section that shows logs from all workers, and will expose these logs as warnings rather than errors.

r: @boyuanzz

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---
XLang	---	---	---		---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[boyuanzz]

Thanks Pablo!

[boyuanzz]

Can we log these info as warning for debugging purpose?

[boyuanzz]

Also could you please add a JIRA issue to track this improvement?

[pabloem]

This removes the streaming subclass implementation that logs as a Dataflow system error. The Dataflow implementation already logs this as a warning log coming from the worker:

beam/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java

Lines 252 to 268 in b5737cb

	@Override
	public void reportLull(Thread trackedThread, long millis) {
	// If we're not logging warnings, nothing to report.
	if (!LOG.isWarnEnabled()) {
	return;
	}
	// Since the lull reporting executes in the sampler thread, it won't automatically inherit the
	// context of the current step. To ensure things are logged correctly, we get the currently
	// registered DataflowWorkerLoggingHandler and log directly in the desired context.
	LogRecord logRecord =
	new LogRecord(Level.WARNING, getLullMessage(trackedThread, Duration.millis(millis)));
	logRecord.setLoggerName(DataflowOperationContext.LOG.getName());
	// Publish directly in the context of this specific ExecutionState.
	DataflowWorkerLoggingInitializer.getLoggingHandler().publish(this, logRecord);
	}

Also filed BEAM-9242 : )

[boyuanzz]

Thanks Pablo! Just curious are we going to have the similar change in batch, or we are just ok now?

[pabloem]

thanks @boyuanzz - Batch does not have the speciall handling that streaming has. Now Batch/Streaming work the same.