logs are incorrectly elevated to error level

[snallapa]

When upgrading Beam from 2.9 to 2.10 and up there seems to be a logging issue that arises. Any log is elevated to error level, which floods the logs for a job and can cost money. This seems to be caused by https://cloud.google.com/dataflow/docs/support/sdk-version-support-status as in the 2.10 versions and up there seem to be issues with the dataflow runner and logging. The fix mentioned in that link resolved the issue for us, namely we added an exclusion rule to the build.sbt for the original slf4j and included the slf4j-jdk14 library instead. The logging worked fine after that change

[nevillelyh]

Which Scio version are you on? Looks like this issue affects Beam 2.10 to 2.12, which affects Scio 0.7.2-0.7.4. Latest master (0.8) is on Beam 2.13 already. So we either have to do patch release of the 0.7 branch, or roll forward and fix it for 0.8.

@regadas FYI, we should probably fix dependencies for the internal 0.7 sbt settings.

[snallapa]

We are on 0.7.4 but I think I saw it happen on 0.7.3 as well

[katielee-joinhoney]

I am experiencing the same issue with version 0.7.4

[nevillelyh]

@katielee-joinhoney it should be fixable by excluding slf4j-* bindings and include slf4j-jdk14 only?

[katielee-joinhoney]

yes, it worked. Thanks!

[regadas]

with beam >= 2.13.0 this should be a non issue and Dataflow should have the right log config in place https://cloud.google.com/dataflow/docs/support/sdk-version-support-status

[nevillelyh]

Yeah let's close this.