[BEAM-4176] move some artifact-staging logs from info to debug

[ryan-williams]

these were generating a somewhat-inordinate-seeming amount of log-spam (cf. example VR test output)

R: @angoenka

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---	---	---	---	---
Java
Python		---		---	---	---	---

[angoenka]

LGTM
Please make sure that we have an info log on the sdk side listing staged artifacts so that the user can see which files are staged.

[ryan-williams]

Finally coming back to this; I tried adding an SDK-side log as you suggested, but it feels equivalently spammy to me at first glance:

do you think it's important to INFO-log every artifact? I'd have said these should be DEBUGs on both ends

perhaps two INFOs saying "staging X artifacts" / "staged X artifacts"?

[angoenka]

Yes, they seem to be a bit spammy.
Seems reasonable. Lets move them to debug.

[lukecwik]

I suggest you follow the same pattern like Dataflow has been doing:

beam/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java

Line 315 in fef40fc

LOG.info(

[angoenka]

Thanks Ryan.
LGTM
Please squash the commits

[ryan-williams]

several tests failed that I don't think are related to my changes here:

:beam-sdks-java-io-hadoop-input-format:test:

19:51:52     localhost/127.0.0.1:7000 is in use by another process.  Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services
19:51:52     Fatal configuration error; unable to start server.  See log for stacktrace.
19:51:52     Sep 18, 2018 11:51:51 PM org.apache.cassandra.service.CassandraDaemon exitOrFail
19:51:52     SEVERE: Fatal configuration error
19:51:52     org.apache.cassandra.exceptions.ConfigurationException: localhost/127.0.0.1:7000 is in use by another process.  Change listen_address:storage_port in cassandra.yaml to values that do not conflict with other services
19:51:52     	at org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:557)
19:51:52     	at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:501)
19:51:52     	at org.apache.cassandra.net.MessagingService.listen(MessagingService.java:485)
19:51:52     	at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:745)

:beam-sdks-java-io-elasticsearch-tests-2:test:

19:45:44 org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest > testWriteWithIndexFn FAILED
19:45:44     java.lang.AssertionError: Einstein index holds incorrect count expected:<10> but was:<20>
19:45:44         at org.junit.Assert.fail(Assert.java:88)
19:45:44         at org.junit.Assert.failNotEquals(Assert.java:834)
19:45:44         at org.junit.Assert.assertEquals(Assert.java:645)
19:45:44         at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestCommon.testWriteWithIndexFn(ElasticsearchIOTestCommon.java:439)
19:45:44         at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithIndexFn(ElasticsearchIOTest.java:175)

:beam-sdks-java-io-amazon-web-services:test:

19:43:59 org.apache.beam.sdk.io.aws.sqs.SqsIOTest > testRead FAILED
19:43:59     akka.stream.impl.io.ConnectionSourceStage$$anon$2$$anon$1: Bind failed because of Address already in use
19:43:59 
19:43:59         Caused by:
19:43:59         java.net.BindException: Address already in use
19:44:07 
19:44:07 org.apache.beam.sdk.io.aws.sns.SnsIOTest > testRetries STANDARD_ERROR
19:44:07     Sep 18, 2018 11:44:07 PM org.apache.beam.sdk.io.aws.sns.SnsIO$Write$SnsWriterFn processElement
19:44:07     WARNING: Error writing to SNS. Retry attempt[1]
19:44:07     com.amazonaws.services.sns.model.InternalErrorException: Service unavailable (Service: null; Status Code: 0; Error Code: null; Request ID: null)
19:44:07     	at org.apache.beam.sdk.io.aws.sns.AmazonSNSMockErrors.publish(AmazonSNSMockErrors.java:29)
19:44:07     	at org.apache.beam.sdk.io.aws.sns.SnsIO$Write$SnsWriterFn.processElement(SnsIO.java:314)

perhaps the hadoop and AWS modules were conflicting about binding the same port?

I'm not sure what happened to the elasticsearch test

[ryan-williams]

Run Java PreCommit