src/php/bin/run_tests.sh Failed to load file

[AspirinSJL]

https://source.cloud.google.com/results/invocations/008051d6-3566-41f0-8445-6c0d5c421c5e/targets/github%2Fgrpc%2Fphp_macos_dbg_native/tests

Starting test 'PersistentListTest::testPersistentChannelTargetUpperBoundNotZero'.
.
Starting test 'PersistentListTest::testPersistentChannelDefaultOutBound1'.
.[Warning] The number of channel for the target localhost:10011 is maxed out bounded.
[Warning] Target upper bound: 1. Current size: 1.
[Warning] Target localhost:10011 will not be persisted.
Starting test 'PersistentListTest::testPersistentChannelDefaultOutBound2'.
.[Warning] The number of channel for the target localhost:10011 is maxed out bounded.
[Warning] Target upper bound: 1. Current size: 1.
[Warning] Target localhost:10011 will not be persisted.
Starting test 'PersistentListTest::testPersistentChannelDefaultOutBound3'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBound'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBoundOutBound1'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBoundOutBound2'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBoundOutBound3'.
E0817 14:18:34.868681000 123145372471296 security_connector.cc:1228]   load_file: {"created":"@1534540714.868657000","description":"Failed to load file","file":"src/core/lib/iomgr/load_file.cc","file_line":71,"filename":"/usr/local/share/grpc/roots.pem","referenced_errors":[{"created":"@1534540714.868655000","description":"OS Error","errno":2,"file":"src/core/lib/iomgr/load_file.cc","file_line":45,"os_error":"No such file or directory","syscall":"fopen"}]}
E0817 14:18:34.868741000 123145372471296 security_connector.cc:1062]   Could not get default pem root certs.
E0817 14:18:34.868752000 123145372471296 secure_channel_create.cc:121] Failed to create secure subchannel for secure name 'localhost:10011'
E0817 14:18:34.868763000 123145372471296 secure_channel_create.cc:154] Failed to create subchannel arguments during subchannel creation.
E0817 14:18:34.868783000 123145372471296 security_connector.cc:1062]   Could not get default pem root certs.
E0817 14:18:34.868792000 123145372471296 secure_channel_create.cc:121] Failed to create secure subchannel for secure name 'localhost:10011'
E0817 14:18:34.868801000 123145372471296 secure_channel_create.cc:154] Failed to create subchannel arguments during subchannel creation.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBoundOutBound4'.
F
Time: 163 ms, Memory: 12.25MB
There was 1 failure:
1) PersistentListTest::testPersistentChannelTwoUpperBoundOutBound4
Failed asserting that false is true.
/Volumes/BuildData/tmpfs/src/github/grpc/workspace_php_macos_dbg_native/src/php/tests/unit_tests/PersistentChannelTests/PersistentChannelTest.php:46
/Volumes/BuildData/tmpfs/src/github/grpc/workspace_php_macos_dbg_native/src/php/tests/unit_tests/PersistentChannelTests/PersistentChannelTest.php:463
FAILURES!
Tests: 20, Assertions: 79, Failures: 1.

[ZhouyihaiDing]

There should be 2 causes:

1. In php test side, a test only method clearPersistentList hasn't finished clean the persistent list while the new test starts running, which cause the flaky.
2. Is there some changes related to the getConnectivityState recently?
   Based on the BigQuery result, there are tons of flaky tests happens starting on Aug 17th while there is no change in the PHP side.

[AspirinSJL]

Do you mean grpc_channel_check_connectivity_state()? From the history, it's not changed since May.

Well, we do have some changes related to connectivity though. Let's chat offline.

[ZhouyihaiDing]

Based on the BigQuery results within last 3 days, it seems (1. ) is solved by manually changing the port to be different for each test.
(2) is still happening. The flaky happens in the function:

public function waitUntilNotIdle($channel) {
      channel->getConnectivityState($try_to_connect = true);
      for ($i = 0; $i < 10; $i++) {
          $now = Grpc\Timeval::now();
          $deadline = $now->add(new Grpc\Timeval(1000));
          if ($channel->watchConnectivityState(GRPC\CHANNEL_IDLE,
              $deadline)) {
              return true;
          }
      }
      $this->assertTrue(false);  // <== Test fails here
  }

When try_to_connect = true, either READY or TRANSIENT_FAILURE can break the loop. But now it remains IDLE sometimes. It can be 10 loops is too short? I will look into this later.

[ZhouyihaiDing]

According to the bigtable result, this error doesn't happen for 5 days.

[yihuazhang]

https://source.cloud.google.com/results/invocations/e12f8c74-386b-41de-877c-5cfb8505ecf1/targets/github%2Fgrpc%2Fphp_macos_dbg_native/log

`Starting test 'PersistentListTest::testInitHelper'.
.
Starting test 'PersistentListTest::testChannelNotPersist'.
.
Starting test 'PersistentListTest::testPersistentChannelCreateOneChannel'.
.
Starting test 'PersistentListTest::testPersistentChannelCreateMultipleChannels'.
.
Starting test 'PersistentListTest::testPersistentChannelStatusChange'.
.
Starting test 'PersistentListTest::testPersistentChannelCloseChannel'.
.
Starting test 'PersistentListTest::testPersistentChannelSameTarget'.
.
Starting test 'PersistentListTest::testPersistentChannelDifferentTarget'.
.
Starting test 'PersistentListTest::testPersistentChannelSharedChannelClose'.
.
Starting test 'PersistentListTest::testPersistentChannelTargetDefaultUpperBound'.
.
Starting test 'PersistentListTest::testPersistentChannelTargetUpperBoundZero'.
.[Warning] The number of channel for the target localhost:10002 is maxed out bounded.
[Warning] Target upper bound: 0. Current size: 0.
[Warning] Target localhost:10002 will not be persisted.
Starting test 'PersistentListTest::testPersistentChannelTargetUpperBoundNotZero'.
.
Starting test 'PersistentListTest::testPersistentChannelDefaultOutBound1'.
.[Warning] The number of channel for the target localhost:10004 is maxed out bounded.
[Warning] Target upper bound: 1. Current size: 1.
[Warning] Target localhost:10004 will not be persisted.
Starting test 'PersistentListTest::testPersistentChannelDefaultOutBound2'.
.[Warning] The number of channel for the target localhost:10005 is maxed out bounded.
[Warning] Target upper bound: 1. Current size: 1.
[Warning] Target localhost:10005 will not be persisted.
Starting test 'PersistentListTest::testPersistentChannelDefaultOutBound3'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBound'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBoundOutBound1'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBoundOutBound2'.
.
Starting test 'PersistentListTest::testPersistentChannelTwoUpperBoundOutBound3'.
E0314 22:15:55.206738000 140736125797248 ssl_utils.cc:464]             load_file: {"created":"@1552626955.206705000","description":"Failed to load file","file":"src/core/lib/iomgr/load_file.cc","file_line":71,"filename":"/usr/local/share/grpc/roots.pem","referenced_errors":[{"created":"@1552626955.206702000","description":"No such file or directory","errno":2,"file":"src/core/lib/iomgr/load_file.cc","file_line":45,"os_error":"No such file or directory","syscall":"fopen"}]}
E0314 22:15:55.206795000 140736125797248 ssl_utils.cc:304]             Could not get default pem root certs.
src/php/bin/run_tests.sh: line 29:  5193 Segmentation fault: 11  php $extension_dir -d max_execution_time=300 $(which phpunit) -v --debug ../tests/unit_tests/PersistentChannelTests
2019-03-14 22:15:55,209 FAILED: src/php/bin/run_tests.sh [ret=139, pid=5174, time=1.1sec]
2019-03-14 22:15:55,209 FAILED: src/php/bin/run_tests.sh
2019-03-14 22:15:55,222 START: tools/run_tests/helper_scripts/post_tests_php.sh
2019-03-14 22:15:55,231 PASSED: tools/run_tests/helper_scripts/post_tests_php.sh [time=0.0sec, retries=0:0]
2019-03-14 22:15:55,232 FAILED: Some tests failed`