Buck tests are flaky on non englisch locale

[davido]

Trying to run tests on most recent master 4806fdf on two sane platforms (Linux with Java 7 and Mac OS X with Java 8) I'm seeing number of unit test failures:

$ buck test
Not using buckd because watchman isn't installed.
FAILURE com.facebook.buck.util.unit.SizeUnitTest testFormatMegabytes: expected:<1[.]5MB> but was:<1[,]5MB>
FAILURE com.facebook.buck.util.unit.SizeUnitTest testFormatGigabytes: expected:<1[.]5GB> but was:<1[,]5GB>
FAILURE com.facebook.buck.util.unit.SizeUnitTest testFormatTerabytes: expected:<1[.]5TB> but was:<1[,]5TB>
FAILURE com.facebook.buck.util.unit.SizeUnitTest testFormatKilobytes: expected:<-1[.]1KB> but was:<-1[,]1KB>
FAILURE com.facebook.buck.util.TimeFormatTest testFormatForConsole: expected:<  1[.]0s> but was:<  1[,]0s>
FAILURE com.facebook.buck.event.listener.SimpleConsoleEventBusListenerTest testSimpleBuild: expected:<...K FILES...FINISHED 0[.]4s
> but was:<...K FILES...FINISHED 0[,]4s
>
FAILURE com.facebook.buck.event.listener.SuperConsoleEventBusListenerTest testBuildRuleSuspendResumeEvents: 
Expected: <[[-] PROCESSING BUCK FILES...FINISHED 0.0s, [+] BUILDING...0,2s,  |=> IDLE]>
     but: was <[[-] PROCESSING BUCK FILES...FINISHED 0,0s, [+] BUILDING...0,2s,  |=> IDLE]>
FAILURE com.facebook.buck.event.listener.SuperConsoleEventBusListenerTest testSimpleBuildWithProgress: 
Expected: <[[-] PROCESSING BUCK FILES...FINISHED 0,2s, [+] BUILDING...0,1s [0%] (0/10 JOBS, 0 UPDATED, 0.0% CACHE MISS)]>
     but: was <[[-] PROCESSING BUCK FILES...FINISHED 0,2s, [+] BUILDING...0,1s [0%] (0/10 JOBS, 0 UPDATED, 0,0% CACHE MISS)]>

The output above is from OS X 10.10.5 with this locale:

$ locale
LANG="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_CTYPE="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_ALL=

The list of failing tests is:

TESTS FAILED: 8 FAILURES
Failed target: //test/com/facebook/buck/event/listener:listener
FAIL com.facebook.buck.event.listener.SimpleConsoleEventBusListenerTest
FAIL com.facebook.buck.event.listener.SuperConsoleEventBusListenerTest
Failed target: //test/com/facebook/buck/util:util
FAIL com.facebook.buck.util.TimeFormatTest
Failed target: //test/com/facebook/buck/util/unit:unit
FAIL com.facebook.buck.util.unit.SizeUnitTest

[sdwilsh]

These looks like locale-specific failures. @bhamiltoncx, what's the best way to test this type of stuff in a locale-aware way?

[davido]

According to this comment, it seems like this is a regression, that was already handled once:

https://github.com/facebook/buck/blob/master/test/com/facebook/buck/event/listener/SuperConsoleEventBusListenerTest.java#L110-L124

/**
   * Formats a string with times passed in in seconds.
   *
   * Used to avoid these tests failing if the user's locale doesn't use '.' as the decimal
   * separator, as was the case in https://github.com/facebook/buck/issues/58.
   */
  private static String formatConsoleTimes(String template, Double... time) {
    return String.format(template, (Object[]) FluentIterable.from(ImmutableList.copyOf(time))
        .transform(new Function<Double, String>() {
          @Override
          public String apply(Double input) {
            return timeFormatter.format(input);
          }
        }).toArray(String.class));
  }

And that now in addition to time elapsed, that is handled correctly, at least in this test file, cache hit/missed ratio was added, that is locale specific:

Expected: <0.0% CACHE MISS>
but: was <0,0% CACHE MISS>

[bhamiltoncx]

Yeah, we need to always pass in a Locale object to any method which formats times, dates, amounts, etc. SizeUnit is implicitly using the default Locale but the test is not. Same with SuperConsoleEventBusListener.

I'll fix it.

[bhamiltoncx]

The last of the fixes for this issue are out for review internally. I hope to land them today.

[bhamiltoncx]

Apologies for the delay. The fixes were reviewed last night and are landing now.

[davido]

Thanks. Trying again on most recent master (ec75fa1) still some tests are failing here:

$ buck test
Shutting down nailgun server...
Using watchman.
Waiting for Watchman command [/home/davido/bin/watchman version]...
Timed out after 10000 ms waiting for Watchman command [/home/davido/bin/watchman version]. Disabling Watchman.
FAILURE com.facebook.buck.event.listener.BuildThreadStateRendererTest withMissingInformation: 
Expected: is <[ |=> //:target3...  2.6s (checking local cache),  |=> IDLE,  |=> IDLE]>
     but: was <[ |=> //:target3...  2,6s (checking local cache),  |=> IDLE,  |=> IDLE]>
FAILURE com.facebook.buck.event.listener.BuildThreadStateRendererTest commonCase: 
Expected: is <[ |=> //:target2...  4.4s (running step A[2.7s]),  |=> //:target3...  2.6s (checking local cache),  |=> //:target1...  3.3s (checking local cache),  |=> IDLE,  |=> //:target4...  1.2s (running step B[0.5s])]>
     but: was <[ |=> //:target2...  4,4s (running step A[2,7s]),  |=> //:target3...  2,6s (checking local cache),  |=> //:target1...  3,3s (checking local cache),  |=> IDLE,  |=> //:target4...  1,2s (running step B[0,5s])]>
FAILURE com.facebook.buck.event.listener.TestThreadStateRendererTest withMissingInformation: 
Expected: is <[ |=> //:target3...  2.6s,  |=> //:target1...  3.3s,  |=> IDLE,  |=> IDLE]>
     but: was <[ |=> //:target3...  2,6s,  |=> //:target1...  3,3s,  |=> IDLE,  |=> IDLE]>
FAILURE com.facebook.buck.event.listener.TestThreadStateRendererTest commonCase: 
Expected: is <[ |=> //:target2...  4.4s (running step A[2.7s]),  |=> //:target3...  2.6s (running Test A[2.6s]),  |=> //:target1...  3.3s,  |=> IDLE,  |=> //:target4...  1.2s (running Test B[0.4s])]>
     but: was <[ |=> //:target2...  4,4s (running step A[2,7s]),  |=> //:target3...  2,6s (running Test A[2,6s]),  |=> //:target1...  3,3s,  |=> IDLE,  |=> //:target4...  1,2s (running Test B[0,4s])]>
FAILURE com.facebook.buck.python.PythonBinaryIntegrationTest nativeLibsEnvVarIsPreserved[INPLACE]: 
Expected: "None"
     but: was "/usr/lib64/mpi/gcc/openmpi/lib64"
FAILURE com.facebook.buck.python.PythonBinaryIntegrationTest nativeLibsEnvVarIsPreserved[STANDALONE]: 
Expected: "None"
     but: was "/usr/lib64/mpi/gcc/openmpi/lib64"

Here is the list of failing tests:

Failed target: //test/com/facebook/buck/event/listener:listener
FAIL com.facebook.buck.event.listener.BuildThreadStateRendererTest
FAIL com.facebook.buck.event.listener.TestThreadStateRendererTest
Failed target: //test/com/facebook/buck/python:python
FAIL com.facebook.buck.python.PythonBinaryIntegrationTest

Here is more verbose log:

http://paste.openstack.org/show/481381.

[k21]

The *ThreadStateRendererTest failures are my fault, I have just added them before the @bhamiltoncx's fix and they are using the system default locale. I will make sure to fix them soon.

[davido]

Thx.

[bhamiltoncx]

Thanks @k21. I should have re-run all the tests in German (I forgot to do it after your diff).

@davido, the PythonBinaryIntegrationTest failure appears to be a separate problem.

[davido]

@bhamiltoncx Thanks for the analysis. I filed new issue to track this failure: #565. Does it work for you?

I'm a bit confused. CI is broken, I cannot get test passed on my Linux machine and getting negative reviews because my PR #558 broke some tests. But how should I be able to track down this breakage, when tests are inherently broken on most recent master (ec75fa1) anyway? Or this is something that environment dependent?

[davido]

Fixed in: [1].

• [1] Disable broken tests #567

[davido]

Fixed by: 447e59a