[Testing] Mark FixedPointConversion tests as "long_test"

[davezarzycki]

As of 14d89ec (GitHub #23220), these tests now take 2.25x longer than the entire validation test suite on fast machines. Mark them as "long_test" for now until they can be broken into smaller concurrent tests.

[davezarzycki]

@swift-ci please smoke test

[stephentyrone]

Thanks Dave!

[stephentyrone]

I think that we should be able to rework these in the long-term to avoid the problem, but this is a perfectly acceptable quick fix.

[davezarzycki]

Great! Thanks

[drodriguez]

@davezarzycki: thanks for the quick fix, in #23220 you say it takes 2.25x the next longest test, in here is 2.25x than the entire validation test. I hope the original statement is the right one. In any case, how are you measuring the numbers? Is there a command in the repo that I can use? I want to check what's taking that long, and why. Also, are we talking about Darwin machines, Linux machines? Did you noticed if it was the Release or the Debug test?

The test might not have been “long” before because I think it was not running a large part of the tests, but I wasn’t expecting such a huge change. I was testing this before, and while it wasn’t the fastest test that I have ever run, it took way less than the 2 minutes I just seen in a beefy machine.

[davezarzycki]

@drodriguez – The "problem" is that with enough CPU cores, the time to run the entire test suite "equals" the time needed run the longest individual test. The change in #23220 made one of the FixedPointConversion tests the new longest test (by 2.25x) and therefore the whole test suite took 2.25x longer.

[drodriguez]

I have been looking a little bit more into this.

The executables themselves take around 2 seconds to run (Debug slightly slower than Release. Averages 2.33 and 2.19). Gyb takes less than 0.2 seconds to run.

The bunch of the time is the compilation of the file. The file is 14k lines (many of them are empty or comments). This are the stats for the compilation in my machine (quite beefy):

===-------------------------------------------------------------------------===
                               Swift compilation
===-------------------------------------------------------------------------===
  Total Execution Time: 21.6956 seconds (21.6968 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
   7.5099 ( 34.9%)   0.0729 ( 35.5%)   7.5828 ( 35.0%)   7.5829 ( 34.9%)  AST verification
   5.4131 ( 25.2%)   0.0170 (  8.3%)   5.4301 ( 25.0%)   5.4307 ( 25.0%)  SILGen
   3.1830 ( 14.8%)   0.0052 (  2.5%)   3.1882 ( 14.7%)   3.1874 ( 14.7%)  Parsing
   3.0194 ( 14.1%)   0.0071 (  3.4%)   3.0265 ( 13.9%)   3.0261 ( 13.9%)  Name binding
   1.7238 (  8.0%)   0.0881 ( 42.9%)   1.8119 (  8.4%)   1.8136 (  8.4%)  Type checking and Semantic analysis
   0.2303 (  1.1%)   0.0140 (  6.8%)   0.2443 (  1.1%)   0.2443 (  1.1%)  IRGen
   0.1440 (  0.7%)   0.0010 (  0.5%)   0.1450 (  0.7%)   0.1450 (  0.7%)  SIL optimization
   0.1353 (  0.6%)   0.0000 (  0.0%)   0.1353 (  0.6%)   0.1353 (  0.6%)  SIL verification, pre-optimization
   0.1315 (  0.6%)   0.0000 (  0.0%)   0.1315 (  0.6%)   0.1315 (  0.6%)  SIL verification, post-optimization
  21.4903 (100.0%)   0.2053 (100.0%)  21.6956 (100.0%)  21.6968 (100.0%)  Total

===-------------------------------------------------------------------------===
                            Driver Compilation Time
===-------------------------------------------------------------------------===
  Total Execution Time: 0.0255 seconds (73.6065 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
   0.0046 (100.0%)   0.0208 ( 99.5%)   0.0254 ( 99.6%)  73.5526 ( 99.9%)  {compile: FixedPointConversion-e27eaa.o <= FixedPointConversion.swift}
   0.0000 (  0.0%)   0.0001 (  0.3%)   0.0001 (  0.2%)   0.0471 (  0.1%)  {link: a.out_Release <= FixedPointConversion-e27eaa.o FixedPointConversion-5f5cdf.autolink}
   0.0000 (  0.0%)   0.0000 (  0.2%)   0.0000 (  0.2%)   0.0068 (  0.0%)  {swift-autolink-extract: FixedPointConversion-5f5cdf.autolink <= FixedPointConversion-e27eaa.o}
   0.0046 (100.0%)   0.0209 (100.0%)   0.0255 (100.0%)  73.6065 (100.0%)  Total

73.31user 0.32system 1:13.66elapsed 99%CPU (0avgtext+0avgdata 172864maxresident)k
1696inputs+8736outputs (5major+37274minor)pagefaults 0swaps

It seems to me that it is more like a lot of little functions taking a long time together. Maybe breaking up the test is the only way to go.

[davezarzycki]

Ya… The best and worst thing about gyb is that it makes combinatorial or permutation in a single file easy. That being said, combinatorial or permutation testing is ripe for concurrency, but that requires multiple files in the lit testing system.