Performance regression

[sergey-tihon]

Seems like FCS is slower on Travis in comparison with AppVeyor
and latest versions become slower.

Repro steps

I use FCS to test SwaggerProvider and ensure that TP is compilable on ~390 available Swagger schemas.

I see following test suite execution time

FCS	AppVeyor	Travis (Linux)
11.0.6	15 min 56 sec	40 min 37 sec
11.0.9	24 min 37 sec	> 50 min
11.0.10	29 min 30 sec	> 50 min
12.0.1	26 min 28 sec	> 50 min

Does it expected regression that is start from 11.0.9?
Do you understand why it is slower on Mono?

[dsyme]

@sergey-tihon Well it could be a number of things

• There may indeed have been a performance regression
• Neither AppVeyor nor Travis are reliable performance environment, and their VM configurations may have changed, or you may randomly get allocated a different VM under different load during the day. No one knows.
• AppVeyor and Travis are likely use different VMs and core allocations so are not comparable
• Mono in general is a bit slower

But still, it's concerning and your numbers are worth investigating. Can you re-run them a few more times?

[dsyme]

Looking at this diff between 11.0.6 and 11.0.9 it's hard to see anything at all suspicious w.r.t. performance.

Was your version of Mono identical in each case?

[sergey-tihon]

If you think that this is not FCS issue - feel free to close this issue anytime.

On Travis i use latest Mono, so it is (always the same)

Mono JIT compiler version 4.8.0 (Stable 4.8.0.520/8f6d0f6 Wed Mar 15 16:18:35 UTC 2017)

I've limited number of tests for FSC to 200 in order to fit TravisCI execution time limitation. Here are the results for FCS 12.0.1:

Execution	AppVeyor	AppVeyor - Paket	Travis (Linux)	Travis - Paket
Attempt 1	16 min 10 sec	45 seconds	33 min 10 sec	27 seconds
Attempt 2	18 min 24 sec	42 seconds	34 min 18 sec	28 seconds
Attempt 3	16 min 57 sec	50 seconds	32 min 42 sec	21 seconds
Attempt 4	16 min 28 sec	44 seconds	31 min 27 sec	22 seconds
Attempt 5	18 min 57 sec	44 seconds	34 min 10 sec	22 seconds

We can conclude that

• Travis (Mono/Linux) is ~2 times slower in average than AppVeyor (may be hardware difference)
• Both AppVeyor and Travis are relatively reliable (show similar results between runs)
• Paket restore is consistent before build (so there is no network delays before the build start)
• Network on Travis is about 2 times faster than on AppVeyor (paket restore is faster)

[sergey-tihon]

The same setup (200 FCS tests) with different FCS versions

FCS	FSharp.Core	AppVeyor	Travis (Linux)
11.0.6	4.1.2	16 min 43 sec	33 min 47 sec
11.0.9	4.1.2	16 min 7 sec	33 min 35 sec
11.0.10	4.1.2	16 min 21 sec	34 min 31 sec
12.0.1	4.1.2	16 min 10 sec	33 min 10 sec
12.0.1	4.1.0.2	16 min 38 sec	34 min 16 sec

Seems that regression is not related with version of FCS.
My bad. Sorry 😐

[dsyme]

That's good to know! Thanks for the doing the analysis again!