Release mode is way more slower than debug mode for user defined functions.

[nabinbhandari]

I was trying to benchmark the performance of flutter and compare it with that of Java and C. So I chose a simple function to calculate nth prime number.

The problem is that the time taken to calculate the 20000th prime number in release mode (821 ms) was almost 8 times slower than that in debug mode (106 ms).

The method that I used is given below:

  int nThPrimeNumber(int n) {
    int counter = 0;

    for (var i = 1;; i++) {
      if (isPrime(i)) counter++;

      if (counter == n) {
        return i;
      }
    }
  }

  bool isPrime(var n) {
    if (n < 2) return false;

    for (var i = 2; i * i <= n; i++) {
      if (n % i == 0) return false;
    }

    return true;
  }

Full source code and benchmark result is available in this repository.

I know this is not a big issue and there are some serious issues which needs to be resolved. But I am creating this issue because I found that in debug mode, Flutter code was running faster than C (but only in debug mode).

If this issue is fixed for release mode, some CPU intensive tasks such as image processing could be easily (and quickly too!) performed in Dart without having the need to write them in C/C++.

[mraleph]

Can you specify what kind of phone and which branch of Flutter did you use?

[nabinbhandari]

Flutter Version:

[√] Flutter (Channel beta, v0.5.1, on Microsoft Windows [Version 10.0.17134.167], locale en-US)
    • Flutter version 0.5.1 at C:\Users\Nabin\Flutter\sdk\flutter
    • Framework revision c7ea3ca377 (8 weeks ago), 2018-05-29 21:07:33 +0200
    • Engine revision 1ed25ca7b7
    • Dart version 2.0.0-dev.58.0.flutter-f981f09760

Device: Motorola Moto G5 Plus, I don't think this is device specific issue because I also found a similar issue in Samsung Galaxy A5. I haven't tested it in other devices.

One can simply verify it by running this file.

[sir-boformer]

I can confirm this on a Sony Xperia Z5 Compact:

Debug build:

I/flutter ( 1444): Flutter - 20000th prime number: 224737 
I/flutter ( 1444): time: 72 ms

Release build:

I/flutter ( 2848): Flutter - 20000th prime number: 224737 
I/flutter ( 2848): time: 383 ms

[mraleph]

I took a look: it's a combination of known issues. Here is the graph for isPrime loop in AOT mode.

 18: B9[join]:72 pred(B4, B8) {
      v7 <- phi(v3, v34 T{Type: class '_int64@0150898'}) alive [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
}
 19:     ParallelMove S-3 <- rdx
 20:     CheckStackOverflow:78(depth 1)
 22:     PushArgument(v7)
 24:     PushArgument(v7)
 26:     v9 <- StaticCall:32( *<0> v7, v7)
 27:     ParallelMove rbx <- rax
 28:     ParallelMove S-4 <- rbx
 28:     PushArgument(v9)
 29:     ParallelMove rax <- S+2, rdx <- C, rcx <- C
 30:     AssertAssignable:34(v2, Type: class 'num', '', instantiator_type_args(v0), function_type_args(v0)) T{Type: class 'num'?}
 32:     PushArgument(v2 T{Type: class 'num'?})
 33:     ParallelMove rax <- S-4
 34:     CheckNull:36(v9)
 36:     v11 <- StaticCall:38( <=<0> v9, v2 T{Type: class 'num'?}) T{bool}
 37:     ParallelMove rax <- rax
 38:     Branch if StrictCompare:42(===, v11, v6) goto (5, 6)
 40: B5[target]:46 ParallelMove rax <- C
 42:     PushArgument(v2 T{Type: class 'num'?})
 44:     PushArgument(v7 T{Type: class '_int64@0150898'})
 46:     v13 <- InstanceCall:54( %<0>, v2 T{Type: class 'num'?}, v7 T{Type: class '_int64@0150898'} IC[0: ])
 47:     ParallelMove rax <- rax, rcx <- C
 48:     Branch if CheckedSmiComparison:56(==, v13, v15) T{bool} goto (7, 8)
 50: B7[target]:64
 51:     ParallelMove rax <- C
 52:     Return:70(v21)
 54: B8[target]:66
 56:     ParallelMove rdx <- S-3
 56:     v32 <- UnboxInt64(v7 T{Type: class '_int64@0150898'}) [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
 58:     ParallelMove rdx <- rdx
 58:     v19 <- BinaryInt64Op(+ [tr], v32 T{Type: class '_int64@0150898'}, v40) [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
 60:     v34 <- BoxInt64(v19) [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
 62:     ParallelMove rdx <- rax, rcx <- S+2, rax <- rcx goto:76 B9

The first problem:

• v2 (value of n) is checked for being num even though I would expect TFA to infer this information for us (more precisely it should infer that v2 is a non-nullable int). /cc @alexmarkov

This can be worked around in user code by changing type annotation on n from var n to int n. This brings around 15% performance improvement.

After this change the graph becomes this:

 26: B9[join]:72 pred(B4, B8) {
      v7 <- phi(v3, v42 T{Type: class '_int64@0150898'}) alive [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
}
 27:     ParallelMove S-3 <- rbx
 28:     CheckStackOverflow:78(depth 1)
 30:     PushArgument(v7)
 32:     PushArgument(v7)
 34:     v9 <- StaticCall:32( *<0> v7, v7)
 35:     ParallelMove rax <- rax
 36:     CheckNull:34(v9)
 38:     ParallelMove rax <- rax
 38:     v40 <- UnboxInt64(v9 T{!null}) [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
 39:     ParallelMove rcx <- S-4
 40:     Branch if RelationalOp(<=, v40, v36) T{bool} goto (5, 6)
 42: B5[target]:44 ParallelMove rax <- C
 44:     PushArgument(v2 T{Type: class 'int'})
 46:     PushArgument(v7 T{Type: class '_int64@0150898'})
 48:     v13 <- StaticCall:54( %<0> v2 T{Type: class 'int'}, v7 T{Type: class '_int64@0150898'}) T{Type: class 'num'?}
 49:     ParallelMove rax <- rax, rcx <- C
 50:     Branch if CheckedSmiComparison:56(==, v13, v15) T{bool} goto (7, 8)
 52: B7[target]:64
 53:     ParallelMove rax <- C
 54:     Return:70(v21)
 56: B8[target]:66
 58:     ParallelMove rdx <- S-3
 58:     v38 <- UnboxInt64(v7 T{Type: class '_int64@0150898'}) [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
 60:     ParallelMove rdx <- rdx
 60:     v19 <- BinaryInt64Op(+ [tr], v38 T{Type: class '_int64@0150898'}, v50) [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
 62:     v42 <- BoxInt64(v19) [-9223372036854775808, 9223372036854775807] T{Type: class '_int64@0150898'}
 64:     ParallelMove rbx <- rbx, rax <- S+2, rdx <- rcx, rcx <- S-4 goto:76 B9

Notice that

1. v9 <- StaticCall:32( *<0> v7, v7) is not inlined
2. v13 <- StaticCall:54( %<0> v2, v7 is not inlined
3. v7 phi is not unboxed across the loop.

These are three separate problems.

1. * is not inlined because in TryOptimizeStaticCallUsingStaticTypes we think that v7 can be nullable integer. (later we infer that it can't - but we don't rerun specialization code at later stages - which we should probably do).
2. % is not inlined because we don't have non-speculative implementation for it. I filed upstream bug for this AOT: implement non-speculative int64 versions of MOD and TRUNCDIV dart-lang/sdk#33967
3. v7 is not unboxed across the loop because the heuristic we have in UnboxPhi is too conservative.

/cc @aartbik this probably fits well into your overarching work on arithmetic so I am tentatively assigning this to you.

[aartbik]

On my desktop, with slightly larger request:

JIT vs AOT:
executed in 0:00:05.369510 (verify="500000th prime is 7368787")
executed in 0:00:17.198906 (verify="500000th prime is 7368787")

[aartbik]

I added this to our internally monitored micro benchmarks.

[aartbik]

Note: spin off issues that are still pending:

dart-lang/sdk#33967
dart-lang/sdk#34072

[aartbik]

With some prototype code I have this down to the following. Still, we need to do better.

JIT vs AOT:
executed in 0:00:05.176929 (verify="500000th prime is 7368787")
executed in 0:00:11.191870 (verify="500000th prime is 7368787")

[aartbik]

The 64-bit idiv on x86_64 is rather slow, so it makes sense to fast path this one even in the 64-bit case. With this and prototype code to deal with 1, 2, and 3 above I finally get the performance we want even on my desktop!

JIT vs AOT:
executed in 0:00:05.765761 (verify="500000th prime is 7368787")
executed in 0:00:04.863650 (verify="500000th prime is 7368787")

[aartbik]

Current status. With "bool isPrime(int n)" the performance of AOT is now on par with JIT.

JIT vs AOT:
executed in 0:00:05.290235 (verify="500000th prime is 7368787")
executed in 0:00:04.936496 (verify="500000th prime is 7368787")

With "bool isPrime(var n)", on the other hand, not all of the recent improvements take effect.

JIT vs AOT:
executed in 0:00:05.032289 (verify="500000th prime is 7368787")
executed in 0:00:14.777356 (verify="500000th prime is 7368787")

Although it is not unreasonable to request using "int" for performance critical parts, it would also be nice to delight our customers even more. So over to Alex who had some ideas on how to propagate "int" into the "var" declaration under AOT (or hand back to me with some pointers if you don't have the time).

You will see a big improvement (3x) for the latter AOT case once the "int" is discovered.

[aartbik]

After latest check-in, "bool isPrime(var n)" is now on par too.

JIT vs. AOT:
executed in 0:00:05.356641 (verify="500000th prime is 7368787")
executed in 0:00:05.327590 (verify="500000th prime is 7368787")

[aartbik]

All has landed.

[aartbik]

(to be clear: landed in Dart SDK, next flutter roll will bring this into the flutter environment).

[ejthayerTN]

I have noticed this problem as well and posted about it but nobody seems to know why this happens or even really care, the question seems to always get dismissed as BS. People keep asking for code because for some odd reason they don't understand the problem. CODE DOES NOT MATTER. Compile an app in release mode and it is slower! I found this out in a similar way by running some speed tests.

[mraleph]

@ejthayerTN

I have noticed this problem as well

It is unlikely that you noticed this problem, because this problem is actually fixed.

It is highly possible however that you have encountered some other situation where AOT compiler does not do good optimisation job.

Unfortunately it is impossible to say what exactly is going wrong without seeing the code, because reasons for the slow down are likely highly specific to the code. You might be using RegExp - which are slower in AOT builds, or you might be doing 64-integer math which is slow on ARM32, or you might be hitting a corner case in global type propagation which fails to infer some type which is critical for achieving good performance... The list is endless.

We would be happy to look at the problem and figure if either us or you can do something to resolve it - but for that, unfortunately, you would need to provide more information about what exactly you are trying to do that gets slower.

[eduardofcr]

I am getting the same symptoms, where the code is slower in production than in development, when using in the same code in Android Phone.
The difference is pretty big and the app does not use calculations at all, only https requests to a firebase database.

I was using this commands:

flutter build apk --target-platform android-arm,android-arm64,android-x64 --split-per-abi
(from https://flutter.dev/docs/deployment/android)
this is the worst result from my tests

flutter build apk --release
best results than above

flutter run
(with a real phone connected using USB)
best results, not good at all, but faster than all the other options above

the environment:

$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel master, 1.26.0-13.0.pre.179, on Linux, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
[✓] Android Studio
[✓] VS Code (version 1.52.1)
[✓] Connected device (1 available)

• No issues found!

any tips or fixes?

[mraleph]

@eduardofcr you should profile your code and see where it is spending time (see https://flutter.dev/docs/perf/rendering/ui-performance and https://flutter.dev/docs/development/tools/devtools/cpu-profiler).

Unfortunately without some more concrete details it is hard to give you any guidance.

[eduardofcr]

Problem solved.
It was not the code at all!!
I just use the command:
flutter build apk --target-platform android-arm --split-per-abi

So, the 64 bit version was not generated.

Now, the production version is faster than the development one! ;)

No hangs, no waits!
Problem solved.

[MarcinKrz1]

Problem solved.
It was not the code at all!!
I just use the command:
flutter build apk --target-platform android-arm --split-per-abi

So, the 64 bit version was not generated.

Now, the production version is faster than the development one! ;)

No hangs, no waits!
Problem solved.

So what command should be used to ensure better release performance than debug?

[mraleph]

So what command should be used to ensure better release performance than debug?

It depends on many factors. Are your release builds performing worse than debug? Did you try to profile your release builds?

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.