Poor performance with larger code base #283

davidgiga1993 · 2022-10-07T15:51:20Z

Hi,

I'm using proguard for nearly 10 years now and so far I never had any major performance issues. However at some point proguard got extremely slow - I can't recall the exact time and I assume it's related to the bytecode input. It takes about 25 minutes to obfuscate a java desktop application with about 400kLoc on an Mac Mini M1.
I've also tested it on an Windows 11 Ryzen 5950x and it took 22 minutes. Most of this time only 1 core is used, sometimes 2 for a short period.

The project is currently using com.guardsquare:proguard-gradle:7.2.2 with gradle 7.5 and temurin jdk 17 with 1.8 as compile target.

The app uses quite a bit of autogenerated code with huge switch/case statements, and I somehow suspect that it could be related to that, howver I haven't verified it yet.

My questions are:

Is there a way to profile what exactly takes proguard so long?
Is performance something you want improve in the future?

Happy to get any feedback on this topic. In the meantime I'll try to remove the auto generated code and see if it makes a difference.

For reference, here is the proguard config and output of the M1 build:

-optimizations !code/simplification/cast,!field/*,!class/merging/*
-optimizationpasses 3
-allowaccessmodification

-adaptresourcefilecontents **.properties,META-INF/MANIFEST.MF

# The remainder of this file is identical to the non-optimized version
# of the Proguard configuration file (except that the other file has
# flags to turn off optimization).

-dontusemixedcaseclassnames
-dontskipnonpubliclibraryclasses
-verbose

-keepattributes *Annotation*

-keepclassmembers enum * {
    public static **[] values();
    public static ** valueOf(java.lang.String);
}

-keepclassmembers class **.R$* {
    public static <fields>;
}

# Libgdx lwjgl fix
-keep class org.lwjgl.system.** { *; }

# External libraries
-dontwarn org.xmlpull.v1.**
-dontwarn org.slf4j.**

# Jetty
-keep class org.eclipse.** { *; }
-keepclasseswithmembers class * {
@org.eclipse.** *;
}
-keepclassmembers class * {
@org.eclipse.** *;
}
-dontwarn "org.eclipse.jetty.**"

Explaining why classes and class members are being kept...
Printing usage to [/Users/david/jenkins/workspace/appName/desktop/build/usage.txt]...
Removing unused program classes and class elements...
  Original number of program classes:            11757
  Final number of program classes:               9962
Optimizing (pass 1/3)...
  Number of finalized classes:                   4797
  Number of unboxed enum classes:                0
  Number of vertically merged classes:           0   (disabled)
  Number of horizontally merged classes:         0   (disabled)
  Number of merged wrapper classes:              0   (disabled)
  Number of removed write-only fields:           0   (disabled)
  Number of privatized fields:                   0   (disabled)
  Number of generalized field accesses:          0   (disabled)
  Number of specialized field types:             0   (disabled)
  Number of inlined constant fields:             0   (disabled)
  Number of privatized methods:                  2512
  Number of staticized methods:                  877
  Number of finalized methods:                   21555
  Number of desynchronized methods:              0
  Number of simplified method signatures:        911
  Number of removed method parameters:           1156
  Number of generalized method invocations:      0
  Number of specialized method parameter types:  0
  Number of specialized method return types:     0
  Number of inlined constant parameters:         664
  Number of inlined constant return values:      78
  Number of inlined short method calls:          4674
  Number of inlined unique method calls:         3197
  Number of inlined tail recursion calls:        44
  Number of merged code blocks:                  104
  Number of variable peephole optimizations:     32078
  Number of arithmetic peephole optimizations:   1509
  Number of cast peephole optimizations:         0   (disabled)
  Number of field peephole optimizations:        137
  Number of branch peephole optimizations:       5602
  Number of object peephole optimizations:       2377
  Number of string peephole optimizations:       1822
  Number of math peephole optimizations:         0
  Number of simplified instructions:             2046
  Number of removed instructions:                24610
  Number of removed local variables:             1275
  Number of removed exception blocks:            611
  Number of optimized local variable frames:     9113
Shrinking...
Removing unused program classes and class elements...
  Original number of program classes:            9962
  Final number of program classes:               9736
Optimizing (pass 2/3)...
  Number of finalized classes:                   4
  Number of unboxed enum classes:                0
  Number of vertically merged classes:           0   (disabled)
  Number of horizontally merged classes:         0   (disabled)
  Number of merged wrapper classes:              0   (disabled)
  Number of removed write-only fields:           0   (disabled)
  Number of privatized fields:                   0   (disabled)
  Number of generalized field accesses:          0   (disabled)
  Number of specialized field types:             0   (disabled)
  Number of inlined constant fields:             0   (disabled)
  Number of privatized methods:                  10
  Number of staticized methods:                  104
  Number of finalized methods:                   116
  Number of desynchronized methods:              0
  Number of simplified method signatures:        384
  Number of removed method parameters:           508
  Number of generalized method invocations:      0
  Number of specialized method parameter types:  0
  Number of specialized method return types:     0
  Number of inlined constant parameters:         834
  Number of inlined constant return values:      41
  Number of inlined short method calls:          470
  Number of inlined unique method calls:         193
  Number of inlined tail recursion calls:        1
  Number of merged code blocks:                  3
  Number of variable peephole optimizations:     2895
  Number of arithmetic peephole optimizations:   52
  Number of cast peephole optimizations:         0   (disabled)
  Number of field peephole optimizations:        0
  Number of branch peephole optimizations:       174
  Number of object peephole optimizations:       0
  Number of string peephole optimizations:       430
  Number of math peephole optimizations:         0
  Number of simplified instructions:             975
  Number of removed instructions:                18026
  Number of removed local variables:             707
  Number of removed exception blocks:            0
  Number of optimized local variable frames:     678
Shrinking...
Removing unused program classes and class elements...
  Original number of program classes:            9736
  Final number of program classes:               9732
Optimizing (pass 3/3)...
  Number of finalized classes:                   1
  Number of unboxed enum classes:                0
  Number of vertically merged classes:           0   (disabled)
  Number of horizontally merged classes:         0   (disabled)
  Number of merged wrapper classes:              0   (disabled)
  Number of removed write-only fields:           0   (disabled)
  Number of privatized fields:                   0   (disabled)
  Number of generalized field accesses:          0   (disabled)
  Number of specialized field types:             0   (disabled)
  Number of inlined constant fields:             0   (disabled)
  Number of privatized methods:                  0
  Number of staticized methods:                  44
  Number of finalized methods:                   23
  Number of desynchronized methods:              0
  Number of simplified method signatures:        108
  Number of removed method parameters:           132
  Number of generalized method invocations:      0
  Number of specialized method parameter types:  0
  Number of specialized method return types:     0
  Number of inlined constant parameters:         404
  Number of inlined constant return values:      32
  Number of inlined short method calls:          0
  Number of inlined unique method calls:         7
  Number of inlined tail recursion calls:        0
  Number of merged code blocks:                  2
  Number of variable peephole optimizations:     244
  Number of arithmetic peephole optimizations:   2
  Number of cast peephole optimizations:         0   (disabled)
  Number of field peephole optimizations:        0
  Number of branch peephole optimizations:       7
  Number of object peephole optimizations:       43
  Number of string peephole optimizations:       12
  Number of math peephole optimizations:         0
  Number of simplified instructions:             159
  Number of removed instructions:                4741
  Number of removed local variables:             127
  Number of removed exception blocks:            0
  Number of optimized local variable frames:     43
Shrinking...
Removing unused program classes and class elements...
  Original number of program classes:            9732
  Final number of program classes:               9732
Obfuscating...
Printing mapping to [/Users/david/jenkins/workspace/appName/desktop/build/mapping.txt]...
  Number of obfuscated classes:                  6306
  Number of obfuscated fields:                   18156
  Number of obfuscated methods:                  22881
Preverifying...
Writing output...

The text was updated successfully, but these errors were encountered:

mrjameshamilton · 2022-10-07T16:41:21Z

Hi @davidgiga1993 !

Great to hear you're a long time user of ProGuard!

There are a couple of issues reported with performance reduction already (#79 #91) but there was never any conclusion or reproducible sample.

Is there a way to profile what exactly takes proguard so long?

An easy way to narrow down the timing of each feature is to pass the log output through the ts command to add a timestamp to every line of logging output. That way you can see which parts take longest.

java -jar proguard.jar @config.pro | ts '[%Y-%m-%d %H:%M:%S]' &> build.log

Is performance something you want improve in the future?

We don't have any specific plans but if you can provide some reproducible sample and we can narrow down the issue that will be a good start for solving any performance problems.

davidgiga1993 · 2022-10-07T19:47:14Z

Thanks for the fast reply!
Not sure how helpful the output will be since most of the time is spent after printing Optimizing (pass X/3).... Each pass takes about 6-8 minutes. I even reduced the number of passes, in the past I was using 5 but it just took too long.

Similar to the other issues, providing a sample will be a bit tricky since it's the IP of my company, but since each optimization step takes so long, would it be possible to (privately) provide the application jar after one pass of obfuscation?

Also as a sidenode, when building the app for android, R8 takes about 2 minutes with the same proguard config file.

# This was done on a 5950X, on actual build servers it takes longer
[2022-10-07 21:35:32] Ignoring unused library classes...
[2022-10-07 21:35:32]   Original number of library classes: 13158
[2022-10-07 21:35:32]   Final number of library classes:    1067
[2022-10-07 21:35:32] Marking classes and class members to be kept...
[2022-10-07 21:35:33] Inlining subroutines...
[2022-10-07 21:35:33] Shrinking...
[2022-10-07 21:35:33] Explaining why classes and class members are being kept...
[2022-10-07 21:35:35] Removing unused program classes and class elements...
[2022-10-07 21:35:35]   Original number of program classes:            11757
[2022-10-07 21:35:35]   Final number of program classes:               10039
[2022-10-07 21:35:35] Optimizing (pass 1/3)...
[2022-10-07 21:41:26]   Number of finalized classes:                   4843
[2022-10-07 21:41:26]   Number of unboxed enum classes:                0
[2022-10-07 21:41:26]   Number of vertically merged classes:           0   (disabled)
[2022-10-07 21:41:26]   Number of horizontally merged classes:         0   (disabled)
[2022-10-07 21:41:26]   Number of merged wrapper classes:              0   (disabled)
[2022-10-07 21:41:26]   Number of removed write-only fields:           0   (disabled)
[2022-10-07 21:41:26]   Number of privatized fields:                   0   (disabled)
[2022-10-07 21:41:26]   Number of generalized field accesses:          0   (disabled)
[2022-10-07 21:41:26]   Number of specialized field types:             0   (disabled)
[2022-10-07 21:41:26]   Number of inlined constant fields:             0   (disabled)
[2022-10-07 21:41:26]   Number of privatized methods:                  2528
[2022-10-07 21:41:26]   Number of staticized methods:                  884
[2022-10-07 21:41:26]   Number of finalized methods:                   21830
[2022-10-07 21:41:26]   Number of desynchronized methods:              0
[2022-10-07 21:41:26]   Number of simplified method signatures:        900
[2022-10-07 21:41:26]   Number of removed method parameters:           1151
[2022-10-07 21:41:26]   Number of generalized method invocations:      0
[2022-10-07 21:41:26]   Number of specialized method parameter types:  0
[2022-10-07 21:41:26]   Number of specialized method return types:     0
[2022-10-07 21:41:26]   Number of inlined constant parameters:         652
[2022-10-07 21:41:26]   Number of inlined constant return values:      78
[2022-10-07 21:41:26]   Number of inlined short method calls:          4651
[2022-10-07 21:41:26]   Number of inlined unique method calls:         3182
[2022-10-07 21:41:26]   Number of inlined tail recursion calls:        45
[2022-10-07 21:41:26]   Number of merged code blocks:                  96
[2022-10-07 21:41:26]   Number of variable peephole optimizations:     32212
[2022-10-07 21:41:26]   Number of arithmetic peephole optimizations:   1528
[2022-10-07 21:41:26]   Number of cast peephole optimizations:         0   (disabled)
[2022-10-07 21:41:26]   Number of field peephole optimizations:        140
[2022-10-07 21:41:26]   Number of branch peephole optimizations:       5627
[2022-10-07 21:41:26]   Number of object peephole optimizations:       2322
[2022-10-07 21:41:26]   Number of string peephole optimizations:       2528
[2022-10-07 21:41:26]   Number of math peephole optimizations:         0
[2022-10-07 21:41:26]   Number of simplified instructions:             2048
[2022-10-07 21:41:26]   Number of removed instructions:                25061
[2022-10-07 21:41:26]   Number of removed local variables:             1276
[2022-10-07 21:41:26]   Number of removed exception blocks:            611
[2022-10-07 21:41:26]   Number of optimized local variable frames:     9197
[2022-10-07 21:41:26] Shrinking...
[2022-10-07 21:41:27] Removing unused program classes and class elements...
[2022-10-07 21:41:27]   Original number of program classes:            10039
[2022-10-07 21:41:27]   Final number of program classes:               9813
[2022-10-07 21:41:27] Optimizing (pass 2/3)...
[2022-10-07 21:47:16]   Number of finalized classes:                   4
[2022-10-07 21:47:16]   Number of unboxed enum classes:                0
[2022-10-07 21:47:16]   Number of vertically merged classes:           0   (disabled)
[2022-10-07 21:47:16]   Number of horizontally merged classes:         0   (disabled)
[2022-10-07 21:47:16]   Number of merged wrapper classes:              0   (disabled)
[2022-10-07 21:47:16]   Number of removed write-only fields:           0   (disabled)
[2022-10-07 21:47:16]   Number of privatized fields:                   0   (disabled)
[2022-10-07 21:47:16]   Number of generalized field accesses:          0   (disabled)
[2022-10-07 21:47:16]   Number of specialized field types:             0   (disabled)
[2022-10-07 21:47:16]   Number of inlined constant fields:             0   (disabled)
[2022-10-07 21:47:16]   Number of privatized methods:                  11
[2022-10-07 21:47:16]   Number of staticized methods:                  105
[2022-10-07 21:47:16]   Number of finalized methods:                   114
[2022-10-07 21:47:16]   Number of desynchronized methods:              0
[2022-10-07 21:47:16]   Number of simplified method signatures:        369
[2022-10-07 21:47:16]   Number of removed method parameters:           518
[2022-10-07 21:47:16]   Number of generalized method invocations:      0
[2022-10-07 21:47:16]   Number of specialized method parameter types:  0
[2022-10-07 21:47:16]   Number of specialized method return types:     0
[2022-10-07 21:47:16]   Number of inlined constant parameters:         816
[2022-10-07 21:47:16]   Number of inlined constant return values:      41
[2022-10-07 21:47:16]   Number of inlined short method calls:          471
[2022-10-07 21:47:16]   Number of inlined unique method calls:         181
[2022-10-07 21:47:16]   Number of inlined tail recursion calls:        1
[2022-10-07 21:47:16]   Number of merged code blocks:                  3
[2022-10-07 21:47:16]   Number of variable peephole optimizations:     2893
[2022-10-07 21:47:16]   Number of arithmetic peephole optimizations:   52
[2022-10-07 21:47:16]   Number of cast peephole optimizations:         0   (disabled)
[2022-10-07 21:47:16]   Number of field peephole optimizations:        0
[2022-10-07 21:47:16]   Number of branch peephole optimizations:       177
[2022-10-07 21:47:16]   Number of object peephole optimizations:       34
[2022-10-07 21:47:16]   Number of string peephole optimizations:       484
[2022-10-07 21:47:16]   Number of math peephole optimizations:         0
[2022-10-07 21:47:16]   Number of simplified instructions:             951
[2022-10-07 21:47:16]   Number of removed instructions:                18035
[2022-10-07 21:47:16]   Number of removed local variables:             708
[2022-10-07 21:47:16]   Number of removed exception blocks:            0
[2022-10-07 21:47:16]   Number of optimized local variable frames:     685
[2022-10-07 21:47:16] Shrinking...
[2022-10-07 21:47:16] Removing unused program classes and class elements...
[2022-10-07 21:47:16]   Original number of program classes:            9813
[2022-10-07 21:47:16]   Final number of program classes:               9809
[2022-10-07 21:47:16] Optimizing (pass 3/3)...
[2022-10-07 21:52:59]   Number of finalized classes:                   1
[2022-10-07 21:52:59]   Number of unboxed enum classes:                0
[2022-10-07 21:52:59]   Number of vertically merged classes:           0   (disabled)
[2022-10-07 21:52:59]   Number of horizontally merged classes:         0   (disabled)
[2022-10-07 21:52:59]   Number of merged wrapper classes:              0   (disabled)
[2022-10-07 21:52:59]   Number of removed write-only fields:           0   (disabled)
[2022-10-07 21:52:59]   Number of privatized fields:                   0   (disabled)
[2022-10-07 21:52:59]   Number of generalized field accesses:          0   (disabled)
[2022-10-07 21:52:59]   Number of specialized field types:             0   (disabled)
[2022-10-07 21:52:59]   Number of inlined constant fields:             0   (disabled)
[2022-10-07 21:52:59]   Number of privatized methods:                  0
[2022-10-07 21:52:59]   Number of staticized methods:                  44
[2022-10-07 21:52:59]   Number of finalized methods:                   23
[2022-10-07 21:52:59]   Number of desynchronized methods:              0
[2022-10-07 21:52:59]   Number of simplified method signatures:        101
[2022-10-07 21:52:59]   Number of removed method parameters:           120
[2022-10-07 21:52:59]   Number of generalized method invocations:      0
[2022-10-07 21:52:59]   Number of specialized method parameter types:  0
[2022-10-07 21:52:59]   Number of specialized method return types:     0
[2022-10-07 21:52:59]   Number of inlined constant parameters:         367
[2022-10-07 21:52:59]   Number of inlined constant return values:      32
[2022-10-07 21:52:59]   Number of inlined short method calls:          0
[2022-10-07 21:52:59]   Number of inlined unique method calls:         8
[2022-10-07 21:52:59]   Number of inlined tail recursion calls:        0
[2022-10-07 21:52:59]   Number of merged code blocks:                  2
[2022-10-07 21:52:59]   Number of variable peephole optimizations:     243
[2022-10-07 21:52:59]   Number of arithmetic peephole optimizations:   2
[2022-10-07 21:52:59]   Number of cast peephole optimizations:         0   (disabled)
[2022-10-07 21:52:59]   Number of field peephole optimizations:        0
[2022-10-07 21:52:59]   Number of branch peephole optimizations:       12
[2022-10-07 21:52:59]   Number of object peephole optimizations:       43
[2022-10-07 21:52:59]   Number of string peephole optimizations:       12
[2022-10-07 21:52:59]   Number of math peephole optimizations:         0
[2022-10-07 21:52:59]   Number of simplified instructions:             137
[2022-10-07 21:52:59]   Number of removed instructions:                4807
[2022-10-07 21:52:59]   Number of removed local variables:             127
[2022-10-07 21:52:59]   Number of removed exception blocks:            0
[2022-10-07 21:52:59]   Number of optimized local variable frames:     44
[2022-10-07 21:52:59] Shrinking...
[2022-10-07 21:53:00] Removing unused program classes and class elements...
[2022-10-07 21:53:00]   Original number of program classes:            9809
[2022-10-07 21:53:00]   Final number of program classes:               9808
[2022-10-07 21:53:00] Obfuscating...
[2022-10-07 21:53:03]   Number of obfuscated classes:                  6382
[2022-10-07 21:53:03]   Number of obfuscated fields:                   18376
[2022-10-07 21:53:03]   Number of obfuscated methods:                  23323
[2022-10-07 21:53:03] Preverifying...
[2022-10-07 21:53:51] Writing output...
[2022-10-07 21:53:51] Preparing output jar [test.jar] (filtered)
[2022-10-07 21:53:51]   Copying resources from program jar [desktop-debug.jar] (filtered)

davidgiga1993 · 2022-10-08T08:28:45Z

I did a bit more profiling and it looks like the most time is spent in proguard.evaluation.Variables.generalize(proguard.evaluation.Variables, boolean).
Drilling down a bit 44% of that methods time is spent in proguard.evaluation.value.InstructionOffsetValue.contains(int) which is a simple loop search. Replacing it with an HashMap for fast lookups might be a solution.

mrjameshamilton · 2022-10-08T09:24:03Z

Nice work tracking that down!

EricLafortune · 2022-10-08T09:45:16Z

I originally gave InstructionOffsetValue an array of ints instead of a HashSet of Integers because the array typically only contains one or two elements and there are many instances of them during the abstract evaluation of a method. So my goal was to keep the memory use and garbage collection in check. Automatically generated code with huge switch/case statements breaks the assumption. You should perhaps check the increase in memory use for large methods.

mrjameshamilton · 2023-01-16T08:51:58Z

Hi @davidgiga1993 ! Thanks for your contribution. The fix you made is now in ProGuard 7.3.1.

davidgiga1993 mentioned this issue Oct 8, 2022

Improve InstructionOffsetValue performance Guardsquare/proguard-core#73

Merged

mrjameshamilton closed this as completed Jan 16, 2023

jrhee17 mentioned this issue Feb 14, 2023

Fix flakiness with the CI build line/armeria#4643

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Poor performance with larger code base #283

Poor performance with larger code base #283

davidgiga1993 commented Oct 7, 2022

mrjameshamilton commented Oct 7, 2022

davidgiga1993 commented Oct 7, 2022 •

edited

Loading

davidgiga1993 commented Oct 8, 2022

mrjameshamilton commented Oct 8, 2022

EricLafortune commented Oct 8, 2022 via email

mrjameshamilton commented Jan 16, 2023

Poor performance with larger code base #283

Poor performance with larger code base #283

Comments

davidgiga1993 commented Oct 7, 2022

mrjameshamilton commented Oct 7, 2022

davidgiga1993 commented Oct 7, 2022 • edited Loading

davidgiga1993 commented Oct 8, 2022

mrjameshamilton commented Oct 8, 2022

EricLafortune commented Oct 8, 2022 via email

mrjameshamilton commented Jan 16, 2023

davidgiga1993 commented Oct 7, 2022 •

edited

Loading