Add debug mode compilers by wtwhite · Pull Request #83 · binaryeq/jcompile

wtwhite · 2024-06-14T02:40:46Z

Resolves #81.

First test run using inject-debug-mode-compiler.pl to add the -g option to checkstyle-10.12.3 produces a slightly larger jar:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ls -ltra jars/openjdk-debug-20.0.1/checkstyle-10.12.3*
-rw-rw-r-- 1 wtwhite wtwhite 2094082 Jun 14 14:35 jars/openjdk-debug-20.0.1/checkstyle-10.12.3.jar
-rw-rw-r-- 1 wtwhite wtwhite 3842430 Jun 14 14:35 jars/openjdk-debug-20.0.1/checkstyle-10.12.3-tests.jar
-rw-rw-r-- 1 wtwhite wtwhite     826 Jun 14 14:35 jars/openjdk-debug-20.0.1/checkstyle-10.12.3.jar.generated-sources
-rw-rw-r-- 1 wtwhite wtwhite 1384763 Jun 14 14:35 jars/openjdk-debug-20.0.1/checkstyle-10.12.3.jar.bytecode-features
-rw-rw-r-- 1 wtwhite wtwhite       0 Jun 14 14:35 jars/openjdk-debug-20.0.1/checkstyle-10.12.3.jar.done
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ^-debug
ls -ltra jars/openjdk-20.0.1/checkstyle-10.12.3*
-rw-rw-r-- 1 wtwhite wtwhite 2093468 Jun 13 18:40 jars/openjdk-20.0.1/checkstyle-10.12.3.jar
-rw-rw-r-- 1 wtwhite wtwhite 3842430 Jun 13 18:40 jars/openjdk-20.0.1/checkstyle-10.12.3-tests.jar
-rw-rw-r-- 1 wtwhite wtwhite     826 Jun 13 18:40 jars/openjdk-20.0.1/checkstyle-10.12.3.jar.generated-sources
-rw-rw-r-- 1 wtwhite wtwhite 1384763 Jun 13 18:41 jars/openjdk-20.0.1/checkstyle-10.12.3.jar.bytecode-features
-rw-rw-r-- 1 wtwhite wtwhite       0 Jun 13 18:41 jars/openjdk-20.0.1/checkstyle-10.12.3.jar.done

…9, 17.0.7, 20.0.1)

wtwhite · 2024-06-14T02:46:08Z

But despite that small size difference, the classfiles in the two jars are actually identical except for timestamps:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ diff <(unzip -lv jars/openjdk-20.0.1/checkstyle-10.12.3.jar|grep -F .class|perl -lpe 's/\d\d \d\d:\d\d /DD XX:XX /') <(unzip -lv jars/openjdk-debug-20.0.1/checkstyle-10.12.3.jar|grep -F .class|perl -lpe 's/\d\d \d\d:\d\d /DD XX:XX /')|wc -l
0

wtwhite · 2024-06-14T03:13:37Z

The problem seems to be that checkstyle-10.12.3 already specifies <plugin> sections for maven-compiler-plugin (multiple ones in fact -- probably under different profiles), and these probably aren't merged with the section the script injects. Detecting this situation and fixing it requires slightly more clever XML wrangling.

Since inject-ecj-compiler.pl works in a very similar way, I suspect that it might be broken in the same way on projects like this.

…s that don't already have them

…in>s but not all

…xecution>...</execution></executions>

… .bak

wtwhite · 2024-06-17T02:00:43Z

The following snippet culled from checkstyle-10.12.3's POM hits many cases that we need to handle:

Existing <configuration> within an <execution> that lacks <compilerArgs>
Existing <configuration> within an <execution> that already has <compilerArgs>
Missing <configuration> directly within a maven-compiler-plugin <plugin>

wtwhite@wtwhite-vuw-vm:~/code/jcompile/tools$ node inject-debug-mode-compiler.js test.xml
/home/wtwhite/code/jcompile/tools/inject-debug-mode-compiler.js starting. Will process test.xml, renaming the original to test.xml.bak.bak.
Found existing <plugins> element within a <build> element!
Found existing maven-compiler-plugin element!
Removing its parent from the list of <plugins> elements without a compiler plugin.
Will inject 0 maven-compiler-plugin plugins into existing <plugins> elements that don't have them yet:
Will modify 1 maven-compiler-plugin <plugin> elements by adding '-g' arguments:
/home/wtwhite/code/jcompile/tools/inject-debug-mode-compiler.js finished.
wtwhite@wtwhite-vuw-vm:~/code/jcompile/tools$ sdiff -W test.xml.bak test.xml
<?xml version="1.0" encoding="UTF-8"?>				<?xml version="1.0" encoding="UTF-8"?>
<project>							<project>
  <profiles>							  <profiles>
    <profile>							    <profile>
      <id>error-prone-compile</id>				      <id>error-prone-compile</id>
      <build>							      <build>
        <plugins>						        <plugins>
          <plugin>						          <plugin>
            <groupId>org.apache.maven.plugins</groupId>		            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>	            <artifactId>maven-compiler-plugin</artifactId>
            <version>${maven.compiler.plugin.version}</versio	            <version>${maven.compiler.plugin.version}</versio
            <executions>					            <executions>
              <!-- Skip the default-compile execution as we d	              <!-- Skip the default-compile execution as we d
              <execution>					              <execution>
                <id>default-compile</id>			                <id>default-compile</id>
                <phase>compile</phase>				                <phase>compile</phase>
                <goals>						                <goals>
                  <goal>compile</goal>				                  <goal>compile</goal>
                </goals>					                </goals>
                <configuration>					                <configuration>
                  <skipMain>true</skipMain>			                  <skipMain>true</skipMain>
							      >	                  <!--Inserted automatically by /home/wtwhite
							      >	                  <compilerArgs>
							      >	                    <arg>-g</arg>
							      >	                  </compilerArgs>
                </configuration>				                </configuration>
              </execution>					              </execution>
              <execution>					              <execution>
                <id>error-prone-compile</id>			                <id>error-prone-compile</id>
                <phase>compile</phase>				                <phase>compile</phase>
                <goals>						                <goals>
                  <goal>compile</goal>				                  <goal>compile</goal>
                </goals>					                </goals>
                <configuration>					                <configuration>
                  <failOnError>false</failOnError>		                  <failOnError>false</failOnError>
                  <source>${java.version}</source>		                  <source>${java.version}</source>
                  <target>${java.version}</target>		                  <target>${java.version}</target>
                  <compilerArgs>				                  <compilerArgs>
                    <arg>-Xpkginfo:always</arg>			                    <arg>-Xpkginfo:always</arg>
                    <arg>-XDcompilePolicy=simple</arg>		                    <arg>-XDcompilePolicy=simple</arg>
                    <arg>					                    <arg>
                      -Xplugin:ErrorProne			                      -Xplugin:ErrorProne
                    </arg>					                    </arg>
							      >	                    <!--Inserted automatically by /home/wtwhi
							      >	                    <arg>-g</arg>
                  </compilerArgs>				                  </compilerArgs>
                  <annotationProcessorPaths>			                  <annotationProcessorPaths>
                    <path>					                    <path>
                      <groupId>com.google.errorprone</groupId	                      <groupId>com.google.errorprone</groupId
                      <artifactId>error_prone_core</artifactI	                      <artifactId>error_prone_core</artifactI
                      <version>${error-prone.version}</versio	                      <version>${error-prone.version}</versio
                    </path>					                    </path>
                  </annotationProcessorPaths>			                  </annotationProcessorPaths>
                </configuration>				                </configuration>
              </execution>					              </execution>
            </executions>					            </executions>
							      >	            <!--Inserted automatically by /home/wtwhite/code/
							      >	            <configuration>
							      >	              <compilerArgs>
							      >	                <arg>-g</arg>
							      >	              </compilerArgs>
							      >	            </configuration>
          </plugin>						          </plugin>
        </plugins>						        </plugins>
      </build>							      </build>
    </profile>							    </profile>
  </profiles>							  </profiles>
</project>							</project>

Another snippet from the same POM hits the case of needing to inject a complete <plugin> into a <build><plugins>...</plugins></build> that has no existing maven-compiler-plugin in it:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/tools$ node inject-debug-mode-compiler.js test_whole_new_plugin.xml 
/home/wtwhite/code/jcompile/tools/inject-debug-mode-compiler.js starting. Will process test_whole_new_plugin.xml, renaming the original to test_whole_new_plugin.xml.bak.bak.
Found existing <plugins> element within a <build> element!
Will inject 1 maven-compiler-plugin plugins into existing <plugins> elements that don't have them yet:
Will modify 0 maven-compiler-plugin <plugin> elements by adding '-g' arguments:
/home/wtwhite/code/jcompile/tools/inject-debug-mode-compiler.js finished.
wtwhite@wtwhite-vuw-vm:~/code/jcompile/tools$ sdiff -W test_whole_new_plugin.xml.bak test_whole_new_plugin.xml
<?xml version="1.0" encoding="UTF-8"?>				<?xml version="1.0" encoding="UTF-8"?>
<project>							<project>
  <profiles>							  <profiles>
    <profile>							    <profile>
      <!-- To be used during development. Run the command -->	      <!-- To be used during development. Run the command -->
      <!-- mvn -Pno-validations site -->			      <!-- mvn -Pno-validations site -->
      <id>no-validations</id>					      <id>no-validations</id>
      <properties>						      <properties>
        <skipTests>true</skipTests>				        <skipTests>true</skipTests>
        <checkstyle.ant.skip>true</checkstyle.ant.skip>		        <checkstyle.ant.skip>true</checkstyle.ant.skip>
        <pmd.skip>true</pmd.skip>				        <pmd.skip>true</pmd.skip>
        <spotbugs.skip>true</spotbugs.skip>			        <spotbugs.skip>true</spotbugs.skip>
        <xml.skip>true</xml.skip>				        <xml.skip>true</xml.skip>
        <forbiddenapis.skip>true</forbiddenapis.skip>		        <forbiddenapis.skip>true</forbiddenapis.skip>
        <jacoco.skip>true</jacoco.skip>				        <jacoco.skip>true</jacoco.skip>
        <maven.javadoc.skip>true</maven.javadoc.skip>		        <maven.javadoc.skip>true</maven.javadoc.skip>
        <linkcheck.skip>true</linkcheck.skip>			        <linkcheck.skip>true</linkcheck.skip>
        <jdepend.skip>true</jdepend.skip>			        <jdepend.skip>true</jdepend.skip>
        <modernizer.skip>true</modernizer.skip>			        <modernizer.skip>true</modernizer.skip>
        <tidy.skip>true</tidy.skip>				        <tidy.skip>true</tidy.skip>
      </properties>						      </properties>
      <build>							      <build>
        <plugins>						        <plugins>
          <!-- disable json validation plugin since it has no	          <!-- disable json validation plugin since it has no
          <plugin>						          <plugin>
            <groupId>com.groupon.maven.plugin.json</groupId>	            <groupId>com.groupon.maven.plugin.json</groupId>
            <artifactId>json-schema-validator</artifactId>	            <artifactId>json-schema-validator</artifactId>
            <version>${json-schema-validator.version}</versio	            <version>${json-schema-validator.version}</versio
            <executions>					            <executions>
              <execution>					              <execution>
                <phase>none</phase>				                <phase>none</phase>
              </execution>					              </execution>
            </executions>					            </executions>
          </plugin>						          </plugin>
							      >	          <!--Inserted automatically by /home/wtwhite/code/jc
							      >	          <plugin>
							      >	            <artifactId>maven-compiler-plugin</artifactId>
							      >	            <configuration>
							      >	              <compilerArgs>
							      >	                <arg>-g</arg>
							      >	              </compilerArgs>
							      >	            </configuration>
							      >	          </plugin>
        </plugins>						        </plugins>
      </build>							      </build>
    </profile>							    </profile>
  </profiles>							  </profiles>
</project>							</project>
wtwhite@wtwhite-vuw-vm:~/code/jcompile/tools$

wtwhite · 2024-06-17T02:10:18Z

I thought we were done, but it now complains about the syntax:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ time make jars/openjdk-debug-20.0.1/checkstyle-10.12.3.jar.done 
--snip--
[INFO] --- tidy:1.2.0:check (validate) @ checkstyle ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.159 s
[INFO] Finished at: 2024-06-17T01:48:04Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:tidy-maven-plugin:1.2.0:check (validate) on project checkstyle: The POM violates the code style. Please format it by running `mvn tidy:pom`. -> [Help 1]

Running mvn tidy:pom downloads some packages and then runs happily, so I think I'll just always run it afterwards.

wtwhite · 2024-06-17T02:40:51Z

After all this, there is apparently still no difference between the regular and debug builds:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ diff <(unzip -lv jars/openjdk-20.0.1/checkstyle-10.12.3.jar|grep -F .class|perl -lpe 's/\d\d \d\d:\d\d /DD XX:XX /') <(unzip -lv jars/openjdk-debug-20.0.1/checkstyle-10.12.3.jar|grep -F .class|perl -lpe 's/\d\d \d\d:\d\d /DD XX:XX /')|wc -l
0

😡

wtwhite · 2024-06-17T06:10:23Z

Same thing with commons-csv-1.10.0:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ diff <(unzip -lv jars/openjdk-20.0.1/commons-csv-1.10.0.jar|grep -F .class|perl -lpe 's/\d\d \d\d:\d\d /DD XX:XX /') <(unzip -lv jars/openjdk-debug-20.0.1/commons-csv-1.10.0.jar|grep -F .class|perl -lpe 's/\d\d \d\d:\d\d /DD XX:XX /')|wc -l
0

Turns out the regular build includes local variable info too:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ javap -l -classpath jars/openjdk-20.0.1/commons-csv-1.10.0.jar org.apache.commons.csv.CSVRecord|head -20
Compiled from "CSVRecord.java"
public final class org.apache.commons.csv.CSVRecord implements java.io.Serializable, java.lang.Iterable<java.lang.String> {
  org.apache.commons.csv.CSVRecord(org.apache.commons.csv.CSVParser, java.lang.String[], java.lang.String, long, long);
    LineNumberTable:
      line 60: 0
      line 61: 4
      line 62: 10
      line 63: 25
      line 64: 30
      line 65: 35
      line 66: 41
    LocalVariableTable:
      Start  Length  Slot  Name   Signature
          0      42     0  this   Lorg/apache/commons/csv/CSVRecord;
          0      42     1 parser   Lorg/apache/commons/csv/CSVParser;
          0      42     2 values   [Ljava/lang/String;
          0      42     3 comment   Ljava/lang/String;
          0      42     4 recordNumber   J
          0      42     6 characterPosition   J

This is despite the fact that man javac says, among other things:

       -g
              Generates all debugging information, including local variables. By default, only line number and source file information is generated.

Maven's compilation probably uses javax.tools.JavaCompiler instead of javac, and probably configures it with -g or equivalent (or it's configured that way by default). Either way, what we need to do is produce a build with (at least some of) the debugging turned off.

wtwhite · 2024-06-18T00:13:11Z

Obsolete -- see #85.

This XML-munging machinery may still be useful in improving the existing ECJ compiler injection though, which currently uses a brittle regex-based approach.

wtwhite added 3 commits June 14, 2024 14:23

First attempt at adding the -g debug flag to compilation

a9bfa55

Add -g debug mode compilation for 4 OpenJDK versions (8.0.372, 11.0.1…

0f33c45

…9, 17.0.7, 20.0.1)

Fix debug mode injection script

976e811

wtwhite self-assigned this Jun 14, 2024

wtwhite mentioned this pull request Jun 14, 2024

Make inject-ecj-compiler.pl smarter #84

Open

wtwhite added 10 commits June 14, 2024 17:47

WIP on new, smarter, JS-based script for injecting -g into pom.xml

c62b649

Inject maven-compiler-plugin <plugin> elements into <plugins> element…

f0f414a

…s that don't already have them

Modify existing maven-compiler-plugin <plugin> entries to add -g

ca27671

Fix modification of some existing <plugin>maven-compiler-plugin</plug…

30f0e6d

…in>s but not all

Fix bug in adding plugin config and also add it inside <executions><e…

3acc278

…xecution>...</execution></executions>

Add comment before each inserted piece of XML

fcdaf65

Tidy up

05c415c

Take filename or dir argument and modify it, renaming the original to…

9cc89bf

… .bak

Fix extra <plugin> nesting level

8ad4109

Simplify building XML with convenience functions

085cc0e

Run mvn tidy:pom after injecting -g for debug mode

b9ea068

wtwhite mentioned this pull request Jun 17, 2024

Add 4 OpenJDK compilers with debugging turned off #85

Merged

wtwhite closed this Jun 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add debug mode compilers#83

Add debug mode compilers#83
wtwhite wants to merge 14 commits into
mainfrom
add-debug-mode-compilers

wtwhite commented Jun 14, 2024

Uh oh!

wtwhite commented Jun 14, 2024 •

edited

Loading

Uh oh!

wtwhite commented Jun 14, 2024

Uh oh!

wtwhite commented Jun 17, 2024

Uh oh!

wtwhite commented Jun 17, 2024

Uh oh!

wtwhite commented Jun 17, 2024

Uh oh!

wtwhite commented Jun 17, 2024 •

edited

Loading

Uh oh!

wtwhite commented Jun 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

wtwhite commented Jun 14, 2024

Uh oh!

wtwhite commented Jun 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wtwhite commented Jun 14, 2024

Uh oh!

wtwhite commented Jun 17, 2024

Uh oh!

wtwhite commented Jun 17, 2024

Uh oh!

wtwhite commented Jun 17, 2024

Uh oh!

wtwhite commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wtwhite commented Jun 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wtwhite commented Jun 14, 2024 •

edited

Loading

wtwhite commented Jun 17, 2024 •

edited

Loading