Need PMD Apex language support. Would contribute to it...

[rsoesemann]

I contributed a PMD language module for the Salesforce.com Apex language and created a PMD wrapper for CodeClimate https://github.com/Up2Go/codeclimate-apexmetrics.

I would love to have Apex support in Codacy and this this could be easily done as you already have a PMD version.

I'd be more than happy to help.

[rtfpessoa]

Hi @up2go-rsoesemann

We are currently finishing the support for multi-language tools (should be ready next week).
I did not get much time to dive into the details, but I think we probably could just tweak a bit the code and this docker could run for all the supported languages, like Apex, JS, Java, JSP, etc.

Still not sure if it would be better to have a separate docker for each language.

Let me know what you think regarding this. Would you prefer a separate docker?

In the meanwhile you can check the details about our dockers specs here

If you have a way to generate the docs for out patterns.json and/or descriptions.json automatically it would be great help.

[rsoesemann]

@rtfpessoa I fully agree that a unified PMD docker is better. For CodeClimate we simply decided to go with a separate engine for two reasons. Another person was working on a general PMD engine (never finished it ;-) and agility. With our own engine, we could easily bundle PMD snapshots to get out our rules.)

So you plan sound perfect. Please keep me in the loop so I can test it and tweet about ;-)

Sorry for the question, but why should I check the specs? Which kind of help do you want from me?

[rsoesemann]

Are those two JSOn files to configure the PMD internal settings (like the ruleset.xml)? How are you doing it for PMD Java in your current engine? My language module works exactely the same. There is a rulset for it. I also added some CodeClimate specific params to PMD so I'm quite sure we can do the same for Codacy.

Maybe we should have a Skype chat about that.

[rtfpessoa]

Currently we try to keep some docs for each pattern. If Apex patterns have some docs it would be great if we could get an easy way to generate them automatically.

For example, in Java PMD have this https://pmd.github.io/pmd-5.5.2/pmd-java/rules/index.html that we map here https://github.com/codacy/codacy-pmdjava/blob/master/src/main/resources/docs

Usually the pattern specs go in https://github.com/codacy/codacy-pmdjava/blob/master/src/main/resources/docs/patterns.json the descriptions go in https://github.com/codacy/codacy-pmdjava/blob/master/src/main/resources/docs/description/description.json and the extended descriptions go in their own markdowns in https://github.com/codacy/codacy-pmdjava/blob/master/src/main/resources/docs/description.

Do you have anything similar for Apex?

[rtfpessoa]

@up2go-rsoesemann just noticed we wrote a message at the same time. Just sent you an email so we can talk about the skype.

[rsoesemann]

We have the same for the Apex rules here: https://pmd.github.io/latest/pmd-apex/rules/index.html
Looking forward to skype with you. Next week?

[rtfpessoa]

Yes. Monday works for me. I sent you an email to confirm the details.

[rtfpessoa]

@up2go-rsoesemann did you get my email?

[rsoesemann]

Sure. Sorry for not replying yet. But now. ;-)

[rtfpessoa]

Nice to talk with you.

Thanks for the resources:

• Docs: https://github.com/pmd/pmd/tree/master/pmd-apex/src/main/resources/rulesets/apex
• All rules: https://github.com/pmd/pmd/blob/master/pmd-apex/src/main/resources/rulesets/apex/ruleset.xml
• Online version: https://pmd.github.io/latest/pmd-apex/rules/index.html

We are finishing tool multi-language support. As soon as we finish it, we will start updating this docker to support Apex.

Also, we will start to evaluate the support for release channels for tool, (e.g. latest, stable, etc) and maybe speed up the release process.

[rsoesemann]

Dito. I can't wait until Apex is recognized by your PMD engine. Please let me be you beta tester.

[rtfpessoa]

Hi @up2go-rsoesemann,

After an initial delay we finally started working on PMD support for the remaining languages, and I have a couple more questions.

I am using PMD from Scala code, and I am having one issue when I depend on Apex (seems related to the special parser).
Since I am using SBT I only need to add something like "net.sourceforge.pmd" % "pmd-xml" % pmdVersion for the other languages, but Apex fails. Do you have any idea how I can get the remaining deps from a maven repo? If it is not available from maven, can you point me to all the jars needed to run with PMD 5.5.3?

Also do you know if I can get the description XML contents from the PMD programatic api?

[rsoesemann]

@rtfpessoa great that you started. Another great tool using PMD.

I need to pass your question to the two main PMD experts @jsotuyod and @adangel.
Maybe they can help you faster. I need to dig into that as well. What is SBT?

[rtfpessoa]

@up2go-rsoesemann sbt is like maven but for Scala. Since it is fully compatible with Maven dependencies it should not be a problem.

[jsotuyod]

@rtfpessoa @up2go-rsoesemann I guess the dependencies you are missing are apex jorje dependencies.

They can be found here. They haven't been published to Maven Central yet by the Salesforce team behind it. They probably will once they open source it.

If you have any further issues, please let me know.

[rsoesemann]

Now I got what you are talking about @rtfpessoa. Yes, we are using a custom closed source parser from the Salesforce Eclipse IDE.

Salesforce will eventually open source their parser and then its available via Maven repos. Before that they need to be bundled with PMD.

[rtfpessoa]

@jsotuyod thank you. About the PMD description XMLs. Is there any way to read them programmatically from Java/Scala?

[jsotuyod]

@rtfpessoa All rules descriptions are available in the ruleset file that defines them (the PMD web is actually generated from these by Maven). For instance, java-unusedcode

You could load the xml as a resource from the proper PMD jar (pmd-java, pmd-apex, ...), and parse them just like you are currently parsing the results xml.

[adangel]

@rtfpessoa About the needed dependency for the Apex parser: We do not directly use the linked jar files, but we needed to shade them - there are conflicting versions of asm in the classpath, which would either break apex or other languages.

When we package PMD, we use therefore not these dependencies, but the shaded one. You can get the shaded apex-jorje parser jar from here:
https://repo.maven.apache.org/maven2/net/sourceforge/pmd/pmd-apex/5.5.3/pmd-apex-5.5.3-apex-jorje-shaded.jar or net.sourceforge.pmd:pmd-apex:5.5.3:apex-jorje-shaded. Once the Apex parser is open sourced, we can break it apart to see the transitive dependencies and see the conflicts during dependency resolution rather than at runtime :)

[rtfpessoa]

@adangel @jsotuyod I think I still did not got this. Can you be more precise about exactly what jars do I need to have Apex working? I tried multiple combinations but I always end up with problems.
The main one is that I seem to have a binary conflict with Jackson.

One of my deps has Jackson as a dependency. Does PMD or Apex have Jackson as a dependency? I could not find it.

[rsoesemann]

@rtfpessoa maybe you overlooked this link: https://github.com/pmd/pmd/tree/master/pmd-apex/repo

This is a as I understand it local/fake Maven repo to get the 2 needed jars. This page also contains links to the original files in the Salesforce repo.

[rtfpessoa]

@up2go-rsoesemann should I use those two jars plus the pmd-apex jar in here? Or do I also need the shaded jar here?

Any thoughts about the Jackson problem?

[adangel]

@rtfpessoa It depends on the context: If you are building a tool, that only supports apex and not more, then you would be probably fine to use the two apex jars directly. However, since these two jars are "über-jars", which contain all sorts of outdated libraries, you very soon get class loading problems at runtime (NoSuchMethodError, IncompatibleClassChangeError, etc.). That's why I created the shaded jar, which is used, when we package PMD.
I found problems, when using the pmd-java module while pmd-apex was on the classpath - since the original apex jars shipped an older, incompatible version of asm, pmd-java was broken.

So, in short: you'll need to use the shaded jar instead of the two original apex jars.

About the Jackson problem: I don't see, that PMD has a dependency on it. The apex "über-jar" seems to contain some version of apache-commons-httpclient, javax-SOAP api, SOAP implementation from IBM, javax WSDL api, ... many others.... and com.fasterxml.jackson ... It seems, it is version 2.2.3 (com.fasterxml.jackson.core:jackson-core:2.2.3).

If you want use a different version of jackson, you'll probably need to shade this package, too. See here for how I relocated asm. Since the apex jorje jars are self-contained (they bring every dependency, they need), this should work without problems.

Once you've verified, that this is the issue, feel free to open a PR/Issue at PMD, so that we can relocate more packages and provide an improved apex shaded jar.

For reference, these seem to be the dependencies of the apex jars:

$ zipinfo pmd-apex-5.5.3-apex-jorje-shaded.jar |grep pom.xml
-rw----     2.0 fat     6150 bl defN 17-Jan-28 14:55 META-INF/maven/com.orientechnologies/orientdb-graphdb/pom.xml
-rw----     2.0 fat     4075 bl defN 17-Jan-28 14:55 META-INF/maven/com.orientechnologies/orientdb-server/pom.xml
-rw----     2.0 fat     2258 bl defN 17-Jan-28 14:55 META-INF/maven/com.orientechnologies/orientdb-client/pom.xml
-rw----     2.0 fat     1904 bl defN 17-Jan-28 14:55 META-INF/maven/com.orientechnologies/orientdb-enterprise/pom.xml
-rw----     2.0 fat     2829 bl defN 17-Jan-28 14:55 META-INF/maven/com.orientechnologies/orientdb-tools/pom.xml
-rw----     2.0 fat     4951 bl defN 17-Jan-28 14:55 META-INF/maven/com.orientechnologies/orientdb-object/pom.xml
-rw----     2.0 fat     6012 bl defN 17-Jan-28 14:55 META-INF/maven/org.hibernate.javax.persistence/hibernate-jpa-2.0-api/pom.xml
-rw----     2.0 fat     8985 bl defN 17-Jan-28 14:55 META-INF/maven/com.orientechnologies/orientdb-core/pom.xml
-rw----     2.0 fat    11865 bl defN 17-Jan-28 14:55 META-INF/maven/org.xerial.snappy/snappy-java/pom.xml
-rw----     2.0 fat    14593 bl defN 17-Jan-28 14:55 META-INF/maven/com.googlecode.concurrentlinkedhashmap/concurrentlinkedhashmap-lru/pom.xml
-rw----     2.0 fat    12403 bl defN 17-Jan-28 14:55 META-INF/maven/commons-collections/commons-collections/pom.xml
-rw----     2.0 fat     3603 bl defN 17-Jan-28 14:55 META-INF/maven/com.tinkerpop.blueprints/blueprints-core/pom.xml
-rw----     2.0 fat     4330 bl defN 17-Jan-28 14:55 META-INF/maven/org.codehaus.jettison/jettison/pom.xml
-rw----     2.0 fat     5380 bl defN 17-Jan-28 14:55 META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml
-rw----     2.0 fat     1257 bl defN 17-Jan-28 14:55 META-INF/maven/com.fasterxml.jackson.core/jackson-annotations/pom.xml
-rw----     2.0 fat     5976 bl defN 17-Jan-28 14:55 META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.xml
-rw----     2.0 fat    14894 bl defN 17-Jan-28 14:55 META-INF/maven/com.carrotsearch/hppc/pom.xml
-rw----     2.0 fat    16628 bl defN 17-Jan-28 14:55 META-INF/maven/commons-configuration/commons-configuration/pom.xml
-rw----     2.0 fat    13976 bl defN 17-Jan-28 14:55 META-INF/maven/commons-lang/commons-lang/pom.xml
-rw----     2.0 fat     3524 bl defN 17-Jan-28 14:55 META-INF/maven/com.tinkerpop.gremlin/gremlin-java/pom.xml
-rw----     2.0 fat     9569 bl defN 17-Jan-28 14:55 META-INF/maven/com.tinkerpop.gremlin/gremlin-groovy/pom.xml
-rw----     2.0 fat     6672 bl defN 17-Jan-28 14:55 META-INF/maven/com.tinkerpop/pipes/pom.xml
-rw----     2.0 fat     1289 bl defN 17-Jan-28 14:55 META-INF/maven/org.fusesource.hawtjni/hawtjni-runtime/pom.xml
-rw----     2.0 fat    10901 bl defN 17-Jan-28 14:55 META-INF/maven/org.fusesource.jansi/jansi-native/pom.xml
-rw----     2.0 fat    10172 bl defN 17-Jan-28 14:55 META-INF/maven/org.fusesource.jansi/jansi/pom.xml
-rw----     2.0 fat     6359 bl defN 17-Jan-28 14:55 META-INF/maven/jline/jline/pom.xml
-rw----     2.0 fat     9585 bl defN 17-Jan-28 14:55 META-INF/maven/org.javassist/javassist/pom.xml
-rw----     2.0 fat     5666 bl defN 17-Jan-28 14:55 META-INF/maven/com.google.guava/guava/pom.xml
-rw----     2.0 fat     2646 bl defN 17-Jan-28 14:55 META-INF/maven/com.salesforce.apex/apex-jorje-semantic/pom.xml

[rtfpessoa]

Thanks for the help, I think I am closer to the solution. How to I generate the shaded version? Do I need to run the whole PMD build?

[rtfpessoa]

I think I got the shaded jar but now seems like since the version jackson had in apex might have changed the APIs.

EDIT: I only shaded the jackson packages. Do I need to shade apache-commons-httpclient, javax-SOAP api, SOAP too?

[error] (run-main-0) java.lang.VerifyError: Bad type on operand stack
[error] Exception Details:
[error]   Location:
[error]     play/api/libs/json/jackson/JacksonJson$.<init>()V @34: invokespecial
[error]   Reason:
[error]     Type 'com/fasterxml/jackson/databind/ObjectMapper' (current frame, stack[3]) is not assignable to 'com/fasterxml/jackson/core/ObjectCodec'
[error]   Current Frame:
[error]     bci: @34
[error]     flags: { }
[error]     locals: { 'play/api/libs/json/jackson/JacksonJson$' }
[error]     stack: { 'play/api/libs/json/jackson/JacksonJson$', uninitialized 26, uninitialized 26, 'com/fasterxml/jackson/databind/ObjectMapper' }
[error]   Bytecode:
[error]     0x0000000: 2ab7 009d 2ab3 009f 2abb 003e 59b7 00a0
[error]     0x0000010: b200 a5b6 00a9 b500 132a bb00 1d59 2ab7
[error]     0x0000020: 0038 b700 acb5 0017 b1
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    play/api/libs/json/jackson/JacksonJson$.<init>()V @34: invokespecial
  Reason:
    Type 'com/fasterxml/jackson/databind/ObjectMapper' (current frame, stack[3]) is not assignable to 'com/fasterxml/jackson/core/ObjectCodec'
  Current Frame:
    bci: @34
    flags: { }
    locals: { 'play/api/libs/json/jackson/JacksonJson$' }
    stack: { 'play/api/libs/json/jackson/JacksonJson$', uninitialized 26, uninitialized 26, 'com/fasterxml/jackson/databind/ObjectMapper' }
  Bytecode:
    0x0000000: 2ab7 009d 2ab3 009f 2abb 003e 59b7 00a0
    0x0000010: b200 a5b6 00a9 b500 132a bb00 1d59 2ab7
    0x0000020: 0038 b700 acb5 0017 b1

        at play.api.libs.json.Json$.parse(Json.scala:21)
        at codacy.pmd.DocGenerator$$anonfun$codacy$pmd$DocGenerator$$parseParameters$2$$anonfun$17$$anonfun$apply$16.apply(DocGenerator.scala:185)
        at codacy.pmd.DocGenerator$$anonfun$codacy$pmd$DocGenerator$$parseParameters$2$$anonfun$17$$anonfun$apply$16.apply(DocGenerator.scala:185)
        at scala.util.Try$.apply(Try.scala:192)
        at codacy.pmd.DocGenerator$$anonfun$codacy$pmd$DocGenerator$$parseParameters$2$$anonfun$17.apply(DocGenerator.scala:185)
        at codacy.pmd.DocGenerator$$anonfun$codacy$pmd$DocGenerator$$parseParameters$2$$anonfun$17.apply(DocGenerator.scala:184)
        at scala.Option.map(Option.scala:146)
        at codacy.pmd.DocGenerator$$anonfun$codacy$pmd$DocGenerator$$parseParameters$2.apply(DocGenerator.scala:184)
        at codacy.pmd.DocGenerator$$anonfun$codacy$pmd$DocGenerator$$parseParameters$2.apply(DocGenerator.scala:177)

Any ideas to fix this?

[adangel]

Make sure, you have excluded the original apex libraries, when you put everything together. Something like this:

libraryDependencies += "net.sourceforge.pmd" % "pmd-apex" % pmdVersion withSources() exclude("apex", "*")
libraryDependencies += "net.sourceforge.pmd" % "pmd-apex" % pmdVersion classifier "apex-jorje-shaded"

[rtfpessoa]

Since I am creating a jar with Jackson shaded I am just doping the jorje jar in the classpath and I already have this error.

[rtfpessoa]

Actually I might have an idea. I think it might me the databind package not being shaded. Need to check it later.

BTW thanks with the classifier syntax I was not finding It

[rtfpessoa]

Thanks a lot for all your help, I already have it running locally.
I will now finish the tests and make a PR for PMD to shade Jackson.

[rtfpessoa]

Build and tests are passing in https://circleci.com/gh/codacy/codacy-pmdjava/139

[jsotuyod]

@rtfpessoa awesome. Let us know if you need anything else!

[adangel]

@rtfpessoa Btw - I've released yesterday the version 5.5.4. You should now be able to use the shaded jorje jar as a dependency with the classifier from here.

[rtfpessoa]

Amazing @adangel, just created #25 where I got rid of the jars.

[rsoesemann]

@rtfpessoa Looks like you are nearly done. Feel free to test the Apex language module using our public Apex repos at https://github.com/up2go.

[rtfpessoa]

@up2go-rsoesemann I will ping you as soon as we put this online. Should not take much more now that the tool side is complete and the backend should also be ready.

[rtfpessoa]

@up2go-rsoesemann we just deployed PMD with Apex. When you add the projects make sure you are either using a configuration file ruleset.xml or that you enable the Apex patterns you your account before adding the project.

[rsoesemann]

@rtfpessoa Why is codacy not getting the rule descriptions right? For all PMD rules it is just linking the PMD pages. On the right side there is CodeClimate. For their engine I output the full description for every single issue.

How are you doing it?

[rtfpessoa]

I think I might have forgotten to generate the long descriptions. I will take a look into that when I start working on Visualforce, should not be hard.

[rtfpessoa]

Should be fixed in #31