[BUG] UnexpectedException on v3 DFA on large ISV codebase

[rsoesemann]

Describe the bug
Ran v3 Data Flow on HUGE ISV Codebase using

sfdx scanner:run:dfa --target '/Users/rsoesemann/dev/aquivalabs/XXX/' --projectdir '/Users/rsoesemann/dev/aquivalabs/XXX/sfdx-source/xxx' --format html

lead to this error after a few minutes

Evaluating SFGE rules. See /Users/rsoesemann/.sfdx-scanner/sfge.log for logs... Error
ERROR running scanner:run:dfa: Jun 30, 2022 11:53:03 AM apex.jorje.parser.impl.BaseApexLexer dedupe
INFO: Deduped array ApexLexer.DFA22_transition. Found 7075528 shorts which is 13MB not including array overhead. Removed 6418975 shorts which is 12MB not counting array overhead. Took 8 ms.
Exception in thread "main" com.salesforce.exception.UnexpectedException: Vertex is terminal. vertex=ReturnStatement{DefiningType=PriceCategoryTypeSelector, BeginLine=58, EndLine=61}
at com.salesforce.graph.build.MethodPathBuilderVisitor.addEdge(MethodPathBuilderVisitor.java:595)
at com.salesforce.graph.build.MethodPathBuilderVisitor.addEdgeFromPreviousSibling(MethodPathBuilderVisitor.java:589)
at com.salesforce.graph.build.MethodPathBuilderVisitor.visit(MethodPathBuilderVisitor.java:176)
at com.salesforce.graph.build.MethodPathBuilderVisitor._visit(MethodPathBuilderVisitor.java:148)
at com.salesforce.graph.build.MethodPathBuilderVisitor._visit(MethodPathBuilderVisitor.java:157)
at com.salesforce.graph.build.MethodPathBuilderVisitor._visit(MethodPathBuilderVisitor.java:157)
at com.salesforce.graph.build.MethodPathBuilderVisitor._visit(MethodPathBuilderVisitor.java:157)
at com.salesforce.graph.build.MethodPathBuilderVisitor._visit(MethodPathBuilderVisitor.java:157)
at com.salesforce.graph.build.MethodPathBuilderVisitor.apply(MethodPathBuilderVisitor.java:95)
at com.salesforce.graph.build.AbstractApexVertexBuilder.afterInsert(AbstractApexVertexBuilder.java:104)
at com.salesforce.graph.build.AbstractApexVertexBuilder.buildVertices(AbstractApexVertexBuilder.java:91)
at com.salesforce.graph.build.AbstractApexVertexBuilder.buildVertices(AbstractApexVertexBuilder.java:89)
at com.salesforce.graph.build.AbstractApexVertexBuilder.buildVertices(AbstractApexVertexBuilder.java:40)
at com.salesforce.graph.build.CustomerApexVertexBuilder.build(CustomerApexVertexBuilder.java:21)
at com.salesforce.graph.build.Util.buildGraph(Util.java:39)
at com.salesforce.graph.ops.GraphUtil.loadSourceFolders(GraphUtil.java:158)
at com.salesforce.Main.execute(Main.java:127)
at com.salesforce.Main.process(Main.java:80)
at com.salesforce.Main.main(Main.java:59)

[rsoesemann]

FYI @rmohan20 @jbartolotta-sfdc @jfeingold35

[rmohan20]

@rsoesemann Thanks for letting us know.

Looks like this codebase has a structure/flow that SFGE doesn't understand yet. The problem seems to be at a method between lines 58 to 61 in PriceCategoryTypeSelector class and possibly impacted by the methods invoked from here.

Would it be possible to share with us this code? If the code can be privately shared but not publicly, could you send it to me over email? Just the affected classes and the methods invoked from 58 to 61 would be a good starting point - if not the actual code, a StackOverflow-style alternative that repros this issue would help us too. Thanks!

[rsoesemann]

Sure @rmohan20 this custom Apex

calls this OSS fflib Query Builder

https://github.com/apex-enterprise-patterns/fflib-apex-common/blob/master/sfdx-source/apex-common/main/classes/fflib_QueryFactory.cls

[rmohan20]

Thanks @rsoesemann - I'll try to get a local repro with your information.

[rmohan20]

@rsoesemann For the "Vertex is terminal" error, I could get a repro. My guess is that there's a line of code right after the return statement before the curly braces close. Possibly at line 62, which is not present in the code snippet image above. SFGE is complaining about unreachable code that Apex compiler did not seem to mind. Could you please confirm if this is true?

If yes:

• SFGE should give a more meaningful feedback.
• This appears to be a code issue that the code owner can address.

If no, could you share some more of the code that follows line 62 so that we can continue troubleshooting?

This issue also highlights two more independent items:

1. SFGE has a limitation that it can only build context from the code provided as a part of the --projectdir parameter. It does not have an insight into code in external package dependencies. In long term, we should improve it to a point where it understands named package dependencies, gets access, and builds paths that include these external files.

2. I removed the unreachable code, created method stubs to replace the external dependency code, and was able to move past the "Vertex is terminal" error. However, I hit another issue where SFGE does not handle a method call parameter to Database.query(). This is certainly an actionable item and we'll address this scenario.

As a side note, we noticed that in your input, --projectdir is not a superset of --target. While --projectdir is supposed to provide a context of all the code to analyze, the actual entry points are selected from --target. The overall run might be faster and more accurate if you swap these values.

[git2gus]

Error while creating work item!

[rsoesemann]

@rmohan20 there is a second return. I updated the original screenshot above.

Regard how to call the command you recommend to replace

sfdx scanner:run:dfa --target '/Users/rsoesemann/dev/aquivalabs/XXX/' --projectdir '/Users/rsoesemann/dev/aquivalabs/XXX/sfdx-source/xxx' --format HTML

with

sfdx scanner:run:dfa --projectdir '/Users/rsoesemann/dev/aquivalabs/XXX/' --target '/Users/rsoesemann/dev/aquivalabs/XXX/sfdx-source/xxx' --format HTML

right?

[rmohan20]

@rsoesemann Thanks for confirming. The second return shouldn't be an issue. Though from the updated screenshot, it seems pretty clear that the assignment to true statement at line 62 would be unreachable the way it is today. Moving that line before line 58 would make it reachable and this exception would go away. I've created a work item for us to return a more helpful message from SFGE when this happens.

As for the parameters, yes, your updated command should be more accurate. Thanks!

[rsoesemann]

@rmohan20 Maybe you should also report that to the Apex compiler team. I think Java doesn't allow you to create unreachable code.

[rmohan20]

The latest 3.4.0 version has related fixes. Closing this issue.