[Bug] Erroneous AST parsing in JSSRC

[pr0me]

Parsing a (possibly ill-formatted) TypeScript file results in a java.lang.NumberFormatException.

To Reproduce
Steps to reproduce the behavior:

1. import the attached example with importCode in joern
   example.zip

2. Observe the following errors:

2023-03-30 13:49:34.946 WARN AstGenRunner$: 	- failed to parse '/home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/retroactiveVesting.ts': 'Unexpected token, expected "}" (39:6)'
java.lang.NumberFormatException: 100000000000000000000
	at upickle.core.Util$.parseLong(Util.scala:83)
	at upickle.core.Util$.parseIntegralNum(Util.scala:48)
	at ujson.Value$.visitFloat64StringParts(Value.scala:213)

Full Stack Trace

Using generator for language: JSSRC: JsSrcCpgGenerator
Creating project `0xlenin_saddle-contract2` for code at `/home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/`
=======================================================================================================
Invoking CPG generator in a separate process. Note that the new process will consume additional memory.
If you are importing a large codebase (and/or running into memory issues), please try the following:
1) exit joern
2) invoke the frontend: /opt/joern/joern-cli/jssrc2cpg.sh -J-Xmx30688m /home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/ --output /home/pr0me/workspace/0xlenin_saddle-contract2/cpg.bin.zip
3) start joern, import the cpg: `importCpg("path/to/cpg")`
=======================================================================================================

2023-03-30 13:49:34.946 WARN AstGenRunner$: 	- failed to parse '/home/pr0me/type-inference-models/experiments/tmp_code/3/0xlenin_saddle-contract/retroactiveVesting.ts': 'Unexpected token, expected "}" (39:6)'
java.lang.NumberFormatException: 100000000000000000000
	at upickle.core.Util$.parseLong(Util.scala:83)
	at upickle.core.Util$.parseIntegralNum(Util.scala:48)
	at ujson.Value$.visitFloat64StringParts(Value.scala:213)
	at ujson.Value$.visitFloat64StringParts(Value.scala:129)
	at ujson.CharParser.visitFloat64StringPartsWithWrapper(CharParser.scala:161)
	at ujson.CharParser.parseNum(CharParser.scala:148)
	at ujson.CharParser.parseNested(CharParser.scala:409)
	at ujson.CharParser.parseTopLevel0(CharParser.scala:324)
	at ujson.CharParser.parseTopLevel(CharParser.scala:308)
	at ujson.CharParser.parse(CharParser.scala:59)
	at ujson.StringParser$.transform(StringParser.scala:28)
	at ujson.StringParser$.transform(StringParser.scala:27)
	at ujson.Readable$fromTransformer.transform(Readable.scala:13)
	at ujson.package$.transform(package.scala:4)
	at ujson.package$.$anonfun$read$1(package.scala:10)
	at upickle.core.TraceVisitor$.withTrace(TraceVisitor.scala:18)
	at ujson.package$.read(package.scala:10)
	at io.joern.jssrc2cpg.parser.BabelJsonParser$.readFile(BabelJsonParser.scala:31)
	at io.joern.jssrc2cpg.passes.AstCreationPass.$anonfun$runOnPart$1(AstCreationPass.scala:46)
	at io.joern.jssrc2cpg.utils.TimeUtils$.time(TimeUtils.scala:11)
	at io.joern.jssrc2cpg.passes.AstCreationPass.runOnPart(AstCreationPass.scala:45)
	at io.joern.jssrc2cpg.passes.AstCreationPass.runOnPart(AstCreationPass.scala:22)
	at io.shiftleft.passes.ConcurrentWriterCpgPass.$anonfun$createApplySerializeAndStore$1(ParallelCpgPass.scala:95)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
moving cpg.bin.zip to cpg.bin because it is already a database file
Creating working copy of CPG to be safe
Loading base CPG from: /home/pr0me/workspace/0xlenin_saddle-contract2/cpg.bin.tmp
Code successfully imported. You can now query it using `cpg`.
For an overview of all imported code, type `workspace`.
Missing metaData block
Adding default overlays to base CPG
Missing metaData block
Missing metaData block
Missing metaData block
Missing metaData block
Missing metaData block
2023-03-30 13:49:36.204 WARN ControlFlow: io.joern.x2cpg.layers.ControlFlow depends on List(base) but CPG only has Set() - skipping creation
Missing metaData block
Missing metaData block
2023-03-30 13:49:36.207 WARN TypeRelations: io.joern.x2cpg.layers.TypeRelations depends on List(base) but CPG only has Set() - skipping creation
Missing metaData block
Missing metaData block
2023-03-30 13:49:36.207 WARN CallGraph: io.joern.x2cpg.layers.CallGraph depends on List(typerel) but CPG only has Set() - skipping creation
The graph has been modified. You may want to use the `save` command to persist changes to disk.  All changes will also be saved collectively on exit
Missing metaData block
Missing metaData block
2023-03-30 13:49:36,359 main ERROR An exception occurred processing Appender Console org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.NoSuchMethodError: 'java.util.Deque org.apache.logging.log4j.util.StackLocatorUtil.getCurrentStackTrace()'

[...]

3. From the stack trace:
   Note that there are multiple missing blocks and structures, probably due to the failure in parsing the AST.
   (The error at the bottom is discussed in issue [Bug] AppenderLoggingException in log4j logging #2439 )

Expected behavior
AST generation should succeed, fail gracefully if possible or abort CPG creation completely if unrecoverable.

Desktop (please complete the following information):

• OS: Linux 5.15.0-1030-gcp 37-Ubuntu
• Joern Version: 1.1.1552
• javac 11.0.18

[xavierpinho]

Smaller sample

console.log(1e20)

Diagnosis

Babel (and therefore astgen) outputs the following fragment:

"arguments": [
              {
                "type": "NumericLiteral",
                "extra": {
                  "rawValue": 100000000000000000000,
                  "raw": "1e20"
                },
                "value": 100000000000000000000
              }
            ]

When slurping this in, upickle throws a NumberFormatException.

Workarounds

We likely need to implement a custom visitor (for ujson.transform), but @max-leuthaeuser should have a better guess.

See also: com-lihaoyi/upickle#240