SIMPLE_IDENTIFIER in comments

[EgorBu]

Hi,
I found that java-driver assigns SIMPLE_IDENTIFIER to something in comments:
Ex:

// SIMPLE_IDENTIFIER in comments
// -------------------- b'java-driver/native/src/main/java/bblfsh/EclipseParser.java'
package bblfsh;

import org.eclipse.jdt.core.JavaCore;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTParser;
import org.eclipse.jdt.core.dom.CompilationUnit;

import java.io.IOException;

import java.util.Map;

/**
 * Java AST parser based on Eclipse JDT.
 * <p>
 * This class is not thread-safe.
 * </p>
 */
public class EclipseParser {
    /**
     * Creates a new EclipseParser
     */
    public EclipseParser() {
        this.parser = ASTParser.newParser(AST.JLS8);
        parser.setKind(ASTParser.K_COMPILATION_UNIT);
    }

    /**
     *
     * @param source String to parses
     * @return CompilationUnit of the AST
     * @throws IOException if anything related to I/O or Json generation failed
     */
    public CompilationUnit parse(final String source) throws IOException {
        parser.setSource(source.toCharArray());
        Map options = JavaCore.getOptions();
        JavaCore.setComplianceOptions(JavaCore.VERSION_1_8, options);
        parser.setCompilerOptions(options);
        return (CompilationUnit) parser.createAST(null);
    }
}

Vis

[abeaumont]

Thanks for the report @EgorBu. Please note that you can use GH's markdown syntax highlighting instead of attaching an image of the code. Could you attach the generated UAST instead and indicate which part you think is wrong?

[EgorBu]

Thanks @abeaumont , it's faster to insert screenshot than manually tag everything

UAST:

uast {
  internal_type: "CompilationUnit"
  children {
    internal_type: "LineComment"
    properties {
      key: "internalRole"
      value: "comments"
    }
    start_position {
      line: 1
      col: 1
    }
    end_position {
      offset: 32
      line: 1
      col: 33
    }
    roles: COMMENT
  }
  children {
    internal_type: "LineComment"
    properties {
      key: "internalRole"
      value: "comments"
    }
    start_position {
      offset: 33
      line: 2
      col: 1
    }
    end_position {
      offset: 128
      line: 2
      col: 96
    }
    roles: COMMENT
  }
  children {
    internal_type: "PackageDeclaration"
    properties {
      key: "internalRole"
      value: "package"
    }
    children {
      internal_type: "SimpleName"
      properties {
        key: "internalRole"
        value: "name"
      }
      token: "bblfsh"
      start_position {
        offset: 137
        line: 3
        col: 9
      }
      end_position {
        offset: 143
        line: 3
        col: 15
      }
      roles: SIMPLE_IDENTIFIER
      roles: EXPRESSION
    }
    token: "package"
    roles: PACKAGE_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              token: "org"
              start_position {
                offset: 153
                line: 5
                col: 8
              }
              end_position {
                offset: 156
                line: 5
                col: 11
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "eclipse"
              start_position {
                offset: 157
                line: 5
                col: 12
              }
              end_position {
                offset: 164
                line: 5
                col: 19
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "jdt"
            start_position {
              offset: 165
              line: 5
              col: 20
            }
            end_position {
              offset: 168
              line: 5
              col: 23
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "core"
          start_position {
            offset: 169
            line: 5
            col: 24
          }
          end_position {
            offset: 173
            line: 5
            col: 28
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "JavaCore"
        start_position {
          offset: 174
          line: 5
          col: 29
        }
        end_position {
          offset: 182
          line: 5
          col: 37
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 146
      line: 5
      col: 1
    }
    end_position {
      offset: 183
      line: 5
      col: 38
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "org"
                start_position {
                  offset: 191
                  line: 6
                  col: 8
                }
                end_position {
                  offset: 194
                  line: 6
                  col: 11
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "eclipse"
                start_position {
                  offset: 195
                  line: 6
                  col: 12
                }
                end_position {
                  offset: 202
                  line: 6
                  col: 19
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "jdt"
              start_position {
                offset: 203
                line: 6
                col: 20
              }
              end_position {
                offset: 206
                line: 6
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "core"
            start_position {
              offset: 207
              line: 6
              col: 24
            }
            end_position {
              offset: 211
              line: 6
              col: 28
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "dom"
          start_position {
            offset: 212
            line: 6
            col: 29
          }
          end_position {
            offset: 215
            line: 6
            col: 32
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "AST"
        start_position {
          offset: 216
          line: 6
          col: 33
        }
        end_position {
          offset: 219
          line: 6
          col: 36
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 184
      line: 6
      col: 1
    }
    end_position {
      offset: 220
      line: 6
      col: 37
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "org"
                start_position {
                  offset: 228
                  line: 7
                  col: 8
                }
                end_position {
                  offset: 231
                  line: 7
                  col: 11
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "eclipse"
                start_position {
                  offset: 232
                  line: 7
                  col: 12
                }
                end_position {
                  offset: 239
                  line: 7
                  col: 19
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "jdt"
              start_position {
                offset: 240
                line: 7
                col: 20
              }
              end_position {
                offset: 243
                line: 7
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "core"
            start_position {
              offset: 244
              line: 7
              col: 24
            }
            end_position {
              offset: 248
              line: 7
              col: 28
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "dom"
          start_position {
            offset: 249
            line: 7
            col: 29
          }
          end_position {
            offset: 252
            line: 7
            col: 32
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "ASTParser"
        start_position {
          offset: 253
          line: 7
          col: 33
        }
        end_position {
          offset: 262
          line: 7
          col: 42
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 221
      line: 7
      col: 1
    }
    end_position {
      offset: 263
      line: 7
      col: 43
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "org"
                start_position {
                  offset: 271
                  line: 8
                  col: 8
                }
                end_position {
                  offset: 274
                  line: 8
                  col: 11
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "eclipse"
                start_position {
                  offset: 275
                  line: 8
                  col: 12
                }
                end_position {
                  offset: 282
                  line: 8
                  col: 19
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "jdt"
              start_position {
                offset: 283
                line: 8
                col: 20
              }
              end_position {
                offset: 286
                line: 8
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "core"
            start_position {
              offset: 287
              line: 8
              col: 24
            }
            end_position {
              offset: 291
              line: 8
              col: 28
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "dom"
          start_position {
            offset: 292
            line: 8
            col: 29
          }
          end_position {
            offset: 295
            line: 8
            col: 32
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "CompilationUnit"
        start_position {
          offset: 296
          line: 8
          col: 33
        }
        end_position {
          offset: 311
          line: 8
          col: 48
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 264
      line: 8
      col: 1
    }
    end_position {
      offset: 312
      line: 8
      col: 49
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          token: "java"
          start_position {
            offset: 321
            line: 10
            col: 8
          }
          end_position {
            offset: 325
            line: 10
            col: 12
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "io"
          start_position {
            offset: 326
            line: 10
            col: 13
          }
          end_position {
            offset: 328
            line: 10
            col: 15
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "IOException"
        start_position {
          offset: 329
          line: 10
          col: 16
        }
        end_position {
          offset: 340
          line: 10
          col: 27
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 314
      line: 10
      col: 1
    }
    end_position {
      offset: 341
      line: 10
      col: 28
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          token: "java"
          start_position {
            offset: 350
            line: 12
            col: 8
          }
          end_position {
            offset: 354
            line: 12
            col: 12
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "util"
          start_position {
            offset: 355
            line: 12
            col: 13
          }
          end_position {
            offset: 359
            line: 12
            col: 17
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "Map"
        start_position {
          offset: 360
          line: 12
          col: 18
        }
        end_position {
          offset: 363
          line: 12
          col: 21
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 343
      line: 12
      col: 1
    }
    end_position {
      offset: 364
      line: 12
      col: 22
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "TypeDeclaration"
    properties {
      key: "interface"
      value: "false"
    }
    properties {
      key: "internalRole"
      value: "types"
    }
    children {
      internal_type: "Javadoc"
      properties {
        key: "internalRole"
        value: "javadoc"
      }
      children {
        internal_type: "TagElement"
        properties {
          key: "internalRole"
          value: "tags"
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "Java AST parser based on Eclipse JDT."
          }
          start_position {
            offset: 373
            line: 15
            col: 4
          }
          end_position {
            offset: 410
            line: 15
            col: 41
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "<p>"
          }
          start_position {
            offset: 414
            line: 16
            col: 4
          }
          end_position {
            offset: 417
            line: 16
            col: 7
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "This class is not thread-safe."
          }
          start_position {
            offset: 421
            line: 17
            col: 4
          }
          end_position {
            offset: 451
            line: 17
            col: 34
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "</p>"
          }
          start_position {
            offset: 455
            line: 18
            col: 4
          }
          end_position {
            offset: 459
            line: 18
            col: 8
          }
          roles: INCOMPLETE
        }
        roles: INCOMPLETE
      }
      roles: DOCUMENTATION
      roles: COMMENT
    }
    children {
      internal_type: "Modifier"
      properties {
        key: "internalRole"
        value: "modifiers"
      }
      token: "public"
      start_position {
        offset: 464
        line: 20
        col: 1
      }
      end_position {
        offset: 470
        line: 20
        col: 7
      }
      roles: VISIBLE_FROM_WORLD
    }
    children {
      internal_type: "SimpleName"
      properties {
        key: "internalRole"
        value: "name"
      }
      token: "EclipseParser"
      start_position {
        offset: 477
        line: 20
        col: 14
      }
      end_position {
        offset: 490
        line: 20
        col: 27
      }
      roles: SIMPLE_IDENTIFIER
      roles: EXPRESSION
    }
    children {
      internal_type: "MethodDeclaration"
      properties {
        key: "constructor"
        value: "true"
      }
      properties {
        key: "internalRole"
        value: "bodyDeclarations"
      }
      children {
        internal_type: "Javadoc"
        properties {
          key: "internalRole"
          value: "javadoc"
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: "Creates a new EclipseParser"
            }
            start_position {
              offset: 508
              line: 22
              col: 8
            }
            end_position {
              offset: 535
              line: 22
              col: 35
            }
            roles: INCOMPLETE
          }
          roles: INCOMPLETE
        }
        roles: DOCUMENTATION
        roles: COMMENT
      }
      children {
        internal_type: "Modifier"
        properties {
          key: "internalRole"
          value: "modifiers"
        }
        token: "public"
        start_position {
          offset: 548
          line: 24
          col: 5
        }
        end_position {
          offset: 554
          line: 24
          col: 11
        }
        roles: VISIBLE_FROM_WORLD
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "EclipseParser"
        start_position {
          offset: 555
          line: 24
          col: 12
        }
        end_position {
          offset: 568
          line: 24
          col: 25
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
        roles: FUNCTION_DECLARATION_NAME
      }
      children {
        internal_type: "Block"
        properties {
          key: "internalRole"
          value: "body"
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "Assignment"
            properties {
              key: "internalRole"
              value: "expression"
            }
            properties {
              key: "operator"
              value: "="
            }
            children {
              internal_type: "FieldAccess"
              properties {
                key: "internalRole"
                value: "leftHandSide"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "parser"
                start_position {
                  offset: 586
                  line: 25
                  col: 14
                }
                end_position {
                  offset: 592
                  line: 25
                  col: 20
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "ThisExpression"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                roles: THIS
                roles: EXPRESSION
              }
              roles: ASSIGNMENT_VARIABLE
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "rightHandSide"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "ASTParser"
                start_position {
                  offset: 595
                  line: 25
                  col: 23
                }
                end_position {
                  offset: 604
                  line: 25
                  col: 32
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "newParser"
                start_position {
                  offset: 605
                  line: 25
                  col: 33
                }
                end_position {
                  offset: 614
                  line: 25
                  col: 42
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              children {
                internal_type: "QualifiedName"
                properties {
                  key: "internalRole"
                  value: "arguments"
                }
                children {
                  internal_type: "SimpleName"
                  properties {
                    key: "internalRole"
                    value: "qualifier"
                  }
                  token: "AST"
                  start_position {
                    offset: 615
                    line: 25
                    col: 43
                  }
                  end_position {
                    offset: 618
                    line: 25
                    col: 46
                  }
                  roles: SIMPLE_IDENTIFIER
                  roles: EXPRESSION
                }
                children {
                  internal_type: "SimpleName"
                  properties {
                    key: "internalRole"
                    value: "name"
                  }
                  token: "JLS8"
                  start_position {
                    offset: 619
                    line: 25
                    col: 47
                  }
                  end_position {
                    offset: 623
                    line: 25
                    col: 51
                  }
                  roles: SIMPLE_IDENTIFIER
                  roles: EXPRESSION
                }
                roles: QUALIFIED_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_POSITIONAL_ARGUMENT
              }
              roles: CALL
              roles: EXPRESSION
              roles: ASSIGNMENT_VALUE
            }
            start_position {
              offset: 581
              line: 25
              col: 9
            }
            end_position {
              offset: 624
              line: 25
              col: 52
            }
            roles: ASSIGNMENT
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "parser"
              start_position {
                offset: 634
                line: 26
                col: 9
              }
              end_position {
                offset: 640
                line: 26
                col: 15
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setKind"
              start_position {
                offset: 641
                line: 26
                col: 16
              }
              end_position {
                offset: 648
                line: 26
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "ASTParser"
                start_position {
                  offset: 649
                  line: 26
                  col: 24
                }
                end_position {
                  offset: 658
                  line: 26
                  col: 33
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "K_COMPILATION_UNIT"
                start_position {
                  offset: 659
                  line: 26
                  col: 34
                }
                end_position {
                  offset: 677
                  line: 26
                  col: 52
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        roles: FUNCTION_DECLARATION_BODY
        roles: BLOCK_SCOPE
        roles: BLOCK
        roles: STATEMENT
      }
      start_position {
        offset: 497
        line: 21
        col: 5
      }
      end_position {
        offset: 685
        line: 27
        col: 6
      }
      roles: VISIBLE_FROM_WORLD
      roles: FUNCTION_DECLARATION
    }
    children {
      internal_type: "MethodDeclaration"
      properties {
        key: "constructor"
        value: "false"
      }
      properties {
        key: "internalRole"
        value: "bodyDeclarations"
      }
      children {
        internal_type: "Javadoc"
        properties {
          key: "internalRole"
          value: "javadoc"
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          properties {
            key: "tagName"
            value: "@param"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            token: "source"
            start_position {
              offset: 716
              line: 31
              col: 15
            }
            end_position {
              offset: 722
              line: 31
              col: 21
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: " String to parses"
            }
            start_position {
              offset: 722
              line: 31
              col: 21
            }
            end_position {
              offset: 739
              line: 31
              col: 38
            }
            roles: INCOMPLETE
          }
          start_position {
            offset: 709
            line: 31
            col: 8
          }
          end_position {
            offset: 739
            line: 31
            col: 38
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          properties {
            key: "tagName"
            value: "@return"
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: " CompilationUnit of the AST"
            }
            start_position {
              offset: 754
              line: 32
              col: 15
            }
            end_position {
              offset: 781
              line: 32
              col: 42
            }
            roles: INCOMPLETE
          }
          start_position {
            offset: 747
            line: 32
            col: 8
          }
          end_position {
            offset: 781
            line: 32
            col: 42
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          properties {
            key: "tagName"
            value: "@throws"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            token: "IOException"
            start_position {
              offset: 797
              line: 33
              col: 16
            }
            end_position {
              offset: 808
              line: 33
              col: 27
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: " if anything related to I/O or Json generation failed"
            }
            start_position {
              offset: 808
              line: 33
              col: 27
            }
            end_position {
              offset: 861
              line: 33
              col: 80
            }
            roles: INCOMPLETE
          }
          start_position {
            offset: 789
            line: 33
            col: 8
          }
          end_position {
            offset: 861
            line: 33
            col: 80
          }
          roles: INCOMPLETE
        }
        roles: DOCUMENTATION
        roles: COMMENT
      }
      children {
        internal_type: "Modifier"
        properties {
          key: "internalRole"
          value: "modifiers"
        }
        token: "public"
        start_position {
          offset: 874
          line: 35
          col: 5
        }
        end_position {
          offset: 880
          line: 35
          col: 11
        }
        roles: VISIBLE_FROM_WORLD
      }
      children {
        internal_type: "SimpleType"
        properties {
          key: "internalRole"
          value: "returnType2"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "CompilationUnit"
          start_position {
            offset: 881
            line: 35
            col: 12
          }
          end_position {
            offset: 896
            line: 35
            col: 27
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: INCOMPLETE
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "parse"
        start_position {
          offset: 897
          line: 35
          col: 28
        }
        end_position {
          offset: 902
          line: 35
          col: 33
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
        roles: FUNCTION_DECLARATION_NAME
      }
      children {
        internal_type: "SingleVariableDeclaration"
        properties {
          key: "internalRole"
          value: "parameters"
        }
        properties {
          key: "varargs"
          value: "false"
        }
        children {
          internal_type: "Modifier"
          properties {
            key: "internalRole"
            value: "modifiers"
          }
          token: "final"
          start_position {
            offset: 903
            line: 35
            col: 34
          }
          end_position {
            offset: 908
            line: 35
            col: 39
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "SimpleType"
          properties {
            key: "internalRole"
            value: "type"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "String"
            start_position {
              offset: 909
              line: 35
              col: 40
            }
            end_position {
              offset: 915
              line: 35
              col: 46
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "source"
          start_position {
            offset: 916
            line: 35
            col: 47
          }
          end_position {
            offset: 922
            line: 35
            col: 53
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
          roles: FUNCTION_DECLARATION_ARGUMENT_NAME
        }
        start_position {
          offset: 903
          line: 35
          col: 34
        }
        end_position {
          offset: 922
          line: 35
          col: 53
        }
        roles: FUNCTION_DECLARATION_ARGUMENT
        roles: INCOMPLETE
      }
      children {
        internal_type: "SimpleType"
        properties {
          key: "internalRole"
          value: "thrownExceptionTypes"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "IOException"
          start_position {
            offset: 931
            line: 35
            col: 62
          }
          end_position {
            offset: 942
            line: 35
            col: 73
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: INCOMPLETE
      }
      children {
        internal_type: "Block"
        properties {
          key: "internalRole"
          value: "body"
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "parser"
              start_position {
                offset: 953
                line: 36
                col: 9
              }
              end_position {
                offset: 959
                line: 36
                col: 15
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setSource"
              start_position {
                offset: 960
                line: 36
                col: 16
              }
              end_position {
                offset: 969
                line: 36
                col: 25
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "source"
                start_position {
                  offset: 970
                  line: 36
                  col: 26
                }
                end_position {
                  offset: 976
                  line: 36
                  col: 32
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "toCharArray"
                start_position {
                  offset: 977
                  line: 36
                  col: 33
                }
                end_position {
                  offset: 988
                  line: 36
                  col: 44
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              roles: CALL_POSITIONAL_ARGUMENT
              roles: CALL
              roles: EXPRESSION
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "VariableDeclarationStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "SimpleType"
            properties {
              key: "internalRole"
              value: "type"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "Map"
              start_position {
                offset: 1001
                line: 37
                col: 9
              }
              end_position {
                offset: 1004
                line: 37
                col: 12
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: INCOMPLETE
          }
          children {
            internal_type: "VariableDeclarationFragment"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "options"
              start_position {
                offset: 1005
                line: 37
                col: 13
              }
              end_position {
                offset: 1012
                line: 37
                col: 20
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "initializer"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "JavaCore"
                start_position {
                  offset: 1015
                  line: 37
                  col: 23
                }
                end_position {
                  offset: 1023
                  line: 37
                  col: 31
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "getOptions"
                start_position {
                  offset: 1024
                  line: 37
                  col: 32
                }
                end_position {
                  offset: 1034
                  line: 37
                  col: 42
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              roles: CALL
              roles: EXPRESSION
            }
            roles: INCOMPLETE
          }
          roles: STATEMENT
          roles: INCOMPLETE
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "JavaCore"
              start_position {
                offset: 1046
                line: 38
                col: 9
              }
              end_position {
                offset: 1054
                line: 38
                col: 17
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setComplianceOptions"
              start_position {
                offset: 1055
                line: 38
                col: 18
              }
              end_position {
                offset: 1075
                line: 38
                col: 38
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "JavaCore"
                start_position {
                  offset: 1076
                  line: 38
                  col: 39
                }
                end_position {
                  offset: 1084
                  line: 38
                  col: 47
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "VERSION_1_8"
                start_position {
                  offset: 1085
                  line: 38
                  col: 48
                }
                end_position {
                  offset: 1096
                  line: 38
                  col: 59
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              token: "options"
              start_position {
                offset: 1098
                line: 38
                col: 61
              }
              end_position {
                offset: 1105
                line: 38
                col: 68
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "parser"
              start_position {
                offset: 1116
                line: 39
                col: 9
              }
              end_position {
                offset: 1122
                line: 39
                col: 15
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setCompilerOptions"
              start_position {
                offset: 1123
                line: 39
                col: 16
              }
              end_position {
                offset: 1141
                line: 39
                col: 34
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              token: "options"
              start_position {
                offset: 1142
                line: 39
                col: 35
              }
              end_position {
                offset: 1149
                line: 39
                col: 42
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "ReturnStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "CastExpression"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleType"
              properties {
                key: "internalRole"
                value: "type"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "CompilationUnit"
                start_position {
                  offset: 1168
                  line: 40
                  col: 17
                }
                end_position {
                  offset: 1183
                  line: 40
                  col: 32
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: INCOMPLETE
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "expression"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "parser"
                start_position {
                  offset: 1185
                  line: 40
                  col: 34
                }
                end_position {
                  offset: 1191
                  line: 40
                  col: 40
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "createAST"
                start_position {
                  offset: 1192
                  line: 40
                  col: 41
                }
                end_position {
                  offset: 1201
                  line: 40
                  col: 50
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              children {
                internal_type: "NullLiteral"
                properties {
                  key: "internalRole"
                  value: "arguments"
                }
                token: "null"
                roles: NULL_LITERAL
                roles: EXPRESSION
                roles: CALL_POSITIONAL_ARGUMENT
              }
              roles: CALL
              roles: EXPRESSION
            }
            roles: EXPRESSION
            roles: INCOMPLETE
          }
          roles: RETURN
          roles: STATEMENT
        }
        roles: FUNCTION_DECLARATION_BODY
        roles: BLOCK_SCOPE
        roles: BLOCK
        roles: STATEMENT
      }
      start_position {
        offset: 691
        line: 29
        col: 5
      }
      end_position {
        offset: 1214
        line: 41
        col: 6
      }
      roles: VISIBLE_FROM_WORLD
      roles: FUNCTION_DECLARATION
    }
    start_position {
      offset: 366
      line: 14
      col: 1
    }
    end_position {
      offset: 1216
      line: 4294967295
    }
    roles: VISIBLE_FROM_WORLD
    roles: TYPE_DECLARATION
  }
  roles: FILE
}

BTW: line: 4294967295 - something suspicious here

[abeaumont]

@EgorBu could you please indicate what do you think is wrong in that UAST?

[abeaumont]

Also note that screenshots are only noise, please tag code properly instead

[abeaumont]

@EgorBu I cannot reproduce the line number issue. In fact, the file is smaller than 1216B, so I wonder how that offset may be possible. Are you sure the UAST is generated from exactly the code you pasted?

[vmarkovtsev]

@abeaumont

1. Egor clearly stated what the problem is: tokens in comments are recognized as identifiers.
2. It is clearly visible on the provided screenshot. I am very far from any bblfsh driver development and yet it took me 5 seconds to figure this out.
3. Why do you need anything else in this report? Egor's tool is not capable of producing any other type of output.

Anyway, I grepped this for you:

properties {
            key: "tagName"
            value: "@param"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            token: "source"
            start_position {
              offset: 716
              line: 31
              col: 15
            }
            end_position {
              offset: 722
              line: 31
              col: 21
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }

[EgorBu]

@abeaumont
yes, it's UAST from the correct file. Here it's commands that I used

egor@egor-sourced:~/workspace/uast_playground$ cat test_data/java_driver_issue42/ex.java
// SIMPLE_IDENTIFIER in comments
// -------------------- b'test_data/java-driver/native/src/main/java/bblfsh/EclipseParser.java'
package bblfsh;

import org.eclipse.jdt.core.JavaCore;
import org.eclipse.jdt.core.dom.AST;
import org.eclipse.jdt.core.dom.ASTParser;
import org.eclipse.jdt.core.dom.CompilationUnit;

import java.io.IOException;

import java.util.Map;

/**
 * Java AST parser based on Eclipse JDT.
 * <p>
 * This class is not thread-safe.
 * </p>
 */
public class EclipseParser {
    /**
     * Creates a new EclipseParser
     */
    public EclipseParser() {
        this.parser = ASTParser.newParser(AST.JLS8);
        parser.setKind(ASTParser.K_COMPILATION_UNIT);
    }

    /**
     *
     * @param source String to parses
     * @return CompilationUnit of the AST
     * @throws IOException if anything related to I/O or Json generation failed
     */
    public CompilationUnit parse(final String source) throws IOException {
        parser.setSource(source.toCharArray());
        Map options = JavaCore.getOptions();
        JavaCore.setComplianceOptions(JavaCore.VERSION_1_8, options);
        parser.setCompilerOptions(options);
        return (CompilationUnit) parser.createAST(null);
    }
}
egor@egor-sourced:~/workspace/uast_playground$ python3 -m bblfsh -f test_data/java_driver_issue42/ex.java
uast {
  internal_type: "CompilationUnit"
  children {
    internal_type: "LineComment"
    properties {
      key: "internalRole"
      value: "comments"
    }
    start_position {
      line: 1
      col: 1
    }
    end_position {
      offset: 32
      line: 1
      col: 33
    }
    roles: COMMENT
  }
  children {
    internal_type: "LineComment"
    properties {
      key: "internalRole"
      value: "comments"
    }
    start_position {
      offset: 33
      line: 2
      col: 1
    }
    end_position {
      offset: 128
      line: 2
      col: 96
    }
    roles: COMMENT
  }
  children {
    internal_type: "PackageDeclaration"
    properties {
      key: "internalRole"
      value: "package"
    }
    children {
      internal_type: "SimpleName"
      properties {
        key: "internalRole"
        value: "name"
      }
      token: "bblfsh"
      start_position {
        offset: 137
        line: 3
        col: 9
      }
      end_position {
        offset: 143
        line: 3
        col: 15
      }
      roles: SIMPLE_IDENTIFIER
      roles: EXPRESSION
    }
    token: "package"
    roles: PACKAGE_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              token: "org"
              start_position {
                offset: 153
                line: 5
                col: 8
              }
              end_position {
                offset: 156
                line: 5
                col: 11
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "eclipse"
              start_position {
                offset: 157
                line: 5
                col: 12
              }
              end_position {
                offset: 164
                line: 5
                col: 19
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "jdt"
            start_position {
              offset: 165
              line: 5
              col: 20
            }
            end_position {
              offset: 168
              line: 5
              col: 23
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "core"
          start_position {
            offset: 169
            line: 5
            col: 24
          }
          end_position {
            offset: 173
            line: 5
            col: 28
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "JavaCore"
        start_position {
          offset: 174
          line: 5
          col: 29
        }
        end_position {
          offset: 182
          line: 5
          col: 37
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 146
      line: 5
      col: 1
    }
    end_position {
      offset: 183
      line: 5
      col: 38
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "org"
                start_position {
                  offset: 191
                  line: 6
                  col: 8
                }
                end_position {
                  offset: 194
                  line: 6
                  col: 11
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "eclipse"
                start_position {
                  offset: 195
                  line: 6
                  col: 12
                }
                end_position {
                  offset: 202
                  line: 6
                  col: 19
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "jdt"
              start_position {
                offset: 203
                line: 6
                col: 20
              }
              end_position {
                offset: 206
                line: 6
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "core"
            start_position {
              offset: 207
              line: 6
              col: 24
            }
            end_position {
              offset: 211
              line: 6
              col: 28
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "dom"
          start_position {
            offset: 212
            line: 6
            col: 29
          }
          end_position {
            offset: 215
            line: 6
            col: 32
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "AST"
        start_position {
          offset: 216
          line: 6
          col: 33
        }
        end_position {
          offset: 219
          line: 6
          col: 36
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 184
      line: 6
      col: 1
    }
    end_position {
      offset: 220
      line: 6
      col: 37
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "org"
                start_position {
                  offset: 228
                  line: 7
                  col: 8
                }
                end_position {
                  offset: 231
                  line: 7
                  col: 11
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "eclipse"
                start_position {
                  offset: 232
                  line: 7
                  col: 12
                }
                end_position {
                  offset: 239
                  line: 7
                  col: 19
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "jdt"
              start_position {
                offset: 240
                line: 7
                col: 20
              }
              end_position {
                offset: 243
                line: 7
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "core"
            start_position {
              offset: 244
              line: 7
              col: 24
            }
            end_position {
              offset: 248
              line: 7
              col: 28
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "dom"
          start_position {
            offset: 249
            line: 7
            col: 29
          }
          end_position {
            offset: 252
            line: 7
            col: 32
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "ASTParser"
        start_position {
          offset: 253
          line: 7
          col: 33
        }
        end_position {
          offset: 262
          line: 7
          col: 42
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 221
      line: 7
      col: 1
    }
    end_position {
      offset: 263
      line: 7
      col: 43
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "QualifiedName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          children {
            internal_type: "QualifiedName"
            properties {
              key: "internalRole"
              value: "qualifier"
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "qualifier"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "org"
                start_position {
                  offset: 271
                  line: 8
                  col: 8
                }
                end_position {
                  offset: 274
                  line: 8
                  col: 11
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "eclipse"
                start_position {
                  offset: 275
                  line: 8
                  col: 12
                }
                end_position {
                  offset: 282
                  line: 8
                  col: 19
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "jdt"
              start_position {
                offset: 283
                line: 8
                col: 20
              }
              end_position {
                offset: 286
                line: 8
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: QUALIFIED_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "core"
            start_position {
              offset: 287
              line: 8
              col: 24
            }
            end_position {
              offset: 291
              line: 8
              col: 28
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: QUALIFIED_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "dom"
          start_position {
            offset: 292
            line: 8
            col: 29
          }
          end_position {
            offset: 295
            line: 8
            col: 32
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "CompilationUnit"
        start_position {
          offset: 296
          line: 8
          col: 33
        }
        end_position {
          offset: 311
          line: 8
          col: 48
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 264
      line: 8
      col: 1
    }
    end_position {
      offset: 312
      line: 8
      col: 49
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          token: "java"
          start_position {
            offset: 321
            line: 10
            col: 8
          }
          end_position {
            offset: 325
            line: 10
            col: 12
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "io"
          start_position {
            offset: 326
            line: 10
            col: 13
          }
          end_position {
            offset: 328
            line: 10
            col: 15
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "IOException"
        start_position {
          offset: 329
          line: 10
          col: 16
        }
        end_position {
          offset: 340
          line: 10
          col: 27
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 314
      line: 10
      col: 1
    }
    end_position {
      offset: 341
      line: 10
      col: 28
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "ImportDeclaration"
    properties {
      key: "internalRole"
      value: "imports"
    }
    properties {
      key: "onDemand"
      value: "false"
    }
    properties {
      key: "static"
      value: "false"
    }
    children {
      internal_type: "QualifiedName"
      properties {
        key: "internalRole"
        value: "name"
      }
      children {
        internal_type: "QualifiedName"
        properties {
          key: "internalRole"
          value: "qualifier"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "qualifier"
          }
          token: "java"
          start_position {
            offset: 350
            line: 12
            col: 8
          }
          end_position {
            offset: 354
            line: 12
            col: 12
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "util"
          start_position {
            offset: 355
            line: 12
            col: 13
          }
          end_position {
            offset: 359
            line: 12
            col: 17
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: QUALIFIED_IDENTIFIER
        roles: EXPRESSION
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "Map"
        start_position {
          offset: 360
          line: 12
          col: 18
        }
        end_position {
          offset: 363
          line: 12
          col: 21
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
      }
      roles: QUALIFIED_IDENTIFIER
      roles: EXPRESSION
      roles: IMPORT_PATH
    }
    start_position {
      offset: 343
      line: 12
      col: 1
    }
    end_position {
      offset: 364
      line: 12
      col: 22
    }
    roles: IMPORT_DECLARATION
  }
  children {
    internal_type: "TypeDeclaration"
    properties {
      key: "interface"
      value: "false"
    }
    properties {
      key: "internalRole"
      value: "types"
    }
    children {
      internal_type: "Javadoc"
      properties {
        key: "internalRole"
        value: "javadoc"
      }
      children {
        internal_type: "TagElement"
        properties {
          key: "internalRole"
          value: "tags"
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "Java AST parser based on Eclipse JDT."
          }
          start_position {
            offset: 373
            line: 15
            col: 4
          }
          end_position {
            offset: 410
            line: 15
            col: 41
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "<p>"
          }
          start_position {
            offset: 414
            line: 16
            col: 4
          }
          end_position {
            offset: 417
            line: 16
            col: 7
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "This class is not thread-safe."
          }
          start_position {
            offset: 421
            line: 17
            col: 4
          }
          end_position {
            offset: 451
            line: 17
            col: 34
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TextElement"
          properties {
            key: "internalRole"
            value: "fragments"
          }
          properties {
            key: "text"
            value: "</p>"
          }
          start_position {
            offset: 455
            line: 18
            col: 4
          }
          end_position {
            offset: 459
            line: 18
            col: 8
          }
          roles: INCOMPLETE
        }
        roles: INCOMPLETE
      }
      roles: DOCUMENTATION
      roles: COMMENT
    }
    children {
      internal_type: "Modifier"
      properties {
        key: "internalRole"
        value: "modifiers"
      }
      token: "public"
      start_position {
        offset: 464
        line: 20
        col: 1
      }
      end_position {
        offset: 470
        line: 20
        col: 7
      }
      roles: VISIBLE_FROM_WORLD
    }
    children {
      internal_type: "SimpleName"
      properties {
        key: "internalRole"
        value: "name"
      }
      token: "EclipseParser"
      start_position {
        offset: 477
        line: 20
        col: 14
      }
      end_position {
        offset: 490
        line: 20
        col: 27
      }
      roles: SIMPLE_IDENTIFIER
      roles: EXPRESSION
    }
    children {
      internal_type: "MethodDeclaration"
      properties {
        key: "constructor"
        value: "true"
      }
      properties {
        key: "internalRole"
        value: "bodyDeclarations"
      }
      children {
        internal_type: "Javadoc"
        properties {
          key: "internalRole"
          value: "javadoc"
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: "Creates a new EclipseParser"
            }
            start_position {
              offset: 508
              line: 22
              col: 8
            }
            end_position {
              offset: 535
              line: 22
              col: 35
            }
            roles: INCOMPLETE
          }
          roles: INCOMPLETE
        }
        roles: DOCUMENTATION
        roles: COMMENT
      }
      children {
        internal_type: "Modifier"
        properties {
          key: "internalRole"
          value: "modifiers"
        }
        token: "public"
        start_position {
          offset: 548
          line: 24
          col: 5
        }
        end_position {
          offset: 554
          line: 24
          col: 11
        }
        roles: VISIBLE_FROM_WORLD
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "EclipseParser"
        start_position {
          offset: 555
          line: 24
          col: 12
        }
        end_position {
          offset: 568
          line: 24
          col: 25
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
        roles: FUNCTION_DECLARATION_NAME
      }
      children {
        internal_type: "Block"
        properties {
          key: "internalRole"
          value: "body"
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "Assignment"
            properties {
              key: "internalRole"
              value: "expression"
            }
            properties {
              key: "operator"
              value: "="
            }
            children {
              internal_type: "FieldAccess"
              properties {
                key: "internalRole"
                value: "leftHandSide"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "parser"
                start_position {
                  offset: 586
                  line: 25
                  col: 14
                }
                end_position {
                  offset: 592
                  line: 25
                  col: 20
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "ThisExpression"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                roles: THIS
                roles: EXPRESSION
              }
              roles: ASSIGNMENT_VARIABLE
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "rightHandSide"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "ASTParser"
                start_position {
                  offset: 595
                  line: 25
                  col: 23
                }
                end_position {
                  offset: 604
                  line: 25
                  col: 32
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "newParser"
                start_position {
                  offset: 605
                  line: 25
                  col: 33
                }
                end_position {
                  offset: 614
                  line: 25
                  col: 42
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              children {
                internal_type: "QualifiedName"
                properties {
                  key: "internalRole"
                  value: "arguments"
                }
                children {
                  internal_type: "SimpleName"
                  properties {
                    key: "internalRole"
                    value: "qualifier"
                  }
                  token: "AST"
                  start_position {
                    offset: 615
                    line: 25
                    col: 43
                  }
                  end_position {
                    offset: 618
                    line: 25
                    col: 46
                  }
                  roles: SIMPLE_IDENTIFIER
                  roles: EXPRESSION
                }
                children {
                  internal_type: "SimpleName"
                  properties {
                    key: "internalRole"
                    value: "name"
                  }
                  token: "JLS8"
                  start_position {
                    offset: 619
                    line: 25
                    col: 47
                  }
                  end_position {
                    offset: 623
                    line: 25
                    col: 51
                  }
                  roles: SIMPLE_IDENTIFIER
                  roles: EXPRESSION
                }
                roles: QUALIFIED_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_POSITIONAL_ARGUMENT
              }
              roles: CALL
              roles: EXPRESSION
              roles: ASSIGNMENT_VALUE
            }
            start_position {
              offset: 581
              line: 25
              col: 9
            }
            end_position {
              offset: 624
              line: 25
              col: 52
            }
            roles: ASSIGNMENT
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "parser"
              start_position {
                offset: 634
                line: 26
                col: 9
              }
              end_position {
                offset: 640
                line: 26
                col: 15
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setKind"
              start_position {
                offset: 641
                line: 26
                col: 16
              }
              end_position {
                offset: 648
                line: 26
                col: 23
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "ASTParser"
                start_position {
                  offset: 649
                  line: 26
                  col: 24
                }
                end_position {
                  offset: 658
                  line: 26
                  col: 33
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "K_COMPILATION_UNIT"
                start_position {
                  offset: 659
                  line: 26
                  col: 34
                }
                end_position {
                  offset: 677
                  line: 26
                  col: 52
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        roles: FUNCTION_DECLARATION_BODY
        roles: BLOCK_SCOPE
        roles: BLOCK
        roles: STATEMENT
      }
      start_position {
        offset: 497
        line: 21
        col: 5
      }
      end_position {
        offset: 685
        line: 27
        col: 6
      }
      roles: VISIBLE_FROM_WORLD
      roles: FUNCTION_DECLARATION
    }
    children {
      internal_type: "MethodDeclaration"
      properties {
        key: "constructor"
        value: "false"
      }
      properties {
        key: "internalRole"
        value: "bodyDeclarations"
      }
      children {
        internal_type: "Javadoc"
        properties {
          key: "internalRole"
          value: "javadoc"
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          properties {
            key: "tagName"
            value: "@param"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            token: "source"
            start_position {
              offset: 716
              line: 31
              col: 15
            }
            end_position {
              offset: 722
              line: 31
              col: 21
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: " String to parses"
            }
            start_position {
              offset: 722
              line: 31
              col: 21
            }
            end_position {
              offset: 739
              line: 31
              col: 38
            }
            roles: INCOMPLETE
          }
          start_position {
            offset: 709
            line: 31
            col: 8
          }
          end_position {
            offset: 739
            line: 31
            col: 38
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          properties {
            key: "tagName"
            value: "@return"
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: " CompilationUnit of the AST"
            }
            start_position {
              offset: 754
              line: 32
              col: 15
            }
            end_position {
              offset: 781
              line: 32
              col: 42
            }
            roles: INCOMPLETE
          }
          start_position {
            offset: 747
            line: 32
            col: 8
          }
          end_position {
            offset: 781
            line: 32
            col: 42
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          properties {
            key: "tagName"
            value: "@throws"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            token: "IOException"
            start_position {
              offset: 797
              line: 33
              col: 16
            }
            end_position {
              offset: 808
              line: 33
              col: 27
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          children {
            internal_type: "TextElement"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            properties {
              key: "text"
              value: " if anything related to I/O or Json generation failed"
            }
            start_position {
              offset: 808
              line: 33
              col: 27
            }
            end_position {
              offset: 861
              line: 33
              col: 80
            }
            roles: INCOMPLETE
          }
          start_position {
            offset: 789
            line: 33
            col: 8
          }
          end_position {
            offset: 861
            line: 33
            col: 80
          }
          roles: INCOMPLETE
        }
        roles: DOCUMENTATION
        roles: COMMENT
      }
      children {
        internal_type: "Modifier"
        properties {
          key: "internalRole"
          value: "modifiers"
        }
        token: "public"
        start_position {
          offset: 874
          line: 35
          col: 5
        }
        end_position {
          offset: 880
          line: 35
          col: 11
        }
        roles: VISIBLE_FROM_WORLD
      }
      children {
        internal_type: "SimpleType"
        properties {
          key: "internalRole"
          value: "returnType2"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "CompilationUnit"
          start_position {
            offset: 881
            line: 35
            col: 12
          }
          end_position {
            offset: 896
            line: 35
            col: 27
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: INCOMPLETE
      }
      children {
        internal_type: "SimpleName"
        properties {
          key: "internalRole"
          value: "name"
        }
        token: "parse"
        start_position {
          offset: 897
          line: 35
          col: 28
        }
        end_position {
          offset: 902
          line: 35
          col: 33
        }
        roles: SIMPLE_IDENTIFIER
        roles: EXPRESSION
        roles: FUNCTION_DECLARATION_NAME
      }
      children {
        internal_type: "SingleVariableDeclaration"
        properties {
          key: "internalRole"
          value: "parameters"
        }
        properties {
          key: "varargs"
          value: "false"
        }
        children {
          internal_type: "Modifier"
          properties {
            key: "internalRole"
            value: "modifiers"
          }
          token: "final"
          start_position {
            offset: 903
            line: 35
            col: 34
          }
          end_position {
            offset: 908
            line: 35
            col: 39
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "SimpleType"
          properties {
            key: "internalRole"
            value: "type"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "name"
            }
            token: "String"
            start_position {
              offset: 909
              line: 35
              col: 40
            }
            end_position {
              offset: 915
              line: 35
              col: 46
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }
          roles: INCOMPLETE
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "source"
          start_position {
            offset: 916
            line: 35
            col: 47
          }
          end_position {
            offset: 922
            line: 35
            col: 53
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
          roles: FUNCTION_DECLARATION_ARGUMENT_NAME
        }
        start_position {
          offset: 903
          line: 35
          col: 34
        }
        end_position {
          offset: 922
          line: 35
          col: 53
        }
        roles: FUNCTION_DECLARATION_ARGUMENT
        roles: INCOMPLETE
      }
      children {
        internal_type: "SimpleType"
        properties {
          key: "internalRole"
          value: "thrownExceptionTypes"
        }
        children {
          internal_type: "SimpleName"
          properties {
            key: "internalRole"
            value: "name"
          }
          token: "IOException"
          start_position {
            offset: 931
            line: 35
            col: 62
          }
          end_position {
            offset: 942
            line: 35
            col: 73
          }
          roles: SIMPLE_IDENTIFIER
          roles: EXPRESSION
        }
        roles: INCOMPLETE
      }
      children {
        internal_type: "Block"
        properties {
          key: "internalRole"
          value: "body"
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "parser"
              start_position {
                offset: 953
                line: 36
                col: 9
              }
              end_position {
                offset: 959
                line: 36
                col: 15
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setSource"
              start_position {
                offset: 960
                line: 36
                col: 16
              }
              end_position {
                offset: 969
                line: 36
                col: 25
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "source"
                start_position {
                  offset: 970
                  line: 36
                  col: 26
                }
                end_position {
                  offset: 976
                  line: 36
                  col: 32
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "toCharArray"
                start_position {
                  offset: 977
                  line: 36
                  col: 33
                }
                end_position {
                  offset: 988
                  line: 36
                  col: 44
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              roles: CALL_POSITIONAL_ARGUMENT
              roles: CALL
              roles: EXPRESSION
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "VariableDeclarationStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "SimpleType"
            properties {
              key: "internalRole"
              value: "type"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "Map"
              start_position {
                offset: 1001
                line: 37
                col: 9
              }
              end_position {
                offset: 1004
                line: 37
                col: 12
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            roles: INCOMPLETE
          }
          children {
            internal_type: "VariableDeclarationFragment"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "options"
              start_position {
                offset: 1005
                line: 37
                col: 13
              }
              end_position {
                offset: 1012
                line: 37
                col: 20
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "initializer"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "JavaCore"
                start_position {
                  offset: 1015
                  line: 37
                  col: 23
                }
                end_position {
                  offset: 1023
                  line: 37
                  col: 31
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "getOptions"
                start_position {
                  offset: 1024
                  line: 37
                  col: 32
                }
                end_position {
                  offset: 1034
                  line: 37
                  col: 42
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              roles: CALL
              roles: EXPRESSION
            }
            roles: INCOMPLETE
          }
          roles: STATEMENT
          roles: INCOMPLETE
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "JavaCore"
              start_position {
                offset: 1046
                line: 38
                col: 9
              }
              end_position {
                offset: 1054
                line: 38
                col: 17
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setComplianceOptions"
              start_position {
                offset: 1055
                line: 38
                col: 18
              }
              end_position {
                offset: 1075
                line: 38
                col: 38
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "QualifiedName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "qualifier"
                }
                token: "JavaCore"
                start_position {
                  offset: 1076
                  line: 38
                  col: 39
                }
                end_position {
                  offset: 1084
                  line: 38
                  col: 47
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "VERSION_1_8"
                start_position {
                  offset: 1085
                  line: 38
                  col: 48
                }
                end_position {
                  offset: 1096
                  line: 38
                  col: 59
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: QUALIFIED_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              token: "options"
              start_position {
                offset: 1098
                line: 38
                col: 61
              }
              end_position {
                offset: 1105
                line: 38
                col: 68
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "ExpressionStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "MethodInvocation"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "expression"
              }
              token: "parser"
              start_position {
                offset: 1116
                line: 39
                col: 9
              }
              end_position {
                offset: 1122
                line: 39
                col: 15
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_RECEIVER
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "name"
              }
              token: "setCompilerOptions"
              start_position {
                offset: 1123
                line: 39
                col: 16
              }
              end_position {
                offset: 1141
                line: 39
                col: 34
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_CALLEE
            }
            children {
              internal_type: "SimpleName"
              properties {
                key: "internalRole"
                value: "arguments"
              }
              token: "options"
              start_position {
                offset: 1142
                line: 39
                col: 35
              }
              end_position {
                offset: 1149
                line: 39
                col: 42
              }
              roles: SIMPLE_IDENTIFIER
              roles: EXPRESSION
              roles: CALL_POSITIONAL_ARGUMENT
            }
            roles: CALL
            roles: EXPRESSION
          }
          roles: STATEMENT
        }
        children {
          internal_type: "ReturnStatement"
          properties {
            key: "internalRole"
            value: "statements"
          }
          children {
            internal_type: "CastExpression"
            properties {
              key: "internalRole"
              value: "expression"
            }
            children {
              internal_type: "SimpleType"
              properties {
                key: "internalRole"
                value: "type"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "CompilationUnit"
                start_position {
                  offset: 1168
                  line: 40
                  col: 17
                }
                end_position {
                  offset: 1183
                  line: 40
                  col: 32
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
              }
              roles: INCOMPLETE
            }
            children {
              internal_type: "MethodInvocation"
              properties {
                key: "internalRole"
                value: "expression"
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "expression"
                }
                token: "parser"
                start_position {
                  offset: 1185
                  line: 40
                  col: 34
                }
                end_position {
                  offset: 1191
                  line: 40
                  col: 40
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_RECEIVER
              }
              children {
                internal_type: "SimpleName"
                properties {
                  key: "internalRole"
                  value: "name"
                }
                token: "createAST"
                start_position {
                  offset: 1192
                  line: 40
                  col: 41
                }
                end_position {
                  offset: 1201
                  line: 40
                  col: 50
                }
                roles: SIMPLE_IDENTIFIER
                roles: EXPRESSION
                roles: CALL_CALLEE
              }
              children {
                internal_type: "NullLiteral"
                properties {
                  key: "internalRole"
                  value: "arguments"
                }
                token: "null"
                roles: NULL_LITERAL
                roles: EXPRESSION
                roles: CALL_POSITIONAL_ARGUMENT
              }
              roles: CALL
              roles: EXPRESSION
            }
            roles: EXPRESSION
            roles: INCOMPLETE
          }
          roles: RETURN
          roles: STATEMENT
        }
        roles: FUNCTION_DECLARATION_BODY
        roles: BLOCK_SCOPE
        roles: BLOCK
        roles: STATEMENT
      }
      start_position {
        offset: 691
        line: 29
        col: 5
      }
      end_position {
        offset: 1214
        line: 41
        col: 6
      }
      roles: VISIBLE_FROM_WORLD
      roles: FUNCTION_DECLARATION
    }
    start_position {
      offset: 366
      line: 14
      col: 1
    }
    end_position {
      offset: 1216
      line: 4294967295
    }
    roles: VISIBLE_FROM_WORLD
    roles: TYPE_DECLARATION
  }
  roles: FILE
}

[abeaumont]

Thanks @vmarkovtsev. The UAST has more than 2500 lines and clearly @EgorBu has a clear idea of where the ones he thinks are wrong ones are located. I don't.
That UAST you posted seems legit. Annotations in Java are not normal comments, they have semantic meaning and thus tokens with roles.

[vmarkovtsev]

@abeaumont It is not an annotation. The whole thing is surrounded with /* */. Here is what you get just a few lines above:

children {
        internal_type: "Javadoc"
        properties {
          key: "internalRole"
          value: "javadoc"
        }
        children {
          internal_type: "TagElement"
          properties {
            key: "internalRole"
            value: "tags"
          }
          properties {
            key: "tagName"
            value: "@param"
          }
          children {
            internal_type: "SimpleName"
            properties {
              key: "internalRole"
              value: "fragments"
            }
            token: "source"
            start_position {
              offset: 716
              line: 31
              col: 15
            }
            end_position {
              offset: 722
              line: 31
              col: 21
            }
            roles: SIMPLE_IDENTIFIER
            roles: EXPRESSION
          }

[abeaumont]

Right, not an annotation but tag elements inside a Javadoc. This semantic information is provided by the native driver and may be of use. If you don't want identifiers inside a documentation comment, you can filter them out.