Segmentation fault when parsing many files

[mjambon]

See #64 for some context and ask @aryx for details on how to reproduce the crash.

keywords: segfault, crash, memory leak

[mjambon]

@aryx can you provide instructions on how to reproduce the crash?

[aryx]

Actually, it's either you get a segfault, or a giant memory leak. To get the segfault, which should be easier to debug, you need
first to modify semgrep/ocaml-tree-sitter/src/bindings/lib/bindings.c which right now contains:

static void finalize_tree(value v) {
  tree_W *p;
  p = (tree_W *)Data_custom_val(v);
  //TODO: ts_tree_delete(p->tree);
  // this caused some segfaults, probably during Gc after
  // analyzing many Ruby files. We go around this bug by
  // running the Ruby parser in a separate process so segfaults
  // or here memory leak do not reach the main semgrep-core process.
}

step1: uncomment the call to ts_tree_delete(p->tree).

then step2, in many of the semgrep-core/parsing/tree_sitter/Parse_xxx_tree_sitter.ml there will be code like

let parse file =
  H.wrap_parser
    (fun () ->
       Parallel.backtrace_when_exn := false;
       Parallel.invoke Tree_sitter_java.Parse.file file ()
    )
    (fun cst ->
       let env = { H.file; conv = H.line_col_to_pos file; extra = () } in
       program env cst
    )

which use Parallel.invoke to call Tree_sitter_java.Parse.file under a fork. If you call directly
Tree_sitter_java.Parse.file file, without the Parallel.invoke, and you use semgrep on a big java repository, at some point you'll get a segfault.

[mjambon]

Fixed by semgrep/ocaml-tree-sitter-core#11