Transpiler gen-test-proof got killed on large(?) circuits

[color-typea]

Problem: When invoked with gen-test-proof, transpiler stops (after ~30-50 minutes), printing "Killed" to terminal.

Steps to reproduce:

• Use examples/cpp/balance_tree.cpp
  ** Change validators_amount_log2 to a larger number (for me, 8 still worked, 9 and 10 failed)
  ** Adjust examples/input/balance_tree.inp to contain corresponding number of values in the first argument (second and third can remain unchanged, but I've tested with the correct values too).
  ** Complete change: color-typea@c687802
• Compile and assign circuit: cmake --build build --target balances_tree_cpp_example_assign
• Transpile: mkdir output && time ./build/bin/transpiler/transpiler -i examples/inputs/balances_tree.inp -t build/exmples/cpp/balances_tree.tbl -c build/exmples/cpp/balances_tree.crct -m gen-test-proof -o output --optimize-gates

Expected result: Transpiler generates test proof (proof.bin) in output folder.
Actual result: Transpiler prints "Killed" to the console in 30-50 minutes and do not generate a test proof.

Toolchain versions:
All built from source
zkkllvm - v0.0.86, d66350b
zkllvm/libs/circifier - v0.0.33, c67458ea4d9f80a9bf61a37ccbef9b60ed11575a
zkllvm/libs/assigner - v0.0.39, d66350b
zkllvm libs/transpiler - v0.0.9. b5fb9115404fa6a581dddd88c4068b2844ea43ad

./build/libs/circifier/llvm/bin/clang-16 --version
clang version 16.0.0 (git@github.com:NilFoundation/zkllvm-circifier.git c67458ea4d9f80a9bf61a37ccbef9b60ed11575a)

P.S. this looks like an act of OOM Killer. For full details, it run on a 32Gb memory (and disabled swap) machine, with ~10Gb used by other processes - so it should have had at least 20Gb. The transpiler memory consumption fluctuated between 5.7Gb and 10.9Gb as it run, but I only monitored it occasionally

[nkaskov]

We will soon remove gen-test-proof option since we are developing a standalone prover, which can be used for test purposes as well: NilFoundation/proof-market-toolchain#89

There also will be a multithreaded version for huge circuit proving.

[nkaskov]

This will be completely solved by #404