Print stack trace on fuzzer crashes in CI

[mbasmanova]

Description

When Fuzzer crashes in CI, we don't get any stack trace. We also don't get INFO logs, hence, don't know which seed triggered a crash.

https://github.com/facebookincubator/velox/actions/runs/9214079180/job/25355158577?pr=9911

*** Aborted at 1716505116 (Unix time, try 'date -d @1716505116') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 81 (pthread TID 0x7fb4994834c0) (linux TID 81) (code: address not mapped to object), stack trace: ***

(error retrieving stack trace)

/__w/_temp/bcbd91e6-d2[37](https://github.com/facebookincubator/velox/actions/runs/9214079180/job/25355158577?pr=9911#step:6:38)-4866-b07d-325693ddfa49.sh: line 18:    81 Segmentation fault      (core dumped) ./velox_expression_fuzzer_test --seed ${RANDOM} --enable_variadic_signatures --velox_fuzzer_enable_complex_types --lazy_vector_generation_ratio 0.2 --velox_fuzzer_enable_column_reuse --velox_fuzzer_enable_expression_reuse --max_expression_trees_per_step 2 --retry_with_try --enable_dereference --duration_sec $DURATION --logtostderr=1 --minloglevel=1 --repro_persist_path=/tmp/fuzzer_repro

CC: @assignUser @kgpai

[mbasmanova]

@assignUser @kgpai Any update?

[assignUser]

We do have a step setup that should upload an repro material in repro_persist_path but it seems that nothing was found at that location. I don't know enough about how the fuzzers work to say anything more @kgpai what do you think?

If the repro only get's saved on a fail maybe we want save the initial state before the actual run and remove it after if there was no fail? That way we would still have it in a case like this?

[kgpai]

In this case @assignUser the fuzzer job crashes, before it can save state, that is why nothing is found . In this case maybe we need to catch segfault and dump what we can to repro the fuzzer.

[kgpai]

As a short term we need to increase the fuzzer logging.

[kgpai]

We could also try this : #6402
Should be as easy as setting this flag. velox_experimental_save_input_on_fatal_signal = true for our fuzzers.

[assignUser]

We reduced the logging level because it increased the runtime by 2-3x but I just looked at the logging options and we can also just log into a file. Currently testing if that keeps the duration in an expected range.

We can out put the file after the fuzzer step.