fix: probabilistic test failures

[laysakura]

use single thread for tests asserting some values?

[laysakura]

Latest 5 test failures and possible causes

Take away: related to window operation?

• https://github.com/SpringQL/SpringQL/runs/5825844630?check_suite_focus=true

  test test_feat_left_outer_join ... FAILED
      Diff < left / right > :
      <2
      >3
  
  
      ': springql-core/tests/feat_join.rs:152
         0: log_panics::init::{{closure}}
  

• https://github.com/SpringQL/SpringQL/runs/5825844880?check_suite_focus=true

  [2022-04-05T00:13:11Z ERROR panic] thread 'GenericWorker' panicked at 'task id Pump { id: "pu_join", input_type: Window } does not have upstream source_city_temperature': springql-core/src/stream_engine/autonomous_executor/task_graph.rs:111
  
  ...
  
  [2022-04-05T00:13:12Z ERROR panic] thread 'test_feat_left_outer_join' panicked at 'failed to wait for worker thread to finish its job: SendError { .. }': springql-core/src/stream_engine/autonomous_executor/worker/worker_handle.rs:42
  

• https://github.com/SpringQL/SpringQL/runs/5825770143?check_suite_focus=true

  test test_e2e_sampling_sliding_window ... FAILED
  
      Diff < left / right > :
  failures:
      <4
  
      >5
  
      ': springql-core/tests/e2e_sampling.rs:204
  

• https://github.com/SpringQL/SpringQL/runs/5825770404?check_suite_focus=true

  [2022-04-05T00:06:01Z ERROR panic] thread 'test_e2e_sampling_fixed_window' panicked at 'assertion failed: `(left == right)`
  
      Diff < left / right > :
      <30
      >20
  
      ': springql-core/tests/e2e_sampling.rs:122
  

• https://github.com/SpringQL/SpringQL/runs/5825477547?check_suite_focus=true

  [2022-04-04T23:31:33Z ERROR panic] thread 'test_e2e_sampling_sliding_window' panicked at 'assertion failed: `(left == right)`
  
      Diff < left / right > :
      <3
      >5
  
      ': springql-core/tests/e2e_sampling.rs:204
  

[laysakura]

does not reproduce with:

    config.worker.n_generic_worker_threads = 1;
    // config.worker.n_source_worker_threads = 1;

reproduce with:

    config.worker.n_generic_worker_threads = 1;
    config.worker.n_source_worker_threads = 1;

    // config.worker.n_generic_worker_threads = 1;
    config.worker.n_source_worker_threads = 1;

[laysakura]

Cause (high confidence)

1. multiple source workers may change the order of source inputs

2. but tests expected the source inputs are in-order:

   

3. input rows come out-of-order. Early input row can close a window too early.

[laysakura]

Remaining issues

The above cause cannot explain the following issue:

• https://github.com/SpringQL/SpringQL/runs/5825844880?check_suite_focus=true

  [2022-04-05T00:13:11Z ERROR panic] thread 'GenericWorker' panicked at 'task id Pump { id: "pu_join", input_type: Window } does not have upstream source_city_temperature': springql-core/src/stream_engine/autonomous_executor/task_graph.rs:111
  
  ...
  
  [2022-04-05T00:13:12Z ERROR panic] thread 'test_feat_left_outer_join' panicked at 'failed to wait for worker thread to finish its job: SendError { .. }': springql-core/src/stream_engine/autonomous_executor/worker/worker_handle.rs:42
  

Created: #80

[laysakura]

Solution

• Make n_*_threads = 1 as default to keep in-order processing.
• Documents n_*_threads > 1 can lead to out-of-order processing.