chore: use single query for getting jobs

[atzoum]

Description

Router iterator, batchrouter & stash can now perform a single SQL query for retrieving jobs instead of multiple.
The above behaviour is toggleable through a feature flag, JobsDB.useSingleGetJobsQuery. If the flag is disabled, rudder-server falls back to its previous behaviour.

Querying

• For unprocessed jobs we are using a left join with the status table
• For all other job states we are using an inner join with the latest status view
• When the query needs to query both types of states we are using a left join with the latest status view

Caching

• The no jobs cache is checked for all requested job states separately. e.g. if you ask for failed and unprocessed and an entry exists in the cache for failed, the final query will only contain predicates for unprocessed jobs.
• The no jobs cache is updated for separate job states as well, e.g.
  • If no jobs are returned by the query, entries will be added in the cache for all states
  • If only unprocessed jobs are returned for a request issued about both failed and unprocessed, an entry will be added in the no jobs cache for failed state.

Linear Ticket

PIPE-219

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

Patch coverage: 55.55% and project coverage change: -0.25% ⚠️

Comparison is base (3a34e06) 68.92% compared to head (c72ab15) 68.68%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3820      +/-   ##
==========================================
- Coverage   68.92%   68.68%   -0.25%     
==========================================
  Files         352      352              
  Lines       52604    52677      +73     
==========================================
- Hits        36259    36181      -78     
- Misses      14015    14174     +159     
+ Partials     2330     2322       -8     

Files Changed	Coverage Δ
processor/stash/stash.go	47.90% <18.42%> (-17.18%)	⬇️
router/batchrouter/handle.go	60.80% <25.00%> (-2.42%)	⬇️
router/handle.go	72.57% <75.00%> (+0.02%)	⬆️
jobsdb/jobsdb.go	70.99% <88.52%> (-1.46%)	⬇️
router/internal/jobiterator/jobiterator.go	100.00% <100.00%> (ø)

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.