Set Jobs Limit for scheduling

[aolfat]

Description

Added a few configurations that limit the # of jobs for a cronjob to exist. I was getting OOM errors if there were a lot of concurrent jobs running

For some context, before, we'd see a huggggeee list of the below and they'd all be running

Type of change

Does this correspond to an open issue?

Select type(s) of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist:

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have fixed any merge conflicts

[codecov]

Codecov Report

Merging #794 (bc10021) into main (6f7b498) will decrease coverage by 2.94%.
The diff coverage is 69.56%.

@@            Coverage Diff             @@
##             main     #794      +/-   ##
==========================================
- Coverage   59.71%   56.77%   -2.94%     
==========================================
  Files          63       66       +3     
  Lines       12408    13020     +612     
==========================================
- Hits         7409     7392      -17     
- Misses       4232     4910     +678     
+ Partials      767      718      -49     

Impacted Files	Coverage Δ
helpers/compression/compression.go	0.00% <ø> (ø)
helpers/env.go	87.50% <ø> (-12.50%)	⬇️
kubernetes/kubernetes.go	20.12% <ø> (-0.56%)	⬇️
metadata/metadata.go	71.02% <ø> (+2.05%)	⬆️
provider/bigquery.go	70.00% <ø> (-0.18%)	⬇️
provider/filepath.go	59.55% <ø> (ø)
provider/filestore.go	25.54% <ø> (-17.39%)	⬇️
provider/k8s.go	9.30% <ø> (-20.90%)	⬇️
provider/provider_config/spark_config.go	60.60% <ø> (+3.55%)	⬆️
provider/provider_type/provider_type.go	100.00% <ø> (ø)
... and 8 more

... and 14 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more