v1.0: Code Cleanup / Streamlining

[Innixma]

We should ensure all primary components in the code-base are cleaned up to not be overly complicated / hacky, and to enhance ease of contribution and maintenance, while minimizing code-dupe.

• Running new benchmarks / configs (AutoGluon-Benchmark, AutoMLBenchmark)
  • Simplify generation of BenchmarkContexts for new benchmarks.
  • Standardize S3 file location + naming
  • Automate retries on failed runs
  • Repeated cross-validation support for reducing simulation overfitting
  • Run configs in nested bagging mode to eliminate validation overfitting
  • Add extra model config results: KNN, NN, XGB, FT-Transformer, TabPFN, Linear, VowpalWabbit, etc.
  • Expand search space for key models (CAT, FASTAI)
  • Augment datasets for more information (subsample rows for example)
  • Collect more high quality datasets (Ex: Kaggle, MachineHack)
• Downloading input files from benchmarks for simulation & loading them (Missing results in metrics #15)
• Handling & Accessing Simulation Contexts (Code to reproduce paper figures #18)
• Handling & Accessing Model Predictions (WIP: Add time_limit support #13)
• Running & Scoring Portfolios (ConfigurationListScorer)
• Conducting Zeroshot Simulations (ZeroshotConfigGenerator, ZeroshotConfigGeneratorCV)
• Generating and evaluating baselines + comparing against simulation results
  • Oracle baselines
  • Simple baselines
• Storing results, extracting analytics, and comparison results from simulations (PortfolioCV, AutoGluon-Benchmark integration)
  • Properly calculate inference speed per task for portfolios
  • Properly calculate scores for portfolios when time_limit is specified.
  • Properly calculate scores for portfolios when infer_limit is specified.
  • Enable extra logic such as temperature scaling to match what occurs in AutoGluon.
• Automate running portfolios on the benchmark to verify results + test additional features like repeated bagging + stacking for learned portfolios.
  • Do so without overfitting (cross-validated portfolios)