feat: Establish new API encapsulation for dataset creation and prediction providers

[cau-git]

This PR introduces a new class design for:

• Dataset builders (class interface and implementations to encapsulate all of the create.py logic)
• Prediction providers (class interface to generalize prediction creation beyond a docling converter, e.g for using hyperscalers SaaS, etc.)
• Record data model, superseeding scattered codes for building a record (BenchmarkColumns enum, HF dataset features, etc)

All modules are kept in a different root directory for now: docling_eval_next.

This PR will stay WIP until the current codebase is fully ported to it and its design is evaluated.

Dataset builder porting

• DPBench (e2e, tables)
• OmniDocBench (e2e, tables)
• DocLayNet v1 (e2e)
• DocLayNet v2 (e2e) 🚧
• FUNSD (key-value)
• XFUND (key-value)
• OTSL Table datasets (FTN, P1M, PTN) (see also Add table evaluation for table datasets, and hyperscaler predictors #47)
• CVAT
• PixParse (see Hyperscalers OCR Evaluation #46)

Prediction providers

• Docling
• TableFormer
  - [ ] Reading Order stays in evaluators
• Azure Document Intelligence (see Add table evaluation for table datasets, and hyperscaler predictors #47)
• AWS Textract
• Google Document AI
• SmolDocling

TODO:

• Entirely decouple PredictionProvider from DatasetBuilder. Have DatasetBuilder only produce GT columns, then read the produced parquet files in a secondary step on the prediction providers to augment it.
• Create a PredictionDatasetRecord subclass, which carries prediction related fields, and remove these from the DatasetRecord class.
• Add record field to carry the raw prediction output of any provider (similar to the original for the GT)
• Provide the full DatasetRecord to the PredictionProvider.predict method instead of some select fields out of it.
• Clean up path handling @cau-git
• Reinstate visualization, move to prediction provider @cau-git
• Factor out more common code between the dataset builders @cau-git
• Adopt bugfixes from stacked PRs @praveenmidde @samiuc
• Add new prediction provider class which can read from a directory with prediction files @nikos-livathinos
• Upgrade evaluator classes to use DatasetRecordWithPrediction model instead of direct column data access (then, deprecate BenchmarkColumns) @nikos-livathinos
• Update CLI to use new API @nikos-livathinos

[praveenmidde]

@cau-git Just a thought, since the new design helps in extending the framework for different datasets and models, can we create a library wheel for docling-eval so that one can install and extend it?

[nikos-livathinos]

Looks good to me!