Refactor dataset creation flow

[Creylay]

Summary

This PR refactors the dataset creation and processing flow to ensure consistency between the frontend and backend.

Previously, the frontend simulated dataset creation by adding a temporary object as soon as a file was uploaded. This fake dataset had no real id, and its status was managed through timers, which often caused tracking issues and desynchronization with the backend.

Now, the flow is aligned with other components of the system (e.g., explorers, converters). Datasets are created in the database first, assigned a valid id, and updated through jobs. The frontend only polls the backend for real status updates, eliminating inconsistencies.

---

Type of change

• Front end new feature.
• Back end new feature.
• Refactoring.
• Bug fix (tracking/desync issues).

---

Changes

Backend

Old flow:

1. Frontend → POST /v1/job/ (with file + parameters)
2. Job creates dataset + processes file in a single step
3. Dataset appears in DB only at the end (success or failure)

New flow:

1. Frontend → POST /v1/datasets/ (only dataset name)
   • Dataset is created immediately in the DB with status = NOT_STARTED.
2. Frontend → POST /v1/job/ (with dataset_id + file)
   • Job processes file and updates status.
3. Job updates:
   NOT_STARTED → STARTED → FINISHED / ERROR

Summary of backend changes:

• Added status column in datasets table.
• Introduced two-step flow: dataset creation → job execution.
• Job is now responsible for updating the dataset status.

---

Frontend

• Removed temporary dataset objects.
• Frontend now always works with a valid dataset.id from the backend.
• Polling is performed against the real backend status.

---

Tests

All API tests that previously created datasets through jobs were updated to match the new flow.

Modified files include:

• conftest.py
• test_dataset_api.py
• test_experiments_api.py
• test_explainer_jobs.py
• test_jobs.py
• test_predict_api.py
• test_runs_api.py

---

How to Test

1. Start the backend and frontend.
2. From the frontend, upload a dataset file.
3. Confirm that:
   • A dataset is immediately created in the DB with status = NOT_STARTED.
   • The job updates the dataset’s status (STARTED → FINISHED/ERROR).
   • The frontend correctly displays the real dataset status (no fake objects).
4. Run the test suite:

   pytest tests/

[Copilot]

Pull Request Overview

This PR refactors the dataset creation flow to establish a two-step process where datasets are first created in the database with a status, then processed by jobs. This replaces the previous approach where datasets only appeared in the database after job completion, eliminating frontend synchronization issues with temporary objects.

Key changes:

• Backend now creates datasets with status tracking before job processing
• Frontend uses real dataset objects from the database instead of temporary placeholders
• Test fixtures updated to use the new two-step dataset creation flow

Reviewed Changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
DashAI/back/core/enums/status.py	Adds DatasetStatus enum for tracking dataset processing states
DashAI/back/dependencies/database/models.py	Adds status column and status management methods to Dataset model
DashAI/back/job/dataset_job.py	Updates job to work with existing dataset records and manage status transitions
DashAI/back/api/api_v1/endpoints/datasets.py	Adds dataset creation endpoint and status validation for operations
DashAI/back/api/api_v1/schemas/datasets_params.py	Adds status field and creation parameters schema
DashAI/front/src/utils/datasetStatus.js	Utility function for mapping status numbers to readable strings
DashAI/front/src/types/dataset.ts	Adds status field to dataset interface
DashAI/front/src/pages/datasets/Datasets.jsx	Updates polling logic to check real dataset status
DashAI/front/src/components/notebooks/datasetCreation/ConfigureAndUploadDataset.jsx	Implements two-step creation flow
DashAI/front/src/api/job.ts	Updates job API to use dataset_id instead of name
DashAI/front/src/api/datasets.ts	Adds createDataset function for the new endpoint
tests/back/api/*.py	Updates test fixtures to use new creation flow

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}