Classification : train and test data to s3 #343

nishika26 · 2025-08-24T17:21:37Z

Summary

Target issue is #341

Checklist

Before submitting a pull request, please ensure that you mark these task.

Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
If you've fixed a bug or added code that is tested and has test cases.

Changes Made

app/core/finetune/preprocessing.py

S3 Upload Support

Added upload_csv_to_s3() method to upload train/test CSV splits directly to S3.
Added detailed logging for both successful and failed uploads.

File Naming

Standardized naming for train/test splits using pct (split ratio %) and a unique uuid.
Generates both:
- .csv → uploaded to S3
- .jsonl → stored locally in temp (for fine-tuning)

Preprocessor class now returns

train_csv_url → S3 path of training split CSV
test_csv_url → S3 path of test split CSV
train_jsonl_path → local path of training JSONL

app/api/routes/fine_tuning.py

Removed test data upload to OpenAI
- Test split will no longer be uploaded to OpenAI.
- For evaluation, test data will instead be fetched directly from S3.
DB is now updated with:
- train_data_url → S3 URL of the training split.
- test_data_url → S3 URL of the test split.

coderabbitai · 2025-08-24T17:21:46Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbit review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch enhancement/data_to_s3

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbit in a new review comment at the desired location with your query.
PR comments: Tag @coderabbit in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbit gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbit read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbit help to get the list of available commands.

Other keywords and placeholders

Add @coderabbit ignore or @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbit summary or @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbit or @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

Visit our Status Page to check the current availability of CodeRabbit.
Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

AkhileshNegi · 2025-08-27T05:37:30Z

backend/app/api/routes/fine_tuning.py

            result = preprocessor.process()
-            train_path = result["train_file"]
-            test_path = result["test_file"]
+            train_path = result["train_jsonl_path"]


how is train_path different than train_data_url? and do we need it?

train_path is the local temp path where the training data is saved i think and train_data_url is the s3 path

@nishika26 maybe we can have more descriptive names? like training_data_tmp_file_path and training_data_object_store_url - they are longer for sure, but better represent what the vars mean

exactly as what @kartpop stated here, and maybe instead of using over descriptive names, I could add a comment along the lines of the varaibles? that would be better I think

made the variable names descriptive

AkhileshNegi · 2025-08-27T05:38:49Z

backend/app/core/finetune/preprocessing.py


        self.system_message = {"role": "system", "content": system_message.strip()}

+    def upload_csv_to_s3(self, df, filename: str) -> str:


do we need separate function to upload? why can't we use existing function to upload file?

We are already using the existing put function for the actual upload inside this function. The upload_csv_to_s3 method is just a wrapper that converts a DataFrame into CSV bytes, wraps it in an UploadFile (the format that put(source: UploadFile, basename: Path) expects), and then delegates the upload to put.

AkhileshNegi · 2025-08-27T05:39:34Z

backend/app/core/finetune/preprocessing.py

-            f"train_data_{int(self.split_ratio * 100)}_{uuid.uuid4().hex}.jsonl"
-        )
-        test_file = f"test_data_{int(self.split_ratio * 100)}_{uuid.uuid4().hex}.jsonl"
+        uuids = uuid.uuid4().hex


uuids or uuid ?

uuids, because if I name the variable uuid, it will shadow the imported uuid module, which can confuse both the interpreter and anyone reading the code. But i guess "uuids" is confusing as well so I will name it something better

AkhileshNegi · 2025-08-27T05:39:54Z

backend/app/core/finetune/preprocessing.py

-        )
-        test_file = f"test_data_{int(self.split_ratio * 100)}_{uuid.uuid4().hex}.jsonl"
+        uuids = uuid.uuid4().hex
+        pct = int(self.split_ratio * 100)  # percentage of data


name variables in a way that is self explainatory

kartpop

approving with comments; can be merged after comments by @AkhileshNegi are resolved

kartpop · 2025-08-27T14:51:52Z

backend/app/api/routes/fine_tuning.py

            document_crud = DocumentCrud(session=session, owner_id=current_user.id)
            document = document_crud.read_one(request.document_id)
            preprocessor = DataPreprocessor(
                document, storage, ratio, request.system_prompt


the storage = AmazonCloudStorage(current_user) will later change to using project_id instead

this is just FYI for now, this change need not be part of the current PR - can be part of a future PR

yes, as of now (Thu, 26 Aug, 12:10 PM), the PR that changes storage initialization from using user_id to project_id hasn’t been merged yet, so I can’t incorporate that change in my PR at this moment or in any other PR till that PR is not merged to main

kartpop · 2025-08-27T15:04:07Z

backend/app/api/routes/fine_tuning.py

            result = preprocessor.process()
-            train_path = result["train_file"]
-            test_path = result["test_file"]
+            train_path = result["train_jsonl_path"]


train_path is the local temp path where the training data is saved i think and train_data_url is the s3 path

@nishika26 maybe we can have more descriptive names? like training_data_tmp_file_path and training_data_object_store_url - they are longer for sure, but better represent what the vars mean

kartpop · 2025-08-27T15:04:47Z

backend/app/core/finetune/preprocessing.py

-        )
-        test_file = f"test_data_{int(self.split_ratio * 100)}_{uuid.uuid4().hex}.jsonl"
+        uuids = uuid.uuid4().hex
+        pct = int(self.split_ratio * 100)  # percentage of data


AkhileshNegi · 2025-08-28T08:34:34Z

backend/app/api/routes/fine_tuning.py


            try:
-                with open(train_path, "rb") as train_f:
+                with open(train_data_temp_filepath, "rb") as train_f:


temp_train_data_filepath looks cleaner

what's rb?

"rb" means read the file in binary mode, which is required here since we’re uploading the file object to OpenAI’s API — the Files API specifically accepts a binary stream as input.

* Classification: db models and migration script (#305) * db models and migration script * Classification: Fine tuning Initiation and retrieve endpoint (#315) * Fine-tuning core, initiation, and retrieval * seperate session for bg task, and formating fixes * fixing alembic revision * Classification : Model evaluation of fine tuned models (#326) * Model evaluation of fine tuned models * fixing alembic revision * alembic revision fix * Classification : train and test data to s3 (#343) * alembic file for adding and removing columns * train and test s3 url column * updating alembic revision * formatting fix * Classification : retaining prediction and fetching data from s3 for model evaluation (#359) * adding new columns to model eval table * test data and prediction data s3 url changes * single migration file * status enum columns * document seeding * Classification : small fixes and storage related changes (#365) * first commit covering all * changing model name to fine tuned model in model eval * error handling in get cloud storage and document not found error handling * fixing alembic revision * uv lock * new uv lock file * updated uv lock file * coderabbit suggestions and removing unused imports * changes in uv lock file * making csv a supported file format, changing uv lock and pyproject toml

alembic file for adding and removing columns

6fb4e5f

nishika26 changed the base branch from main to feature/classification August 24, 2025 17:22

train and test s3 url column

6f6fadb

nishika26 changed the title ~~Enhancement/data to s3~~ train and test data to s3 Aug 24, 2025

nishika26 added the enhancement New feature or request label Aug 25, 2025

nishika26 linked an issue Aug 25, 2025 that may be closed by this pull request

Classification : Enhance pipeline to save train and test datasets to s3 for reuse #341

Closed

nishika26 self-assigned this Aug 25, 2025

nishika26 changed the title ~~train and test data to s3~~ Classification : train and test data to s3 Aug 25, 2025

only keeping fine tuning table changes in alembic migration

c7a0f2f

nishika26 marked this pull request as ready for review August 25, 2025 09:05

nishika26 requested review from AkhileshNegi and kartpop August 25, 2025 09:05

nishika26 added the ready-for-review label Aug 25, 2025

AkhileshNegi requested changes Aug 27, 2025

View reviewed changes

kartpop approved these changes Aug 27, 2025

View reviewed changes

nishika26 added 2 commits August 28, 2025 13:39

PR review fixes

77ce4c4

making column name descriptive

6278d34

AkhileshNegi approved these changes Aug 28, 2025

View reviewed changes

nishika26 merged commit 9407df0 into feature/classification Aug 28, 2025
1 check passed

nishika26 deleted the enhancement/data_to_s3 branch August 28, 2025 08:49


		self.system_message = {"role": "system", "content": system_message.strip()}

		def upload_csv_to_s3(self, df, filename: str) -> str:

Uh oh!

Classification : train and test data to s3 #343

Classification : train and test data to s3 #343

Uh oh!

Conversation

nishika26 commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Changes Made

Uh oh!

coderabbitai bot commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR/Issue comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Status, Documentation and Community

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nishika26 Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kartpop left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nishika26 commented Aug 24, 2025 •

edited

Loading

coderabbitai bot commented Aug 24, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

nishika26 Aug 28, 2025 •

edited

Loading