chore: batch & fine-tune improvements #1089

b4s36t4 · 2025-05-09T13:57:47Z

Description

Fine-tune support for vertex
Batch embedding support for vertex
preserve custom_id for the vertex batch output.
streamlined responses for vertex with OAI.

Motivation

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Testing

Screenshots (if applicable)

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

matter-code-review · 2025-05-09T13:57:51Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the PR
/matter release-notes: Generate AI release-notes for the PR
/matter <ask-question>: Chat with your PR with Matter AI Agent
/matter remember <recommendation>: Generate AI memories for the PR

matter-code-review · 2025-05-09T13:58:16Z

Summary By MatterAI

🔄 What Changed

Added BatchEndpoints enum to standardize batch operation endpoints
Implemented Fireworks AI fine-tuning capabilities (create, cancel, list fine-tunes)
Enhanced file upload functionality for Fireworks AI
Improved Google Vertex AI batch operations with support for embeddings
Added line trimming in stream handler utils
Updated package.json dependencies order

🔍 Impact of the Change

This PR enhances the platform's capabilities by adding comprehensive fine-tuning support for Fireworks AI and improving batch operations across providers. The changes enable more efficient model training workflows and standardize batch operation endpoints, making the system more consistent and maintainable.

📁 Total Files Changed

17 files modified with 814 additions and 68 deletions
New files: src/providers/fireworks-ai/cancelFinetune.ts, src/providers/fireworks-ai/createFinetune.ts, src/providers/fireworks-ai/listFinetune.ts
Major modifications to Google Vertex AI and Fireworks AI provider implementations

🧪 Test Added

N/A - No explicit tests mentioned in the PR

🔒Security Vulnerabilities

N/A - No security vulnerabilities detected

Motivation

To improve batch operations and add fine-tuning capabilities for Fireworks AI, enhancing the platform's model training and inference capabilities.

Type of Change

New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Testing

Screenshots (if applicable)

N/A

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

N/A

Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway
    participant FireworksAI
    participant GoogleVertexAI
    
    %% Batch Operations Flow
    Client->>Gateway: Request batch operation
    Gateway->>Gateway: Check BatchEndpoints enum
    Note over Gateway: New BatchEndpoints enum added with:
    Note over Gateway: CHAT_COMPLETIONS, COMPLETIONS, EMBEDDINGS
    
    %% Fireworks AI Fine-tuning Flow
    Client->>Gateway: Create fine-tune request
    Gateway->>FireworksAI: POST /supervisedFineTuningJobs
    Note over Gateway,FireworksAI: Transform using FireworksFinetuneCreateConfig
    FireworksAI-->>Gateway: Fine-tune job response
    Gateway-->>Client: Transformed OpenAI-compatible response
    
    Client->>Gateway: Cancel fine-tune request
    Gateway->>FireworksAI: DELETE /supervisedFineTuningJobs/{jobId}
    FireworksAI-->>Gateway: Cancelled job response
    Gateway-->>Client: Transformed OpenAI-compatible response
    
    Client->>Gateway: List fine-tunes request
    Gateway->>FireworksAI: GET /supervisedFineTuningJobs?pageToken={page}&pageSize={size}
    FireworksAI-->>Gateway: Fine-tune jobs list
    Gateway-->>Client: Transformed OpenAI-compatible response
    
    %% File Upload Flow
    Client->>Gateway: Upload file request
    Gateway->>FireworksAI: Create dataset
    FireworksAI-->>Gateway: Dataset created
    Gateway->>FireworksAI: Get upload endpoint
    FireworksAI-->>Gateway: Pre-signed URL
    Gateway->>FireworksAI: PUT file to pre-signed URL
    Gateway->>FireworksAI: Validate dataset
    FireworksAI-->>Gateway: Validation response
    Gateway-->>Client: File upload response
    
    %% Google Vertex AI Batch Operations
    Client->>Gateway: Batch operation with vertexBatchEndpoint
    Gateway->>GoogleVertexAI: Process batch based on endpoint type
    Note over Gateway,GoogleVertexAI: Support for embeddings and chat completions
    GoogleVertexAI-->>Gateway: Batch operation response
    Gateway-->>Client: Transformed response

matter-code-review · 2025-05-09T14:00:43Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the PR
/matter release-notes: Generate AI release-notes for the PR
/matter <ask-question>: Chat with your PR with Matter AI Agent
/matter remember <recommendation>: Generate AI memories for the PR

matter-code-review · 2025-05-09T14:01:08Z

Summary By MatterAI

🔄 What Changed

Added support for batch operations with new BatchEndpoints enum
Implemented fine-tuning improvements for Fireworks AI provider
Added new handlers for Fireworks AI fine-tuning operations (create, list, cancel)
Enhanced Google Vertex AI to support different batch endpoints (chat completions, embeddings)
Improved line splitting in stream handler to trim lines
Updated package.json dependencies ordering

🔍 Impact of the Change

This PR enhances the platform's capabilities for batch processing and fine-tuning operations. It adds comprehensive support for Fireworks AI fine-tuning workflows and improves Google Vertex AI batch operations to handle different endpoint types. These changes make the system more flexible and capable of handling various AI model training and batch inference scenarios.

📁 Total Files Changed

17 files modified with 814 additions and 68 deletions
New files: src/providers/fireworks-ai/cancelFinetune.ts, src/providers/fireworks-ai/createFinetune.ts, src/providers/fireworks-ai/listFinetune.ts
Major modifications to Google Vertex AI and Fireworks AI provider implementations

🧪 Test Added

N/A - No explicit tests mentioned in the PR

🔒Security Vulnerabilities

N/A - No security vulnerabilities detected

Motivation

To enhance the platform's capabilities for batch processing and fine-tuning operations across different AI providers, particularly Fireworks AI and Google Vertex AI.

Type of Change

New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Testing

Screenshots (if applicable)

N/A

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

N/A

Sequence Diagram

sequenceDiagram
    participant Client
    participant BatchEndpoints
    participant FireworksAI
    participant GoogleVertexAI
    
    Note over BatchEndpoints: New enum added with endpoints
    Note over BatchEndpoints: CHAT_COMPLETIONS, COMPLETIONS, EMBEDDINGS
    
    Client->>FireworksAI: Upload File
    FireworksAI-->>Client: Return dataset ID
    
    Client->>FireworksAI: Create Finetune
    Note over FireworksAI: Uses FireworksFinetuneCreateConfig
    Note over FireworksAI: Transforms request with FireworksRequestTransform
    FireworksAI-->>Client: Return finetune job
    
    Client->>FireworksAI: List Finetunes
    Note over FireworksAI: Uses FireworkListFinetuneResponseTransform
    FireworksAI-->>Client: Return list of finetunes
    
    Client->>FireworksAI: Cancel Finetune
    Note over FireworksAI: Uses FireworksCancelFinetuneRequestHandler
    FireworksAI-->>Client: Return cancelled finetune
    
    Client->>GoogleVertexAI: Create Batch
    Note over GoogleVertexAI: Now supports embeddings endpoint
    Note over GoogleVertexAI: Uses vertexBatchEndpoint parameter
    GoogleVertexAI-->>Client: Return batch job
    
    Client->>GoogleVertexAI: Get Batch Output
    Note over GoogleVertexAI: Handles different batch endpoints
    Note over GoogleVertexAI: Uses different prediction file paths
    GoogleVertexAI-->>Client: Return batch results

…ch-improvements

matter-code-review · 2025-05-26T11:18:37Z

Description

# Summary By MatterAI

🔄 What Changed

Introduced BatchEndpoints enum for standardizing batch API endpoints (/v1/chat/completions, /v1/completions, /v1/embeddings).
Added comprehensive support for Fireworks AI finetuning, including creation, listing, retrieval, and cancellation of finetune jobs.
Implemented Fireworks AI file upload functionality, involving dataset creation, signed URL retrieval, and validation.
Refactored Google Vertex AI file upload to support different purposes (batch, fine-tune, direct upload) and dynamically apply transformations based on the specified batch endpoint (chat completions or embeddings).
Enhanced Google Vertex AI batch output retrieval to correctly handle embedding models and their specific output file names (000000000000.jsonl vs predictions.jsonl).
Introduced generateSignedURL utility for secure Google Cloud Storage uploads within the Vertex AI context.
Added new request headers x-portkey-provider-batch-endpoint (for Vertex AI batch endpoint specification) and x-portkey-file-upload-size (for Fireworks AI file length) to Options.
Fixed a minor bug in createLineSplitter to ensure enqueued lines are trimmed, preventing empty strings.

🔍 Impact of the Change

Significantly expands the gateway's capabilities by enabling full finetuning and file management workflows for Fireworks AI, aligning with OpenAI's API.
Improves the flexibility and correctness of Google Vertex AI batch processing, making it compatible with both chat completion and embedding models.
Enhances the robustness of file upload mechanisms across providers by handling pre-signed URLs and different data transformation requirements.
Provides a more configurable and feature-rich platform for AI model interactions, supporting a wider range of use cases.

📁 Total Files Changed

13

🧪 Test Added

No dedicated unit or integration test files were added in this pull request.
Manual testing would be required to verify the new Fireworks AI finetuning and file upload flows, as well as the updated Google Vertex AI batch and file upload functionalities.

🔒Security Vulnerabilities

The generateSignedURL function handles Google Cloud service account private keys for generating signed URLs. It is critical that the serviceAccountInfo.private_key is securely managed and never exposed in client-side code or logs. The implementation itself appears to use standard cryptographic practices for signing, but the secure handling of the private key in the environment is paramount.

Motivation

Expanded provider capabilities and improved existing batch/file handling.

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Testing

Screenshots (if applicable)

N/A

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

.

Tip

Quality Recommendations

Add comprehensive unit and integration tests for the new Fireworks AI finetuning and file upload functionalities, covering various success and failure scenarios, especially around data transformations and API interactions.
Enhance documentation for the new x-portkey-provider-batch-endpoint and x-portkey-file-upload-size headers, including their expected values and impact on request processing.
Review the usage of fireworksFileLength in uploadFile for Fireworks AI. If content-length header is reliably available from the client, it might be more robust to use that directly instead of a custom header, or clarify the scenarios where fireworksFileLength is necessary.
Ensure consistent error response structures across all new and modified handlers, particularly for provider-specific errors, to provide a uniform experience to the client.
Consider adding more specific type definitions for response and error objects in FireworkCancelFinetuneResponseTransform and FireworkFinetuneTransform to improve type safety and readability.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway
    participant FireworksAI_API as Fireworks AI API
    participant GoogleVertexAI_API as Google Vertex AI API
    participant GCS as Google Cloud Storage

    Client->>Gateway: POST /v1/files (Upload File)
    activate Gateway
    Note over Gateway: Headers: x-portkey-provider-file-name, x-portkey-file-upload-size (Fireworks), x-portkey-vertex-storage-bucket-name, x-portkey-provider-model, x-portkey-provider-batch-endpoint (Vertex)
    Gateway->>Gateway: Parse Headers (handlerUtils.ts)
    Gateway->>Gateway: Determine Provider (Fireworks/Vertex)
    alt Fireworks AI File Upload
        Gateway->>FireworksAI_API: POST /datasets (Create Dataset)
        activate FireworksAI_API
        FireworksAI_API-->>Gateway: Dataset ID
        deactivate FireworksAI_API
        Gateway->>FireworksAI_API: POST /datasets/{id}:getUploadEndpoint (Get Signed URL)
        activate FireworksAI_API
        FireworksAI_API-->>Gateway: Pre-signed GCS URL
        deactivate FireworksAI_API
        Gateway->>GCS: PUT/POST Pre-signed URL (Upload File Stream)
        activate GCS
        GCS-->>Gateway: Upload Status
        deactivate GCS
        Gateway->>FireworksAI_API: POST /datasets/{id}:validateUpload (Validate Dataset)
        activate FireworksAI_API
        FireworksAI_API-->>Gateway: Validation Status
        deactivate FireworksAI_API
        Gateway-->>Client: File Upload Response (id, bytes, status)
    else Google Vertex AI File Upload
        Gateway->>Gateway: Determine upload method (PUT/POST) based on purpose
        alt PUT Method (Direct GCS Upload)
            Gateway->>GCS: PUT /bucket/object (Upload File Stream)
            activate GCS
            Note over GCS: Headers: Authorization, Content-Type: application/octet-stream
            GCS-->>Gateway: Upload Status
            deactivate GCS
        else POST Method (Signed URL Upload)
            Gateway->>Gateway: Generate Signed URL (utils.ts: generateSignedURL)
            Gateway->>GCS: POST Signed URL (Upload File Stream)
            activate GCS
            Note over GCS: Headers: Content-Type, x-goog-content-length-range
            GCS-->>Gateway: Upload Status
            deactivate GCS
        end
        Gateway->>Gateway: Transform request body per model/endpoint (chatComplete, embed)
        Gateway-->>Client: File Upload Response (id, bytes, status)
    end
    deactivate Gateway

    Client->>Gateway: POST /v1/fine_tuning/jobs (Create Finetune - Fireworks)
    activate Gateway
    Note over Gateway: Headers: x-portkey-fireworks-account-id
    Gateway->>Gateway: Transform Request Body (createFinetune.ts)
    Gateway->>FireworksAI_API: POST /supervisedFineTuningJobs
    activate FireworksAI_API
    FireworksAI_API-->>Gateway: Finetune Job Response
    deactivate FireworksAI_API
    Gateway->>Gateway: Transform Response (utils.ts: fireworkFinetuneToOpenAIFinetune)
    Gateway-->>Client: Finetune Job Response
    deactivate Gateway

    Client->>Gateway: POST /v1/fine_tuning/jobs/{jobId}/cancel (Cancel Finetune - Fireworks)
    activate Gateway
    Gateway->>FireworksAI_API: DELETE /supervisedFineTuningJobs/{jobId}
    activate FireworksAI_API
    FireworksAI_API-->>Gateway: Cancellation Response
    deactivate FireworksAI_API
    Gateway->>Gateway: Transform Response (cancelFinetune.ts)
    Gateway-->>Client: Cancellation Response
    deactivate Gateway

    Client->>Gateway: GET /v1/fine_tuning/jobs (List Finetunes - Fireworks)
    activate Gateway
    Note over Gateway: Query Params: limit, after
    Gateway->>FireworksAI_API: GET /supervisedFineTuningJobs?pageToken={page}&pageSize={size}
    activate FireworksAI_API
    FireworksAI_API-->>Gateway: List Finetunes Response
    deactivate FireworksAI_API
    Gateway->>Gateway: Transform Response (listFinetune.ts)
    Gateway-->>Client: List Finetunes Response
    deactivate Gateway

    Client->>Gateway: GET /v1/batches/{batchId}/output (Get Batch Output - Vertex AI)
    activate Gateway
    Note over Gateway: Headers: x-portkey-provider-model, x-portkey-provider-batch-endpoint
    Gateway->>GoogleVertexAI_API: GET /v1/projects/{project}/locations/{location}/batchPredictionJobs/{batchId}
    activate GoogleVertexAI_API
    GoogleVertexAI_API-->>Gateway: Batch Job Details (output URI, model)
    deactivate GoogleVertexAI_API
    Gateway->>GCS: GET /bucket/output_file (predictions.jsonl or 000000000000.jsonl)
    activate GCS
    GCS-->>Gateway: Batch Output Stream
    deactivate GCS
    Gateway->>Gateway: Process stream line by line (streamHandlerUtils.ts, getBatchOutput.ts)
    Gateway->>Gateway: Transform each row based on model/endpoint (chatComplete, embed)
    Gateway-->>Client: Streamed Batch Output
    deactivate Gateway

matter-code-review · 2025-05-26T11:18:37Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

src/globals.ts

src/handlers/handlerUtils.ts

src/providers/fireworks-ai/api.ts

src/providers/google-vertex-ai/utils.ts

…ch-improvements

matter-code-review · 2025-06-19T09:10:07Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

matter-code-review · 2025-06-19T10:09:55Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

matter-code-review · 2025-07-02T07:17:44Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the latest commit in the PR
/matter review-full: Generate AI Reviews for the complete PR
/matter release-notes: Generate AI release-notes for the PR
/matter : Chat with your PR with Matter AI Agent
/matter remember : Generate AI memories for the PR
/matter explain: Get an explanation of the PR
/matter help: Show the list of available commands and documentation
Need help? Join our Discord server: https://discord.gg/fJU5DvanU3

chore: batch & fine-tune improvements

122c0da

chore: revert package updates

3f74159

b4s36t4 added 2 commits May 26, 2025 16:12

Merge branch 'main' of github.com:Portkey-AI/gateway into feat/ft-bat…

a9527a6

…ch-improvements

chore: add support for normal file upload for inference purposes

c67d71d

narengogi reviewed May 27, 2025

View reviewed changes

b4s36t4 added 2 commits June 19, 2025 14:38

Merge branch 'main' of github.com:Portkey-AI/gateway into feat/ft-bat…

26e1d77

…ch-improvements

chore: remove unused import

d872db0

chore: add comment

d525f53

Merge branch 'main' into feat/ft-batch-improvements

a9dd0f9

narengogi approved these changes Jun 26, 2025

View reviewed changes

Merge branch 'main' into feat/ft-batch-improvements

b8ecbc6

chore: batch & fine-tune improvements #1089

Are you sure you want to change the base?

chore: batch & fine-tune improvements #1089

Uh oh!

Conversation

b4s36t4 commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Uh oh!

matter-code-review bot commented May 9, 2025

PR Review Skipped

Command List

Uh oh!

matter-code-review bot commented May 9, 2025

Summary By MatterAI

🔄 What Changed

🔍 Impact of the Change

📁 Total Files Changed

🧪 Test Added

🔒Security Vulnerabilities

Motivation

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Sequence Diagram

Uh oh!

matter-code-review bot commented May 9, 2025

PR Review Skipped

Command List

Uh oh!

matter-code-review bot commented May 9, 2025

Summary By MatterAI

🔄 What Changed

🔍 Impact of the Change

📁 Total Files Changed

🧪 Test Added

🔒Security Vulnerabilities

Motivation

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Sequence Diagram

Uh oh!

matter-code-review bot commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

# Summary By MatterAI

🔄 What Changed

🔍 Impact of the Change

📁 Total Files Changed

🧪 Test Added

🔒Security Vulnerabilities

Motivation

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Quality Recommendations

Sequence Diagram

Uh oh!

matter-code-review bot commented May 26, 2025

PR Review Skipped

Command List

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

b4s36t4 commented May 9, 2025 •

edited

Loading

matter-code-review bot commented May 26, 2025 •

edited

Loading