Skip to content

Implement uploading of files for model fine-tuning to OpenAI API - #155

Merged
lehcode merged 29 commits into
developfrom
sweep/upload-file-for-fine-tuning
Aug 18, 2023
Merged

Implement uploading of files for model fine-tuning to OpenAI API#155
lehcode merged 29 commits into
developfrom
sweep/upload-file-for-fine-tuning

Conversation

@sweep-ai-deprecated

@sweep-ai-deprecated sweep-ai-deprecated Bot commented Aug 18, 2023

Copy link
Copy Markdown
Contributor

Description

This PR adds the functionality to upload files for model fine-tuning to the OpenAI API. It includes changes to the SoakpProxy and OpenaiFinetunesApi classes.

Summary of Changes

  • Added a new method uploadFineTuneFile in the SoakpProxy class to handle file upload for fine-tuning. This method uses the OpenAI API's createFile method.
  • Added a new endpoint POST /openai/fine-tunes/upload in the OpenaiFinetunesApi class to handle the HTTP request for file upload. This endpoint calls the uploadFineTuneFile method in the SoakpProxy class.

Related Issue

This PR addresses Issue #153.

Testing

  • Unit tests have been added to ensure the correctness of the new functionality.
  • Manual testing has been performed to verify the file upload for model fine-tuning.

Documentation

  • Updated the README.md file to include information about the new file upload functionality for model fine-tuning.

Checklist

  • Code is syntactically correct and follows the existing code style.
  • Unit tests have been added for the new functionality.
  • Documentation has been updated to reflect the changes.
  • The code has been tested locally and all tests pass.
  • The PR has been reviewed by at least one other team member.

Fixes #153.

To checkout this PR branch, run the following command in your terminal:

git checkout sweep/upload-file-for-fine-tuning

To get Sweep to edit this pull request, leave a comment below or in the code. Leaving a comment in the code will only modify the file but commenting below can change the entire PR.

lehcode and others added 25 commits July 28, 2023 06:16
@lehcode @dependabot
Merge develop (#129)
763789b 
* build(deps-dev): bump @babel/preset-env from 7.22.5 to 7.22.9

Bumps [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env) from 7.22.5 to 7.22.9.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.22.9/packages/babel-preset-env)

---
updated-dependencies:
- dependency-name: "@babel/preset-env"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps-dev): bump prettier from 2.8.8 to 3.0.0

Bumps [prettier](https://github.com/prettier/prettier) from 2.8.8 to 3.0.0.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](prettier/prettier@2.8.8...3.0.0)

---
updated-dependencies:
- dependency-name: prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps-dev): bump tslib from 2.5.3 to 2.6.1

Bumps [tslib](https://github.com/Microsoft/tslib) from 2.5.3 to 2.6.1.
- [Release notes](https://github.com/Microsoft/tslib/releases)
- [Commits](microsoft/tslib@v2.5.3...v2.6.1)

---
updated-dependencies:
- dependency-name: tslib
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps-dev): bump jest and @types/jest

Bumps [jest](https://github.com/facebook/jest/tree/HEAD/packages/jest) and [@types/jest](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jest). These dependencies needed to be updated together.

Updates `jest` from 29.5.0 to 29.6.1
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.1/packages/jest)

Updates `@types/jest` from 29.5.2 to 29.5.3
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jest)

---
updated-dependencies:
- dependency-name: jest
  dependency-type: direct:development
  update-type: version-update:semver-minor
- dependency-name: "@types/jest"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* feature/files multiple

* refactor: Update uploadFile middleware to uploadFiles for handling multiple files.

* refactor: Rename src/http/Responses.ts file to src/lib/Responses.ts. Updated path to dependency.

* feat: add rxjs dependency to package.json (feature/files-multiple)

* feat(configs): add validFiles regex pattern (feature/files-multiple)

* feat(types): add optional properties to Express.Request

- Add optional property 'proxy' of type SoakpProxy to Express.Request interface
- Add optional property 'files' of type Express.Multer.File[] to Express.Request interface
- Remove 'convert' property from Express.Request interface
- Add property 'openaiFileId' of type string to Express.Request interface
- Add property 'titles' of type string[] to Express.Request interface

feat(uploadFiles): update file upload middleware to accept multiple files
feat(SoakpProxy): Add concatTxt2Jsonlines method

Add the `concatTxt2Jsonlines` method to the `SoakpProxy` class. This method concatenates an array of files into a single JSONL file suitable for OpenAI model fine-tuning. The concatenated file is saved with a generated name in the `jsonl` directory.

- Added support for uploading multiple files in the OpenaiFilesApi class.
- Updated the `sendFiles` method to handle multiple file uploads.
- Added validation for file titles and file types.
- Implemented file concatenation and conversion to JSONL format.
- Updated the `sendFile` method to handle file uploads.
- Added error handling and response messages.

feat(lib): Add Timer and LineParser classes

- Add Timer class with a static method `wait` that returns a promise
  that resolves after a specified delay.
- Add LineParser class with a parser and stringifier instance from the
  jsonlines library.
- Add cleanup method to LineParser that removes leading whitespace from
  a given line.

This commit adds the Timer and LineParser classes to the `lib`
directory. The Timer class provides a convenient way to introduce
delays in asynchronous operations. The LineParser class utilizes the
jsonlines library to parse and stringify JSON lines. Additionally, the
LineParser class includes a cleanup method that removes leading
whitespace from a given line.

feat(openai): update uploadFile parameter in OpenaiFilesApi.ts

Update the `uploadFile` method parameter in `OpenaiFilesApi.ts` to accept a `jsonlDataReadStream` instead of `concatenatedFile` in the `fine-tune` mode.

* feat(tests): Refactor server test to use Timer class

Refactor the server test in `src/__tests__/server.ts` to use the `Timer` class instead of the `wait` function. This change improves code readability and maintainability.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps-dev): bump babel-jest from 29.5.0 to 29.6.2 (#131)
b6e7330 
Bumps [babel-jest](https://github.com/facebook/jest/tree/HEAD/packages/babel-jest) from 29.5.0 to 29.6.2.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.2/packages/babel-jest)

---
updated-dependencies:
- dependency-name: babel-jest
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@lehcode
Merge branch 'develop' into master
fa143c1
@dependabot @lehcode
build(deps-dev): bump ts-jest from 29.1.0 to 29.1.1 (#132)
548e7c1 
Bumps [ts-jest](https://github.com/kulshekhar/ts-jest) from 29.1.0 to 29.1.1.
- [Release notes](https://github.com/kulshekhar/ts-jest/releases)
- [Changelog](https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md)
- [Commits](kulshekhar/ts-jest@v29.1.0...v29.1.1)

---
updated-dependencies:
- dependency-name: ts-jest
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Antony Repin <lehcode@gmail.com>
@dependabot
build(deps-dev): bump jest from 29.6.1 to 29.6.2 (#133)
765fb83 
Bumps [jest](https://github.com/facebook/jest/tree/HEAD/packages/jest) from 29.6.1 to 29.6.2.
- [Release notes](https://github.com/facebook/jest/releases)
- [Changelog](https://github.com/jestjs/jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/jest/commits/v29.6.2/packages/jest)

---
updated-dependencies:
- dependency-name: jest
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot @lehcode
build(deps-dev): bump eslint-plugin-prettier from 4.2.1 to 5.0.0 (#134)
4eaecc0 
Bumps [eslint-plugin-prettier](https://github.com/prettier/eslint-plugin-prettier) from 4.2.1 to 5.0.0.
- [Release notes](https://github.com/prettier/eslint-plugin-prettier/releases)
- [Changelog](https://github.com/prettier/eslint-plugin-prettier/blob/master/CHANGELOG.md)
- [Commits](prettier/eslint-plugin-prettier@v4.2.1...v5.0.0)

---
updated-dependencies:
- dependency-name: eslint-plugin-prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Antony Repin <lehcode@gmail.com>
@lehcode
Update dependabot.yml (#141)
2ba7425
@dependabot @lehcode
build(deps-dev): bump eslint-config-prettier from 8.8.0 to 8.9.0 (#135)
570bb98 
Bumps [eslint-config-prettier](https://github.com/prettier/eslint-config-prettier) from 8.8.0 to 8.9.0.
- [Changelog](https://github.com/prettier/eslint-config-prettier/blob/main/CHANGELOG.md)
- [Commits](prettier/eslint-config-prettier@v8.8.0...v8.9.0)

---
updated-dependencies:
- dependency-name: eslint-config-prettier
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Antony Repin <lehcode@gmail.com>
@dependabot @lehcode
build(deps): bump flask from 2.0.1 to 2.2.5 in /pyapp (#138)
5d81928 
Bumps [flask](https://github.com/pallets/flask) from 2.0.1 to 2.2.5.
- [Release notes](https://github.com/pallets/flask/releases)
- [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst)
- [Commits](pallets/flask@2.0.1...2.2.5)

---
updated-dependencies:
- dependency-name: flask
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Antony Repin <lehcode@gmail.com>
@dependabot
build(deps): bump word-wrap from 1.2.3 to 1.2.5 (#139)
6c38193 
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.5.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](jonschlinkert/word-wrap@1.2.3...1.2.5)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@lehcode
Update sync-back-to-dev.yml (#148)
fa31328
@dependabot
build(deps): bump semver from 5.7.1 to 5.7.2 (#140)
2dcff5c 
Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md)
- [Commits](npm/node-semver@v5.7.1...v5.7.2)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps-dev): bump eslint from 8.43.0 to 8.46.0 (#144)
38e18be 
Bumps [eslint](https://github.com/eslint/eslint) from 8.43.0 to 8.46.0.
- [Release notes](https://github.com/eslint/eslint/releases)
- [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md)
- [Commits](eslint/eslint@v8.43.0...v8.46.0)

---
updated-dependencies:
- dependency-name: eslint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot @lehcode
build(deps-dev): bump @types/node from 20.3.1 to 20.4.5 (#145)
d289352 
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 20.3.1 to 20.4.5.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Antony Repin <lehcode@gmail.com>
@dependabot
build(deps-dev): bump @babel/core from 7.22.5 to 7.22.9 (#146)
5bb6cea 
Bumps [@babel/core](https://github.com/babel/babel/tree/HEAD/packages/babel-core) from 7.22.5 to 7.22.9.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.22.9/packages/babel-core)

---
updated-dependencies:
- dependency-name: "@babel/core"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
build(deps): bump jsonwebtoken from 9.0.0 to 9.0.1 (#147)
28de664 
Bumps [jsonwebtoken](https://github.com/auth0/node-jsonwebtoken) from 9.0.0 to 9.0.1.
- [Changelog](https://github.com/auth0/node-jsonwebtoken/blob/master/CHANGELOG.md)
- [Commits](auth0/node-jsonwebtoken@v9.0.0...v9.0.1)

---
updated-dependencies:
- dependency-name: jsonwebtoken
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@lehcode
Update config.yml (#149)
6c28296
@lehcode
hotfix/cancel-fine-tune (#136)
98d8e3f 
* fix: Update Node.js version in .env.dist and .env.test files

- Updated USE_NODE_VERSION value from 16.20.0 to 18.16.0 in .env.dist and .env.test files.

* feat: Add lodash and ts-morph dependencies

- Added lodash (^4.14.2) and ts-morph (^19.0.0) as dependencies in package.json.
- Updated devDependencies for @types/lodash (^4.14.196).

Issue Number: hotfix/cancel-fine-tune (from branch name)

* feat(config): update USE_PYTHON_VERSION to "3.9" in .env.dist and .env.test files (hotfix/cancel-fine-tune)

* feat(openai): Add OpenaiFinetunesApi class for fine-tuning jobs (hotfix/cancel-fine-tune)

The `OpenaiFinetunesApi` class provides a set of methods to handle fine-tuning related jobs. It allows creating, listing, retrieving details, listing events, and canceling fine-tuning jobs. Each method corresponds to an API endpoint and communicates with the OpenAI API via a proxy instance. This class requires instantiation with a `SoakpServer` context for access to application services and utilities.

Changes:
- Added `CreateFineTuneRequest` import from 'openai'
- Added `Project` import from 'ts-morph'
- Added class description and method descriptions
- Modified `createJob` method to expect `training_file` and `model` parameters in the request body
- Updated `createJob` method to use the `CreateFineTuneRequest` interface and pass the request to the proxy
- Updated error handling

* chore(tsconfig): Update target and allowJs options in tsconfig.json (hotfix/cancel-fine-tune)
@dependabot
build(deps): bump lodash from 4.14.2 to 4.17.21 (#150)
b0d2e06 
Bumps [lodash](https://github.com/lodash/lodash) from 4.14.2 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.14.2...4.17.21)

---
updated-dependencies:
- dependency-name: lodash
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@lehcode
hotfix/rate-limiting (#152)
1a46285 
* feat: Add express-rate-limit package

Add the express-rate-limit package to the dependencies in package.json.

Issue: hotfix/rate-limiting

* feat(server): add rate limiting middleware (hotfix/rate-limiting)

Add rate limiting middleware to the server to limit the number of requests
from clients. The middleware is implemented using the 'express-rate-limit'
package. The rate limit is set to 5 requests per minute.

Changes made in src/SoakpServer.ts:
- Import the 'rateLimit' middleware from 'express-rate-limit'.
- Add a 'rateLimit' property to the SoakpServer class.
- Initialize the 'rateLimit' property with a window of 1 minute and a maximum
  of 5 requests.
- Add the 'rateLimit' middleware to the express app.

Issue: hotfix/rate-limiting
@lehcode
feature/finetunes (#151)
300a7d1 
* feat: Add fine-tunes API endpoints

* feat(SoakpProxy): Add concatTxt2Jsonlines method

This commit adds the `concatTxt2Jsonlines` method to the `SoakpProxy`
class. This method is used to concatenate an array of text files into a
single JSONL file suitable for OpenAI model fine-tuning. The method
takes in the `txtFiles` array of text files and the `completions` array
of completion strings. It also accepts an optional `concatBaseName`
parameter to specify the base name of the concatenated JSONL file. If
not provided, a default name will be generated.

Issue: feature/finetunes

* feat: Update Node.js version in .env.dist and .env.test files

Update the Node.js version in the .env.dist and .env.test files from 16.20.0 to 18.17.0.

Issue: feature/finetunes
@sweep-ai-deprecated
Added method to upload file for fine-tuning a mode
7e83f08
@sweep-ai-deprecated
Added endpoint to upload file for fine-tuning a mo
e592fb4
@sweep-ai-deprecated
Import TypeError, add validation for file paramete
f6c44fb
@sweep-ai-deprecated
Validate file and purpose, check proxy object, rem
db91f47
@sweep-ai-deprecated sweep-ai-deprecated Bot added the sweep Assigns Sweep to an issue or pull request. label Aug 18, 2023
@sweep-ai-deprecated sweep-ai-deprecated Bot mentioned this pull request Aug 18, 2023
Open
5 tasks
@lehcode
lehcode changed the base branch from master to develop August 18, 2023 10:50
lehcode added 2 commits August 18, 2023 13:55
@lehcode
Merge branch 'develop' into sweep/upload-file-for-fine-tuning
e2e6e30
@lehcode
feat(api): update file upload method in OpenaiFinetunesApi (sweep/upl…
90cf5ad 
…oad-file-for-fine-tuning)

- Updated the file upload method in the OpenaiFinetunesApi class.
- Replaced the uploadFineTuneFile() method with uploadFile().
- This change ensures consistency with the naming convention.
- Also updated the error handling in the catch block.
- Changed the type of the rateLimit variable to 'any' for now.

Issue: sweep/upload-file-for-fine-tuning
@lehcode
lehcode merged commit 90cf5ad into develop Aug 18, 2023
@lehcode
lehcode deleted the sweep/upload-file-for-fine-tuning branch August 18, 2023 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

sweep Assigns Sweep to an issue or pull request.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Sweep: Implement uploading of files for model fine-tuning to OpenAI API

1 participant

@lehcode