add .json
to SUPPORTED_EXTENSIONS
#1114
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This PR allows us to create a finetuning dataset from
.json
files.What Happened:
I was trying to finetune
llama2-7b-chat
on the OpenHermes-2.5 dataset but got an error because the dataset is a.json
file, not a.jsonl
file.Looking into it, I discovered that when
is_safe=True
, we check that the dataset files all have extensions inSUPPORTED_EXTENSIONS
(source). Currently,and because the OpenHermes-2.5 dataset consists of a
.json
file, which is not inSUPPORTED_EXTENSIONS
, we get an error.My Solution:
So I added
.json
to the SUPPORTED_EXTENSIONS and now I can finetune on this. This PR is really pretty simple. But I tested this on some runs to make sure everything works.Testing:
My runs (below) show that when you currently finetune on a
.json
files it works whenis_safe=False
(run 2) but fails whenis_safe=True
(run 1). With my changes in theeitan-patch-json
branch, you can finetune on a.json
file whenis_safe=True
(run 3).Runs:
llama2-7b-chat-open-hermes-ft-DKoAb6
onmain
branch -- I successfully finetuned on the.json
in OpenHermes-2.5 whenis_safe=False
.llama2-7b-chat-open-hermes-ft-CY6yNJ
onmain
branch -- I got an error finetuning on the.json
file in OpenHermes-2.5 whenis_safe=True
.llama2-7b-chat-open-hermes-ft-5xoXNv
oneitan-patch-json
branch -- I successfully finetuned on the.json
in OpenHermes-2.5 whenis_safe=True
by running on my brancheitan-patch-json
.Note: I stopped runs 1. and 3. after training for 10+ batches so the error you see in the logs is from me stopping the run early, not from preparing the dataset. To view the logs of run 2. do
mcli logs llama2-7b-chat-open-hermes-ft-CY6yNJ --resumption 0
. The error from this run looks like: