ERROR in ready_any_format validator: file does not appear to be in valid JSON Format

[rmajidy]

Describe the bug

When I try to use the prepare data tool for fine tuning I always get the following error that i'm not using a valid JSON file. I have even used a valid JSON validator which comes back as valid. I have a feeling this has to do with my environment more than anything.

from my understanding this tool accepts any data format, even JSON to convert to JSONL

I'm new to python so please excuse my lack of experience with the language.

I even installed it with just pip install openai and still had no luck. along with this I tried using python with pyenv but still had no luck.

And again i'm new to programming so this is one of my first ever big reports, sorry if it's not in the right format if you will

To Reproduce

Latest version of python3 was installed using home brew with
brew install python
pip3 install openai

export OPENAI_API_KEY="<OPENAI_API_KEY>"
openai tools fine_tunes.prepare_data -f <LOCAL_FILE>

pip v23.0.1

Code snippets

user@user-MBP training_models % openai tools fine_tunes.prepare_data -f prompt_completion_pairs_manual.json
Analyzing...


ERROR in read_any_format validator: Your file `prompt_completion_pairs_manual.json` does not appear to be in valid JSON format. Please ensure your file is formatted as a valid JSON file.

Aborting...%

OS

macOS Ventura v13.0.1

Python version

Python 3.11.2

Library version

openai-0.27.1

[rmajidy]

Describe the bug

When I try to use the prepare data tool for fine tuning I always get the following error that i'm not using a valid JSON file. I have even used a valid JSON validator which comes back as valid. I have a feeling this has to do with my environment more than anything.

from my understanding this tool accepts any data format, even JSON to convert to JSONL

I'm new to python so please excuse my lack of experience with the language.

I even installed it with just pip install openai and still had no luck. along with this I tried using python with pyenv but still had no luck.

And again i'm new to programming so this is one of my first ever big reports, sorry if it's not in the right format if you will

To Reproduce

Latest version of python3 was installed using home brew with brew install python pip3 install openai

export OPENAI_API_KEY="<OPENAI_API_KEY>" openai tools fine_tunes.prepare_data -f <LOCAL_FILE>

pip v23.0.1

Code snippets

user@user-MBP training_models % openai tools fine_tunes.prepare_data -f prompt_completion_pairs_manual.json
Analyzing...


ERROR in read_any_format validator: Your file `prompt_completion_pairs_manual.json` does not appear to be in valid JSON format. Please ensure your file is formatted as a valid JSON file.

Aborting...%

OS

macOS Ventura v13.0.1

Python version

Python 3.11.2

Library version

openai-0.27.1

Interesting, i saw on a deep thread somewhere where they downgraded their openai-CLI version to v0.25.x and it worked. Uninstalled it and installed it with pip3 specifying that version and it worked!

Anyone else have this issue? is this a complication with my python3 / pip3 versions? would love to know

[leo-paz]

I'm seeing this issue as well

EDIT:

Downgrading to 0.25 worked for me

[samedovzaur]

Thanks Leo it worked for me too! for Downgrading: pip install openai==0.25.0

[rmajidy]

awesome! I'm curious as to why no one has addressed this issue yet

[rando76]

Even after downgrading to version 0.25.0 this problem hasn't gone away for me and now the error message is slightly different

ERROR in read_any_format validator: File C:/Users/ryana/Documents/Python/TrainingDataV1.json does not exist.

it says "does not exist" rather than "does not appear to be in valid JSON format. Please ensure your file is formatted as a valid JSON file."
My file clearly exists but then again I'm a newbie at code and maybe there is something fundamental and obvious that I am missing

any help would be appreciated

[fishygeek91]

Downgrading worked for me thank you

[hallacy]

Hi @rmajidy,

Any chance you have a sample file I could use to replicate this issue?

[JAEHOON-1993]

Thanks Leo it worked for me too! for Downgrading: pip install openai==0.25.0

Thank you!

[hallacy]

I'm guessing the issue was from a regression fixed in #389. Since that's merged, closing this