Feature/carter qol -- finetuning cancelation support and retry logic #23
Conversation
|
retry logic for data synthesis (sometimes the models dont emit valid json), and adding a signal handler to cancel finetuning when you exit the program |
|
Right now even if one datapoint has an issue (leads to unparseable JSON), the whole process in restarted. I think this is fine for now, but maybe in the future we want to have all the other datapoints that are still formatted correctly to still be used and just the bad datapoints ignored. |
|
while testing, I have been stuck doing generation for last 10 minutes because one datapoint has a parsing issue |
|
@abhigya-sodani, thats some terrible luck! its set to fail after two tries by default, did you hit that limit or ? |
|
I’m happy to add some more robust formatting tricks to this pr. Or are you comfortable with this being merged in as is ? |
|
Cntrl+c quitting was not working for me |
|
Ok that’s weird. What os, terminal, and shell are you using? |
|
Anaconda Prompt, windows 10 |
|
Ok I think interrupts/signals aren’t supported on windows. I’ll check out if there’s a simple equivalent. If it’s easy for you to whip up a handler for native windows , I’ll stack that on |
No description provided.