Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

colab example train failed with no log in https://app.lamini.ai/train #24

Closed
yangcheng opened this issue Sep 27, 2023 · 5 comments
Closed

Comments

@yangcheng
Copy link

yangcheng commented Sep 27, 2023

I try to follow the https://colab.research.google.com/drive/1QMeGzR9FnhNJJFmcHtm9RhFP3vrwIkFn?usp=sharing on readme

The train step

start=time.time()
finetune_model.train(enable_peft=True)
print(f"Time taken: {time.time()-start} seconds")

always failed with

Training job submitted! Check status of job 3459 here: https://app.lamini.ai/train
Job failed: {'job_id': 3459, 'status': 'FAILED', 'start_time': '2023-09-27T12:52:33.304092', 'model_name': None, 'custom_model_name': None, 'is_public': None}
Time taken: 35.00070023536682 seconds

I tried to specify different model names but it does not help. What make it harder is the log tab on https://app.lamini.ai/train is also empty.

Screenshot 2023-09-27 at 8 55 35 PM

how should I go from here? any suggestions are greatly appreciated!

@edamamez
Copy link
Contributor

Hello!! Checking this out now 👀

Thank you for bringing this to our attention!

@edamamez
Copy link
Contributor

Fixed!! Please try again 🙏 The logs should appear as well.

(We recently rotated some keys and missed a spot 😅 )

@yangcheng
Copy link
Author

Just tried again, now the job is in queued state, fingers crossed

@yangcheng
Copy link
Author

My last run finished successfully, with eval results and log in dashboard. Thanks for the quick turn around

@GZDXGeorge
Copy link

I had also met this kind of problem.Help.What I should do

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants