Repository navigation
Request for technical feedback on an independent language model (CatAI / Purr 1.0) #1931
Unanswered
bananakitssu
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hello Qwen team and community!
I'm an independent developer working on CatAI, an experimental decoder-only language model. Its current model generation is called Purr 1.0.
Repository: https://github.com/bananakitssu/catai
I've been building the model and its training pipeline independently, and I'm looking for technical feedback from people with experience developing and training language models.
Current architecture and training setup
CatAI currently uses:
The recommended
smallpreset has approximately 6.95 million parameters, with a pretraining budget of up to 200 million tokens.The training pipeline also includes supervised fine-tuning using OpenAssistant conversation data (OASST1 and OASST2), along with additional curated conversational datasets.
The problem
Despite the improvements to the architecture, tokenizer, dataset preparation, and training pipeline, CatAI still produces incoherent responses.
For example, when asked what HTTP 404 means, an earlier checkpoint responded:
This is the main issue I'm trying to solve. I want the model to generate coherent, relevant, and natural responses rather than merely producing text that resembles language.
What I'd appreciate guidance on
I'm not asking the Qwen team to adopt or integrate CatAI. I'm simply hoping to learn from the experience of people who have worked on language-model development.
Any technical feedback, recommended resources, or pointers to relevant research would be greatly appreciated.
Thank you for your time!
All reactions