-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
关于lora_b矩阵的更新问题 #30
Comments
可以看看uie_trainer里的正则化项? |
一个新的任务的参数变化计算方式:C=AB,C的列向量可由A的列向量线性表示,C的列向量张成的空间是A的列向量张成空间的子空间,所以我们希望在优化时能让新的lora参数A’和旧的参数A正交(内积=0->每个列向量都正交->正交),这样使得新的参数C'与旧的参数C也正交,使得两个任务的参数变化发生在不同的空间里 |
哦哦我明白了,我理解错了,万分感谢! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
作者您好,我想请教一下关于工作里面lora_b矩阵更新的问题。论文里只写了它也是由lora_a张成的子空间中的向量构成,但是在训练过程中是如何保证这一点并更新加权系数的呢?我在代码中只找到了这样的片段,没看明白具体是如何完成要求的
The text was updated successfully, but these errors were encountered: