Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

看起来参数是随机丢弃了 #10

Closed
ReactiveCJ opened this issue Nov 28, 2023 · 2 comments
Closed

看起来参数是随机丢弃了 #10

ReactiveCJ opened this issue Nov 28, 2023 · 2 comments

Comments

@ReactiveCJ
Copy link

看起来参数是随机丢弃了, 有没有对比实验,不丢弃参数,但是只scale参数了

@yule-BUAA
Copy link
Owner

这个实验我们没有尝试过,但模型的表现很大概率上会变差。

从模型的计算角度讲,不丢弃参数只rescale参数会导致模型的输出期望发生变化(相当于整体放大或者缩小了模型的输出),很有可能导致较差的结果。我们的rescale操作是为了保证在drop一部分参数的后,模型的输出期望大致不变。

@yule-BUAA
Copy link
Owner

我先关闭这个issue啦。

如果后续有问题可以随时重启这个issue。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants