-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
小模型自监督效果 #59
Comments
|
再请教一个问题,小模型在自监督训练的时候,会出现loss突然增大的情况,您有遇到过类似情况吗? |
印象中没有遇到。用fp16了吗?我猜测也有可能是batchsize或learning rate过大 |
没有用fp16,batchsize是1000左右,比默认的4096小,learning rate是您的代码里面计算得到的 |
可以开源下您resnet50的训练日志吗 |
您有对比过400或者800轮相比1600轮的效果吗 |
可见我们paper里的ablation部分,另外您或许可以调整--base_lr=1e-4,我们的默认值2e-4或许对您dataset来说过大 |
数据集同样使用的imagenet,只是模型是mobilenet级别的网络,学习率还建议调小吗 |
不是很确定,或许可以尝试调整一下变大变小;另外如果网络中有特殊算子,可能需要手动定义一下它的sparse形式,因为 https://github.com/keyu-tian/SparK/blob/main/pretrain/encoder.py#L39-L110 中只定义了conv2d,maxpooling,avgpooling,bn2d,syncbn,layernorm的sparse形式。例如如果网络中间有linear层也是需要定义一下sparse的(因为输入中有0,经过linear之后0+bias变成非0了,需要对output进行mask归0) |
@leoxxxxxD Hi, I have a small model, like Mobilenet. Have you finally concluded whether small models are suitable for training with self-supervised methods such as MAE? Thank you! |
我们尝试了小模型自监督的效果,结果不如有监督训练,gold-yolo这篇论文也是类似的结论,模型越小提升幅度越小,您对小模型用spark方法有什么看法吗?
The text was updated successfully, but these errors were encountered: