Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Align ViT hyperparam #226

Merged
merged 8 commits into from
Apr 2, 2022
Merged

Align ViT hyperparam #226

merged 8 commits into from
Apr 2, 2022

Conversation

rentainhe
Copy link
Contributor

TODO

仔细校对了一下ViT的训练settings，对齐官方settings，并且重构了一下ViT的config，利用了一些可复用的参数，减少出错概率

rentainhe
rentainhe commented Mar 27, 2022
View reviewed changes
libai/models/vision_transformer.py
@@ -131,6 +131,9 @@ def _init_weights(self, m):
nn.init.constant_(m.bias, 0)
nn.init.constant_(m.weight, 1.0)

def no_weight_decay(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个部分我感觉可以不加也行

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以删了

@rentainhe rentainhe mentioned this pull request Mar 27, 2022
Closed
8 tasks
@rentainhe
Copy link
Contributor Author

rentainhe commented Mar 28, 2022 •
edited
Loading

这个PR等正确性验证完后再合并~，目前在CIFAR100下没有问题，在ImageNet下需要跑完整训练（已经在训练）

@rentainhe

This comment was marked as duplicate.

Sign in to view
rentainhe
rentainhe commented Apr 2, 2022
View reviewed changes
docs/source/tutorials/get_started/quick_run.md
@@ -93,7 +93,7 @@ dataloader.test[0].dataset.root = "/path/to/imagenet"
```
- To train `vit_tiny_patch16_224` model on ImageNet on a single node with 8 GPUs for 300 epochs, run:
```bash
bash tools/train.sh configs/vit_imagenet.py 8
bash tools/train.sh tools/train_net.py configs/vit_imagenet.py 8
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix了文档部分的问题

configs/vit_imagenet.py
@@ -1,5 +1,5 @@
from libai.config import LazyCall
from .common.models.vit.vit_tiny_patch16_224 import model
from .common.models.vit.vit_base_patch16_224 import model
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

替换默认模型为base，避免了tiny模型tensor并行无法执行的尴尬情况

@lixiang007666
Copy link
Contributor

目前存在一个文档问题: Markdown的表格语法显示错误, 可能需要 @lixiang007666 帮忙康康

1648867717(1)

这个是sphinx-rtd-theme这个主题不支持md语法的问题，我去查了下doc，他们支持的表格方式怪复杂，而且不支持写在markdown里。
所以目前可以用下面这种解决方式，样式可调：

'<table border="2" align="center">
    <tr>
        <td align="center">Model</td>
        <td align="center">Pretrain</td>
        <td align="center">Resolution</td>
        <td align="center">Acc@1</td>
        <td align="center">Acc@5</td>
        <td align="center">Download</td>
    </tr>
    <tr>
        <td align="center">ViT-Tiny</td>
        <td align="center">ImageNet-1K</td>
        <td align="center">224x224</td>
        <td align="center">72.7</td>
        <td align="center">91.0</td>
        <td align="center"><a href="https://oneflow-public.oss-cn-beijing.aliyuncs.com/model_zoo/LiBai/ImageNet/vit_tiny_patch16_224/config.yaml">Config</a> ｜ <a href="https://oneflow-public.oss-cn-beijing.aliyuncs.com/model_zoo/LiBai/ImageNet/vit_tiny_patch16_224/model_best.zip">Checkpoint</a></td>
    </tr>
</table>'

后面我再仔细看下～

@rentainhe
Copy link
Contributor Author

目前存在一个文档问题: Markdown的表格语法显示错误, 可能需要 @lixiang007666 帮忙康康

1648867717(1)

已经通过另一种html语法解决:
image

@lixiang007666 lixiang007666 mentioned this pull request Apr 2, 2022
Closed
@rentainhe
Copy link
Contributor Author

目前存在一个文档问题: Markdown的表格语法显示错误, 可能需要 @lixiang007666 帮忙康康
1648867717(1)

这个是sphinx-rtd-theme这个主题不支持md语法的问题，我去查了下doc，他们支持的表格方式怪复杂，而且不支持写在markdown里。 所以目前可以用下面这种解决方式，样式可调：

'<table border="2" align="center">
    <tr>
        <td align="center">Model</td>
        <td align="center">Pretrain</td>
        <td align="center">Resolution</td>
        <td align="center">Acc@1</td>
        <td align="center">Acc@5</td>
        <td align="center">Download</td>
    </tr>
    <tr>
        <td align="center">ViT-Tiny</td>
        <td align="center">ImageNet-1K</td>
        <td align="center">224x224</td>
        <td align="center">72.7</td>
        <td align="center">91.0</td>
        <td align="center"><a href="https://oneflow-public.oss-cn-beijing.aliyuncs.com/model_zoo/LiBai/ImageNet/vit_tiny_patch16_224/config.yaml">Config</a> ｜ <a href="https://oneflow-public.oss-cn-beijing.aliyuncs.com/model_zoo/LiBai/ImageNet/vit_tiny_patch16_224/model_best.zip">Checkpoint</a></td>
    </tr>
</table>'

后面我再仔细看下～

可以的，已经修改为这种形式了~

@rentainhe rentainhe requested review from oneflow-ci-bot and removed request for oneflow-ci-bot April 2, 2022 03:21
@rentainhe rentainhe merged commit b3b70e9 into main Apr 2, 2022
@rentainhe rentainhe deleted the refine_vit_model branch April 2, 2022 03:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CPFLAME CPFLAME CPFLAME approved these changes

@L1aoXingyu L1aoXingyu L1aoXingyu approved these changes

@oneflow-ci-bot oneflow-ci-bot Awaiting requested review from oneflow-ci-bot

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rentainhe @lixiang007666 @CPFLAME @L1aoXingyu