-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update new reader for resnet, mobilenet; test=develop #4685
Conversation
dygraph/mobilenet/train.py
Outdated
test_reader = imagenet_reader.val(settings=args) | ||
train_data_loader.set_sample_list_generator(train_reader, place) | ||
test_data_loader.set_sample_list_generator(test_reader, place) | ||
#train_reader = imagenet_reader.train(settings=args) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
没用的注释要不要清理掉?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
places=place, | ||
shuffle=True, | ||
drop_last=True, | ||
num_workers=10) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这个用多进程一般情况下docker应该直接会崩,CE会不会被卡?或者要不要加点注释提醒下用户?虽然文档和报错里也都有,但QA可能会因为这个问题来找
例如:
Note:
- Docker
/dev/shm
size: DataLoader use mmap(Memory Map) to implement shared memory. You need to confirm if your shared memory is big enough, otherwise a bus error will occur. The docker default/dev/shm
size is 64M, you can set it by docker--shm-size
parameter, for example,docker run --name [Name of container] -it --shm-size=2g -v $PWD:/paddle <imagename> /bin/bash
, you can see docker's shared memory size bydf -h
- You need to make sure your
/dev/shm
size is larger than the following calculation results, otherwise, it will report anBus error
because of insufficient shared memory
DataLoader Num * queue capacity * 1 batch data size
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
好 我改下
use_double_buffer=True, | ||
iterable=True, | ||
return_list=True, | ||
use_multiprocess=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
同上,use_multiprocess
打开,默认创建的docker直接会崩
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
好 我改下
先merge了,还有需要修改的地方,下个PR再改吧。 |
更新mobile net reader,使用更高效的DataLoader,
更新resnet reader,支持image net的数据格式