Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add batch-level processing pipeline with xmap_reader for acceleration. #231

Open
wants to merge 1 commit into
base: dev-static
Choose a base branch
from

Conversation

xinghai-sun
Copy link
Contributor

Resolve 2nd part of #230

DS2 的 _padding_batch 函数以 pipeline方式优化(此前仅instance preprocess 以 pipeline并行化,但尚未在batch 层面做)

@lispc
Copy link

lispc commented Sep 6, 2017

虽然我没有实测证据,但是经验上略担心用线程而不是进程可能速度overhead更大。
以前用的一个Tensorflow上层库使用 多进程 + zeromq传递数据,做到了直接磁盘读jpg(而不是recordio格式)来训练,还是挺惊艳的。

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants