fix pserver sub-blocks #11585
fix pserver sub-blocks #11585
Conversation
|
Should we remove these sub blocks from blk_list in the RunAsyncLoop method of listen_and_serv_op, like what we do in the RunSyncLoop method? |
May not, RunAsyncLoop use |
| if not op.has_attr('sub_block'): | ||
| return | ||
| return -1 |
There was a problem hiding this comment.
Who uses the return value?
| PADDLE_ENFORCE_GE(num_blocks, 2, | ||
| "server program should have at least 2 blocks"); | ||
|
|
||
| std::vector<int> optimize_block_id_list; | ||
| for (int blkid = 1; blkid < num_blocks; ++blkid) { | ||
| if (std::find(prefetch_block_id_list.begin(), prefetch_block_id_list.end(), | ||
| blkid) == prefetch_block_id_list.end()) { | ||
| blkid) == prefetch_block_id_list.end() && |
There was a problem hiding this comment.
Is there a way to make it beautiful?
There was a problem hiding this comment.
Done, please follow the comment: #11585 (review)
| @@ -101,13 +101,16 @@ void ListenAndServOp::RunSyncLoop( | |||
| framework::Scope *recv_scope, | |||
| const std::vector<int> &prefetch_block_id_list) const { | |||
| size_t num_blocks = program->Size(); | |||
| auto skip_sub_blks = Attr<std::vector<int>>("skip_sub_blks"); | |||
There was a problem hiding this comment.
Record only the start block id and end block id for listen_and_serv op to run parallelly. Sub-blocks append to the end to make the program in a strict order.
There was a problem hiding this comment.
Another way is that we record the parallel block ids, the benefit is don't need insert block to the specified idx, just append block to pserver_program and record the optimize block id.
There was a problem hiding this comment.
Reuse that deprecated optimize_block attribute could be cool.
…_blocks fix pserver sub-blocks
Fixed #11429