Add design doc for lookup remote table in Fluid #9068

Yancey1989 · 2018-03-14T09:30:40Z

typhoonzero · 2018-03-14T12:58:53Z

doc/fluid/design/dist_train/large_model.md

@@ -0,0 +1,44 @@
+# Design Doc: Large Model


Need a more meaningful name, like "remote large parameter prefetching"?

Done, maybe Prefetching Parameter From Parameter Server sounds good?

typhoonzero · 2018-03-14T13:10:13Z

doc/fluid/design/dist_train/large_model.md

@@ -0,0 +1,44 @@
+# Design Doc: Large Model
+
+## Abstract


Need to tell about the background, why we need this feature.

typhoonzero · 2018-03-14T13:10:39Z

doc/fluid/design/dist_train/large_model.md

+
+### Split Large Parameter
+
+<img src="src/split_parameter.png" width="400" />


Seems the picture's number's are wrong.

abhinavarora

Hi @Yancey1989 The document required some polishing in the grammar. I have directly pushed a commit to this PR to refine and polish the document.

abhinavarora · 2018-03-14T18:04:16Z

doc/fluid/design/dist_train/prefetch_parameter.md

+
+## Abstract
+
+We propose an approach to prefetch parameter from Parameter


pre-fetch

It should be:
We propose an approach to pre-fetch the parameters from a Parameter Server while distributed training so that Fluid is able to train a model with a large number of parameters that cannot be stored in one trainer's memory.

helinwang · 2018-03-14T21:31:52Z

Thanks for the design doc! Curious how much time can prefetch save in our use case?

From my understand this is how prefetch save time:

pre-fetch-OP -> some-OP-A -> some-OP-B -> OP-that-use-the-prefetched-value

prefetch could overlap fetch with the time "some-OP-A" and "some-OP-B" take.

However, is it true that in our case the "OP-that-use-the-prefetched-value" is in the very beginning of the ProgramDesc, thus even if we insert the "pre-fetch-OP" at the beginning of the ProgramDesc, it would not save much time? E.g.,

pre-fetch-OP -> few-OPs-that-does-not-take-long-to-run -> OP-that-use-the-prefetched-value

typhoonzero · 2018-03-15T01:43:50Z

@helinwang Well, I think prefetch does not intend to save time but to make training with very large parameter distributed on many pservers(because the feature space is very large). This feature could definitely slow down the training but very useful for some CTR models.

Yancey1989 · 2018-03-15T01:44:37Z

Thanks @abhinavarora, that looks better😆😆

helinwang · 2018-03-15T02:05:15Z

@typhoonzero I see, that makes sense, thanks for the reply!

jacquesqiao · 2018-03-15T02:30:30Z

Please review this design #9075 first, for it has something to do with Abacus migration. We should consider being compatible with both sides.

Yancey1989 · 2018-03-15T10:52:30Z

I changed the title to Lookup Remote Table because the specific application scenario description would be clearer. And this feature is mainly applied to embedding layer.

…odel_design_doc

typhoonzero · 2018-07-04T10:56:03Z

doc/fluid/design/dist_train/distributed_lookup_table_design.md

+For another design, we can implement a distributed sparse table in Fluid,
+and don't need to maintain an external storage component while training.
+
+Prior to reading this design, it would be useful for the reader to make themselves


You may need to read ... before going on.

typhoonzero · 2018-07-04T10:59:03Z

doc/fluid/design/dist_train/distributed_lookup_table_design.md

+![fluid lookup remote table](./src/fluid_lookup_remote_table.png)
+
+Partition a large table into multiple pserver instances
+1. `DistributeTranspiler` would split the table partitioned into some small


I think we only use mod for now, but never mind, it's a design.

The mod is used for finding the right pserver according to the input Id, and we also need to initialize the shape of the table block on each PServer by split.

typhoonzero

LGTM, we may need to remove old design sections later.

jacquesqiao

I will update the outdated part of this design.

…gn_doc Add design doc for lookup remote table in Fluid

Yancey1989 added 2 commits March 14, 2018 17:27

Add large model design doc

cb7891a

update png

af8c728

Yancey1989 requested review from jacquesqiao and typhoonzero March 14, 2018 09:30

add note message

b382747

typhoonzero reviewed Mar 14, 2018

View reviewed changes

Yancey1989 added 2 commits March 14, 2018 23:54

update by comment

f839e91

update by comment

302136e

Yancey1989 requested a review from helinwang March 14, 2018 15:57

Refine the prefetch parameter document

5948fd2

abhinavarora reviewed Mar 14, 2018

View reviewed changes

typhoonzero mentioned this pull request Mar 15, 2018

Add distributed lookup table design #9075

Merged

Yancey1989 added this to To do in distributed lookup table Mar 15, 2018

Yancey1989 changed the title ~~Large model design doc~~ Large parameter distributed training Mar 15, 2018

update by lookup remote table

1b4db80

Yancey1989 changed the title ~~Large parameter distributed training~~ Add design doc for lookup remote table Mar 15, 2018

Yancey1989 added 2 commits March 15, 2018 18:53

modify the picture size

282aa96

update figure

cc1e949

This was referenced Mar 19, 2018

Add table design doc #9210

Closed

Support Distribute Lookup Table #9211

Closed

jacquesqiao moved this from To do to In progress in distributed lookup table Mar 19, 2018

Yancey1989 added 2 commits July 4, 2018 11:24

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into large_m…

1c5616b

…odel_design_doc

merge fluid lookup table into abacus one

285e7ac

Yancey1989 changed the title ~~Add design doc for lookup remote table~~ Add design doc for lookup remote table in Fluid Jul 4, 2018

typhoonzero reviewed Jul 4, 2018

View reviewed changes

polish sentense

e343afb

typhoonzero approved these changes Jul 4, 2018

View reviewed changes

jacquesqiao approved these changes Jul 4, 2018

View reviewed changes

Yancey1989 merged commit 845618e into PaddlePaddle:develop Jul 5, 2018

distributed lookup table automation moved this from In progress to Done Jul 5, 2018

Yancey1989 deleted the large_model_design_doc branch July 5, 2018 01:14

kuke pushed a commit to kuke/Paddle that referenced this pull request Aug 25, 2018

Merge pull request PaddlePaddle#9068 from Yancey1989/large_model_desi…

a67a80d

…gn_doc Add design doc for lookup remote table in Fluid

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add design doc for lookup remote table in Fluid #9068

Add design doc for lookup remote table in Fluid #9068

Yancey1989 commented Mar 14, 2018

typhoonzero Mar 14, 2018

Yancey1989 Mar 14, 2018

typhoonzero Mar 14, 2018

Yancey1989 Mar 14, 2018

typhoonzero Mar 14, 2018

abhinavarora left a comment

abhinavarora Mar 14, 2018

abhinavarora Mar 14, 2018

helinwang commented Mar 14, 2018 •

edited

typhoonzero commented Mar 15, 2018

Yancey1989 commented Mar 15, 2018

helinwang commented Mar 15, 2018

jacquesqiao commented Mar 15, 2018

Yancey1989 commented Mar 15, 2018

typhoonzero Jul 4, 2018

typhoonzero Jul 4, 2018

Yancey1989 Jul 4, 2018

typhoonzero left a comment

jacquesqiao left a comment


		### Split Large Parameter

		<img src="src/split_parameter.png" width="400" />


		## Abstract

		We propose an approach to prefetch parameter from Parameter

Add design doc for lookup remote table in Fluid #9068

Add design doc for lookup remote table in Fluid #9068

Conversation

Yancey1989 commented Mar 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abhinavarora left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helinwang commented Mar 14, 2018 • edited

typhoonzero commented Mar 15, 2018

Yancey1989 commented Mar 15, 2018

helinwang commented Mar 15, 2018

jacquesqiao commented Mar 15, 2018

Yancey1989 commented Mar 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

typhoonzero left a comment

Choose a reason for hiding this comment

jacquesqiao left a comment

Choose a reason for hiding this comment

helinwang commented Mar 14, 2018 •

edited