Add distributed lookup table design #9075
Conversation
jacquesqiao
changed the title
|
Thanks to @jacquesqiao for this design. After talking with Xu laoshi, I put our comments in jacquesqiao#4. Please review it. Thanks! |
Update distributed lookup table design doc
jacquesqiao
changed the title
|
Related: #9068 |
|
|
||
| <!-- | ||
| Note: please update the following URL when update this digraph. | ||
| <img src='https://g.gravizo.com/svg? |
There was a problem hiding this comment.
One awesome tool for pasting figures!
|
|
||
| - Pro: parameter servers do not retrieve W(x) from the storage | ||
| service, thus saves half network communication. | ||
| - Con: the storage service needs to be able to run the optimization |
There was a problem hiding this comment.
Does this means we actually have two kinds of servers when doing training:
- original parameter server
- storage server can run optimization
There was a problem hiding this comment.
yes, storage server can run optimization will only handle large-scale embedding table, other parameters still use the fluid optimization operators.
There was a problem hiding this comment.
Yes, if we go this way. But let us go the other way first.
| ### Storage Service Doesn't Optimize | ||
|
|
||
| In this design, we use highly-optimized distributed storage, e.g., | ||
| memcached, as the storage service, and we run the optimization |
There was a problem hiding this comment.
@wangkuiyi After discussing with @ldmod LiDong, We think that a standalone distributed KV service like Memcached maybe not a good solution. It's better that parameter and the corresponding optimization happened at the same place. For large scale model training and optimization, we need to use the asynchronous update, in this condition, we need to make sure every optimizer should update the latest parameter, even when the gradient is calculated by the old parameter. If we use a standalone KV service, we need to add a lock to the parameter when some optimizer is updating it. If the parameter and optimization op is at the same place, the solution will be very simple, for example, use one thread to do the optimization.
There was a problem hiding this comment.
But still, we can have two option, one is using the current optimization operators, but the parameter will be maintained by the pserver. The other one is using a distributed Storage Service which can do Optimization.
There was a problem hiding this comment.
Discussed with @Yancey1989 yesterday that we agree with this idea, mixing two servers (original pserver and embedding table pserver) is a perfect way to embed an external parameter server:
- Add design doc for lookup remote table in Fluid #9068 is trying to describe how to implement this opensource.
- embed Abucus parameter server for advanced industry model training
we can carry on this two methods at the same time, then both opensource users and industry users can have their choice.
There was a problem hiding this comment.
To using distributed Storage Service, PServer would also communicate with the storage, this would make the double traffic than using embedding table pserver. And I agree with @typhoonzero , we can carry on this two methods at the same time.
There was a problem hiding this comment.
Let us have a baseline solution as early as possible. @jacquesqiao @ldmod.
There was a problem hiding this comment.
@wangkuiyi I agree. We can implement a baseline solution without Memcached.
|
How to implement sparse regularization? |
|
|
||
| ## Conclusion | ||
|
|
||
| Let us do the "storage service does not optimize" solution first, as a |
There was a problem hiding this comment.
Sorry I don't understand why we need to use memcached given that our current parameter server implementation should be faster in key lookup.
memcached is a general purpose in-memory key value store, from its website description:
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.
We only need a special case in-memory key value store (few keys, large values). Currently our parameter server (recv operator in fluid) does exactly this. In terms of the key lookup speed, our parameter server should be faster or at least same speed comparing to memcached.
The design doc mentioned:
Let us do the "storage service does not optimize" solution first
Our parameter server already does this (and does optimization as well), why we have to make another implementation with memcached again, is it because of performance reasons?
There was a problem hiding this comment.
@helinwang I am not so clear about the table lookup implementation currently, for a large scale lookup table, the keys may be discrete in a very big range, so we need a key-value lookup module.
|
Will issue a detailed design in another PR. |
No description provided.