2018 milestones

[reyoung]

Fluid supports multi-GPUs and cluster, and high usability

Deadline:

KPI:

1. Make fluid supports all models in PaddlePaddle/Book, PaddlePaddle/models. Complete the inference framework of Fluid on linux and mobile.
   • Make Baidu teams (Speech, NLP, Image, Abacus) use fluid to train and inference models.
2. The speed of training models in PaddlePaddle/book is not slower than TF in MultiGPUs and cluster.
3. The memory consumption of training models in PaddlePaddle/book is not larger than TF in MultiGPUs and a cluster.

Fluid distributed computing

Deadline:

KPI:

1. Make Fluid support EDL (Elastic Deep Learning). Make cluster training of Fluid can adapt to workload changes by provisioning and de-provisioning resources in an autonomic manner.
2. Support model parallelism as well as data parallelism.
3. Make Fluid support OpenMPI APIs to do distributed all-reduce.
4. Make Fluid support GPU direct when possible.

Compatible with ONNX

Deadline:

KPI:

1. Make ProgramDesc can be converted to ONNX model files.
2. Make ONNX can be converted into ProgramDesc, and make Fluid can train ONNX model.

Support CSP program model and imperative programming

Deadline:

KPI:

1. Users only use Python as a compiler frontend and produce the ProgramDesc, and an interpreter will execute the ProgramDesc.
2. The ProgramDesc includes IfElse operator and While, and supports auto diff.
3. Saving and loading model, printing metrics are all configured in ProgramDesc. Deeply integrate with VisualDL to give a GUI.
4. Support to configure CSP(coroutines, channel, select) in ProgramDesc. Use CSP to implement multi GPUs and cluster training.

[helinwang]

Should we specify a TensorFlow version (e.g., the latest release v1.6.0) for performance comparison? Otherwise if after we surpass TensorFlow's performance, they release a new well-optimized version, we would be in an awkward position.

Not saying that we should not aim at being better than the latest TensorFlow, my point is maybe we should focus on a fixed target first.

[helinwang]

Support model parallelism as well as data parallelism

Is there a need for model parallelism other than large embedding lookups. If so we may want to change to "Support large embedding lookups as well as data parallelism"

[helinwang]

Make Fluid support OpenMPI APIs to do distributed all-reduce.

Currently, we use the parameter server architecture (via send/recv operator) for parameter update, it's a completely different architecture with all-reduce. From my understanding their theoretical network throughput consumption and time consumption for each step are similar. We already support parameter server architecture, what is the reason that we need to support another approach with similar performance?

Support to configure CSP(coroutines, channel, select) in ProgramDesc. Use CSP to implement multi GPUs and cluster training.

If we use CSP for cluster training, it looks more like the parameter server architecture than the all-reduce architecture.

[wangkuiyi]

I agree with @helinwang that MPI AllReduce is NOT part of the milestones of PaddlePaddle. I am open to someone out of PaddlePaddle team to try that approach, but it makes sense only if they could do it when they run PaddlePaddle jobs in containers.

[wangkuiyi]

Thanks to @reyoung and @helinwang and others for this list.

I tried to summarize the milestones as the follows. @PaddlePaddle/paddle team please comment

note from @panyx0718 about eng time: 6 fulltime months could mean 3 people spend 2 months fulltime working on something

Version	ETA	Features	eng time
0.11.2	end-March	Support FP16	2+ fulltime months
0.11.3	mid-April	Improve the performance of the single-node-multi-GPU setting	5+ fulltime months
0.11.4	end-April	Improve the performance of the multi-node setting	5+ fulltime months
		Implement the distributed lookup table using memcached	?
0.12	end-April	ONNX-compatible and Fluid-to-ONNX converter	?
		Support the imperative programming paradigm	30+ fulltime months
0.12.1	end-May	Improve the Fluid API and make it stable	15+ fulltime months
0.13	end-June	Fluid on Kubernetes and EDL	?
0.13.1	end-July	Develop the transpiler for CUDA	?
0.13.2	mid-August	Develop the transpiler for ARM	?
0.13.3	end-August	Develop the transpiler for ROCm (depends on AMD team's work)	?
0.13.4	mid-September	Develop the transpiler for Intel Movidius	?
0.14	mid-October	Fluid Debugger	5 fulltime monts
1.0	end-October	Wrap up the above work and officially release the new set of tutorials, documentation, Websites	?
1.0.1	end-November	Fluid on EDL Kubernetes stable and performant for cloud service deployment	?

[varunarora]

This is a solid list, thanks to everyone who worked on it. (more comments coming soon)

[typhoonzero]

@helinwang @wangkuiyi The reason for supporting MPI all-reduce is that latest openmpi implement can support GPU direct if the hardware supports it. This is the fastest way to implement very high performance distributed GPU training. Anyway, we can still try the time-consuming way: implement GPU direct using CUDA libs directly. What do you think?

[PaddleCI]

The principle is that we need to make sure that the distributed training system is reasonably easy to use. Given that most AI systems would depend on not only PaddlePaddle, but application-specific third party software, e.g., OpenCV for vision, we’d prefer to run AI applications inside containers. Another invariant is that we don’t want to loss the capability of fault recovery. Given above two rules, I would like to see an application of MPI with PaddlePaddle Fluid. I think the capability of starting a distributed job using MPI is more urgent than the speed-up, because it seems that many teams inside Baidu are using MPI. On Mar 15, 2018, at 6:50 PM, 武毅 <notifications@github.com<mailto:notifications@github.com>> wrote: @helinwang<https://github.com/helinwang> @wangkuiyi<https://github.com/wangkuiyi> The reason for supporting MPI all-reduce is that latest openmpi implement can support GPU direct if the hardware supports it. This is the fastest way to implement very high performance distributed training. Anyway, we can still try the time-consuming way: implement GPU direct using CUDA libs directly. What do you think? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub<#9108 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AbwsoChN6ePfsLVvjT8Bau2lEFFv0nC2ks5texpggaJpZM4Srsy->.

[panyx0718]

@wangkuiyi Thanks for the milestone.

I suggest we add 2 more columns: 1. the number of full time engineers and 2. the number of months spent developing them. For example:
0.13.1 | end-July | Develop the transpiler for CUDA | fulltime N people | develop for M months

[wangkuiyi]

Good point! @panyx0718 Please feel free to add these columns.

[helinwang]

@typhoonzero @wangkuiyi @PaddleCI thanks for the comments! Good to know the MPI need from Baidu, as well as the GPU direct support from openmpi.

Given that we already have NCCL all-reduce, the development time for integrating openmpi (or NCCL2) maybe not that high, plus the additional benefit of already tuned communication and GPU direct support. That could save us a lot of effort. Fault recovery can be added by checkpointing, maybe fault tolerance on MPI can be added with some special peer-aware logic that creates a new MPI communicator when some node left or joined.