add auto_parallel pass

* feat(AutoParallel): support remove parallel_cast ops

* feat(AutoParallel): export enable_auto_parallel_prune_parallel_cast_ops

* format code

* feat(Conv): add grad computation cost

* fix ConvDataGrad computation cost

* update conv grad cost

* refine

* Try to speed up sbp collector.
However, throughput drop

* Shrink the parallel candidates for the proxy node

* Print out some information and then refine

* Store the sbp set for each consumer

* Update binary set intersection

* Remove impossible parallel candidates from sbp proxy

* Refine binary set

* Add a Clear() in binary set

* Filter out those proxy candidates containing two
sbps from the same unique group

* refine

* Check spells

* Clip useless edges

* feat(AutoParallel): mainstem algorithm add mutable_op_ctrl_edge

* use if instead std::max

* fix(AutoParallel): fix auto parallel dump uniform sbp bug

* refine source op judgement

* fix(AutoParallel): fix auto parallel dump uniform sbp bug

* feat(AutoParallel): add inferface for op to dump nd_sbp to op_conf

* refactor(AutoParallel): refactor DumpNdSbpSignatureForOpConfFn

* refactor(AP): refactor SbpEdge

* Rename variables

* Add const for some functions

Co-authored-by: Yipeng Li <jamesonli1313@gmail.com>

* Rename

* Code clean up

* Code clean up

* Code clean up and package up

* Rename

* Add const for some functions

* Code clean up

* Package up

* Code clean up and package up in SbpNode and SbpEdge

* Rename

* Rename

* Rename mainstem to trunk

* Typo, small bugs and rename

* Rename and of format

* Package up SbpCollector

* Add const for SbpGraph

* Add const for SbpNode

* Add const for SbpEdge

* Add const for SbpCollector

* Add const, rename, and package up for BinarySet

* Rename for BinarySet

* Rename for SbpCollector

* Rename for SbpCollector

* Rename for algorithm utils

* Fix a bug for an unused function AddEntries()

* Rename for BinarySet

* Rename for SbpConstructor

* Rename for BoxingCollector

* Add const for sbp utils

* Remove template for sbp signature

* Remove _H_ from cpp files

* Remove namespace specifier oneflow::

* Remove namespace specifier oneflow::

* Of format

* Move the inline functions to cpp files

* Can not add inline specifier?

* Update oneflow/core/auto_parallel/sbp_graph.h

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Of format

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Delete copy/move constructor/operator

* Move the deconstructor of SbpEdge to the cpp file

* Equal by address for Sbp data structor

* Replace sbp_sig_list_ with sbp_sig_obj_list_

* Check the output shape for operator in auto parallel

* Return infinity for different sbps while is_mutable

* Update oneflow/core/auto_parallel/sbp_constructor.cpp

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Update oneflow/core/operator/operator.cpp

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* with output -> check output

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Prune a line of parallel cast ops

* Avoid repeated pruning

* Code clean up

* Remove identity op

* Update oneflow/core/job_rewriter/auto_parallel.cpp

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

* Speed up after pruning identity

* Slight changes

* Of format

* Use const auto &

* Of format and rename

* Re-compute cost if steals sbp signatures

* doc(AutoParallel): add auto parallel document framework

* docs(AutoParallel): add document

* fix typo

* refine document

* refine documentation

* test(AutoParallel): test alexnet for auto_parallel

* test(AutoParallel): test model add auto_parallel config

* Fix the bug of missing sbp for uniform op

* Speed up

* Add the mising sbp for optional input UserSourceOpTickInput

* Remove the repeated all-B sbp signature

* Add sbp for undefined UserSourceOpTickInput

fix(AutoParallel): disabled ZeRO when enabled AutoParallel

…s.cpp

GetComputationCostFn -> GetComputationCost

Co-authored-by: Yinggang Wang <wyg19970408@gmail.com>

fix(AutoParallel): skip 1n1d sbp agreement check

* Use B for inplace op and remove the check for sbp
while truning the auto prallelism on

* Slight change

* Not using B as the constrain

* Address comments

…nto feat-auto_parallel

…ignore_user_sbp_config