[autoparallel] add rotor C version #1658
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Merge ColossalAI
Daily merge
…ossalAI into feature/add_rotor_c_version
FrankLeeeee
reviewed
Sep 27, 2022
setup.py
Outdated
| @@ -191,6 +192,12 @@ def cuda_ext_helper(name, sources, extra_cuda_flags, extra_cxx_flags=[]): | |||
| extra_cxx_flags = ['-std=c++14', '-lcudart', '-lcublas', '-g', '-Wno-reorder', '-fopenmp', '-march=native'] | |||
| ext_modules.append(cuda_ext_helper('cpu_adam', ['cpu_adam.cpp'], extra_cuda_flags, extra_cxx_flags)) | |||
|
|
|||
| if build_auto_ckpt_ext: | |||
There was a problem hiding this comment.
There is no way to specify this parameter by the user.
FrankLeeeee
reviewed
Sep 27, 2022
setup.py
Outdated
| @@ -191,6 +192,12 @@ def cuda_ext_helper(name, sources, extra_cuda_flags, extra_cxx_flags=[]): | |||
| extra_cxx_flags = ['-std=c++14', '-lcudart', '-lcublas', '-g', '-Wno-reorder', '-fopenmp', '-march=native'] | |||
| ext_modules.append(cuda_ext_helper('cpu_adam', ['cpu_adam.cpp'], extra_cuda_flags, extra_cxx_flags)) | |||
|
|
|||
| if build_auto_ckpt_ext: | |||
| print(os.path.join(this_dir, "colossalai/fx/passes/algorithms/dynamic_programs.c")) | |||
There was a problem hiding this comment.
Remove this print statement.
FrankLeeeee
reviewed
Sep 27, 2022
setup.py
Outdated
| if build_auto_ckpt_ext: | ||
| print(os.path.join(this_dir, "colossalai/fx/passes/algorithms/dynamic_programs.c")) | ||
| ext_modules.append( | ||
| Extension("c_version_dp", |
There was a problem hiding this comment.
do not name this variable way, it is quite meaningless.
There was a problem hiding this comment.
I've fixed those problems, currently we will build the C extension the first time user run the rotor solver, only if C extension building fail will we use the python version of dynamic programming.
|
I will build a unit test to compare the solution of python and C version dynamic programs. |
|
I will convert this PR to draft until it is ready for review again. |
|
|
FrankLeeeee
reviewed
Sep 29, 2022
Comment on lines
9
to
27
| try: | ||
| from .dynamic_programs_C_version import persistent_compute_table | ||
| CVERSION = True | ||
| except ModuleNotFoundError: | ||
| import subprocess | ||
| import os | ||
| print("dynamic_programs_C_version hasn't been built! Building library...") | ||
| this_dir = os.path.dirname(os.path.abspath(__file__)) | ||
| result = subprocess.Popen(f'python {os.path.join(this_dir, "build_c_ext.py")} build_ext --build-lib={this_dir}', | ||
| stdout=subprocess.PIPE, | ||
| stderr=subprocess.PIPE, | ||
| shell=True) | ||
| if result.wait() == 0: | ||
| print("dynamic_programs_C_version has been built!") | ||
| from .dynamic_programs_C_version import persistent_compute_table | ||
| CVERSION = True | ||
| else: | ||
| print("dynamic_programs_C_version built failed! Using python version!") | ||
| CVERSION = False |
There was a problem hiding this comment.
This should be put inside the solver_rotor function.
FrankLeeeee
reviewed
Sep 29, 2022
| except ModuleNotFoundError: | ||
| import subprocess | ||
| import os | ||
| print("dynamic_programs_C_version hasn't been built! Building library...") |
There was a problem hiding this comment.
can replace print with colossalai logger.
…ossalAI into feature/add_rotor_c_version
super-dainiu
approved these changes
Oct 3, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's New?
In this PR I add C extension of rotor dynamic programming table calculation and remove repeated
MetaInfoPropin rotor solver, right now you should execute the following code before running solver rotorAs we might use rotor for simplified offload strategy, this modification is crucial as we might run solver over the same graph several time for auto capper heuristic, we don't need to run this MetaInfoProp over and over again. Actually with the new C version dynamic programming, the
MetaInfoPropitself consumes more time than simply run the dynamic programming.