Distributed Compilation as an option to DaCe Program #1555
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Option to activate/deactivate Distributed Compilation.
This small PR is based on the following comment (DAPP/DaCe Mattermost channel):
I have an unexpected behaviour in DaCe distributed compilation.
Currently, if you have an MPI program, distributed compilation is the default behaviour (as seen in this file). I was expecting that after the loading of the compiled sdfg every rank would do symbol specialization.
Although, this is not the case, i.e. every rank uses the compiled sdfg from rank 0, which specializes its symbols with the values corresponding to rank 0. Therefore, the compiled sdfg loaded by all the other ranks use a wrong sdfg (symbols are not specialized with the values of the correct rank).
To validate this behaviour, I have de-activated the distributed compilation and set
dace.config.Config.set("cache", value="unique"). Indeed, this approach works without any issue.Is there a way to change this unexpected behaviour, i.e. to have by default the distributed compilation but every rank to perform symbol specialization.
To give a bit more context, I am generating an sdfg that uses closures heavily, i.e. all the gt4py fields are defined externally to the sdfg (could that be an issue)?