[ROCm] AMDGPU compiler fixes #41641
[ROCm] AMDGPU compiler fixes #41641
Conversation
Deleting temporary files after compilation
| g_hsacoCache.request_count++; | ||
| if (hit) g_hsacoCache.hit_count++; | ||
| if (!(g_hsacoCache.request_count % 50)) | ||
| VLOG(0) << "HSACO cache: " << g_hsacoCache.request_count << " requests, " |
There was a problem hiding this comment.
Can you bump this to VLOG(1)?
| @@ -584,18 +640,29 @@ StatusOr<std::vector<uint8>> EmitModuleToHsaco( | |||
| std::string tempdir_name = tempdir_vector.front(); | |||
| VLOG(1) << "Compile-time artifacts located at: " << tempdir_name; | |||
|
|
|||
| bool keep_tempfiles = false; | |||
| TF_CHECK_OK(tensorflow::ReadBoolFromEnvVar("TF_ROCM_XLA_TEMPFILES", | |||
There was a problem hiding this comment.
Would TF_ROCM_KEEP_XLA_TEMPFILES be clearer?
| @@ -584,18 +640,29 @@ StatusOr<std::vector<uint8>> EmitModuleToHsaco( | |||
| std::string tempdir_name = tempdir_vector.front(); | |||
There was a problem hiding this comment.
Would it be possible (and better?) to pick a new sub-directory each time, instead of different file names?
There was a problem hiding this comment.
Not sure what's the benefit in that. More library calls, and all the files are going to be deleted anyway. Also, there's a theoretical possibility that we won't be able to delete the subdirectory if it is still open in some subprocess when we get to the end of the function.
In any event, that is an additional feature and shouldn't go into this PR, I think.
| std::string ir_str; | ||
| llvm::raw_string_ostream stream(ir_str); | ||
| stream << *module; | ||
| std::string str = stream.str(); |
There was a problem hiding this comment.
You can use ir_str, no need to copy it into str.
| std::string str = stream.str(); | ||
| // Delete the first two lines, since they usually vary even when the rest of | ||
| // the code is the same (but verify that they are what we expect). | ||
| if (str.size() >= 13 && str.substr(0, 13) == "; ModuleID = ") { |
There was a problem hiding this comment.
Would it make sense to use llvm string processing helpers here, or even llvm::Regex?
There was a problem hiding this comment.
The code does the job as written. Why bring in nonstandard APIs and add more complexity?
| if (dump_lls) { | ||
| static int hsaco_count = 0; | ||
| char name[256]; | ||
| sprintf(name, "/tmp/%d.ll", hsaco_count); |
There was a problem hiding this comment.
I don't think we should use C libraries here. Can you please modernize this block?
|
All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the ℹ️ Googlers: Go here for more info. |
ekuznetsov139
force-pushed
the
google-upstream-hsaco-cache
branch
from
499cdbd
to
a190fee
Compare
|
CLAs look good, thanks! ℹ️ Googlers: Go here for more info. |
rthadur
removed
awaiting review
google-ml-butler
bot
added
kokoro:force-run
gbaned
added
ready to pull
This PR:
Ensures that AMDGPU compiler's temporary files are all different from instance to instance (important for multi-GPU training, e.g. with Horovod) and that they are deleted after compilation
Adds a HSACO cache, to speed up the compilation process when compilation of multiple identical IR's is requested