Make hpctoolkitv4 reader sparse #544
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lithomas1
added
status: ready for review
ocnkr
reviewed
Apr 22, 2024
| @@ -144,9 +147,10 @@ def from_hpctoolkit(dirname): | |||
| from .readers.hpctoolkit_v4_reader import HPCToolkitV4Reader | |||
|
|
|||
| if "experiment.xml" in os.listdir(dirname): | |||
| # TODO: Make old hpctoolkit outputs sparse? | |||
There was a problem hiding this comment.
I don't think we should make the old hpctoolkit outputs sparse. We can discuss this with Abhinav.
| value=0, | ||
| ) | ||
|
|
||
| if not self.sparse_format: |
There was a problem hiding this comment.
We can move this if statement to line 1609 because we don't need to use not_visited_nodes in the sparse format.
hatchet/tests/hpctoolkit.py
Outdated
| @@ -76,6 +80,11 @@ def test_graphframe(data_dir, calc_pi_hpct_db): | |||
| elif col in ("name", "type", "file", "module", "node"): | |||
| assert gf.dataframe[col].dtype == object | |||
|
|
|||
| # In case of sparse format check to make sure we are not inserting dummy values | |||
| # into the dataframe | |||
| if sparse_format: | |||
There was a problem hiding this comment.
We know how many rows for each node we should have in the dense format = number of ranks * number of threads. To test the sparse format, maybe we can check if there are some nodes with fewer number of rows? What do you think?
There was a problem hiding this comment.
Good idea, you actually caught an issue in my test (I was using old < v4 hpctoolkit data :D)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I made the hpctoolkit reader have the option to output in sparse format, and gated the option behind a keyword.
When we get the rest of the methods working, we should flip the default value to True from False.
Then, before the release we should delete the keyword.