Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manual rocprof join breaks in ROCm 5.2.x #139

Closed
coleramos425 opened this issue Jun 8, 2023 · 1 comment
Closed

Manual rocprof join breaks in ROCm 5.2.x #139

coleramos425 opened this issue Jun 8, 2023 · 1 comment
Assignees
Labels
bug Something isn't working Profiling Related to the profiling done in Omniperf
Milestone

Comments

@coleramos425
Copy link
Collaborator

I noticed that profiling applications in ROCm 5.2.x causes issues. A peak at verbose debug logs shows
that we crash when checking for arch_vgpr and accum_vgpr (two counters added in ROCm 5.3).

It's not the concern I expressed in the original ticket (#117 (comment)), but it'll be an easy fix

ROCPRofiler: 167 contexts collected, output directory 
/tmp/rpl_data_230608_132620_2929822/input_results_230608_132620
File '/home/colramos/GitHub/omniperf-pub/workloads/mix_all/mi200/timestamps.csv' is generating
Successfully joined gpu in pmc_perf.csv
Successfully joined grd in pmc_perf.csv
Successfully joined wgr in pmc_perf.csv
Successfully joined lds in pmc_perf.csv
Successfully joined scr in pmc_perf.csv
Traceback (most recent call last):
  File "./src/omniperf", line 917, in <module>
    main()
  File "./src/omniperf", line 812, in main
    omniperf_profile(args, VER)
  File "./src/omniperf", line 698, in omniperf_profile
    join_prof(workload_dir, args.join_type, log, args.verbose)
  File "/home/colramos/GitHub/omniperf-pub/src/utils/perfagg.py", line 136, in join_prof
    if not test_df_column_equality(_df):
  File "/home/colramos/GitHub/omniperf-pub/src/utils/perfagg.py", line 92, in test_df_column_equality
    return df.eq(df.iloc[:, 0], axis=0).all(1).all()
  File "/home/colramos/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 961, in __getitem__
    return self._getitem_tuple(key)
  File "/home/colramos/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1458, in _getitem_tuple
    tup = self._validate_tuple_indexer(tup)
  File "/home/colramos/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 769, in _validate_tuple_indexer
    self._validate_key(k, i)
  File "/home/colramos/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1361, in _validate_key
    self._validate_integer(key, axis)
  File "/home/colramos/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1452, in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds

https://github.com/AMDResearch/omniperf/blob/a346db7646b0a935f4cac51d131b4a585f065c05/src/utils/perfagg.py#L123-L133

@coleramos425 coleramos425 added bug Something isn't working Profiling Related to the profiling done in Omniperf labels Jun 8, 2023
@coleramos425 coleramos425 self-assigned this Jun 8, 2023
coleramos425 added a commit that referenced this issue Jun 9, 2023
Signed-off-by: colramos-amd <colramos@amd.com>
@coleramos425 coleramos425 added this to the v.1.1.0 milestone Jun 12, 2023
@coleramos425
Copy link
Collaborator Author

Fixed.

coleramos425 added a commit that referenced this issue Aug 15, 2023
Signed-off-by: colramos-amd <colramos@amd.com>
feizheng10 pushed a commit to feizheng10/omniperf that referenced this issue Dec 6, 2023
Signed-off-by: colramos-amd <colramos@amd.com>
Signed-off-by: fei.zheng <fei.zheng@amd.com>
feizheng10 pushed a commit to feizheng10/omniperf that referenced this issue Dec 20, 2023
Signed-off-by: colramos-amd <colramos@amd.com>
Signed-off-by: fei.zheng <fei.zheng@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Profiling Related to the profiling done in Omniperf
Projects
None yet
Development

No branches or pull requests

1 participant