Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update error reporting #597

Merged
merged 6 commits into from
Apr 22, 2024
Merged

update error reporting #597

merged 6 commits into from
Apr 22, 2024

Conversation

debermudez
Copy link
Contributor

@debermudez debermudez commented Apr 19, 2024

Attempt to improve the logging for genai-perf.

  • Adds a logging config
  • creates a logging file for warns and up
  • outputs logging statements to console as well
  • some boy scout clean up as well

Some example output
Using a modified profile_export.json that removed response timestamps from the first request entry.
Before:
genai-perf - ERROR - 'response_timestamps'

After:

Traceback (most recent call last):
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/main.py", line 147, in run
    data_parser = calculate_metrics(args, tokenizer)
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/main.py", line 89, in calculate_metrics
    return LLMProfileDataParser(
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/llm_metrics.py", line 475, in __init__
    super().__init__(filename)
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/llm_metrics.py", line 406, in __init__
    self._parse_profile_data(data)
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/llm_metrics.py", line 416, in _parse_profile_data
    metrics = self._parse_requests(requests)
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/llm_metrics.py", line 489, in _parse_requests
    res_timestamps = request["response_timestamps"]
KeyError: 'response_timestamps'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/main.py", line 157, in main
    run()
  File "/workspace/client/src/c++/perf_analyzer/genai-perf/genai_perf/main.py", line 151, in run
    raise GenAIPerfException(e)
genai_perf.exceptions.GenAIPerfException: 'response_timestamps'
2024-04-19 16:35 [ERROR] genai_perf.main:161 - 'response_timestamps'

The error message now contains time info, module where the error occurred, and line numbers.
The console will also display the traceback for easier debugging.

@rmccorm4
Copy link
Contributor

rmccorm4 commented Apr 19, 2024

Couple requests to make it easier to grok the cosmetic and functional changes:

  1. Can you add before and after outputs of stdout logging for the happy path to the description?

  2. Can you also show an example of output from an error? Does it only go to file, stdout, or both?

@debermudez
Copy link
Contributor Author

debermudez commented Apr 19, 2024

(edited by Ryan): Moved to thread: #597 (comment)

Copy link
Contributor

@dyastremsky dyastremsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank so much for making this change, Elias! This will speed up debugging during development significantly.

@debermudez debermudez merged commit 4174b92 into main Apr 22, 2024
3 checks passed
@debermudez debermudez deleted the dbermudez-update-error-reporting branch April 22, 2024 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants