Skip to content

[Jobs] append error trace to job driver logs - #40380

Merged
architkulkarni merged 5 commits into
ray-project:masterfrom
GeneDer:jobs-append-error-trace-to-logs
Oct 17, 2023
Merged

[Jobs] append error trace to job driver logs#40380
architkulkarni merged 5 commits into
ray-project:masterfrom
GeneDer:jobs-append-error-trace-to-logs

Conversation

@GeneDer

@GeneDer GeneDer commented Oct 16, 2023

Copy link
Copy Markdown
Member

Why are these changes needed?

Log traceback to job driver log.

Related issue number

Closes #35997

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@GeneDer
WIP: added logics to log message, still need to write unit tests
552faa1 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
Merge branch 'master' into jobs-append-error-trace-to-logs
b66a52e
@GeneDer
added tests to ensure the error message are logged to the job logs
bf7d8f3 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
linting
4f5b603 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer
fix tests
574954f 
Signed-off-by: Gene Su <e870252314@gmail.com>
@GeneDer

GeneDer commented Oct 16, 2023

Copy link
Copy Markdown
Member Author

Tested locally by running ray job submit --runtime-env-json='{"pip": ["requests==3333.26.0"]}' -- python script.py and seeing the job driver log file created correctly with the correct traceback

image

@GeneDer GeneDer added release-blocker P0 Issue that blocks the release ray 2.8 labels Oct 16, 2023
@GeneDer GeneDer changed the title Jobs append error trace to logs [Jobs] append error trace to job driver logs Oct 16, 2023
@GeneDer
GeneDer marked this pull request as ready for review October 17, 2023 04:22
architkulkarni
architkulkarni approved these changes Oct 17, 2023
View reviewed changes

@architkulkarni architkulkarni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice enhancement!

@GeneDer GeneDer added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Oct 17, 2023
@architkulkarni
architkulkarni merged commit bde327f into ray-project:master Oct 17, 2023
@GeneDer
GeneDer deleted the jobs-append-error-trace-to-logs branch October 17, 2023 16:20
@scottsun94

scottsun94 commented Oct 17, 2023

Copy link
Copy Markdown
Contributor

Great to see this! Nice work! @GeneDer
This will make the runtime debugging experience much easier!

@rickyyx rickyyx mentioned this pull request Oct 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@architkulkarni architkulkarni architkulkarni approved these changes

Assignees

No one assigned

Labels

release-blocker P0 Issue that blocks the release tests-ok The tagger certifies test failures are unrelated and assumes personal liability.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Jobs] Append error traceback to job driver log

3 participants

@GeneDer @scottsun94 @architkulkarni