feat(integrations): add usage metrics and table logging to OpenAI autologger #5521
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
May 10, 2023 11:44
# Conflicts: # wandb/integration/openai/openai.py # wandb/integration/openai/resolver.py
parambharat
commented
May 10, 2023
parambharat
commented
May 10, 2023
parambharat
commented
May 10, 2023
parambharat
commented
May 10, 2023
dmitryduev
reviewed
May 10, 2023
…ic autolog monkey patch to accommodate requests that come from args/kwargs and adjust response to allow for a dictionary of wandb allowable media types or metrics for various packages or APIs. This commit refactors the autologging classes PatchLLMAPI and AutologLLMAPI to PatchAPI and AutologAPI, respectively. Additionally, the argument resolver protocol RequestResponseResolver is changed to ArgumentResponseResolver. The request parameter is changed to args and kwargs, and the response parameter is changed to a dictionary of wandb media objects or metrics. This makes the code more concise and easier to understand.
… clarity and readability of the autologging module.
dmitryduev
requested review from
a team
and removed request for
raubitsj and
kptkin
…ample_method and SampleClass - Use sample_method and SampleClass for testing - Test enabling and disabling autologging API - Test patching and logging for sample_method and SampleClass.sample_class_method - Test argument and response resolver functionality
added 2 commits
May 31, 2023 12:58
kptkin
reviewed
Jun 1, 2023
tests/functional_tests/t0_main/llm/t4_llm_openai_chat_completion.py
Outdated
Show resolved
Hide resolved
kptkin
reviewed
Jun 1, 2023
dmitryduev
changed the title
dmitryduev
approved these changes
Jun 2, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
🤖 Generated by Copilot at 68189f9
This pull request adds usage metrics and stats table logging for OpenAI requests and responses in
wandb/integration/openai/resolver.py. It also updates and adds yea tests for the OpenAI integration scripts intests/functional_tests/t0_main/openai/, and renames some of the scripts to make them consistent.Sample test run:
https://wandb.ai/dimaduev/openai_logging/runs/s9ko6mbs?workspace=user-dimaduev
Testing
How was this PR tested?
Checklist
🤖 Generated by Copilot at 3657255