Markdown JSON Mode #246

Anmol6 · 2023-12-01T17:21:02Z

Addresses Issue #235

Summary by CodeRabbit

New Features
- Introduced a new Circle class for enhanced image processing.
- Added a new mode MD_JSON for extended functionality in handling responses.
Enhancements
- Improved the draw_circle function to include image encoding to base64.
Bug Fixes
- Adjusted the response logic for better accuracy and performance.
Tests
- Modified existing assertions and added new tests to ensure reliability of the new features.

coderabbitai · 2023-12-01T17:21:07Z

Warning

Rate Limit Exceeded

@Anmol6 has exceeded the limit for the number of files or commits that can be reviewed per hour. Please wait 12 minutes and 54 seconds before requesting another review.

How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.
Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.
Please see our FAQ for further information.

Commits

Files that changed from the base of the PR and between 336d3b3 and b46b536.

Walkthrough

The codebase has been updated to include new imports, a Circle class, and modifications to function signatures and logic. A new mode MD_JSON has been added to an enum and handled in various functions, impacting how messages are processed and responses are created. Tests have been updated to reflect these changes, with new assertions and a test function added to ensure proper functionality.

Changes

File Path	Change Summary
`examples/vision/run.py`	Added imports, `Circle` class, modified `draw_circle` function, and updated `response` assignment.
`instructor/function_calls.py` `instructor/patch.py`	Imported `re` module, added `MD_JSON` mode, updated method signatures, and logic for message handling.
`tests/openai/test_modes.py`	Modified assertions in `test_json_mode` and added new `test_markdown_json_mode` function.

🐇✨
To code's rhythm, we hop and tweak,
With every line, perfection we seek.
New modes to juggle, tests to pass,
In binary burrows, we raise our glass! 🥂
🎉🐰

Tips

Chat with CodeRabbit Bot (`@coderabbitai`)

If you reply to a review comment from CodeRabbit, the bot will automatically respond.
To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
Note: Review comments are made on code diffs or files, not on the PR overview.
Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
The JSON schema for the configuration file is available here.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

jxnl

we should motify the messages so tthat

the last message has a ```json
the stop token is ```
add the modes in our evals (lets do the user details eval)

instructor/patch.py

instructor/function_calls.py

coderabbitai

Review Status

Actionable comments generated: 5

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between d59ed26 and 56ae384.

Files selected for processing (4)

examples/vision/run.py (1 hunks)
instructor/function_calls.py (3 hunks)
instructor/patch.py (2 hunks)
tests/openai/test_modes.py (1 hunks)

Additional comments: 6

examples/vision/run.py (2)

28-28: The radius for the circles is hardcoded to 100, which removes the randomization aspect. Verify if this is intentional or if it should be reverted to the commented-out randomization logic.

43-61: The client.chat.completions.create method includes a response_model parameter set to Circle. Verify that this parameter is supported by the custom instructor.patch function and that it correctly handles the Circle model.

instructor/function_calls.py (2)

17-17: The addition of MD_JSON to the Mode enum is consistent with the summary.

20-20: The summary mentions a modification to the from_response method of the openai_function class to handle the new MD_JSON mode, but this change is not reflected in the provided code. Please verify if the changes were made in the correct class or if they are missing.

instructor/patch.py (1)

79-105: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [79-117]

The implementation of the Mode.MD_JSON in the handle_response_model function appears to be correct and aligns with the summary provided. The addition of a new message block and stop token for Mode.MD_JSON is implemented as expected. The logic for inserting a system message and appending the message to the first system message if it exists is also handled properly. The code is well-structured and follows good practices for error handling by raising a ValueError for an invalid mode.

tests/openai/test_modes.py (1)

66-83: The addition of test_markdown_json_mode is a good practice to ensure the new feature works as expected.

coderabbitai · 2023-12-01T19:30:11Z

tests/openai/test_modes.py

@@ -58,7 +58,28 @@ def test_json_mode():
        ],
    )
    user = UserExtract.from_response(response, mode=Mode.JSON)
-    assert user.name.lower() == "jason"
+    assert user.name.lower()


The assertion assert user.name.lower() will pass even if user.name is an empty string. It should explicitly check for a non-empty string.

- assert user.name.lower() + assert user.name.lower() != ''

Committable suggestion

❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change

assert user.name.lower()

assert user.name.lower() != ''

coderabbitai · 2023-12-01T19:30:12Z

tests/openai/test_modes.py

+        ],
+    )
+    user = UserExtract.from_response(response, mode=Mode.MD_JSON)
+    assert user.name.lower()


The assertion assert user.name.lower() will pass even if user.name is an empty string. It should explicitly check for a non-empty string.

- assert user.name.lower() + assert user.name.lower() != ''

Committable suggestion

❗ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change

assert user.name.lower()

assert user.name.lower() != ''

coderabbitai · 2023-12-01T19:30:12Z

instructor/function_calls.py

@@ -1,4 +1,5 @@
 import json
+import re 


The re module is imported but not used in the provided code. If it's not used elsewhere, consider removing the import to avoid unnecessary dependencies.

coderabbitai · 2023-12-01T19:30:12Z

instructor/function_calls.py

                context=validation_context,
                strict=strict,
            )
+        elif mode == Mode.MD_JSON:
+            return cls.model_validate_json(
+                message.content,
+                context=validation_context,
+                strict=strict,
+            )


The from_response method in the OpenAISchema class has been updated to handle the MD_JSON mode. However, the logic for MD_JSON is identical to the JSON mode. If the handling for MD_JSON is supposed to be different, this needs to be implemented, otherwise, the MD_JSON case can be combined with the JSON case to avoid code duplication.

coderabbitai · 2023-12-01T19:30:12Z

examples/vision/run.py

+response = client.chat.completions.create(
+    model="gpt-4-vision-preview",
+    max_tokens=1800,
+    response_model=Circle,
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": 'find the circle'},
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": f"data:image/jpeg;base64,{base64_image}"
+                    },
+                },
+            ],
+        }
+    ],
+)
+
+print(f"Found circle with center at x: {response.x}, y: {response.y} and color: {response.color}")


There is no error handling for the API call or file operations. Consider adding try-except blocks to handle potential exceptions and provide a graceful error message to the user.

jxnl · 2023-12-01T19:36:38Z

added some tests and found that reask does not work cause it'll apologize ands hit.

pydantic_core._pydantic_core.ValidationError: 1 validation error for UserDetails
E             Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='Apologies for the confus...ven JSON schema is:\n\n', input_type=str]
E               For further information visit https://errors.pydantic.dev/2.5/v/json_invalid

coderabbitai

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 56ae384 and de5dd17.

Files selected for processing (1)

instructor/patch.py (4 hunks)

Additional comments: 4

instructor/patch.py (4)

79-105: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [79-117]

The implementation of Mode.MD_JSON in the handle_response_model function appears to be consistent with the PR objective and the summary. The addition of a new message block and stop token for Mode.MD_JSON is correctly implemented.

185-197: The handling for Mode.MD_JSON in the retry_async function is consistent with the PR objective and the summary. The addition of the ```json message block upon retry is correctly implemented.

229-241: The handling for Mode.MD_JSON in the retry_sync function is consistent with the PR objective and the summary. The addition of the ```json message block upon retry is correctly implemented.

94-94: The implementation includes the stop token and the ```json message block as suggested in the existing comments. This supports streaming natively and avoids the need for JSON parsing.

jxnl · 2023-12-02T00:03:02Z

maybe in patching add a note that like if you want to try the vision model it only supports md_mode, and add a

!!! warning "Experimental"

     This is not recommended, and may not be supported in the future, this is left so support vision models

jxnl

looks good, can we update docs and add a warning?

coderabbitai

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between de5dd17 and eeb16a4.

Files selected for processing (1)

docs/concepts/patching.md (1 hunks)

Additional comments: 3

docs/concepts/patching.md (3)

40-56: The summary suggests that "Markdown JSON Mode" replaces "JSON Mode," but the hunk shows that both modes are documented, with "Markdown JSON Mode" added as a new section. Please verify if "JSON Mode" is indeed replaced or if both modes are meant to coexist.

45-47: The warning about the experimental nature of "Markdown JSON Mode" is clear and important for users to understand the potential risks of using this feature.

54-54: The code for initializing the client with Mode.MD_JSON is correct and aligns with the PR objective to introduce Markdown JSON handling.

first commit

a5d9e7d

s

abd6bae

jxnl requested changes Dec 1, 2023

View reviewed changes

instructor/patch.py Show resolved Hide resolved

instructor/function_calls.py Outdated Show resolved Hide resolved

jxnl and others added 5 commits December 1, 2023 12:55

Merge branch 'main' into add_vision_md_json

f3fe7a7

Merge branch 'main' into add_vision_md_json

20c80ec

fix

f9a930c

add test

9a261d8

s

56ae384

Anmol6 marked this pull request as ready for review December 1, 2023 19:26

Anmol6 requested a review from jxnl December 1, 2023 19:28

coderabbitai bot reviewed Dec 1, 2023

View reviewed changes

fix retry

de5dd17

coderabbitai bot reviewed Dec 1, 2023

View reviewed changes

Anmol6 changed the title ~~Markdown JSON for vision~~ Markdown JSON Mode Dec 1, 2023

Merge branch 'main' into add_vision_md_json

d35803a

jxnl requested changes Dec 2, 2023

View reviewed changes

update docs

eeb16a4

Anmol6 requested a review from jxnl December 2, 2023 00:26

coderabbitai bot reviewed Dec 2, 2023

View reviewed changes

Anmol6 added 3 commits December 1, 2023 16:27

format

336d3b3

Merge branch 'main' into add_vision_md_json

f037b04

s

b46b536

jxnl approved these changes Dec 2, 2023

View reviewed changes

jxnl merged commit ae59ed4 into jxnl:main Dec 2, 2023
4 checks passed

Anmol6 deleted the add_vision_md_json branch December 4, 2023 02:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Markdown JSON Mode #246

Markdown JSON Mode #246

Anmol6 commented Dec 1, 2023 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Dec 1, 2023 •

edited

Loading

Rate Limit Exceeded

Chat with CodeRabbit Bot (`@coderabbitai`)

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

jxnl left a comment

coderabbitai bot left a comment

coderabbitai bot Dec 1, 2023

coderabbitai bot Dec 1, 2023

coderabbitai bot Dec 1, 2023

coderabbitai bot Dec 1, 2023

coderabbitai bot Dec 1, 2023

jxnl commented Dec 1, 2023

coderabbitai bot left a comment

jxnl commented Dec 2, 2023

jxnl left a comment

coderabbitai bot left a comment

Markdown JSON Mode #246

Markdown JSON Mode #246

Conversation

Anmol6 commented Dec 1, 2023 • edited by coderabbitai bot Loading

Summary by CodeRabbit

coderabbitai bot commented Dec 1, 2023 • edited Loading

Rate Limit Exceeded

Walkthrough

Changes

Chat with CodeRabbit Bot (@coderabbitai)

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

jxnl left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot Dec 1, 2023

Choose a reason for hiding this comment

coderabbitai bot Dec 1, 2023

Choose a reason for hiding this comment

coderabbitai bot Dec 1, 2023

Choose a reason for hiding this comment

coderabbitai bot Dec 1, 2023

Choose a reason for hiding this comment

coderabbitai bot Dec 1, 2023

Choose a reason for hiding this comment

jxnl commented Dec 1, 2023

coderabbitai bot left a comment

Choose a reason for hiding this comment

jxnl commented Dec 2, 2023

jxnl left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

Anmol6 commented Dec 1, 2023 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Dec 1, 2023 •

edited

Loading

Chat with CodeRabbit Bot (`@coderabbitai`)

CodeRabbit Configration File (`.coderabbit.yaml`)