Parse Any HTML-esh Style Tags #2046
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2046 +/- ##
===========================================
+ Coverage 36.73% 48.92% +12.18%
===========================================
Files 69 69
Lines 7134 7190 +56
Branches 1557 1700 +143
===========================================
+ Hits 2621 3518 +897
+ Misses 4280 3371 -909
- Partials 233 301 +68
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
sharsha315
pushed a commit
to sharsha315/autogen
that referenced
this pull request
Mar 29, 2024
* tried implementing my own regex * improves tests * finally works * removes prints * fixed test * adds start and end * delete unused imports * refactored to use new tool * significantly improved algo * tag content -> tag attr * fix tests + adds new field * return full match * return remove start and end * update docstrings * update docstrings * update docstrings --------- Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Apr 3, 2024
* DOC FIX - Formatted Docstrings for the retrieve_user_proxy_agent.py and Added first single line for the class RetrieveUserProxyAgent. * DOC FIX - Formatted Docstrings for theinitiate_chats functiion of ChatResult class in autogen/agentchat/chat.py * Add vision capability (#2025) * Add vision capability * Configurate: description_prompt * Print warning instead of raising issues for type * Skip vision capability test if dependencies not installed * Append "vision" to agent's system message when enabled VisionCapability * GPT-4V notebook update with ConversableAgent * Clean GPT-4V notebook * Add vision capability test to workflow * Lint import * Update system message for vision capability * Add a `custom_caption_func` to VisionCapability * Add custom function example for vision capability * Skip test Vision capability custom func * GPT-4V notebook metadata to website * Remove redundant files * The custom caption function takes more inputs now * Add a more complex example of custom caption func * Remove trailing space --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Native tool call support for Mistral AI API and topic notebook. (#2135) * Support for Mistral AI API and topic notebook. * formatting * formatting * New conversational chess notebook using nested chats and tool use (#2137) * add chess notebook * update * update * Update notebook with figure * Add example link * redirect * Clean up example format * address gagan's comments * update references * fix links * add webarena in samples (#2114) * add webarena in samples/tools * Update samples/tools/webarena/README.md Co-authored-by: gagb <gagb@users.noreply.github.com> * Update samples/tools/webarena/README.md Co-authored-by: gagb <gagb@users.noreply.github.com> * Update samples/tools/webarena/README.md Co-authored-by: gagb <gagb@users.noreply.github.com> * update installation instructions * black formatting * Update README.md --------- Co-authored-by: gagb <gagb@users.noreply.github.com> Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * context to kwargs (#2064) * context to kwargs * add tag * add test * text to kwargs --------- Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /website (#2131) Bumps [webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) from 5.3.3 to 5.3.4. - [Release notes](https://github.com/webpack/webpack-dev-middleware/releases) - [Changelog](https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md) - [Commits](webpack/webpack-dev-middleware@v5.3.3...v5.3.4) --- updated-dependencies: - dependency-name: webpack-dev-middleware dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * Parse Any HTML-esh Style Tags (#2046) * tried implementing my own regex * improves tests * finally works * removes prints * fixed test * adds start and end * delete unused imports * refactored to use new tool * significantly improved algo * tag content -> tag attr * fix tests + adds new field * return full match * return remove start and end * update docstrings * update docstrings * update docstrings --------- Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Integrate AgentOptimizer (#1767) * draft agent optimizer * refactor * remove * change openai config interface * notebook * update blog * add test * clean up * redir * update * update interface * change model name * move to contrib * Update autogen/agentchat/contrib/agent_optimizer.py Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> --------- Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”> Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Jieyu Zhang <jieyuz2@cs.washington.edu> Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> * Introducing IOStream protocol and adding support for websockets (#1551) * Introducing IOStream * bug fixing * polishing * refactoring * refactoring * refactoring * wip: async tests * websockets added * wip * merge with main * notebook added * FastAPI example added * wip * merge * getter/setter to iostream added * website/blog/2024-03-03-AutoGen-Update/img/dalle_gpt4v.png: convert to Git LFS * website/blog/2024-03-03-AutoGen-Update/img/gaia.png: convert to Git LFS * website/blog/2024-03-03-AutoGen-Update/img/teach.png: convert to Git LFS * add SSL support * wip * wip * exception handling added to on_connect() * refactoring: default iostream is being set in a context manager * test fix * polishing * polishing * polishing * fixed bug with new thread * polishing * a bit of refactoring and docs added * notebook added to docs * type checking added to CI * CI fix * CI fix * CI fix * polishing * obsolete todo comment removed * fixed precommit error --------- Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * [CAP] [Feature] Get list of actors from directory service. (#2073) * Search directory for list of actors using regex '.*' gets all actors * docs changes * pre-commit fixes * Use ActorInfo from protobuf * pre-commit * Added zmq tests to work on removing sleeps * minor refactor of zmq tests * 1) Change DirSvr to user Broker. 2) Add req-router to broker 3) In ActorConnector use handshake and req/resp to remove sleep * 1) Change DirSvr to user Broker. 2) Add req-router to broker 3) In ActorConnector use handshake and req/resp to remove sleep * move socket creation to thread with recv * move socket creation to thread with recv * Better logging for DirectorySvc * better logging for directory svc * Use logging config * Start removing sleeps * pre-commit * Cleanup monitor socket * Mark cache as a protocol and update type hints to reflect (#2168) * Mark cache as a protocl and update type hints to reflect * int * undo init change modified: autogen/agentchat/chat.py * fix(): fix word spelling errors (#2171) * Implement User Defined Functions for Local CLI Executor (#2102) * Implement user defined functions feature for local cli exec, add docs * add tests, update docs * fixes * fix test * add pandas test dep * install test * provide template as func * formatting * undo change * address comments * add test deps * formatting * test only in 1 env * formatting * remove test for local only --------- Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * simplify getting-started; update news (#2175) * simplify getting-started; update news * bug fix * update (#2178) Co-authored-by: AnonymousRepoSub <“shaokunzhang529@outlook.com” > * Fix formatting of admonitions in udf docs (#2188) * Fix iostream on new thread (#2181) * fixed get_stream in new thread by introducing a global default * fixed get_stream in new thread by introducing a global default --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Add link for rendering notebooks docs on website (#2191) * Transform Messages Capability (#1923) * wip * Adds docstrings * fixed spellings * wip * fixed errors * better class names * adds tests * added tests to workflow * improved token counting * improved notebook * improved token counting in test * improved docstrings * fix inconsistencies * changed by mistake * fixed docstring * fixed details * improves tests + adds openai contrib test * fix spelling oai contrib test * clearer docstrings * remove repeated docstr * improved notebook * adds metadata to notebook * Improve outline and description (#2125) * better dir structure * clip max tokens to allowed tokens * more accurate comments/docstrs * add deperecation warning * fix front matter desc * add deperecation warning notebook * undo local notebook settings changes * format notebook * format workflow --------- Co-authored-by: gagb <gagb@users.noreply.github.com> * Bump express from 4.18.2 to 4.19.2 in /website (#2157) Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.19.2. - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](expressjs/express@4.18.2...4.19.2) --- updated-dependencies: - dependency-name: express dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add clarity analytics (#2201) * Docstring formatting fix: Standardize docstrings to adhere to Google style guide, ensuring consistency and clarity. and also fixed the broken link for autogen/agentchat/chat.py * Docstring fix: Reformattted docstrings to adhere to Google style guide, nsuring consistency and clarity. For agentchat/contrib/retrieve_user_proxy_agent.py file * Fixed Pre-Commit Error, Trailing spaces on agentchat/chat.py * Fixed Pre-Commit Error, Trailing spaces on agentchat/chat.py --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Li Jiang <bnujli@gmail.com> Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> Co-authored-by: olgavrou <olgavrou@gmail.com> Co-authored-by: gagb <gagb@users.noreply.github.com> Co-authored-by: Qingyun Wu <qingyun0327@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Wael Karkoub <wael.karkoub96@gmail.com> Co-authored-by: Shaokun Zhang <shaokunzhang529@gmail.com> Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”> Co-authored-by: Jieyu Zhang <jieyuz2@cs.washington.edu> Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> Co-authored-by: Davor Runje <davor@airt.ai> Co-authored-by: Rajan <rajan.chari@yahoo.com> Co-authored-by: calm <1191465097@qq.com> Co-authored-by: AnonymousRepoSub <“shaokunzhang529@outlook.com” >
whiskyboy
pushed a commit
to whiskyboy/autogen
that referenced
this pull request
Apr 17, 2024
* tried implementing my own regex * improves tests * finally works * removes prints * fixed test * adds start and end * delete unused imports * refactored to use new tool * significantly improved algo * tag content -> tag attr * fix tests + adds new field * return full match * return remove start and end * update docstrings * update docstrings * update docstrings --------- Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com>
whiskyboy
pushed a commit
to whiskyboy/autogen
that referenced
this pull request
Apr 17, 2024
* DOC FIX - Formatted Docstrings for the retrieve_user_proxy_agent.py and Added first single line for the class RetrieveUserProxyAgent. * DOC FIX - Formatted Docstrings for theinitiate_chats functiion of ChatResult class in autogen/agentchat/chat.py * Add vision capability (microsoft#2025) * Add vision capability * Configurate: description_prompt * Print warning instead of raising issues for type * Skip vision capability test if dependencies not installed * Append "vision" to agent's system message when enabled VisionCapability * GPT-4V notebook update with ConversableAgent * Clean GPT-4V notebook * Add vision capability test to workflow * Lint import * Update system message for vision capability * Add a `custom_caption_func` to VisionCapability * Add custom function example for vision capability * Skip test Vision capability custom func * GPT-4V notebook metadata to website * Remove redundant files * The custom caption function takes more inputs now * Add a more complex example of custom caption func * Remove trailing space --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Native tool call support for Mistral AI API and topic notebook. (microsoft#2135) * Support for Mistral AI API and topic notebook. * formatting * formatting * New conversational chess notebook using nested chats and tool use (microsoft#2137) * add chess notebook * update * update * Update notebook with figure * Add example link * redirect * Clean up example format * address gagan's comments * update references * fix links * add webarena in samples (microsoft#2114) * add webarena in samples/tools * Update samples/tools/webarena/README.md Co-authored-by: gagb <gagb@users.noreply.github.com> * Update samples/tools/webarena/README.md Co-authored-by: gagb <gagb@users.noreply.github.com> * Update samples/tools/webarena/README.md Co-authored-by: gagb <gagb@users.noreply.github.com> * update installation instructions * black formatting * Update README.md --------- Co-authored-by: gagb <gagb@users.noreply.github.com> Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * context to kwargs (microsoft#2064) * context to kwargs * add tag * add test * text to kwargs --------- Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /website (microsoft#2131) Bumps [webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) from 5.3.3 to 5.3.4. - [Release notes](https://github.com/webpack/webpack-dev-middleware/releases) - [Changelog](https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md) - [Commits](webpack/webpack-dev-middleware@v5.3.3...v5.3.4) --- updated-dependencies: - dependency-name: webpack-dev-middleware dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * Parse Any HTML-esh Style Tags (microsoft#2046) * tried implementing my own regex * improves tests * finally works * removes prints * fixed test * adds start and end * delete unused imports * refactored to use new tool * significantly improved algo * tag content -> tag attr * fix tests + adds new field * return full match * return remove start and end * update docstrings * update docstrings * update docstrings --------- Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Integrate AgentOptimizer (microsoft#1767) * draft agent optimizer * refactor * remove * change openai config interface * notebook * update blog * add test * clean up * redir * update * update interface * change model name * move to contrib * Update autogen/agentchat/contrib/agent_optimizer.py Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> --------- Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”> Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Jieyu Zhang <jieyuz2@cs.washington.edu> Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> * Introducing IOStream protocol and adding support for websockets (microsoft#1551) * Introducing IOStream * bug fixing * polishing * refactoring * refactoring * refactoring * wip: async tests * websockets added * wip * merge with main * notebook added * FastAPI example added * wip * merge * getter/setter to iostream added * website/blog/2024-03-03-AutoGen-Update/img/dalle_gpt4v.png: convert to Git LFS * website/blog/2024-03-03-AutoGen-Update/img/gaia.png: convert to Git LFS * website/blog/2024-03-03-AutoGen-Update/img/teach.png: convert to Git LFS * add SSL support * wip * wip * exception handling added to on_connect() * refactoring: default iostream is being set in a context manager * test fix * polishing * polishing * polishing * fixed bug with new thread * polishing * a bit of refactoring and docs added * notebook added to docs * type checking added to CI * CI fix * CI fix * CI fix * polishing * obsolete todo comment removed * fixed precommit error --------- Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * [CAP] [Feature] Get list of actors from directory service. (microsoft#2073) * Search directory for list of actors using regex '.*' gets all actors * docs changes * pre-commit fixes * Use ActorInfo from protobuf * pre-commit * Added zmq tests to work on removing sleeps * minor refactor of zmq tests * 1) Change DirSvr to user Broker. 2) Add req-router to broker 3) In ActorConnector use handshake and req/resp to remove sleep * 1) Change DirSvr to user Broker. 2) Add req-router to broker 3) In ActorConnector use handshake and req/resp to remove sleep * move socket creation to thread with recv * move socket creation to thread with recv * Better logging for DirectorySvc * better logging for directory svc * Use logging config * Start removing sleeps * pre-commit * Cleanup monitor socket * Mark cache as a protocol and update type hints to reflect (microsoft#2168) * Mark cache as a protocl and update type hints to reflect * int * undo init change modified: autogen/agentchat/chat.py * fix(): fix word spelling errors (microsoft#2171) * Implement User Defined Functions for Local CLI Executor (microsoft#2102) * Implement user defined functions feature for local cli exec, add docs * add tests, update docs * fixes * fix test * add pandas test dep * install test * provide template as func * formatting * undo change * address comments * add test deps * formatting * test only in 1 env * formatting * remove test for local only --------- Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> * simplify getting-started; update news (microsoft#2175) * simplify getting-started; update news * bug fix * update (microsoft#2178) Co-authored-by: AnonymousRepoSub <“shaokunzhang529@outlook.com” > * Fix formatting of admonitions in udf docs (microsoft#2188) * Fix iostream on new thread (microsoft#2181) * fixed get_stream in new thread by introducing a global default * fixed get_stream in new thread by introducing a global default --------- Co-authored-by: Chi Wang <wang.chi@microsoft.com> * Add link for rendering notebooks docs on website (microsoft#2191) * Transform Messages Capability (microsoft#1923) * wip * Adds docstrings * fixed spellings * wip * fixed errors * better class names * adds tests * added tests to workflow * improved token counting * improved notebook * improved token counting in test * improved docstrings * fix inconsistencies * changed by mistake * fixed docstring * fixed details * improves tests + adds openai contrib test * fix spelling oai contrib test * clearer docstrings * remove repeated docstr * improved notebook * adds metadata to notebook * Improve outline and description (microsoft#2125) * better dir structure * clip max tokens to allowed tokens * more accurate comments/docstrs * add deperecation warning * fix front matter desc * add deperecation warning notebook * undo local notebook settings changes * format notebook * format workflow --------- Co-authored-by: gagb <gagb@users.noreply.github.com> * Bump express from 4.18.2 to 4.19.2 in /website (microsoft#2157) Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.19.2. - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](expressjs/express@4.18.2...4.19.2) --- updated-dependencies: - dependency-name: express dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * add clarity analytics (microsoft#2201) * Docstring formatting fix: Standardize docstrings to adhere to Google style guide, ensuring consistency and clarity. and also fixed the broken link for autogen/agentchat/chat.py * Docstring fix: Reformattted docstrings to adhere to Google style guide, nsuring consistency and clarity. For agentchat/contrib/retrieve_user_proxy_agent.py file * Fixed Pre-Commit Error, Trailing spaces on agentchat/chat.py * Fixed Pre-Commit Error, Trailing spaces on agentchat/chat.py --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Li Jiang <bnujli@gmail.com> Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com> Co-authored-by: Chi Wang <wang.chi@microsoft.com> Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com> Co-authored-by: olgavrou <olgavrou@gmail.com> Co-authored-by: gagb <gagb@users.noreply.github.com> Co-authored-by: Qingyun Wu <qingyun0327@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Wael Karkoub <wael.karkoub96@gmail.com> Co-authored-by: Shaokun Zhang <shaokunzhang529@gmail.com> Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”> Co-authored-by: Jieyu Zhang <jieyuz2@cs.washington.edu> Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> Co-authored-by: Davor Runje <davor@airt.ai> Co-authored-by: Rajan <rajan.chari@yahoo.com> Co-authored-by: calm <1191465097@qq.com> Co-authored-by: AnonymousRepoSub <“shaokunzhang529@outlook.com” >
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR introduces an enhancement to the way LMM agents communicate with each other. Currently, agents use HTML-style tags embedded in text messages to share image information (e.g.,
Hi, take a look at this image <img http://example.com/something.jpg>). Expanding the tagging system to audio and video could be beneficial as LMM agents' capabilities expand.The key improvements in this PR include:
Some examples:
<audio prompt="Whisper" text="Hello autogen" task="generate">, the output would be{"tag": "audio", "content": {"prompt": "whisper", "text": "Hello autogen", "task": "generate"}}.<video https://example.com prompt="change all red cars to yellow" tasks="modify">, the output would be{"tag": "video", "content": {"prompt": "change all red cars to yellow", "task": "modify"}}.NOTE: As you may have noticed, the implementation is complex because I don't know what I'm doing when it comes to regex.
I also thought of fully supporting HTML, simplifying the implementation. However, I prioritized backward compatibility instead.
Related issue number
Checks