WebResearchRetriever error handling in urls with connection error #13401

pedro-inf-custodio · 2023-11-15T15:43:55Z

Description: Added a method fetch_valid_documents to WebResearchRetriever class that will test the connection for every url in new_urls and remove those that raise a ConnectionError.
Issue: Previous PR,
Dependencies: None,
Tag maintainer: @efriis

Please make sure your PR is passing linting and testing before submitting. Run make format, make lint and make test to check this locally.

See contribution guidelines for more information on how to write/run tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:

a test for the integration, preferably unit tests that do not rely on network access,
an example notebook showing its use. It lives in docs/extras directory.

If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.

vercel · 2023-11-15T15:43:59Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Nov 17, 2023 10:03am

efriis · 2023-11-15T19:27:54Z

libs/langchain/langchain/retrievers/web_research.py

@@ -154,6 +154,17 @@ def search_tool(self, query: str, num_search_results: int = 1) -> List[dict]:
        query_clean = self.clean_search_query(query)
        result = self.search.results(query_clean, num_search_results)
        return result
+
+    def fetch_valid_documents(self, url_list: List[str]) -> List[str]:


Suggested change

def fetch_valid_documents(self, url_list: List[str]) -> List[str]:

def _fetch_valid_documents(self, url_list: List[str]) -> List[str]:

make it semiprivate

efriis · 2023-11-15T19:34:48Z

libs/langchain/langchain/retrievers/web_research.py

@@ -195,6 +206,9 @@ def _get_relevant_documents(
        # Check for any new urls that we have not processed
        new_urls = list(urls.difference(self.url_database))

+        # Removes urls with connection errors from list
+        new_urls = self.fetch_valid_documents(new_urls)


this will significantly slow down execution.

I think the best option would be a combination approach of what you did here and in #13353, where you add a config param to AsyncHtmlLoader that defaults to False of ignore_load_errors, and you construct the loader here with that set to true.

What do you think?

It does make sense to me. I've done the try ... except ... in the _scrape method to avoid putting a lot of logic in the load method.

efriis · 2023-11-15T19:35:25Z

libs/langchain/langchain/retrievers/web_research.py

+        clean_urls = []
+        for url in url_list:
+            try:
+                Requests().get(url).status_code


let's avoid adding extra requests to the path if we can

efriis

Thanks @pedro-inf-custodio

[![Mend Renovate logo banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [@autoblocks/client](https://togithub.com/autoblocksai/javascript-sdk) | [`^0.0.17` -> `^0.0.20`](https://renovatebot.com/diffs/npm/@autoblocks%2fclient/0.0.17/0.0.20) | [![age](https://developer.mend.io/api/mc/badges/age/npm/@autoblocks%2fclient/0.0.20?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/@autoblocks%2fclient/0.0.20?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/@autoblocks%2fclient/0.0.17/0.0.20?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/@autoblocks%2fclient/0.0.17/0.0.20?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [@types/node](https://togithub.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/node) ([source](https://togithub.com/DefinitelyTyped/DefinitelyTyped)) | [`20.9.0` -> `20.9.2`](https://renovatebot.com/diffs/npm/@types%2fnode/20.9.0/20.9.2) | [![age](https://developer.mend.io/api/mc/badges/age/npm/@types%2fnode/20.9.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/@types%2fnode/20.9.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/@types%2fnode/20.9.0/20.9.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/@types%2fnode/20.9.0/20.9.2?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [ai](https://sdk.vercel.ai/docs) ([source](https://togithub.com/vercel/ai)) | [`2.2.22` -> `2.2.24`](https://renovatebot.com/diffs/npm/ai/2.2.22/2.2.24) | [![age](https://developer.mend.io/api/mc/badges/age/npm/ai/2.2.24?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/ai/2.2.24?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/ai/2.2.22/2.2.24?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/ai/2.2.22/2.2.24?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [eslint](https://eslint.org) ([source](https://togithub.com/eslint/eslint)) | [`8.53.0` -> `8.54.0`](https://renovatebot.com/diffs/npm/eslint/8.53.0/8.54.0) | [![age](https://developer.mend.io/api/mc/badges/age/npm/eslint/8.54.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/eslint/8.54.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/eslint/8.53.0/8.54.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/eslint/8.53.0/8.54.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [eslint-config-next](https://nextjs.org/docs/app/building-your-application/configuring/eslint#eslint-config) ([source](https://togithub.com/vercel/next.js)) | [`14.0.2` -> `14.0.3`](https://renovatebot.com/diffs/npm/eslint-config-next/14.0.2/14.0.3) | [![age](https://developer.mend.io/api/mc/badges/age/npm/eslint-config-next/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/eslint-config-next/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/eslint-config-next/14.0.2/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/eslint-config-next/14.0.2/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [langchain](https://togithub.com/langchain-ai/langchain) | `^0.0.335` -> `^0.0.338` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/langchain/0.0.338?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/langchain/0.0.338?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/langchain/0.0.335/0.0.338?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/langchain/0.0.335/0.0.338?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [langchain](https://togithub.com/langchain-ai/langchainjs) | [`^0.0.186` -> `^0.0.193`](https://renovatebot.com/diffs/npm/langchain/0.0.186/0.0.193) | [![age](https://developer.mend.io/api/mc/badges/age/npm/langchain/0.0.193?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/langchain/0.0.193?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/langchain/0.0.186/0.0.193?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/langchain/0.0.186/0.0.193?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [next](https://nextjs.org) ([source](https://togithub.com/vercel/next.js)) | [`14.0.2` -> `14.0.3`](https://renovatebot.com/diffs/npm/next/14.0.2/14.0.3) | [![age](https://developer.mend.io/api/mc/badges/age/npm/next/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/next/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/next/14.0.2/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/next/14.0.2/14.0.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [openai](https://togithub.com/openai/openai-python) | `1.2.3` -> `1.3.3` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/openai/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/openai/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/openai/1.2.3/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/openai/1.2.3/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [openai](https://togithub.com/openai/openai-python) | `0.28.1` -> `1.3.3` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/openai/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/openai/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/openai/0.28.1/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/openai/0.28.1/1.3.3?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [openai](https://togithub.com/openai/openai-node) | [`4.17.4` -> `4.19.0`](https://renovatebot.com/diffs/npm/openai/4.17.4/4.19.0) | [![age](https://developer.mend.io/api/mc/badges/age/npm/openai/4.19.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/openai/4.19.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/openai/4.17.4/4.19.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/openai/4.17.4/4.19.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>autoblocksai/javascript-sdk (@autoblocks/client)</summary> ### [`v0.0.20`](https://togithub.com/autoblocksai/javascript-sdk/compare/0.0.19...0.0.20) [Compare Source](https://togithub.com/autoblocksai/javascript-sdk/compare/0.0.19...0.0.20) ### [`v0.0.19`](https://togithub.com/autoblocksai/javascript-sdk/compare/0.0.18...0.0.19) [Compare Source](https://togithub.com/autoblocksai/javascript-sdk/compare/0.0.18...0.0.19) ### [`v0.0.18`](https://togithub.com/autoblocksai/javascript-sdk/compare/0.0.17...0.0.18) [Compare Source](https://togithub.com/autoblocksai/javascript-sdk/compare/0.0.17...0.0.18) </details> <details> <summary>vercel/ai (ai)</summary> ### [`v2.2.24`](https://togithub.com/vercel/ai/releases/tag/ai%402.2.24) [Compare Source](https://togithub.com/vercel/ai/compare/ai@2.2.23...ai@2.2.24) ##### Patch Changes - [`69ca8f5`](https://togithub.com/vercel/ai/commit/69ca8f5): ai/react: add experimental_useAssistant hook and experimental_AssistantResponse - [`3e2299e`](https://togithub.com/vercel/ai/commit/3e2299e): experimental_StreamData/StreamingReactResponse: optimize parsing, improve types - [`70bd2ac`](https://togithub.com/vercel/ai/commit/70bd2ac): ai/solid: add experimental_StreamData support to useChat Proper documentation for the new features will be ready in the near future, but in the meantime you can refer to [this document](https://togithub.com/vercel/ai/blob/main/examples/next-openai/app/api/assistant/assistant-setup.md) and the accompanying [example](https://togithub.com/vercel/ai/blob/main/examples/next-openai/app/api/assistant/route.ts) for the Assistants API, and [this example](https://togithub.com/vercel/ai/blob/fbda5b20afe33f1e9b73644a1052954b2d2e7602/examples/next-openai/app/api/chat-with-vision) for working with the new `data` API for vision. Thanks [@lgrammel](https://togithub.com/lgrammel) for the great work in this release! ### [`v2.2.23`](https://togithub.com/vercel/ai/releases/tag/ai%402.2.23) [Compare Source](https://togithub.com/vercel/ai/compare/ai@2.2.22...ai@2.2.23) ##### Patch Changes - [`5a04321`](https://togithub.com/vercel/ai/commit/5a04321): add StreamData support to StreamingReactResponse, add client-side data API to react/use-chat </details> <details> <summary>eslint/eslint (eslint)</summary> ### [`v8.54.0`](https://togithub.com/eslint/eslint/releases/tag/v8.54.0) [Compare Source](https://togithub.com/eslint/eslint/compare/v8.53.0...v8.54.0) #### Features - [`a7a883b`](https://togithub.com/eslint/eslint/commit/a7a883bd6ba4f140b60cbbb2be5b53d750f6c8db) feat: for-direction rule add check for condition in reverse order ([#17755](https://togithub.com/eslint/eslint/issues/17755)) (Angelo Annunziata) - [`1452dc9`](https://togithub.com/eslint/eslint/commit/1452dc9f12c45c05d7c569f737221f0d988ecef1) feat: Add suggestions to no-console ([#17680](https://togithub.com/eslint/eslint/issues/17680)) (Joel Mathew Koshy) - [`21ebf8a`](https://togithub.com/eslint/eslint/commit/21ebf8a811be9f4b009cf70a10be5062d4fdc736) feat: update `no-array-constructor` rule ([#17711](https://togithub.com/eslint/eslint/issues/17711)) (Francesco Trotta) #### Bug Fixes - [`98926e6`](https://togithub.com/eslint/eslint/commit/98926e6e7323e5dd12a9f016cb558144296665af) fix: Ensure that extra data is not accidentally stored in the cache file ([#17760](https://togithub.com/eslint/eslint/issues/17760)) (Milos Djermanovic) - [`e8cf9f6`](https://togithub.com/eslint/eslint/commit/e8cf9f6a524332293f8b2c90a2db4a532e47d919) fix: Make dark scroll bar in dark theme ([#17753](https://togithub.com/eslint/eslint/issues/17753)) (Pavel) - [`3cbeaad`](https://togithub.com/eslint/eslint/commit/3cbeaad7b943c153937ce34365cec2c406f2b98b) fix: Use `cwd` constructor option as config `basePath` in Linter ([#17705](https://togithub.com/eslint/eslint/issues/17705)) (Milos Djermanovic) #### Documentation - [`becfdd3`](https://togithub.com/eslint/eslint/commit/becfdd39b25d795e56c9a13eb3e77af6b9c86e8a) docs: Make clear when rules are removed ([#17728](https://togithub.com/eslint/eslint/issues/17728)) (Nicholas C. Zakas) - [`05d6e99`](https://togithub.com/eslint/eslint/commit/05d6e99153ed6d94eb30f46c57609371918a41f3) docs: update "Submit a Pull Request" page ([#17712](https://togithub.com/eslint/eslint/issues/17712)) (Francesco Trotta) - [`eb2279e`](https://togithub.com/eslint/eslint/commit/eb2279e5148cee8fdea7dae614f4f8af7a2d06c3) docs: display info about deprecated rules ([#17749](https://togithub.com/eslint/eslint/issues/17749)) (Percy Ma) - [`d245326`](https://togithub.com/eslint/eslint/commit/d24532601e64714ac5d08507e05aa5c14ecd1d5a) docs: Correct working in migrating plugin docs ([#17722](https://togithub.com/eslint/eslint/issues/17722)) (Filip Tammergård) #### Chores - [`d644de9`](https://togithub.com/eslint/eslint/commit/d644de9a4b593b565617303a095bc9aa69e7b768) chore: upgrade [@eslint/js](https://togithub.com/eslint/js)[@8](https://togithub.com/8).54.0 ([#17773](https://togithub.com/eslint/eslint/issues/17773)) (Milos Djermanovic) - [`1e6e314`](https://togithub.com/eslint/eslint/commit/1e6e31415cc429a3a9fc64b2ec03df0e0ec0c91b) chore: package.json update for [@eslint/js](https://togithub.com/eslint/js) release (Jenkins) - [`6fb8805`](https://togithub.com/eslint/eslint/commit/6fb8805310afe7476d6c404f172177a6d15fcf11) chore: Fixed grammar in issue_templates/rule_change ([#17770](https://togithub.com/eslint/eslint/issues/17770)) (Joel Mathew Koshy) - [`85db724`](https://togithub.com/eslint/eslint/commit/85db7243ddb8706ed60ab64a7ddf604d0d7de493) chore: upgrade `markdownlint` to 0.31.1 ([#17754](https://togithub.com/eslint/eslint/issues/17754)) (Nitin Kumar) - [`6d470d2`](https://togithub.com/eslint/eslint/commit/6d470d2e74535761bd56dcb1c021b463ef9e8a9c) chore: update dependency recast to ^0.23.0 ([#17736](https://togithub.com/eslint/eslint/issues/17736)) (renovate\[bot]) - [`b7121b5`](https://togithub.com/eslint/eslint/commit/b7121b590d578c9c9b38ee481313317f30e54817) chore: update dependency markdownlint-cli to ^0.37.0 ([#17735](https://togithub.com/eslint/eslint/issues/17735)) (renovate\[bot]) - [`633b9a1`](https://togithub.com/eslint/eslint/commit/633b9a19752b6a22ab4d6c824f27a75ac0e4151b) chore: update dependency regenerator-runtime to ^0.14.0 ([#17739](https://togithub.com/eslint/eslint/issues/17739)) (renovate\[bot]) - [`acac16f`](https://togithub.com/eslint/eslint/commit/acac16fdf8540f7ba86cf637e3c1b253bd35a268) chore: update dependency vite-plugin-commonjs to ^0.10.0 ([#17740](https://togithub.com/eslint/eslint/issues/17740)) (renovate\[bot]) - [`ba8ca7e`](https://togithub.com/eslint/eslint/commit/ba8ca7e3debcba68ee7015b9221cf5acd7870206) chore: add .github/renovate.json5 ([#17567](https://togithub.com/eslint/eslint/issues/17567)) (Josh Goldberg ✨) </details> <details> <summary>vercel/next.js (eslint-config-next)</summary> ### [`v14.0.3`](https://togithub.com/vercel/next.js/compare/v14.0.2...v14.0.3) [Compare Source](https://togithub.com/vercel/next.js/compare/v14.0.2...v14.0.3) </details> <details> <summary>langchain-ai/langchain (langchain)</summary> ### [`v0.0.338`](https://togithub.com/langchain-ai/langchain/releases/tag/v0.0.338) [Compare Source](https://togithub.com/langchain-ai/langchain/compare/v0.0.337...v0.0.338) #### What's Changed - Override Keys Option by [@hinthornw](https://togithub.com/hinthornw) in [https://github.com/langchain-ai/langchain/pull/13537](https://togithub.com/langchain-ai/langchain/pull/13537) - Neptune graph updates by [@3coins](https://togithub.com/3coins) in [https://github.com/langchain-ai/langchain/pull/13491](https://togithub.com/langchain-ai/langchain/pull/13491) - WebResearchRetriever error handling in urls with connection error by [@pedro-inf-custodio](https://togithub.com/pedro-inf-custodio) in [https://github.com/langchain-ai/langchain/pull/13401](https://togithub.com/langchain-ai/langchain/pull/13401) - Add execution time by [@hinthornw](https://togithub.com/hinthornw) in [https://github.com/langchain-ai/langchain/pull/13542](https://togithub.com/langchain-ai/langchain/pull/13542) - Generic LLM wrapper to support chat model interface with configurable chat prompt format by [@krasserm](https://togithub.com/krasserm) in [https://github.com/langchain-ai/langchain/pull/8295](https://togithub.com/langchain-ai/langchain/pull/8295) - Use random seed by [@hinthornw](https://togithub.com/hinthornw) in [https://github.com/langchain-ai/langchain/pull/13544](https://togithub.com/langchain-ai/langchain/pull/13544) - Fix typo/line break in the middle of a word by [@marks](https://togithub.com/marks) in [https://github.com/langchain-ai/langchain/pull/13314](https://togithub.com/langchain-ai/langchain/pull/13314) - Adds support for new OctoAI endpoints by [@AI-Bassem](https://togithub.com/AI-Bassem) in [https://github.com/langchain-ai/langchain/pull/13521](https://togithub.com/langchain-ai/langchain/pull/13521) - fixed `openai_assistant` namespace by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13543](https://togithub.com/langchain-ai/langchain/pull/13543) - move streaming stdout by [@hwchase17](https://togithub.com/hwchase17) in [https://github.com/langchain-ai/langchain/pull/13559](https://togithub.com/langchain-ai/langchain/pull/13559) - update multi index templates by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13569](https://togithub.com/langchain-ai/langchain/pull/13569) - bump 338, exp 42 by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13564](https://togithub.com/langchain-ai/langchain/pull/13564) #### New Contributors - [@pedro-inf-custodio](https://togithub.com/pedro-inf-custodio) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13401](https://togithub.com/langchain-ai/langchain/pull/13401) - [@marks](https://togithub.com/marks) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13314](https://togithub.com/langchain-ai/langchain/pull/13314) **Full Changelog**: https://github.com/langchain-ai/langchain/compare/v0.0.337...v0.0.338 ### [`v0.0.337`](https://togithub.com/langchain-ai/langchain/releases/tag/v0.0.337) [Compare Source](https://togithub.com/langchain-ai/langchain/compare/v0.0.336...v0.0.337) #### What's Changed - Make pirate-speak-configurable template not require env vars for alte… by [@nfcampos](https://togithub.com/nfcampos) in [https://github.com/langchain-ai/langchain/pull/13395](https://togithub.com/langchain-ai/langchain/pull/13395) - Fix a link in docs by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchain/pull/13423](https://togithub.com/langchain-ai/langchain/pull/13423) - updated `clickup` example by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13424](https://togithub.com/langchain-ai/langchain/pull/13424) - DOCS: rag nit by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13436](https://togithub.com/langchain-ai/langchain/pull/13436) - callback refactor by [@hwchase17](https://togithub.com/hwchase17) in [https://github.com/langchain-ai/langchain/pull/13372](https://togithub.com/langchain-ai/langchain/pull/13372) - updated `Activeloop DeepMemory` notebook by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13428](https://togithub.com/langchain-ai/langchain/pull/13428) - updated `semadb` example by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13431](https://togithub.com/langchain-ai/langchain/pull/13431) - Bagatur/chain of note by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13470](https://togithub.com/langchain-ai/langchain/pull/13470) - Update multi-modal RAG cookbook by [@rlancemartin](https://togithub.com/rlancemartin) in [https://github.com/langchain-ai/langchain/pull/13429](https://togithub.com/langchain-ai/langchain/pull/13429) - Update chain of note README.md by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13473](https://togithub.com/langchain-ai/langchain/pull/13473) - docs: `integrations/text_embeddings/` cleanup by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13476](https://togithub.com/langchain-ai/langchain/pull/13476) - fix for `integratons/document_loaders` sidebar by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13471](https://togithub.com/langchain-ai/langchain/pull/13471) - Add ahandle_event to *all* by [@eyurtsev](https://togithub.com/eyurtsev) in [https://github.com/langchain-ai/langchain/pull/13469](https://togithub.com/langchain-ai/langchain/pull/13469) - Astra DB: minor improvements to docstrings and demo notebook by [@hemidactylus](https://togithub.com/hemidactylus) in [https://github.com/langchain-ai/langchain/pull/13449](https://togithub.com/langchain-ai/langchain/pull/13449) - Use List instead of list by [@ifduyue](https://togithub.com/ifduyue) in [https://github.com/langchain-ai/langchain/pull/13443](https://togithub.com/langchain-ai/langchain/pull/13443) - updated `memory` Titles by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13435](https://togithub.com/langchain-ai/langchain/pull/13435) - BUG Fix app_name in cli app new by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13482](https://togithub.com/langchain-ai/langchain/pull/13482) - Lock pydantic v1 in app template, cli 0.0.18 by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13485](https://togithub.com/langchain-ai/langchain/pull/13485) - Add serialisation arguments to Bedrock and ChatBedrock by [@dqbd](https://togithub.com/dqbd) in [https://github.com/langchain-ai/langchain/pull/13465](https://togithub.com/langchain-ai/langchain/pull/13465) - add input_type to VoyageEmbeddings by [@thomas0809](https://togithub.com/thomas0809) in [https://github.com/langchain-ai/langchain/pull/13488](https://togithub.com/langchain-ai/langchain/pull/13488) - Bugfix: OpenAIFunctionsAgentOutputParser doesn't handle functions with no args by [@chrisaffirm](https://togithub.com/chrisaffirm) in [https://github.com/langchain-ai/langchain/pull/13467](https://togithub.com/langchain-ai/langchain/pull/13467) - Allow openai v1 in all templates that require it by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13489](https://togithub.com/langchain-ai/langchain/pull/13489) - updated `async-faiss` example by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13434](https://togithub.com/langchain-ai/langchain/pull/13434) - docs `integrations/vectorstores/` cleanup by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13487](https://togithub.com/langchain-ai/langchain/pull/13487) - Add optional arguments to FalkorDBGraph constructor by [@gkorland](https://togithub.com/gkorland) in [https://github.com/langchain-ai/langchain/pull/13459](https://togithub.com/langchain-ai/langchain/pull/13459) - updated `data_connection` index page by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13426](https://togithub.com/langchain-ai/langchain/pull/13426) - Add Wrapping Library Metadata to MongoDB vector store by [@NoahStapp](https://togithub.com/NoahStapp) in [https://github.com/langchain-ai/langchain/pull/13084](https://togithub.com/langchain-ai/langchain/pull/13084) - \[LLMonitorCallbackHandler] Various improvements by [@hughcrt](https://togithub.com/hughcrt) in [https://github.com/langchain-ai/langchain/pull/13151](https://togithub.com/langchain-ai/langchain/pull/13151) - TEMPLATES: Add multi-index templates by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13490](https://togithub.com/langchain-ai/langchain/pull/13490) - IMPROVEMENT: update assistants output and doc by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13480](https://togithub.com/langchain-ai/langchain/pull/13480) - Runnable with message history by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13418](https://togithub.com/langchain-ai/langchain/pull/13418) - Add VertexAI Chuck Norris template by [@wietsevenema](https://togithub.com/wietsevenema) in [https://github.com/langchain-ai/langchain/pull/13531](https://togithub.com/langchain-ai/langchain/pull/13531) - bump 337 by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13534](https://togithub.com/langchain-ai/langchain/pull/13534) #### New Contributors - [@ifduyue](https://togithub.com/ifduyue) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13443](https://togithub.com/langchain-ai/langchain/pull/13443) - [@chrisaffirm](https://togithub.com/chrisaffirm) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13467](https://togithub.com/langchain-ai/langchain/pull/13467) - [@wietsevenema](https://togithub.com/wietsevenema) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13531](https://togithub.com/langchain-ai/langchain/pull/13531) **Full Changelog**: https://github.com/langchain-ai/langchain/compare/v0.0.336...v0.0.337 ### [`v0.0.336`](https://togithub.com/langchain-ai/langchain/releases/tag/v0.0.336) [Compare Source](https://togithub.com/langchain-ai/langchain/compare/v0.0.335...v0.0.336) #### What's Changed - Add new models to openai callback by [@IsakNyberg](https://togithub.com/IsakNyberg) in [https://github.com/langchain-ai/langchain/pull/13244](https://togithub.com/langchain-ai/langchain/pull/13244) - Update README.md by [@levalencia](https://togithub.com/levalencia) in [https://github.com/langchain-ai/langchain/pull/8570](https://togithub.com/langchain-ai/langchain/pull/8570) - Free knowledge base pod information update by [@mpskex](https://togithub.com/mpskex) in [https://github.com/langchain-ai/langchain/pull/12813](https://togithub.com/langchain-ai/langchain/pull/12813) - Update ollama.py by [@glad4enkonm](https://togithub.com/glad4enkonm) in [https://github.com/langchain-ai/langchain/pull/12895](https://togithub.com/langchain-ai/langchain/pull/12895) - Improve CSV reader which can't call .strip() on NoneType by [@dennisdegreef](https://togithub.com/dennisdegreef) in [https://github.com/langchain-ai/langchain/pull/13079](https://togithub.com/langchain-ai/langchain/pull/13079) - Typo fix to quickstart.mdx by [@marioangst](https://togithub.com/marioangst) in [https://github.com/langchain-ai/langchain/pull/13178](https://togithub.com/langchain-ai/langchain/pull/13178) - dalle add model parameter by [@AzeWZ](https://togithub.com/AzeWZ) in [https://github.com/langchain-ai/langchain/pull/13201](https://togithub.com/langchain-ai/langchain/pull/13201) - Remove `_get_kwarg_value` function by [@Guillem96](https://togithub.com/Guillem96) in [https://github.com/langchain-ai/langchain/pull/13184](https://togithub.com/langchain-ai/langchain/pull/13184) - Update README.md - Added notebook for extraction_openai_tools by [@shauryr](https://togithub.com/shauryr) in [https://github.com/langchain-ai/langchain/pull/13205](https://togithub.com/langchain-ai/langchain/pull/13205) - Add dockerfile template by [@langchain-infra](https://togithub.com/langchain-infra) in [https://github.com/langchain-ai/langchain/pull/13240](https://togithub.com/langchain-ai/langchain/pull/13240) - added system prompt and template fields to ollama by [@Govind-S-B](https://togithub.com/Govind-S-B) in [https://github.com/langchain-ai/langchain/pull/13022](https://togithub.com/langchain-ai/langchain/pull/13022) - Add rag google vertex ai search template by [@juan-calvo-datatonic](https://togithub.com/juan-calvo-datatonic) in [https://github.com/langchain-ai/langchain/pull/13294](https://togithub.com/langchain-ai/langchain/pull/13294) - Add OpenAI API v1 support for ChatAnyscale and fixed a bug with openai_api_key by [@kylehh](https://togithub.com/kylehh) in [https://github.com/langchain-ai/langchain/pull/13237](https://togithub.com/langchain-ai/langchain/pull/13237) - Fix typo in timescalevector.ipynb by [@eltociear](https://togithub.com/eltociear) in [https://github.com/langchain-ai/langchain/pull/13239](https://togithub.com/langchain-ai/langchain/pull/13239) - docs: align custom_tool document headers by [@edwardzjl](https://togithub.com/edwardzjl) in [https://github.com/langchain-ai/langchain/pull/13252](https://togithub.com/langchain-ai/langchain/pull/13252) - chore: bump momento dependency version and refactor search hit usage by [@malandis](https://togithub.com/malandis) in [https://github.com/langchain-ai/langchain/pull/13111](https://togithub.com/langchain-ai/langchain/pull/13111) - Add MyScaleWithoutJSON which allows user to wrap columns into Document's Metadata by [@mpskex](https://togithub.com/mpskex) in [https://github.com/langchain-ai/langchain/pull/13164](https://togithub.com/langchain-ai/langchain/pull/13164) - Ollama pass kwargs as options instead of top by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13280](https://togithub.com/langchain-ai/langchain/pull/13280) - Use endpoint_url if provided with boto3 session for dynamodb by [@chevalmuscle](https://togithub.com/chevalmuscle) in [https://github.com/langchain-ai/langchain/pull/11622](https://togithub.com/langchain-ai/langchain/pull/11622) - Add missing filter to max_marginal_relevance_search inner call to max_marginal_relevance_search_by_vector by [@Frank995](https://togithub.com/Frank995) in [https://github.com/langchain-ai/langchain/pull/13260](https://togithub.com/langchain-ai/langchain/pull/13260) - FIX: 'from_texts' method in Weaviate with non-existent kwargs param by [@takatost](https://togithub.com/takatost) in [https://github.com/langchain-ai/langchain/pull/11604](https://togithub.com/langchain-ai/langchain/pull/11604) - Refine Weaviate docs and add RAG example by [@iamleonie](https://togithub.com/iamleonie) in [https://github.com/langchain-ai/langchain/pull/13057](https://togithub.com/langchain-ai/langchain/pull/13057) - Update error message in evaluation runner by [@hinthornw](https://togithub.com/hinthornw) in [https://github.com/langchain-ai/langchain/pull/13296](https://togithub.com/langchain-ai/langchain/pull/13296) - Fix serialization issue in Matching Engine Vector Store by [@konstantin-spiess](https://togithub.com/konstantin-spiess) in [https://github.com/langchain-ai/langchain/pull/13266](https://togithub.com/langchain-ai/langchain/pull/13266) - Self-query template by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/12694](https://togithub.com/langchain-ai/langchain/pull/12694) - Fix Pinecone cosine relevance score by [@ruiramos](https://togithub.com/ruiramos) in [https://github.com/langchain-ai/langchain/pull/8920](https://togithub.com/langchain-ai/langchain/pull/8920) - add: license file to subproject by [@YYYasin19](https://togithub.com/YYYasin19) in [https://github.com/langchain-ai/langchain/pull/8403](https://togithub.com/langchain-ai/langchain/pull/8403) - IMPROVEMENT self-query template by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13305](https://togithub.com/langchain-ai/langchain/pull/13305) - Cookbook for multi-modal RAG eval by [@rlancemartin](https://togithub.com/rlancemartin) in [https://github.com/langchain-ai/langchain/pull/13272](https://togithub.com/langchain-ai/langchain/pull/13272) - Increase flexibility of ElasticVectorSearch by [@mertkayhan](https://togithub.com/mertkayhan) in [https://github.com/langchain-ai/langchain/pull/6863](https://togithub.com/langchain-ai/langchain/pull/6863) - add cookbook for RAG with baidu QIANFAN and elasticsearch by [@wemysschen](https://togithub.com/wemysschen) in [https://github.com/langchain-ai/langchain/pull/13287](https://togithub.com/langchain-ai/langchain/pull/13287) - IMPROVEMENT redirect root to docs by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13303](https://togithub.com/langchain-ai/langchain/pull/13303) - gpt researcher by [@hwchase17](https://togithub.com/hwchase17) in [https://github.com/langchain-ai/langchain/pull/13062](https://togithub.com/langchain-ai/langchain/pull/13062) - add retrieval agent by [@hwchase17](https://togithub.com/hwchase17) in [https://github.com/langchain-ai/langchain/pull/13317](https://togithub.com/langchain-ai/langchain/pull/13317) - Update main readme by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13298](https://togithub.com/langchain-ai/langchain/pull/13298) - DOCS: cleanup docs directory by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13301](https://togithub.com/langchain-ai/langchain/pull/13301) - Move OAI assistants to langchain and add callbacks by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13236](https://togithub.com/langchain-ai/langchain/pull/13236) - fix litellm openai imports by [@krrishdholakia](https://togithub.com/krrishdholakia) in [https://github.com/langchain-ai/langchain/pull/13307](https://togithub.com/langchain-ai/langchain/pull/13307) - arxiv retrieval agent improvement by [@hwchase17](https://togithub.com/hwchase17) in [https://github.com/langchain-ai/langchain/pull/13329](https://togithub.com/langchain-ai/langchain/pull/13329) - add more reasonable arxiv retriever by [@hwchase17](https://togithub.com/hwchase17) in [https://github.com/langchain-ai/langchain/pull/13327](https://togithub.com/langchain-ai/langchain/pull/13327) - Pgvector template by [@manuel-soria](https://togithub.com/manuel-soria) in [https://github.com/langchain-ai/langchain/pull/13267](https://togithub.com/langchain-ai/langchain/pull/13267) - Fix latest message index by [@billytrend-cohere](https://togithub.com/billytrend-cohere) in [https://github.com/langchain-ai/langchain/pull/13355](https://togithub.com/langchain-ai/langchain/pull/13355) - CLI interactivity by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13148](https://togithub.com/langchain-ai/langchain/pull/13148) - cli 0.0.17 by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13359](https://togithub.com/langchain-ai/langchain/pull/13359) - added `Cookbooks` link by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13078](https://togithub.com/langchain-ai/langchain/pull/13078) - Bump pyarrow from 13.0.0 to 14.0.1 in /libs/langchain by [@dependabot](https://togithub.com/dependabot) in [https://github.com/langchain-ai/langchain/pull/13363](https://togithub.com/langchain-ai/langchain/pull/13363) - feat(llms): support Openai API v1 for Azure OpenAI completions by [@mspronesti](https://togithub.com/mspronesti) in [https://github.com/langchain-ai/langchain/pull/13231](https://togithub.com/langchain-ai/langchain/pull/13231) - Lint Python notebooks with ruff. by [@obi1kenobi](https://togithub.com/obi1kenobi) in [https://github.com/langchain-ai/langchain/pull/12677](https://togithub.com/langchain-ai/langchain/pull/12677) - Bump all libraries to the latest `ruff` version. by [@obi1kenobi](https://togithub.com/obi1kenobi) in [https://github.com/langchain-ai/langchain/pull/13350](https://togithub.com/langchain-ai/langchain/pull/13350) - fmt by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13371](https://togithub.com/langchain-ai/langchain/pull/13371) - more cli interactivity, bugfix by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13360](https://togithub.com/langchain-ai/langchain/pull/13360) - fix cli release by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13373](https://togithub.com/langchain-ai/langchain/pull/13373) - bump openai by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13262](https://togithub.com/langchain-ai/langchain/pull/13262) - `Yi` model from `01.ai` , example by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13375](https://togithub.com/langchain-ai/langchain/pull/13375) - Update `rag-timescale-conversation` to dependencies without CVEs. by [@obi1kenobi](https://togithub.com/obi1kenobi) in [https://github.com/langchain-ai/langchain/pull/13364](https://togithub.com/langchain-ai/langchain/pull/13364) - Update `templates/rag-self-query` with newer dependencies without CVEs. by [@obi1kenobi](https://togithub.com/obi1kenobi) in [https://github.com/langchain-ai/langchain/pull/13362](https://togithub.com/langchain-ai/langchain/pull/13362) - Add limit_to_domains to APIChain based tools by [@fielding](https://togithub.com/fielding) in [https://github.com/langchain-ai/langchain/pull/13367](https://togithub.com/langchain-ai/langchain/pull/13367) - api doc newlines by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13378](https://togithub.com/langchain-ai/langchain/pull/13378) - docs integration cards site by [@leo-gan](https://togithub.com/leo-gan) in [https://github.com/langchain-ai/langchain/pull/13379](https://togithub.com/langchain-ai/langchain/pull/13379) - Add some properties to NotionDBLoader by [@kenta-takeuchi](https://togithub.com/kenta-takeuchi) in [https://github.com/langchain-ai/langchain/pull/13358](https://togithub.com/langchain-ai/langchain/pull/13358) - IMPROVEMENT more research-assistant configurability by [@efriis](https://togithub.com/efriis) in [https://github.com/langchain-ai/langchain/pull/13312](https://togithub.com/langchain-ai/langchain/pull/13312) - Make it easier to subclass RunnableEach by [@nfcampos](https://togithub.com/nfcampos) in [https://github.com/langchain-ai/langchain/pull/13346](https://togithub.com/langchain-ai/langchain/pull/13346) - Agent window management how to by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13033](https://togithub.com/langchain-ai/langchain/pull/13033) - Bedrock cohere embedding support by [@celmore25](https://togithub.com/celmore25) in [https://github.com/langchain-ai/langchain/pull/13366](https://togithub.com/langchain-ai/langchain/pull/13366) - docs: install nit by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13380](https://togithub.com/langchain-ai/langchain/pull/13380) - Bagatur/update rag use case by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13319](https://togithub.com/langchain-ai/langchain/pull/13319) - Passthrough kwargs in runnable lambda by [@nfcampos](https://togithub.com/nfcampos) in [https://github.com/langchain-ai/langchain/pull/13405](https://togithub.com/langchain-ai/langchain/pull/13405) - PGVector needs to close its connection if it is garbage collected by [@Sumukh](https://togithub.com/Sumukh) in [https://github.com/langchain-ai/langchain/pull/13232](https://togithub.com/langchain-ai/langchain/pull/13232) - Fix Runnable Lambda Afunc Repr by [@hinthornw](https://togithub.com/hinthornw) in [https://github.com/langchain-ai/langchain/pull/13413](https://togithub.com/langchain-ai/langchain/pull/13413) - Use secretstr for api keys for javelin-ai-gateway by [@eyurtsev](https://togithub.com/eyurtsev) in [https://github.com/langchain-ai/langchain/pull/13417](https://togithub.com/langchain-ai/langchain/pull/13417) - FIX: Infer runnable agent single or multi action by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13412](https://togithub.com/langchain-ai/langchain/pull/13412) - bump 336, exp 44 by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13420](https://togithub.com/langchain-ai/langchain/pull/13420) - img update by [@baskaryan](https://togithub.com/baskaryan) in [https://github.com/langchain-ai/langchain/pull/13421](https://togithub.com/langchain-ai/langchain/pull/13421) #### New Contributors - [@IsakNyberg](https://togithub.com/IsakNyberg) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13244](https://togithub.com/langchain-ai/langchain/pull/13244) - [@glad4enkonm](https://togithub.com/glad4enkonm) made their first contribution in [https://github.com/langchain-ai/langchain/pull/12895](https://togithub.com/langchain-ai/langchain/pull/12895) - [@dennisdegreef](https://togithub.com/dennisdegreef) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13079](https://togithub.com/langchain-ai/langchain/pull/13079) - [@marioangst](https://togithub.com/marioangst) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13178](https://togithub.com/langchain-ai/langchain/pull/13178) - [@AzeWZ](https://togithub.com/AzeWZ) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13201](https://togithub.com/langchain-ai/langchain/pull/13201) - [@Guillem96](https://togithub.com/Guillem96) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13184](https://togithub.com/langchain-ai/langchain/pull/13184) - [@shauryr](https://togithub.com/shauryr) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13205](https://togithub.com/langchain-ai/langchain/pull/13205) - [@langchain-infra](https://togithub.com/langchain-infra) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13240](https://togithub.com/langchain-ai/langchain/pull/13240) - [@Govind-S-B](https://togithub.com/Govind-S-B) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13022](https://togithub.com/langchain-ai/langchain/pull/13022) - [@juan-calvo-datatonic](https://togithub.com/juan-calvo-datatonic) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13294](https://togithub.com/langchain-ai/langchain/pull/13294) - [@chevalmuscle](https://togithub.com/chevalmuscle) made their first contribution in [https://github.com/langchain-ai/langchain/pull/11622](https://togithub.com/langchain-ai/langchain/pull/11622) - [@Frank995](https://togithub.com/Frank995) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13260](https://togithub.com/langchain-ai/langchain/pull/13260) - [@takatost](https://togithub.com/takatost) made their first contribution in [https://github.com/langchain-ai/langchain/pull/11604](https://togithub.com/langchain-ai/langchain/pull/11604) - [@iamleonie](https://togithub.com/iamleonie) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13057](https://togithub.com/langchain-ai/langchain/pull/13057) - [@konstantin-spiess](https://togithub.com/konstantin-spiess) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13266](https://togithub.com/langchain-ai/langchain/pull/13266) - [@ruiramos](https://togithub.com/ruiramos) made their first contribution in [https://github.com/langchain-ai/langchain/pull/8920](https://togithub.com/langchain-ai/langchain/pull/8920) - [@mertkayhan](https://togithub.com/mertkayhan) made their first contribution in [https://github.com/langchain-ai/langchain/pull/6863](https://togithub.com/langchain-ai/langchain/pull/6863) - [@kenta-takeuchi](https://togithub.com/kenta-takeuchi) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13358](https://togithub.com/langchain-ai/langchain/pull/13358) - [@celmore25](https://togithub.com/celmore25) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13366](https://togithub.com/langchain-ai/langchain/pull/13366) - [@Sumukh](https://togithub.com/Sumukh) made their first contribution in [https://github.com/langchain-ai/langchain/pull/13232](https://togithub.com/langchain-ai/langchain/pull/13232) **Full Changelog**: https://github.com/langchain-ai/langchain/compare/v0.0.335...v0.0.336 </details> <details> <summary>langchain-ai/langchainjs (langchain)</summary> ### [`v0.0.193`](https://togithub.com/langchain-ai/langchainjs/releases/tag/0.0.193) [Compare Source](https://togithub.com/langchain-ai/langchainjs/compare/0.0.192...0.0.193) #### What's Changed - Release 0.0.192 by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3311](https://togithub.com/langchain-ai/langchainjs/pull/3311) - Use .invoke for all agent docs and examples by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3319](https://togithub.com/langchain-ai/langchainjs/pull/3319) - \[AUTO-GENERATED] Add JSDoc examples to classes. by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3309](https://togithub.com/langchain-ai/langchainjs/pull/3309) - updated langchain stack img to be svg by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3324](https://togithub.com/langchain-ai/langchainjs/pull/3324) - \[AUTO-GENERATED] Add JSDoc examples to classes. by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3325](https://togithub.com/langchain-ai/langchainjs/pull/3325) - \[AUTO-GENERATED] Add JSDoc examples to classes. by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3327](https://togithub.com/langchain-ai/langchainjs/pull/3327) - \[AUTO-GENERATED] Add JSDoc examples to classes. by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3329](https://togithub.com/langchain-ai/langchainjs/pull/3329) - \[AUTO-GENERATED] Add JSDoc examples to classes. by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3330](https://togithub.com/langchain-ai/langchainjs/pull/3330) - Update Ollama functions by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3336](https://togithub.com/langchain-ai/langchainjs/pull/3336) - Remove console.log from googlevertexai-connection.ts by [@raioalbano](https://togithub.com/raioalbano) in [https://github.com/langchain-ai/langchainjs/pull/3322](https://togithub.com/langchain-ai/langchainjs/pull/3322) - Add batch size arg by [@hinthornw](https://togithub.com/hinthornw) in [https://github.com/langchain-ai/langchainjs/pull/3310](https://togithub.com/langchain-ai/langchainjs/pull/3310) - feat: Added support of terms filter in OpenSearch vector store by [@faileon](https://togithub.com/faileon) in [https://github.com/langchain-ai/langchainjs/pull/3312](https://togithub.com/langchain-ai/langchainjs/pull/3312) - Improve MessageContent type by [@netzhuffle](https://togithub.com/netzhuffle) in [https://github.com/langchain-ai/langchainjs/pull/3318](https://togithub.com/langchain-ai/langchainjs/pull/3318) - Add missing PrismaVectorStore filter operators by [@Njuelle](https://togithub.com/Njuelle) in [https://github.com/langchain-ai/langchainjs/pull/3321](https://togithub.com/langchain-ai/langchainjs/pull/3321) #### New Contributors - [@raioalbano](https://togithub.com/raioalbano) made their first contribution in [https://github.com/langchain-ai/langchainjs/pull/3322](https://togithub.com/langchain-ai/langchainjs/pull/3322) - [@faileon](https://togithub.com/faileon) made their first contribution in [https://github.com/langchain-ai/langchainjs/pull/3312](https://togithub.com/langchain-ai/langchainjs/pull/3312) - [@netzhuffle](https://togithub.com/netzhuffle) made their first contribution in [https://github.com/langchain-ai/langchainjs/pull/3318](https://togithub.com/langchain-ai/langchainjs/pull/3318) **Full Changelog**: https://github.com/langchain-ai/langchainjs/compare/0.0.192...0.0.193 ### [`v0.0.192`](https://togithub.com/langchain-ai/langchainjs/releases/tag/0.0.192) [Compare Source](https://togithub.com/langchain-ai/langchainjs/compare/0.0.191...0.0.192) #### What's Changed - Release 0.0.191 by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3300](https://togithub.com/langchain-ai/langchainjs/pull/3300) - Delete artifacts by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3305](https://togithub.com/langchain-ai/langchainjs/pull/3305) - Add missing docs by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3290](https://togithub.com/langchain-ai/langchainjs/pull/3290) - Brace/new api refs build by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3303](https://togithub.com/langchain-ai/langchainjs/pull/3303) - Fix broken fetch usage for CFW by [@dqbd](https://togithub.com/dqbd) in [https://github.com/langchain-ai/langchainjs/pull/3302](https://togithub.com/langchain-ai/langchainjs/pull/3302) - Bump Anthropic + OpenAI versions by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3308](https://togithub.com/langchain-ai/langchainjs/pull/3308) - Hotfix pdf by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3306](https://togithub.com/langchain-ai/langchainjs/pull/3306) - Add PrismaVectorStore filter IN operator by [@Njuelle](https://togithub.com/Njuelle) in [https://github.com/langchain-ai/langchainjs/pull/3304](https://togithub.com/langchain-ai/langchainjs/pull/3304) - feat(apify): support Document\[] return type for mapping function by [@omikader](https://togithub.com/omikader) in [https://github.com/langchain-ai/langchainjs/pull/3262](https://togithub.com/langchain-ai/langchainjs/pull/3262) - Integrate Rockset as a vector store by [@kwadhwa18](https://togithub.com/kwadhwa18) in [https://github.com/langchain-ai/langchainjs/pull/3231](https://togithub.com/langchain-ai/langchainjs/pull/3231) - feat: add file-system based cache by [@vdeturckheim](https://togithub.com/vdeturckheim) in [https://github.com/langchain-ai/langchainjs/pull/3089](https://togithub.com/langchain-ai/langchainjs/pull/3089) #### New Contributors - [@Njuelle](https://togithub.com/Njuelle) made their first contribution in [https://github.com/langchain-ai/langchainjs/pull/3304](https://togithub.com/langchain-ai/langchainjs/pull/3304) - [@kwadhwa18](https://togithub.com/kwadhwa18) made their first contribution in [https://github.com/langchain-ai/langchainjs/pull/3231](https://togithub.com/langchain-ai/langchainjs/pull/3231) - [@vdeturckheim](https://togithub.com/vdeturckheim) made their first contribution in [https://github.com/langchain-ai/langchainjs/pull/3089](https://togithub.com/langchain-ai/langchainjs/pull/3089) **Full Changelog**: https://github.com/langchain-ai/langchainjs/compare/0.0.191...0.0.192 ### [`v0.0.191`](https://togithub.com/langchain-ai/langchainjs/releases/tag/0.0.191) [Compare Source](https://togithub.com/langchain-ai/langchainjs/compare/0.0.190...0.0.191) #### What's Changed - Release 0.0.190 by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3298](https://togithub.com/langchain-ai/langchainjs/pull/3298) **Full Changelog**: https://github.com/langchain-ai/langchainjs/compare/0.0.190...0.0.191 ### [`v0.0.190`](https://togithub.com/langchain-ai/langchainjs/releases/tag/0.0.190) [Compare Source](https://togithub.com/langchain-ai/langchainjs/compare/0.0.189...0.0.190) #### What's Changed - Release 0.0.189 by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3278](https://togithub.com/langchain-ai/langchainjs/pull/3278) - Brace/move syntaxtypes up by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3281](https://togithub.com/langchain-ai/langchainjs/pull/3281) - Brace/api refs css by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3282](https://togithub.com/langchain-ai/langchainjs/pull/3282) - Added runnable to xml agent, moved legacy to hidden page by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3287](https://togithub.com/langchain-ai/langchainjs/pull/3287) - redo intro docs page by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3288](https://togithub.com/langchain-ai/langchainjs/pull/3288) - Add better docstrings for runnables by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3291](https://togithub.com/langchain-ai/langchainjs/pull/3291) - Update HTTP response output parser logic by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3295](https://togithub.com/langchain-ai/langchainjs/pull/3295) **Full Changelog**: https://github.com/langchain-ai/langchainjs/compare/0.0.189...0.0.190 ### [`v0.0.189`](https://togithub.com/langchain-ai/langchainjs/releases/tag/0.0.189) [Compare Source](https://togithub.com/langchain-ai/langchainjs/compare/0.0.188...0.0.189) #### What's Changed - Release 0.0.188 by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3276](https://togithub.com/langchain-ai/langchainjs/pull/3276) - Revert Cohere update by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3277](https://togithub.com/langchain-ai/langchainjs/pull/3277) **Full Changelog**: https://github.com/langchain-ai/langchainjs/compare/0.0.188...0.0.189 ### [`v0.0.188`](https://togithub.com/langchain-ai/langchainjs/releases/tag/0.0.188) [Compare Source](https://togithub.com/langchain-ai/langchainjs/compare/0.0.187...0.0.188) #### What's Changed - Release 0.0.187 by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3255](https://togithub.com/langchain-ai/langchainjs/pull/3255) - Break words on api refs sidebar instead of scrolling by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3265](https://togithub.com/langchain-ai/langchainjs/pull/3265) - Use replaceAll instead of replace when generating operationid. by [@Manouchehri](https://togithub.com/Manouchehri) in [https://github.com/langchain-ai/langchainjs/pull/3267](https://togithub.com/langchain-ai/langchainjs/pull/3267) - Brace/bump cohere by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3263](https://togithub.com/langchain-ai/langchainjs/pull/3263) - Added documentation for few shot prompting by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3122](https://togithub.com/langchain-ai/langchainjs/pull/3122) - Allow custom system prompt for Ollama functions by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/langchainjs/pull/3264](https://togithub.com/langchain-ai/langchainjs/pull/3264) - Brace/add ignore with tsmorph by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3271](https://togithub.com/langchain-ai/langchainjs/pull/3271) - Added rag over code example by [@bracesproul](https://togithub.com/bracesproul) in [https://github.com/langchain-ai/langchainjs/pull/3109](https://togithub.com/langchain-ai/langchainjs/pull/3109) - Meta Llama2 support for BedrockChat by [@shafkevi](https://togithub.com/shafkevi) in [https://github.com/langchain-ai/langchainjs/pull/3260](https://togithub.com/langchain-ai/langchainjs/pull/3260) - Adds HTTP output parser to parse chunks into different content types by [@jacoblee93](https://togithub.com/jacoblee93) in [https://github.com/langchain-ai/lang </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on Monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/autoblocksai/autoblocks-examples).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

@efriis

…on error (langchain-ai#13401) - **Description:** Added a method `fetch_valid_documents` to `WebResearchRetriever` class that will test the connection for every url in `new_urls` and remove those that raise a `ConnectionError`. - **Issue:** [Previous PR](langchain-ai#13353), - **Dependencies:** None, - **Tag maintainer:** @efriis Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.

@efriis

…on error (langchain-ai#13401) - **Description:** Added a method `fetch_valid_documents` to `WebResearchRetriever` class that will test the connection for every url in `new_urls` and remove those that raise a `ConnectionError`. - **Issue:** [Previous PR](langchain-ai#13353), - **Dependencies:** None, - **Tag maintainer:** @efriis Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.

Added method to OK url selection

0b45987

dosubot bot added the 🤖:improvement Medium size change to existing code to handle new use-cases label Nov 15, 2023

efriis reviewed Nov 15, 2023

View reviewed changes

Pedro Custodio and others added 4 commits November 16, 2023 12:16

Added attrib to ignore ConnectionError

9911b4a

Requestes import removal

9698e4b

Re-run make format and make lint

2decedc

Merge branch 'master' into feature/web-retriever-error-handling

ae72453

vercel bot deployed to Preview November 17, 2023 09:21 View deployment

Pedro Custodio and others added 2 commits November 17, 2023 10:03

make format & lint on modified files

a5f8124

Merge branch 'master' into feature/web-retriever-error-handling

1719de8

efriis approved these changes Nov 17, 2023

View reviewed changes

efriis merged commit 0fb5f85 into langchain-ai:master Nov 17, 2023
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebResearchRetriever error handling in urls with connection error #13401

WebResearchRetriever error handling in urls with connection error #13401

pedro-inf-custodio commented Nov 15, 2023 •

edited

vercel bot commented Nov 15, 2023 •

edited

efriis Nov 15, 2023

efriis Nov 15, 2023

pedro-inf-custodio Nov 16, 2023

efriis Nov 15, 2023

efriis left a comment

	def fetch_valid_documents(self, url_list: List[str]) -> List[str]:
	def _fetch_valid_documents(self, url_list: List[str]) -> List[str]:

WebResearchRetriever error handling in urls with connection error #13401

WebResearchRetriever error handling in urls with connection error #13401

Conversation

pedro-inf-custodio commented Nov 15, 2023 • edited

vercel bot commented Nov 15, 2023 • edited

efriis Nov 15, 2023

Choose a reason for hiding this comment

efriis Nov 15, 2023

Choose a reason for hiding this comment

pedro-inf-custodio Nov 16, 2023

Choose a reason for hiding this comment

efriis Nov 15, 2023

Choose a reason for hiding this comment

efriis left a comment

Choose a reason for hiding this comment

pedro-inf-custodio commented Nov 15, 2023 •

edited

vercel bot commented Nov 15, 2023 •

edited