langchain[minor]: Couchbase document loader #4364
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| --- | ||
| hide_table_of_contents: true | ||
| sidebar_class_name: node-only | ||
| --- | ||
|
|
||
| # Couchbase | ||
|
|
||
| [Couchbase](http://couchbase.com/) is an award-winning distributed NoSQL cloud database that delivers unmatched versatility, performance, scalability, and financial value for all of your cloud, mobile, AI, and edge computing applications. | ||
|
|
||
| This guide shows how to use load documents from couchbase database. | ||
|
|
||
| # Installation | ||
|
|
||
| ```bash npm2yarn | ||
| npm install couchbase | ||
| ``` | ||
|
|
||
|
|
||
| ## Usage | ||
|
|
||
| ### Querying for Documents from Couchbase | ||
| For more details on connecting to a Couchbase cluster, please check the [Node.js SDK documentation](https://docs.couchbase.com/nodejs-sdk/current/howtos/managing-connections.html#connection-strings). | ||
|
|
||
| For help with querying for documents using SQL++ (SQL for JSON), please check the [documentation](https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/index.html). | ||
|
|
||
| ```typescript | ||
| import { CouchbaseDocumentLoader } from "langchain/document_loaders/web/couchbase"; | ||
| import { Cluster } from "couchbase" | ||
|
|
||
| const connectionString = "couchbase://localhost"; // valid couchbase connection string | ||
| const dbUsername = "Administrator" // valid database user with read access to the bucket being queried | ||
| const dbPassword = "Password" // password for the database user | ||
|
|
||
| // query is a valid SQL++ query | ||
| const query = ` | ||
| SELECT h.* FROM `travel-sample`.inventory.hotel h | ||
| WHERE h.country = 'United States' | ||
| LIMIT 1 | ||
| `; | ||
| ``` | ||
|
|
||
| ### Connect to Couchbase Cluster | ||
|
|
||
| ```typescript | ||
| const couchbaseClient = await Cluster.connect(connectionString, { | ||
|
There was a problem hiding this comment. Formatting is a bit off here (shouldn't be indented) - you could also write this as a Deno notebook if you'd like? There was a problem hiding this comment. I have fixed the formatting |
||
| username: dbUsername, | ||
| password: dbPassword, | ||
| configProfile: "wanDevelopment" | ||
| }); | ||
|
|
||
| ``` | ||
|
|
||
| ### Create the Loader | ||
|
|
||
| ```typescript | ||
| const loader = new CouchbaseDocumentLoader( | ||
| couchbaseClient, // The connected couchbase cluster client | ||
| query, // A valid SQL++ query which will return the required data | ||
| ); | ||
| ``` | ||
|
|
||
| ### Load Documents | ||
|
|
||
| You can fetch the documents by calling the `load` method of the loader. It will return a list with all the documents. If you want to avoid this blocking call, you can call `lazy_load` method that returns an Iterator. | ||
|
|
||
| ```typescript | ||
| // using load method | ||
| docs = await loader.load(); | ||
| console.log(docs) | ||
| ``` | ||
|
|
||
| ```typescript | ||
| // using lazy_load | ||
| for await (const doc of this.lazyLoad()){ | ||
| console.log(doc); | ||
| break; // break based on required condition | ||
| } | ||
| ``` | ||
|
|
||
| ### Specifying Fields with Content and Metadata | ||
| The fields that are part of the Document content can be specified using the `pageContentFields` parameter. | ||
| The metadata fields for the Document can be specified using the `metadataFields` parameter. | ||
|
|
||
| ```typescript | ||
| const loaderWithSelectedFields = new CouchbaseDocumentLoader( | ||
| couchbaseClient, | ||
| query, | ||
| // pageContentFields | ||
| [ | ||
| "address", | ||
| "name", | ||
| "city", | ||
| "phone", | ||
| "country", | ||
| "geo", | ||
| "description", | ||
| "reviews", | ||
| ], | ||
| ["id"] // metadataFields | ||
| ); | ||
|
|
||
| const filtered_docs = await loaderWithSelectedFields.load() | ||
| console.log(filtered_docs) | ||
| ``` | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,38 @@ | ||||||
| import { test, expect } from "@jest/globals"; | ||||||
| import { Cluster } from "couchbase"; | ||||||
| import { CouchbaseDocumentLoader } from "../web/couchbase.js"; | ||||||
|
|
||||||
| test("Test Couchbase Cluster connection ", async () => { | ||||||
| const connectionString = "<enter-valid-couchbase-connection-string>"; | ||||||
| const databaseUsername = "<enter-valid-couchbase-user>"; | ||||||
| const databasePassword = "<enter-valid-couchbase-password>"; | ||||||
| const query = ` | ||||||
| SELECT h.* FROM \`travel-sample\`.inventory.hotel h | ||||||
| WHERE h.country = 'United States' | ||||||
| LIMIT 10 | ||||||
| `; | ||||||
| const validPageContentFields = ["country", "name", "description"]; | ||||||
| const validMetadataFields = ["id"] | ||||||
|
|
||||||
| const couchbaseClient = await Cluster.connect(connectionString, { | ||||||
| username: databaseUsername, | ||||||
| password: databasePassword, | ||||||
| configProfile: "wanDevelopment" | ||||||
| }); | ||||||
| const loader = new CouchbaseDocumentLoader( | ||||||
| couchbaseClient, | ||||||
| query, | ||||||
| validPageContentFields, | ||||||
| validMetadataFields | ||||||
| ); | ||||||
| const docs = await loader.load(); | ||||||
| console.log(docs); | ||||||
|
There was a problem hiding this comment.
Suggested change
I'd remove console.log There was a problem hiding this comment. Removed logging |
||||||
| expect(docs.length).toBeGreaterThan(0); | ||||||
|
|
||||||
| for (const doc of docs) { | ||||||
| console.log(doc); | ||||||
|
There was a problem hiding this comment. Same here - remove console.log There was a problem hiding this comment. Removed logging |
||||||
| expect(doc.pageContent).not.toBe(""); // Assuming valid page content fields | ||||||
| expect(doc.metadata).toHaveProperty('id'); // Assuming metadata has id field | ||||||
| expect(doc.metadata.id).not.toBe(""); | ||||||
| } | ||||||
| }); | ||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| import { Cluster, QueryResult } from "couchbase"; | ||
| import { Document } from "@langchain/core/documents"; | ||
| import { BaseDocumentLoader, DocumentLoader } from "../base.js"; | ||
|
|
||
| /** | ||
| * loader for couchbase document | ||
| */ | ||
| export class CouchbaseDocumentLoader | ||
| extends BaseDocumentLoader | ||
| implements DocumentLoader | ||
| { | ||
| private cluster: Cluster; | ||
|
|
||
| private query: string; | ||
|
|
||
| private pageContentFields?: string[]; | ||
|
|
||
| private metadataFields?: string[]; | ||
|
|
||
| /** | ||
| * construct Couchbase document loader with a requirement for couchbase cluster client | ||
| * @param client { Cluster } [ couchbase connected client to connect to database ] | ||
| * @param query { string } [ query to get results from while loading the data ] | ||
| * @param pageContentFields { Array<string> } [ filters fields of the document and shows these only ] | ||
| * @param metadataFields { Array<string> } [ metadata fields required ] | ||
| */ | ||
| constructor( | ||
| client: Cluster, | ||
| query: string, | ||
| pageContentFields?: string[], | ||
| metadataFields?: string[] | ||
| ) { | ||
| super(); | ||
| if (!client) { | ||
| throw new Error("Couchbase client cluster must be provided."); | ||
| } | ||
| this.cluster = client; | ||
| this.query = query; | ||
| this.pageContentFields = pageContentFields; | ||
| this.metadataFields = metadataFields; | ||
| } | ||
|
|
||
| /** | ||
| * Function to load document based on query from couchbase | ||
| * @returns {Promise<Document[]>} [ Returns a promise of all the documents as array ] | ||
| */ | ||
| async load(): Promise<Document[]> { | ||
| const documents: Document[] = []; | ||
| for await (const doc of this.lazyLoad()) { | ||
| documents.push(doc); | ||
| } | ||
| return documents; | ||
| } | ||
|
|
||
| /** | ||
| * Function to load documents based on iterator rather than full load | ||
| * @returns {AsyncIterable<Document>} [ Returns an iterator to fetch documents ] | ||
| */ | ||
| async *lazyLoad(): AsyncIterable<Document> { | ||
| // Run SQL++ Query | ||
| const result: QueryResult = await this.cluster.query(this.query); | ||
| for await (const row of result.rows) { | ||
| let { metadataFields, pageContentFields } = this; | ||
|
|
||
| if (!pageContentFields) { | ||
| pageContentFields = Object.keys(row); | ||
| } | ||
|
|
||
| if (!metadataFields) { | ||
| metadataFields = []; | ||
| } | ||
|
|
||
| const metadata = metadataFields.reduce( | ||
| (obj, field) => ({ ...obj, [field]: row[field] }), | ||
| {} | ||
| ); | ||
|
|
||
| const document = pageContentFields | ||
| .map((k) => `${k}: ${JSON.stringify(row[k])}`) | ||
| .join("\n"); | ||
|
|
||
| yield new Document({ | ||
| pageContent: document, | ||
| metadata, | ||
| }); | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix formatting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed formatting