Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(apify): pass through Document metadata generic type #3121

Merged
merged 6 commits into from
Nov 7, 2023
Merged

feat(apify): pass through Document metadata generic type #3121

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4713ad8
feat(apify): pass through Document generics
omikader Nov 1, 2023
e218d4f
Merge branch 'main' of https://github.com/hwchase17/langchainjs into …
jacoblee93 Nov 2, 2023
f1c88f8
Fix lint warnings
jacoblee93 Nov 2, 2023
5266138
misc: remove unnecessary export
omikader Nov 6, 2023
bf60fa0
misc: re-export type
omikader Nov 6, 2023
d1cfbef
Merge branch 'main' into main
omikader Nov 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 14 additions & 15 deletions langchain/src/document_loaders/web/apify_dataset.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,27 @@
import {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is flagging the addition of an environment variable requirement via getEnvironmentVariable in the code. Maintainers should review this change to ensure it is implemented correctly and does not introduce any security or compatibility issues.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI I didn't add this call. It was already in the code

ActorCallOptions,
ApifyClient,
ApifyClientOptions,
ActorCallOptions,
TaskCallOptions,
} from "apify-client";

import { Document } from "../../document.js";
import { BaseDocumentLoader, DocumentLoader } from "../base.js";
import { Document } from "../../document.js";
import { getEnvironmentVariable } from "../../util/env.js";

/**
* A type that represents a function that takes a single object (an Apify
* dataset item) and converts it to an instance of the Document class.
*/
export type ApifyDatasetMappingFunction = (
item: Record<string | number, unknown>
) => Document;
export type ApifyDatasetMappingFunction<Metadata extends Record<string, any>> =
(item: Record<string | number, unknown>) => Document<Metadata>;

/**
* A class that extends the BaseDocumentLoader and implements the
* DocumentLoader interface. It represents a document loader that loads
* documents from an Apify dataset.
*/
export class ApifyDatasetLoader
export class ApifyDatasetLoader<Metadata extends Record<string, any>>
extends BaseDocumentLoader
implements DocumentLoader
{
Expand All @@ -32,12 +31,12 @@ export class ApifyDatasetLoader

protected datasetMappingFunction: (
item: Record<string | number, unknown>
) => Document;
) => Document<Metadata>;

constructor(
datasetId: string,
config: {
datasetMappingFunction: ApifyDatasetMappingFunction;
datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>;
clientOptions?: ApifyClientOptions;
}
) {
Expand All @@ -63,7 +62,7 @@ export class ApifyDatasetLoader
* instances.
* @returns An array of Document instances.
*/
async load(): Promise<Document[]> {
async load(): Promise<Document<Metadata>[]> {
const datasetItems = (
await this.apifyClient.dataset(this.datasetId).listItems({ clean: true })
).items;
Expand All @@ -78,15 +77,15 @@ export class ApifyDatasetLoader
* @param options.datasetMappingFunction A function that takes a single object (an Apify dataset item) and converts it to an instance of the Document class.
* @returns An instance of `ApifyDatasetLoader` with the results from the Actor run.
*/
static async fromActorCall(
static async fromActorCall<Metadata extends Record<string, any>>(
actorId: string,
input: Record<string | number, unknown>,
config: {
callOptions?: ActorCallOptions;
clientOptions?: ApifyClientOptions;
datasetMappingFunction: ApifyDatasetMappingFunction;
datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>;
}
): Promise<ApifyDatasetLoader> {
): Promise<ApifyDatasetLoader<Metadata>> {
const apifyApiToken = ApifyDatasetLoader._getApifyApiToken(
config.clientOptions
);
Expand All @@ -110,15 +109,15 @@ export class ApifyDatasetLoader
* @param options.datasetMappingFunction A function that takes a single object (an Apify dataset item) and converts it to an instance of the Document class.
* @returns An instance of `ApifyDatasetLoader` with the results from the task's run.
*/
static async fromActorTaskCall(
static async fromActorTaskCall<Metadata extends Record<string, any>>(
taskId: string,
input: Record<string | number, unknown>,
config: {
callOptions?: TaskCallOptions;
clientOptions?: ApifyClientOptions;
datasetMappingFunction: ApifyDatasetMappingFunction;
datasetMappingFunction: ApifyDatasetMappingFunction<Metadata>;
}
): Promise<ApifyDatasetLoader> {
): Promise<ApifyDatasetLoader<Metadata>> {
const apifyApiToken = ApifyDatasetLoader._getApifyApiToken(
config.clientOptions
);
Expand Down