🦜️🏓 LangServe

This unofficial LangChain server based on Langserve is a work in progress. It is not yet ready for production use. Not all features are implemented

Overview​

LangServe helps developers deploy LangChain runnables and chains as a REST API.

This library used express as a peer dependency to provide a server.

In addition, it provides a client that can be used to call into runnables deployed on a server. A javascript client is available in LangChainJS.

Features​

object, and enforced on every API call, with rich error messages

• /invoke/ endpoint with support for many concurrent requests on a single server
• Playground page at /playground/ with streaming output and intermediate steps
• All built with battle-tested open-source JavaScript libraries like Express.
• Use the client SDK to call a LangServe server as if it was a Runnable running locally (or call the HTTP API directly)
• LangServe Hub

Limitations​

• Client callbacks are not yet supported for events that originate on the server

Security​

Installation​

npm install langserve

Examples​

Get your LangServe instance started quickly with LangChain Templates.

For more examples, see the templates index or the examples directory.

Server​

Here's a server that deploys an OpenAI chat model, an Anthropic chat model, and a chain that uses the Anthropic model to tell a joke about a topic.

import express from "express";
import { ChatPromptTemplate } from "langchain/prompts";
import { ChatAnthropic } from "langchain/chat_models/anthropic";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { add_express_routes as add_routes } from "langserve";

const app = express();

add_routes(app, ChatOpenAI(), { path = "/openai" });
add_routes(app, ChatAnthropic(), { path: "/anthropic" });

const model = ChatAnthropic();
const prompt = ChatPromptTemplate.from_template("tell me a joke about {topic}");

add_routes(app, prompt.pipe(model), { path: "/joke" });

app.listen(8000);

Client​

Javascript SDK

import { SystemMessage, HumanMessage } from "langchain/schema";
import { ChatPromptTemplate } from "langchain/prompts";
import { RunnableMap } from "langchain/schema/runnable";
import { RemoteRunnable } from "langchain/runnables/remote";

const openai = new RemoteRunnable("http://localhost:8000/openai/");
const anthropic = new RemoteRunnable("http://localhost:8000/anthropic/");
const joke_chain = new RemoteRunnable("http://localhost:8000/joke/");

await joke_chain.ainvoke({ topic: "parrots" });

const prompt = [
  SystemMessage((content = "Act like either a cat or a parrot.")),
  HumanMessage((content = "Hello!")),
];

// Supports astream

for await (const msg of anthropic.astream(prompt)) {
  console.log(msg);
}

const prompt = ChatPromptTemplate.from_messages([
  ["system", "Tell me a long story about {topic}"],
]);

//  Can define custom chains

const chain = prompt.pipe(
  RunnableMap({
    openai: openai,
    anthropic: anthropic,
  })
);

chain.batch([{ topic: "parrots" }, { topic: "cats" }]);

In TypeScript (requires LangChain.js version 0.0.166 or later):

import { RemoteRunnable } from "langchain/runnables/remote";

const chain = new RemoteRunnable({
  url: `http://localhost:8000/joke/`,
});
const result = await chain.invoke({
  topic: "cats",
});

Python using requests:

import requests
response = requests.post(
    "http://localhost:8000/joke/invoke",
    json={'input': {'topic': 'cats'}}
)
response.json()

You can also use curl:

curl --location --request POST 'http://localhost:8000/joke/invoke' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "input": {
            "topic": "cats"
        }
    }'

Endpoints​

The following code:

...
add_routes(
  app,
  runnable,
  path="/my_runnable",
)

adds of these endpoints to the server:

• POST /my_runnable/invoke - invoke the runnable on a single input

These endpoints match the LangChain Expression Language interface -- please reference this documentation for more details.

Playground​

You can find a playground page for your runnable at /my_runnable/playground/. This exposes a simple UI to configure and invoke your runnable with streaming output and intermediate steps.