HNSWLib-node not found when using in a AWS Lambda function #943

joaotextor · 2023-04-22T14:15:54Z

I recently made a simple Typescript function to create a VectorStore using HNSWLib-node.
It saves the vector store in a folder and then, in another script file, I load and execute a RetrievalQAChain using OpenAI.

Everything was working fine until I decided to put that in a AWS Lambda Function.

My package.json has the following dependencies:

    "hnswlib-node": "^1.4.2",
    "langchain": "^0.0.59",

Also, I double checked and the hnswlib-node folder is inside "node_modules" folder in my lambda function folder.

However, I keep getting the following error (from CloudWatch Logs):

ERROR	Invoke Error 	{"errorType":"Error","errorMessage":"Please install hnswlib-node as a dependency with,
 e.g. `npm install -S hnswlib-node`",
    "stack":["Error: Please install hnswlib-node as a dependency with, e.g. `npm install -S hnswlib-node`","    
    at Function.imports (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:161:19)","
    at async Function.getHierarchicalNSW (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:38:37)","
    at async Function.load (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:123:23)"," 
    at async AMCompanion (/var/task/index.js:18:29)","    at async Runtime.exports.handler (/var/task/index.js:39:22)"]}

Also, this error is not thrown on importing HNSWLib, but only in the following line of code:

const vectorStore = await HNSWLib.load("data", new OpenAIEmbeddings(
            {
                openAIApiKey: process.env.OPENAI_API_KEY,
            } 
        ))

This is my import:

const { HNSWLib } = require("langchain/vectorstores/hnswlib")

It seems I'm not the only one with this problem. See this post

Tried this and didn't work.

Expeted behavior: code would be executed properly, just like when executed on my local machine.

Actual behavior: the error pasted above.

The text was updated successfully, but these errors were encountered:

sourdragon · 2023-04-28T06:31:49Z

@joaotextor If you found a solution please let me know as I am having a similar issue but in vite React js...

kwkr · 2023-04-28T08:58:58Z

I did some quick research and it looks like the error is a bit misleading. The original error that is being suppressed by a catch clause here is Error: /lib64/libm.so.6: version GLIBC_2.29'. I assume it has something to do with the libraries required by the hnsw bindings that are not available on the Lambda runtime.

I found a way to resolve this issue by installing the library in some runtime similar to a Lambda one using Docker. This produces node_modules that contain all the required libs and I was able to run the code within lambda:

sudo docker run --rm -v "$PWD":/var/task lambci/lambda:build-nodejs12.x npm install langchain hnswlib-node

this command produces the required node_modules that can be then uploaded to a Lambda function. Could you try applying this to your upload script and see if it works?

joaotextor · 2023-04-28T14:11:26Z

@joaotextor If you found a solution please let me know as I am having a similar issue but in vite React js...

I did some quick research and it looks like the error is a bit misleading. The original error that is being suppressed by a catch clause here is Error: /lib64/libm.so.6: version GLIBC_2.29'. I assume it has something to do with the libraries required by the hnsw bindings that are not available on the Lambda runtime.

I found a way to resolve this issue by installing the library in some runtime similar to a Lambda one using Docker. This produces node_modules that contain all the required libs and I was able to run the code within lambda:

sudo docker run --rm -v "$PWD":/var/task lambci/lambda:build-nodejs12.x npm install langchain hnswlib-node

this command produces the required node_modules that can be then uploaded to a Lambda function. Could you try applying this to your upload script and see if it works?

Thanks for the reply. I will try this in a few days. If anyone tries it before me, please let us know here.

joaotextor · 2023-04-28T23:20:14Z

I did some quick research and it looks like the error is a bit misleading. The original error that is being suppressed by a catch clause here is Error: /lib64/libm.so.6: version GLIBC_2.29'. I assume it has something to do with the libraries required by the hnsw bindings that are not available on the Lambda runtime.

I found a way to resolve this issue by installing the library in some runtime similar to a Lambda one using Docker. This produces node_modules that contain all the required libs and I was able to run the code within lambda:

sudo docker run --rm -v "$PWD":/var/task lambci/lambda:build-nodejs12.x npm install langchain hnswlib-node

this command produces the required node_modules that can be then uploaded to a Lambda function. Could you try applying this to your upload script and see if it works?

Same error. Deleted node_modules, just to be sure, before running the command.

Got this on CloudWatch:

2023-04-28T22:36:23.785Z	1434cf73-c02d-4c91-83aa-3e2f2fd012ca	ERROR	Invoke Error 	{
    "errorType": "Error",
    "errorMessage": "Please install hnswlib-node as a dependency with, e.g. `npm install -S hnswlib-node`",
    "stack": [
        "Error: Please install hnswlib-node as a dependency with, e.g. `npm install -S hnswlib-node`",
        "    at Function.imports (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:161:19)",
        "    at async Function.getHierarchicalNSW (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:38:37)",
        "    at async Function.load (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:123:23)",
        "    at async AMCompanion (/var/task/index.js:18:29)",
        "    at async Runtime.exports.handler (/var/task/index.js:39:22)"
    ]
}

kwkr · 2023-04-29T09:28:13Z

I ran this using Node 18, x86_64 Lambda. Do you use the same one? You could also try adding console.log directly here in the node_modules that you are about to upload in order to make sure we are actually getting the same errors.

joaotextor · 2023-04-29T15:15:55Z

My lambda function is running on Node 18 and x86_64 too.

I added a console.log on hnswlib.cjs file which successfully ran on my CloudWatch console, specifically before this line:
/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:162:19

Tried running the command again, and noticed some interesting warnings, like this one:

npm WARN notsup Unsupported engine for langchain@0.0.66: wanted: {"node":">=18"} (current: {"node":"12.20.1","npm":"6.14.10"})
npm WARN notsup Not compatible with your version of node/npm: langchain@0.0.66

the command runs lambci/lambda:build-nodejs12.x, and for what I saw in their Repo there is no newer version.

I'm trying to figure it out how to use the newer AWS Lambda base images that has support to nodejs18.x, as stated in the official Repository. If you figure it out how to make this command work using this base images, let me know.

I'll try here too. Thank you very much so far.

toonverbeek · 2023-05-03T14:02:56Z

I'm using Vercel serverless functions, which under the hood utilize AWS Lambda. Interestingly this used to work for me fine, but it stopped working when I switched over to NextJS 13 Route Handlers, which is currently in beta. I'm pretty sure the new NextJS 13 app router also utilizes Lambda, so I don't know what changed compared to the old NextJS implementation specifically that is causing this to stop working.

If anyone else has this working with NextJS Route handlers, let me know.

manduks · 2023-05-12T00:23:55Z

I found the solution to this on Nextjs, user the following

/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,
    experimental: {
        appDir: true,
    },
    webpack: (config) => {
        config.externals = [...config.externals, "hnswlib-node"]

        return config
    },
}

export default nextConfig

benjaminpeto · 2023-06-26T08:13:31Z

@manduks Cheers mate! After a few days of headache with this issue, your solution worked for me as charm. Awesome stuff! Thank you again!

dmb0058 · 2023-07-11T00:29:40Z

Any chance of some more detail on how to do this for a docker newbie?
Essentially this doesn't work for building my lambda layer because most of the node modules require 18.x.

So, can I "docker pull node" to get the latest version, then do some "docker run docker run --rm -v "$PWD":/var/task nodes npm install langchain hnswlib-node my_other_modules" magic to create a node_modules that I can zip up and use as a lambda layer?

Thanks for any help,

David

ali-habibzadeh · 2023-09-06T23:26:34Z

Same issue here.

Runtime.NODEJS_18_X

"hnswlib-node": "^2.0.0",
"langchain": "^0.0.144",

error:

Could not import hnswlib-node. Please install hnswlib-node as a dependency with, e.g. `npm install -S hnswlib-node`.

Error: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/node_modules/hnswlib-node/build/Release/addon.node)
    at HNSWLib.imports (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:271:19)
    at async HNSWLib.getHierarchicalNSW (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:54:37)
    at async HNSWLib.load (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:213:23)

bennofatius · 2023-09-13T08:34:45Z

Same error when trying to use vectorstores in a Netlify function:

const { HNSWLib } = require("langchain/vectorstores/hnswlib");

error:

Error - Could not import hnswlib-node. Please install hnswlib-node as a dependency with, e.g. `npm install -S hnswlib-node`. Error: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/node_modules/hnswlib-node/build/Release/addon.node)

Stack trace
Error: Could not import hnswlib-node. Please install hnswlib-node as a dependency with, e.g. `npm install -S hnswlib-node`.

Error: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/node_modules/hnswlib-node/build/Release/addon.node)
    at HNSWLib.imports (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:271:19)
    at async HNSWLib.getHierarchicalNSW (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:54:37)
    at async HNSWLib.initIndex (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:68:26)
    at async HNSWLib.addVectors (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:95:9)
    at async HNSWLib.fromDocuments (/var/task/node_modules/langchain/dist/vectorstores/hnswlib.cjs:261:9)
    at async exports.handler (/var/task/functions/test.js:38:25)

digredeth · 2023-09-13T22:53:00Z

I found the solution to this on Nextjs, user the following

/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,
    experimental: {
        appDir: true,
    },
    webpack: (config) => {
        config.externals = [...config.externals, "hnswlib-node"]

        return config
    },
}

export default nextConfig

That worked for me, but when I'm dockerize the app the same error arises, not sure whats going on.

fedematteis · 2023-10-04T08:10:40Z

I found the solution to this on Nextjs, user the following

/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,
    experimental: {
        appDir: true,
    },
    webpack: (config) => {
        config.externals = [...config.externals, "hnswlib-node"]

        return config
    },
}

export default nextConfig

worked perfectly for me as I had the same issue, thanks a lot mate

SimRunBot · 2023-10-12T12:30:20Z

I found the solution to this on Nextjs, user the following

/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,
    experimental: {
        appDir: true,
    },
    webpack: (config) => {
        config.externals = [...config.externals, "hnswlib-node"]

        return config
    },
}

export default nextConfig

also worked for me,
seems like a Next.JS issue then...

shridhar-tl · 2023-10-23T07:18:12Z

Same issue happens to me when included in GastBY API also and I have been trying to fix it since 3 days now and still not able to resolve it.

I try to deploy this function in netlify.

benjick · 2023-11-03T13:36:02Z

No fix yet?

cbothra · 2023-11-21T12:59:59Z

@SimRunBot @fedematteis can you tell me what versions of HNSWLib and Langchain you are using? as I did the suggested change in next,.config.js but still having the same error! BTW, my versions of these libraries are:
"hnswlib-node": "^1.4.2"
"langchain": "^0.0.154"
"next": "13.5.1"

Thanks

Riyaancode · 2023-11-22T17:06:43Z

Guys Any Solution?

Riyaancode · 2023-11-24T11:26:56Z

@joaotextor try this this one works for me

import { MemoryVectorStore } from 'langchain/vectorstores/memory';

const embeddings = new OpenAIEmbeddings();
const store = await MemoryVectorStore.fromDocuments(docs, embeddings);

shridhar-tl · 2023-11-27T04:40:41Z

This is the alternate I have been using and it works fine for my use case.

// For creating a new vector from existing documents
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";

const apiKey = process.env.OPEN_AI_API_KEY;

const embeddings = new OpenAIEmbeddings({ openAIApiKey: apiKey });

// You can use one of the following method to create a vector store based on data available
const vectorStore = await MemoryVectorStore.fromDocuments(docs, embeddings);
// or
const vectorStore = await MemoryVectorStore.fromTexts(textArray, metadataArray, embeddings);

// Once it is created you can persist the data as json file either in S3 or any other persistent storage
const jsonToStore = vectorStore.memoryVectors; // I saved this data as json file

// Next time when needed I load this file again using the following code.
const vectorStoreData = loadDataFromJsonFileStoredAbove();
const vectorStore = await MemoryVectorStore.fromExistingIndex(embeddings);
vectorStore.memoryVectors = vectorStoreData;

// Now do a similarity search 
const result = await vectorStore.similaritySearch(query, 4);

soutot · 2024-01-11T13:21:11Z

I found the solution to this on Nextjs, user the following
/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,
    experimental: {
        appDir: true,
    },
    webpack: (config) => {
        config.externals = [...config.externals, "hnswlib-node"]

        return config
    },
}

export default nextConfig
That worked for me, but when I'm dockerize the app the same error arises, not sure whats going on.

Same issue for me. It works if I run via next dev (pnpm run dev), but it fails if I run via docker. Here's my docker-compose container:

chat:
    container_name: chat
    image: node:20.6.1
    build:
      context: .
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
    ports:
      - 3333:3333
    entrypoint: sh -c "pnpm install && pnpm run build && pnpm run dev"
    working_dir: /chat
    volumes:
      - .:/chat

And my DockerFile:

ARG PNPM_VERSION=8.7.1

FROM node:20.6.1

COPY . /chat
WORKDIR /chat

RUN npm install -g pnpm@${PNPM_VERSION}

ENTRYPOINT  pnpm install && pnpm run build && pnpm start

Any suggestions on how to fix this?

Packages versions:
"next": "13.4.19",
"@langchain/community": "^0.0.16",
"@langchain/core": "^0.1.12",
"hnswlib-node": "^2.1.0",
"langchain": "^0.0.209"

Docker version 24.0.6

soutot · 2024-01-11T14:19:34Z

I could get it to work by updating my docker files as the following:

docker-compose:

chat:
    container_name: chat
    build:
      context: .
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
    ports:
      - 3333:3333
    working_dir: /chat
    volumes:
      - .:/chat
      - chat_node_modules:/chat/node_modules

Dockerfile:

ARG PNPM_VERSION=8.7.1

FROM node:20.6.1

RUN npm install -g pnpm@${PNPM_VERSION}

WORKDIR /chat

COPY . .

RUN pnpm install

RUN pnpm run build

CMD [ "pnpm", "run", "start"]

Then on terminal I first ran

docker-compose build --no-cache

Then

docker-compose run chat sh

I haven't dived deep to check if just a single part of this is enough or if everything is needed, though. But now it's working

dosubot · 2024-04-11T16:01:42Z

Hi, @joaotextor,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. It looks like the issue involves the HNSWLib-node library not being found when the code is executed in an AWS Lambda function, despite being present in the node_modules folder. There were discussions about potential solutions, including the use of Docker to produce node_modules containing all the required libs, but it seems the error persisted for you and other users even after trying the suggested command. Some users found workarounds, while others are still seeking a resolution.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

joaotextor · 2024-04-12T11:59:37Z

Hi, @joaotextor,

I'm helping the langchainjs team manage their backlog and am marking this issue as stale. It looks like the issue involves the HNSWLib-node library not being found when the code is executed in an AWS Lambda function, despite being present in the node_modules folder. There were discussions about potential solutions, including the use of Docker to produce node_modules containing all the required libs, but it seems the error persisted for you and other users even after trying the suggested command. Some users found workarounds, while others are still seeking a resolution.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

I'm not using it anymore, so I can't say if this is still an issue.

dosubot · 2024-04-12T12:01:50Z

Hi @joaotextor,

Thank you for your response. We appreciate your input. Since you're no longer using it, we'll go ahead and close the issue. If you ever encounter similar issues in the future, feel free to reach out. Thank you again!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

gregorym mentioned this issue Jun 27, 2023

HNSWLib not compatible with 0.0.96 #1764

Closed

This was referenced Jul 11, 2023

Using Faiss-node in aws having problem working fine locally ewfian/faiss-node#25

Closed

Langchain doesn't work on AWS Lambda with either faiss or hnswlib #1930

Closed

PierrickLozach mentioned this issue Jan 11, 2024

Still having an issue using FaissStore in AWS lambda #3941

Closed

froggy1014 mentioned this issue Mar 13, 2024

Feature/extract keyword nl Swipe-Jeju/swiping-jeju-frontend_v2#15

Merged

4 tasks

dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Apr 11, 2024

dosubot bot removed the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Apr 12, 2024

dosubot bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HNSWLib-node not found when using in a AWS Lambda function #943

HNSWLib-node not found when using in a AWS Lambda function #943

joaotextor commented Apr 22, 2023

sourdragon commented Apr 28, 2023 •

edited

Loading

kwkr commented Apr 28, 2023 •

edited

Loading

joaotextor commented Apr 28, 2023

joaotextor commented Apr 28, 2023 •

edited

Loading

kwkr commented Apr 29, 2023

joaotextor commented Apr 29, 2023

toonverbeek commented May 3, 2023

manduks commented May 12, 2023

benjaminpeto commented Jun 26, 2023

dmb0058 commented Jul 11, 2023

ali-habibzadeh commented Sep 6, 2023

bennofatius commented Sep 13, 2023

digredeth commented Sep 13, 2023

fedematteis commented Oct 4, 2023

SimRunBot commented Oct 12, 2023

shridhar-tl commented Oct 23, 2023

benjick commented Nov 3, 2023

cbothra commented Nov 21, 2023

Riyaancode commented Nov 22, 2023

Riyaancode commented Nov 24, 2023

shridhar-tl commented Nov 27, 2023 •

edited

Loading

soutot commented Jan 11, 2024 •

edited

Loading

soutot commented Jan 11, 2024 •

edited

Loading

dosubot bot commented Apr 11, 2024

joaotextor commented Apr 12, 2024

dosubot bot commented Apr 12, 2024

About Dosu

HNSWLib-node not found when using in a AWS Lambda function #943

HNSWLib-node not found when using in a AWS Lambda function #943

Comments

joaotextor commented Apr 22, 2023

sourdragon commented Apr 28, 2023 • edited Loading

kwkr commented Apr 28, 2023 • edited Loading

joaotextor commented Apr 28, 2023

joaotextor commented Apr 28, 2023 • edited Loading

kwkr commented Apr 29, 2023

joaotextor commented Apr 29, 2023

toonverbeek commented May 3, 2023

manduks commented May 12, 2023

benjaminpeto commented Jun 26, 2023

dmb0058 commented Jul 11, 2023

ali-habibzadeh commented Sep 6, 2023

bennofatius commented Sep 13, 2023

digredeth commented Sep 13, 2023

fedematteis commented Oct 4, 2023

SimRunBot commented Oct 12, 2023

shridhar-tl commented Oct 23, 2023

benjick commented Nov 3, 2023

cbothra commented Nov 21, 2023

Riyaancode commented Nov 22, 2023

Riyaancode commented Nov 24, 2023

shridhar-tl commented Nov 27, 2023 • edited Loading

soutot commented Jan 11, 2024 • edited Loading

soutot commented Jan 11, 2024 • edited Loading

dosubot bot commented Apr 11, 2024

joaotextor commented Apr 12, 2024

dosubot bot commented Apr 12, 2024

About Dosu

sourdragon commented Apr 28, 2023 •

edited

Loading

kwkr commented Apr 28, 2023 •

edited

Loading

joaotextor commented Apr 28, 2023 •

edited

Loading

shridhar-tl commented Nov 27, 2023 •

edited

Loading

soutot commented Jan 11, 2024 •

edited

Loading

soutot commented Jan 11, 2024 •

edited

Loading