AWS Lambda - Read Only

[Joepetey]

Langchain trys to download the GPT2FastTokenizer when I run a chain. In a Lambda function this doesnt work because the Lambda is read only. Any run into this, or know how to fix this?

[ellisonbg]

@3coins may be able to help with this.

[Joepetey]

That would be great if I could get some help with this, or some documentation!

Thank you!

[juankysoriano]

Ah, I have also seen this issue. As I was mentioning on the langchain discord:

OpenAIChat uses GPT2TokenizerFast.from_pretrained("gpt2") for calculating the number of tokens, while tiktoken.get_encoding("gpt2") uses tiktoken.

The first one is not usable when going serverless (AWS lambdas, or Google Cloud Functions). And it might represent a deal breaker for a more than valid use case. I'd even say a desirable use case, the serverless scenario.

So far I have been able to find a workaround, but it's not ideal and will be likely to suffer breaking changes as they continue updating langchain (which happens very frequently).

The workaround is to create a custom implementation of OpenAIChat overriding the problematic function.

class TikOpenAIChat(OpenAIChat):
    def get_num_tokens(self, text: str) -> int:
        import tiktoken
        enc = tiktoken.get_encoding("gpt2")
        tokenized_text = enc.encode(text)
        return len(tokenized_text)

I hope this helps on your scenario @Joepetey , and let's aim for an official solution from @hwchase17

[juankysoriano]

Alternatively we could continue using GPT2FastTokenizer but we need facilitate a way of selecting where are those models going to be stored, serverless functions has typically some available space on /tmp.

Now, having to download everytime (serverless is stateless) is going to make things very slow I believe.

I am talking btw without being an expert.

[juankysoriano]

I have opened a PR that would solve this issue.

#1457

[Joepetey]

Wow thank you @juankysoriano this is so helpful!

[hwchase17]

thanks @juankysoriano ! merging this in now

[juankysoriano]

solved by #1457

[Joepetey]

Quick question, when can we expect this to be available in a release?

[juankysoriano]

I don't know but the owner is typically very fast on doing releases, they are very frequent. It shouldn't take more than a couple of days in my experience. Will see

[juankysoriano]

@Joepetey if the recent release, 0.0.102 solved your problem, the issue can be closed.

[gmpetrov]

Alternatively we could continue using GPT2FastTokenizer but we need facilitate a way of selecting where are those models going to be stored, serverless functions has typically some available space on /tmp.

Now, having to download everytime (serverless is stateless) is going to make things very slow I believe.

I am talking btw without being an expert.

With AWS you can mount an EFS to a lambda to cache a pre-trained model.
Checkout this example https://github.com/aws-samples/zero-administration-inference-with-aws-lambda-for-hugging-face

Also if you're using HuggingFaceEmbeddings (which uses sentence_transformers.SentenceTransformer) you need to use SENTENCE_TRANSFORMERS_HOME env variable to download the model at a specific location

[Joepetey]

Thank you Juanky, it worked!