llm.batch method fails with max_concurrency in config

[kaihe-stori]

System Info

• langchain version: 0.0.320
• python: 3.9.17

Who can help?

No response

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

Similar to the guide for chat_models.batch, but if we use llm instead, batch and abatch methods will throw exceptions.

Here is an example for batch

model = OpenAI()
prompt = PromptTemplate.from_template("tell me a joke about {topic}")
chain = prompt | model
chain.batch([{"topic": "bears"}, {"topic": "cats"}], config={"max_concurrency": 5})

You get the following error for dead loop exceeding recursion limit

...
--> 300         for output in self.batch(
    301             batch, config=config, return_exceptions=return_exceptions, **kwargs
    302         )
    303     ]

File [/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:273](https://file+.vscode-resource.vscode-cdn.net/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:273), in BaseLLM.batch(self, inputs, config, return_exceptions, **kwargs)
    270 if not inputs:
    271     return []
--> 273 config = get_config_list(config, len(inputs))
    274 max_concurrency = config[0].get("max_concurrency")
    276 if max_concurrency is None:

File [/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/schema/runnable/config.py:115](https://file+.vscode-resource.vscode-cdn.net/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/schema/runnable/config.py:115), in get_config_list(config, length)
    108 def get_config_list(
    109     config: Optional[Union[RunnableConfig, List[RunnableConfig]]], length: int
    110 ) -> List[RunnableConfig]:
    111     """
    112     Helper method to get a list of configs from a single config or a list of
    113     configs, useful for subclasses overriding batch() or abatch().
    114     """
--> 115     if length < 0:
    116         raise ValueError(f"length must be >= 0, but got {length}")
    117     if isinstance(config, list) and len(config) != length:

RecursionError: maximum recursion depth exceeded in comparison

Or if you set max_concurrency to be smaller than the length of the prompts, say 1

chain.batch([{"topic": "bears"}, {"topic": "cats"}], config={"max_concurrency": 1})

ValueError                                Traceback (most recent call last)
...
File [/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/schema/runnable/base.py:1271](https://file+.vscode-resource.vscode-cdn.net/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/schema/runnable/base.py:1271), in RunnableSequence.batch(self, inputs, config, return_exceptions, **kwargs)
   1269     else:
   1270         for i, step in enumerate(self.steps):
-> 1271             inputs = step.batch(
   1272                 inputs,
   1273                 [
   1274                     # each step a child run of the corresponding root run
   1275                     patch_config(
   1276                         config, callbacks=rm.get_child(f"seq:step:{i+1}")
   1277                     )
   1278                     for rm, config in zip(run_managers, configs)
   1279                 ],
   1280             )
   1282 # finish the root runs
   1283 except BaseException as e:

File [/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:297](https://file+.vscode-resource.vscode-cdn.net/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:297), in BaseLLM.batch(self, inputs, config, return_exceptions, **kwargs)
    292 else:
    293     batches = [
    294         inputs[i : i + max_concurrency]
    295         for i in range(0, len(inputs), max_concurrency)
    296     ]
--> 297     return [
    298         output
    299         for batch in batches
    300         for output in self.batch(
    301             batch, config=config, return_exceptions=return_exceptions, **kwargs
    302         )
    303     ]

File [/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:300](https://file+.vscode-resource.vscode-cdn.net/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:300), in <listcomp>(.0)
    292 else:
    293     batches = [
    294         inputs[i : i + max_concurrency]
    295         for i in range(0, len(inputs), max_concurrency)
    296     ]
    297     return [
    298         output
    299         for batch in batches
--> 300         for output in self.batch(
    301             batch, config=config, return_exceptions=return_exceptions, **kwargs
    302         )
    303     ]

File [/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:273](https://file+.vscode-resource.vscode-cdn.net/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/llms/base.py:273), in BaseLLM.batch(self, inputs, config, return_exceptions, **kwargs)
    270 if not inputs:
    271     return []
--> 273 config = get_config_list(config, len(inputs))
    274 max_concurrency = config[0].get("max_concurrency")
    276 if max_concurrency is None:

File [/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/schema/runnable/config.py:118](https://file+.vscode-resource.vscode-cdn.net/usr/local/Caskroom/miniconda/base/envs/chatgpt/lib/python3.9/site-packages/langchain/schema/runnable/config.py:118), in get_config_list(config, length)
    116     raise ValueError(f"length must be >= 0, but got {length}")
    117 if isinstance(config, list) and len(config) != length:
--> 118     raise ValueError(
    119         f"config must be a list of the same length as inputs, "
    120         f"but got {len(config)} configs for {length} inputs"
    121     )
    123 return (
    124     list(map(ensure_config, config))
    125     if isinstance(config, list)
    126     else [patch_config(config, copy_locals=True) for _ in range(length)]
    127 )

ValueError: config must be a list of the same length as inputs, but got 2 configs for 1 inputs

Similar errors for abatch with max_concurrecy.

I believe the bug lies in

langchain/libs/langchain/langchain/llms/base.py

Line 297 in f94e24d

config = [{**c, "max_concurrency": None} for c in config] # type: ignore[misc]

where config list length is inconsistent with the inputs length after chopping up by max_concurrency. The fix could be as simple as

config = {**config[0], "max_concurrency": None}  # no list is needed

Expected behavior

No error, just like with chat_models

[dosubot]

🤖

Hello @kaihe-stori,

I'm Dosu, an AI here to assist you with your LangChain issues while you're waiting for a human maintainer. I'm here to help you solve bugs, answer questions, and guide you on your journey to becoming a contributor.

From your description, it seems like you've encountered a recursion error when using the llm.batch method with the max_concurrency parameter in the config. This issue arises due to the way the max_concurrency parameter is handled in the batch method.

The max_concurrency parameter is used to control the maximum number of concurrent tasks that can be run at a time. If max_concurrency is specified, the inputs will be divided into batches of size max_concurrency, and these batches will be processed sequentially. However, in the current implementation, the max_concurrency value is set to None for the recursive calls, which means that the concurrency limit is not enforced for these calls. This can lead to too many concurrent tasks being created, which can cause the method to fail.

Your proposed fix of setting config = {**config[0], "max_concurrency": None} seems to be a valid solution. This would ensure that the max_concurrency value is not passed to the recursive calls, preventing the creation of too many concurrent tasks.

However, please note that this is a temporary workaround and might not be the best solution in the long run. The LangChain team might need to revisit the handling of the max_concurrency parameter in the batch method to provide a more robust solution.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/llms/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.