ImportError: Could not import transformers python package.

[Qunlexie]

Describe the bug
When running via local llm in Ollama, ( Llama3.2 ) with SmartScraperMultiGraph it keeps throwing this error.

ImportError: Could not import transformers python package. This is needed in order to calculate get_token_ids. Please install it with pip install transformers. I have installed transformers manually with pip in the environment but still encounter this issue.

This is related to this unresolved but closed issue: #719.

[VinciGit00]

does it work with normal smart scraper? can see your config?

[Qunlexie]

I can confirm that this works for SmartScraperGraph but fails for SmartScraperMultiGraph

graph_config = {
   'llm': {
      'model': 'ollama/llama3.2',
      'temperature': 0.0,
      'format': 'json',
      'model_tokens': 2000,
      'base_url': 'http://localhost:11434'
   },
   'verbose': True,
   'headless': False
}

[Qunlexie]

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/base.py", line 62, in get_tokenizer
    from transformers import GPT2TokenizerFast  # type: ignore[import]
ImportError: cannot import name 'GPT2TokenizerFast' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 21, in <module>
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/smart_scraper_multi_graph.py", line 107, in run
    self.final_state, self.execution_info = self.graph.execute(inputs)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 321, in execute
    return self._execute_standard(initial_state)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 270, in _execute_standard
    raise e
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 243, in _execute_standard
    result, node_exec_time, cb_data = self._execute_node(
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 171, in _execute_node
    result = current_node.execute(state)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/nodes/graph_iterator_node.py", line 73, in execute
    state = asyncio.run(self._async_execute(state, batchsize))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/nodes/graph_iterator_node.py", line 136, in _async_execute
    answers = await tqdm.gather(
  File "/usr/local/lib/python3.10/dist-packages/tqdm/asyncio.py", line 79, in gather
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
  File "/usr/local/lib/python3.10/dist-packages/tqdm/asyncio.py", line 79, in <listcomp>
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
  File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/usr/local/lib/python3.10/dist-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/nodes/graph_iterator_node.py", line 126, in _async_run
    return await asyncio.to_thread(graph.run)
  File "/usr/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/smart_scraper_graph.py", line 280, in run
    self.final_state, self.execution_info = self.graph.execute(inputs)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 321, in execute
    return self._execute_standard(initial_state)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 270, in _execute_standard
    raise e
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 243, in _execute_standard
    result, node_exec_time, cb_data = self._execute_node(
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/base_graph.py", line 171, in _execute_node
    result = current_node.execute(state)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/nodes/parse_node.py", line 85, in execute
    chunks = split_text_into_chunks(text=docs_transformed.page_content,
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/utils/split_text_into_chunks.py", line 28, in split_text_into_chunks
    chunks = chunk(text=text,
  File "/usr/local/lib/python3.10/dist-packages/semchunk/semchunk.py", line 149, in chunk
    if token_counter(split) > local_chunk_size:
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/utils/split_text_into_chunks.py", line 24, in count_tokens
    return num_tokens_calculus(text, model)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/utils/tokenizer.py", line 30, in num_tokens_calculus
    num_tokens = num_tokens_fn(string, llm_model)
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/utils/tokenizers/tokenizer_ollama.py", line 26, in num_tokens_ollama
    tokens = llm_model.get_num_tokens(text)
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/base.py", line 366, in get_num_tokens
    return len(self.get_token_ids(text))
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/base.py", line 353, in get_token_ids
    return _get_token_ids_default_method(text)
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/base.py", line 77, in _get_token_ids_default_method
    tokenizer = get_tokenizer()
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/base.py", line 69, in get_tokenizer
    raise ImportError(msg) from e
ImportError: Could not import transformers python package. This is needed in order to calculate get_token_ids. Please install it with `pip install transformers`.

processing graph instances:   0%|          | 0/4 [00:50<?, ?it/s]

Error Output:

[VinciGit00]

please try with the new beta

[PeriniM]

Hey @Qunlexie, this is fixed with the new release, please try it out!

[Qunlexie]

I’ll check it out!