NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

[MRX2005nzr]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from llama_index.core.indices.service_context import ServiceContext
from llama_index.core import Settings
from llama_index.core.indices.knowledge_graph.base import KnowledgeGraphIndex
from llama_index.core.graph_stores import SimpleGraphStore
from llama_index.core import download_loader
from langchain.chat_models import AzureChatOpenAI

os.environ["API_TYPE"] = "azure"
os.environ["OPENAI_API_VERSION"] = "2023-05-15"
os.environ["OPENAI_API_BASE"] = "https://###-app.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "###############"

# define LLM
llm = AzureChatOpenAI(
            temperature=0,
            deployment_name="Test-trans-01",
            #streaming=True,
        )
service_context = ServiceContext.from_defaults(llm=llm, chunk_size=512)

from llama_index.core.indices.loading import load_index_from_storage
from llama_hub.youtube_transcript import YoutubeTranscriptReader

try:

    storage_context = StorageContext.from_defaults(persist_dir='./storage_graph', graph_store=graph_store)
    kg_index = load_index_from_storage(
        storage_context=storage_context,
        service_context=service_context,
        max_triplets_per_chunk=15,
        space_name=space_name,
        edge_types=edge_types,
        rel_prop_names=rel_prop_names,
        tags=tags,
        verbose=True,
    )
    index_loaded = True
except:
    index_loaded = False

if not index_loaded:
    
    WikipediaReader = download_loader("WikipediaReader")
    loader = WikipediaReader()
    wiki_documents = loader.load_data(pages=['Philadelphia Phillies'], auto_suggest=False)
    print(f'Loaded {len(wiki_documents)} documents')

    youtube_loader = YoutubeTranscriptReader()
    youtube_documents = youtube_loader.load_data(ytlinks=['https://www.youtube.com/watch?v=k-HTQ8T7oVw'])    
    print(f'Loaded {len(youtube_documents)} YouTube documents')

    kg_index = KnowledgeGraphIndex.from_documents(
        documents=wiki_documents + youtube_documents,
        storage_context=storage_context,
        max_triplets_per_chunk=15,
        service_context=service_context,
        space_name=space_name,
        edge_types=edge_types,
        rel_prop_names=rel_prop_names,
        tags=tags,
        include_embeddings=True,
    )
    
    kg_index.storage_context.persist(persist_dir='./storage_graph')

Description

I'm trying to load data from Wiki and Youtube and create KG index. But I got problems in the last step. The error information is shown as follows (maybe a little long) :

---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
Cell In[14], line 30
     27 youtube_documents = youtube_loader.load_data(ytlinks=['https://www.youtube.com/watch?v=k-HTQ8T7oVw'])    
     28 print(f'Loaded {len(youtube_documents)} YouTube documents')
---> 30 kg_index = KnowledgeGraphIndex.from_documents(
     31     documents=wiki_documents + youtube_documents,
     32     storage_context=storage_context,
     33     max_triplets_per_chunk=15,
     34     service_context=service_context,
     35     space_name=space_name,
     36     edge_types=edge_types,
     37     rel_prop_names=rel_prop_names,
     38     tags=tags,
     39     include_embeddings=True,
     40 )
     42 kg_index.storage_context.persist(persist_dir='./storage_graph')

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/base.py:142, in BaseIndex.from_documents(cls, documents, storage_context, show_progress, callback_manager, transformations, service_context, **kwargs)
    133     docstore.set_document_hash(doc.get_doc_id(), doc.hash)
    135 nodes = run_transformations(
    136     documents,  # type: ignore
    137     transformations,
    138     show_progress=show_progress,
    139     **kwargs,
    140 )
--> 142 return cls(
    143     nodes=nodes,
    144     storage_context=storage_context,
    145     callback_manager=callback_manager,
    146     show_progress=show_progress,
    147     transformations=transformations,
    148     service_context=service_context,
    149     **kwargs,
    150 )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/knowledge_graph/base.py:99, in KnowledgeGraphIndex.__init__(self, nodes, objects, index_struct, llm, embed_model, storage_context, kg_triple_extract_template, max_triplets_per_chunk, include_embeddings, show_progress, max_object_length, kg_triplet_extract_fn, service_context, **kwargs)
     94 self._llm = llm or llm_from_settings_or_context(Settings, service_context)
     95 self._embed_model = embed_model or embed_model_from_settings_or_context(
     96     Settings, service_context
     97 )
---> 99 super().__init__(
    100     nodes=nodes,
    101     index_struct=index_struct,
    102     service_context=service_context,
    103     storage_context=storage_context,
    104     show_progress=show_progress,
    105     objects=objects,
    106     **kwargs,
    107 )
    109 # TODO: legacy conversion - remove in next release
    110 if (
    111     len(self.index_struct.table) > 0
    112     and isinstance(self.graph_store, SimpleGraphStore)
    113     and len(self.graph_store._data.graph_dict) == 0
    114 ):

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/base.py:91, in BaseIndex.__init__(self, nodes, objects, index_struct, storage_context, callback_manager, transformations, show_progress, service_context, **kwargs)
     89 if index_struct is None:
     90     nodes = nodes or []
---> 91     index_struct = self.build_index_from_nodes(
     92         nodes + objects  # type: ignore
     93     )
     94 self._index_struct = index_struct
     95 self._storage_context.index_store.add_index_struct(self._index_struct)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/base.py:209, in BaseIndex.build_index_from_nodes(self, nodes)
    207 """Build the index from nodes."""
    208 self._docstore.add_documents(nodes, allow_update=True)
--> 209 return self._build_index_from_nodes(nodes)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/knowledge_graph/base.py:217, in KnowledgeGraphIndex._build_index_from_nodes(self, nodes)
    214 if self.include_embeddings:
    215     triplet_texts = [str(t) for t in triplets]
--> 217     embed_outputs = self._embed_model.get_text_embedding_batch(
    218         triplet_texts, show_progress=self._show_progress
    219     )
    220     for rel_text, rel_embed in zip(triplet_texts, embed_outputs):
    221         index_struct.add_to_embedding_dict(rel_text, rel_embed)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/base/embeddings/base.py:255, in BaseEmbedding.get_text_embedding_batch(self, texts, show_progress, **kwargs)
    249 if idx == len(texts) - 1 or len(cur_batch) == self.embed_batch_size:
    250     # flush
    251     with self.callback_manager.event(
    252         CBEventType.EMBEDDING,
    253         payload={EventPayload.SERIALIZED: self.to_dict()},
    254     ) as event:
--> 255         embeddings = self._get_text_embeddings(cur_batch)
    256         result_embeddings.extend(embeddings)
    257         event.on_end(
    258             payload={
    259                 EventPayload.CHUNKS: cur_batch,
    260                 EventPayload.EMBEDDINGS: embeddings,
    261             },
    262         )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/embeddings/openai/base.py:419, in OpenAIEmbedding._get_text_embeddings(self, texts)
    412 """Get text embeddings.
    413 
    414 By default, this is a wrapper around _get_text_embedding.
    415 Can be overridden for batch queries.
    416 
    417 """
    418 client = self._get_client()
--> 419 return get_embeddings(
    420     client,
    421     texts,
    422     engine=self._text_engine,
    423     **self.additional_kwargs,
    424 )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:289, in BaseRetrying.wraps.<locals>.wrapped_f(*args, **kw)
    287 @functools.wraps(f)
    288 def wrapped_f(*args: t.Any, **kw: t.Any) -> t.Any:
--> 289     return self(f, *args, **kw)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:379, in Retrying.__call__(self, fn, *args, **kwargs)
    377 retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)
    378 while True:
--> 379     do = self.iter(retry_state=retry_state)
    380     if isinstance(do, DoAttempt):
    381         try:

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:325, in BaseRetrying.iter(self, retry_state)
    323     retry_exc = self.retry_error_cls(fut)
    324     if self.reraise:
--> 325         raise retry_exc.reraise()
    326     raise retry_exc from fut.exception()
    328 if self.wait:

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:158, in RetryError.reraise(self)
    156 def reraise(self) -> t.NoReturn:
    157     if self.last_attempt.failed:
--> 158         raise self.last_attempt.result()
    159     raise self

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/concurrent/futures/_base.py:451, in Future.result(self, timeout)
    449     raise CancelledError()
    450 elif self._state == FINISHED:
--> 451     return self.__get_result()
    453 self._condition.wait(timeout)
    455 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
    401 if self._exception:
    402     try:
--> 403         raise self._exception
    404     finally:
    405         # Break a reference cycle with the exception in self._exception
    406         self = None

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:382, in Retrying.__call__(self, fn, *args, **kwargs)
    380 if isinstance(do, DoAttempt):
    381     try:
--> 382         result = fn(*args, **kwargs)
    383     except BaseException:  # noqa: B902
    384         retry_state.set_exception(sys.exc_info())  # type: ignore[arg-type]

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/embeddings/openai/base.py:180, in get_embeddings(client, list_of_text, engine, **kwargs)
    176 assert len(list_of_text) <= 2048, "The batch size should not be larger than 2048."
    178 list_of_text = [text.replace("\n", " ") for text in list_of_text]
--> 180 data = client.embeddings.create(input=list_of_text, model=engine, **kwargs).data
    181 return [d.embedding for d in data]

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/resources/embeddings.py:113, in Embeddings.create(self, input, model, dimensions, encoding_format, user, extra_headers, extra_query, extra_body, timeout)
    107         embedding.embedding = np.frombuffer(  # type: ignore[no-untyped-call]
    108             base64.b64decode(data), dtype="float32"
    109         ).tolist()
    111     return obj
--> 113 return self._post(
    114     "/embeddings",
    115     body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams),
    116     options=make_request_options(
    117         extra_headers=extra_headers,
    118         extra_query=extra_query,
    119         extra_body=extra_body,
    120         timeout=timeout,
    121         post_parser=parser,
    122     ),
    123     cast_to=CreateEmbeddingResponse,
    124 )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/_base_client.py:1200, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
   1186 def post(
   1187     self,
   1188     path: str,
   (...)
   1195     stream_cls: type[_StreamT] | None = None,
   1196 ) -> ResponseT | _StreamT:
   1197     opts = FinalRequestOptions.construct(
   1198         method="post", url=path, json_data=body, files=to_httpx_files(files), **options
   1199     )
-> 1200     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/_base_client.py:889, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
    880 def request(
    881     self,
    882     cast_to: Type[ResponseT],
   (...)
    887     stream_cls: type[_StreamT] | None = None,
    888 ) -> ResponseT | _StreamT:
--> 889     return self._request(
    890         cast_to=cast_to,
    891         options=options,
    892         stream=stream,
    893         stream_cls=stream_cls,
    894         remaining_retries=remaining_retries,
    895     )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/_base_client.py:980, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
    977         err.response.read()
    979     log.debug("Re-raising status error")
--> 980     raise self._make_status_error_from_response(err.response) from None
    982 return self._process_response(
    983     cast_to=cast_to,
    984     options=options,
   (...)
    987     stream_cls=stream_cls,
    988 )

NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

I don't know how to solve it. Help!😭

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 22.6.0: Wed Oct 4 21:25:40 PDT 2023; root:xnu-8796.141.3.701.17~4/RELEASE_ARM64_T8103
Python Version: 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:41:52) [Clang 15.0.7 ]

Package Information

langchain_core: 0.1.25
langchain: 0.1.8
langchain_community: 0.0.21
langsmith: 0.1.5

[dosubot[bot]]

🤖

Thank you for providing the detailed information, MRX2005nzr. It's clear that you're encountering a NotFoundError when trying to create a KnowledgeGraphIndex using the from_documents method. To better assist you, I need a bit more information.

1. Could you please provide the full traceback of the error? This will help me understand where exactly the error is occurring.
2. Are you able to successfully load data from Wikipedia and YouTube before the error occurs? If not, could you please specify at which point the data loading fails?
3. Could you confirm if the pages and ytlinks you're trying to load exist and are accessible? The error code 404 usually indicates that a resource was not found, which might be the case if the page or video you're trying to load does not exist or is not publicly accessible.
4. Have you made any modifications to the LangChain or Llama Index libraries that might affect their functionality?

Once I have this additional information, I'll be better equipped to help you resolve this issue.

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[MRX2005nzr]

@dosu-bot what I just provided is the full traceback of the error. And I confirm the pages and Youtube links is accessible. The LangChain and Llama Index libraries are not modified. The pages and links are shown as follows:


And there is the full result of the code:

/var/folders/8d/4zx3rfw52_g9_88bnvmq7bq40000gn/T/ipykernel_25156/285902024.py:21: DeprecationWarning: Call to deprecated function (or staticmethod) download_loader. (`download_loader()` is deprecated. Please install tool using pip install directly instead.)
  WikipediaReader = download_loader("WikipediaReader")
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: llama-index-readers-wikipedia in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (0.1.3)
Requirement already satisfied: llama-index-core<0.11.0,>=0.10.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-readers-wikipedia) (0.10.10)
Requirement already satisfied: PyYAML>=6.0.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (6.0.1)
Requirement already satisfied: SQLAlchemy>=1.4.49 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2.0.22)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.8.6)
Requirement already satisfied: dataclasses-json in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (0.5.7)
Requirement already satisfied: deprecated>=1.2.9.3 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.2.14)
Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.0.8)
Requirement already satisfied: fsspec>=2023.5.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2023.10.0)
Requirement already satisfied: httpx in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (0.25.1)
Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.13 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (0.1.13)
Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.6.0)
Requirement already satisfied: networkx>=3.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.1)
Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.8.1)
Requirement already satisfied: numpy in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.26.3)
Requirement already satisfied: openai>=1.1.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.12.0)
Requirement already satisfied: pandas in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2.2.0)
Requirement already satisfied: pillow>=9.0.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (10.0.0)
Requirement already satisfied: requests>=2.31.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2.31.0)
Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (8.2.3)
Requirement already satisfied: tiktoken>=0.3.3 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (0.6.0)
Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (4.66.1)
Requirement already satisfied: typing-extensions>=4.5.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (4.8.0)
Requirement already satisfied: typing-inspect>=0.8.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (0.9.0)
Requirement already satisfied: attrs>=17.3.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (4.0.3)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.3.1)
Requirement already satisfied: wrapt<2,>=1.10 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from deprecated>=1.2.9.3->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.14.1)
Requirement already satisfied: pydantic>=1.10 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from llamaindex-py-client<0.2.0,>=0.1.13->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2.6.1)
Requirement already satisfied: anyio in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.7.1)
Requirement already satisfied: certifi in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2024.2.2)
Requirement already satisfied: httpcore in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.0.2)
Requirement already satisfied: idna in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.4)
Requirement already satisfied: sniffio in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.3.0)
Requirement already satisfied: click in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (8.1.7)
Requirement already satisfied: joblib in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.3.2)
Requirement already satisfied: regex>=2021.8.3 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2023.12.25)
Requirement already satisfied: distro<2,>=1.7.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from openai>=1.1.0->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.9.0)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.26.18)
Requirement already satisfied: greenlet!=0.4.17 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.0.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.0.0)
Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (3.20.1)
Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.5.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.7 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2023.3)
Requirement already satisfied: exceptiongroup in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from anyio->httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.1.3)
Requirement already satisfied: packaging>=17.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.3.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (23.2)
Requirement already satisfied: annotated-types>=0.4.0 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.13->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (0.6.0)
Requirement already satisfied: pydantic-core==2.16.2 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.13->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (2.16.2)
Requirement already satisfied: six>=1.5 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (1.16.0)
Requirement already satisfied: h11<0.15,>=0.13 in /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages (from httpcore->httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-readers-wikipedia) (0.14.0)
Loaded 1 documents
Loaded 1 YouTube documents
/opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `predict_messages` was deprecated in LangChain 0.1.7 and will be removed in 0.2.0. Use invoke instead.
  warn_deprecated(
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
Cell In[14], line 30
     27 youtube_documents = youtube_loader.load_data(ytlinks=['https://www.youtube.com/watch?v=k-HTQ8T7oVw'])    
     28 print(f'Loaded {len(youtube_documents)} YouTube documents')
---> 30 kg_index = KnowledgeGraphIndex.from_documents(
     31     documents=wiki_documents + youtube_documents,
     32     storage_context=storage_context,
     33     max_triplets_per_chunk=15,
     34     service_context=service_context,
     35     space_name=space_name,
     36     edge_types=edge_types,
     37     rel_prop_names=rel_prop_names,
     38     tags=tags,
     39     include_embeddings=True,
     40 )
     42 kg_index.storage_context.persist(persist_dir='./storage_graph')

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/base.py:142, in BaseIndex.from_documents(cls, documents, storage_context, show_progress, callback_manager, transformations, service_context, **kwargs)
    133     docstore.set_document_hash(doc.get_doc_id(), doc.hash)
    135 nodes = run_transformations(
    136     documents,  # type: ignore
    137     transformations,
    138     show_progress=show_progress,
    139     **kwargs,
    140 )
--> 142 return cls(
    143     nodes=nodes,
    144     storage_context=storage_context,
    145     callback_manager=callback_manager,
    146     show_progress=show_progress,
    147     transformations=transformations,
    148     service_context=service_context,
    149     **kwargs,
    150 )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/knowledge_graph/base.py:99, in KnowledgeGraphIndex.__init__(self, nodes, objects, index_struct, llm, embed_model, storage_context, kg_triple_extract_template, max_triplets_per_chunk, include_embeddings, show_progress, max_object_length, kg_triplet_extract_fn, service_context, **kwargs)
     94 self._llm = llm or llm_from_settings_or_context(Settings, service_context)
     95 self._embed_model = embed_model or embed_model_from_settings_or_context(
     96     Settings, service_context
     97 )
---> 99 super().__init__(
    100     nodes=nodes,
    101     index_struct=index_struct,
    102     service_context=service_context,
    103     storage_context=storage_context,
    104     show_progress=show_progress,
    105     objects=objects,
    106     **kwargs,
    107 )
    109 # TODO: legacy conversion - remove in next release
    110 if (
    111     len(self.index_struct.table) > 0
    112     and isinstance(self.graph_store, SimpleGraphStore)
    113     and len(self.graph_store._data.graph_dict) == 0
    114 ):

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/base.py:91, in BaseIndex.__init__(self, nodes, objects, index_struct, storage_context, callback_manager, transformations, show_progress, service_context, **kwargs)
     89 if index_struct is None:
     90     nodes = nodes or []
---> 91     index_struct = self.build_index_from_nodes(
     92         nodes + objects  # type: ignore
     93     )
     94 self._index_struct = index_struct
     95 self._storage_context.index_store.add_index_struct(self._index_struct)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/base.py:209, in BaseIndex.build_index_from_nodes(self, nodes)
    207 """Build the index from nodes."""
    208 self._docstore.add_documents(nodes, allow_update=True)
--> 209 return self._build_index_from_nodes(nodes)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/indices/knowledge_graph/base.py:217, in KnowledgeGraphIndex._build_index_from_nodes(self, nodes)
    214 if self.include_embeddings:
    215     triplet_texts = [str(t) for t in triplets]
--> 217     embed_outputs = self._embed_model.get_text_embedding_batch(
    218         triplet_texts, show_progress=self._show_progress
    219     )
    220     for rel_text, rel_embed in zip(triplet_texts, embed_outputs):
    221         index_struct.add_to_embedding_dict(rel_text, rel_embed)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/core/base/embeddings/base.py:255, in BaseEmbedding.get_text_embedding_batch(self, texts, show_progress, **kwargs)
    249 if idx == len(texts) - 1 or len(cur_batch) == self.embed_batch_size:
    250     # flush
    251     with self.callback_manager.event(
    252         CBEventType.EMBEDDING,
    253         payload={EventPayload.SERIALIZED: self.to_dict()},
    254     ) as event:
--> 255         embeddings = self._get_text_embeddings(cur_batch)
    256         result_embeddings.extend(embeddings)
    257         event.on_end(
    258             payload={
    259                 EventPayload.CHUNKS: cur_batch,
    260                 EventPayload.EMBEDDINGS: embeddings,
    261             },
    262         )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/embeddings/openai/base.py:419, in OpenAIEmbedding._get_text_embeddings(self, texts)
    412 """Get text embeddings.
    413 
    414 By default, this is a wrapper around _get_text_embedding.
    415 Can be overridden for batch queries.
    416 
    417 """
    418 client = self._get_client()
--> 419 return get_embeddings(
    420     client,
    421     texts,
    422     engine=self._text_engine,
    423     **self.additional_kwargs,
    424 )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:289, in BaseRetrying.wraps.<locals>.wrapped_f(*args, **kw)
    287 @functools.wraps(f)
    288 def wrapped_f(*args: t.Any, **kw: t.Any) -> t.Any:
--> 289     return self(f, *args, **kw)

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:379, in Retrying.__call__(self, fn, *args, **kwargs)
    377 retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)
    378 while True:
--> 379     do = self.iter(retry_state=retry_state)
    380     if isinstance(do, DoAttempt):
    381         try:

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:325, in BaseRetrying.iter(self, retry_state)
    323     retry_exc = self.retry_error_cls(fut)
    324     if self.reraise:
--> 325         raise retry_exc.reraise()
    326     raise retry_exc from fut.exception()
    328 if self.wait:

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:158, in RetryError.reraise(self)
    156 def reraise(self) -> t.NoReturn:
    157     if self.last_attempt.failed:
--> 158         raise self.last_attempt.result()
    159     raise self

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/concurrent/futures/_base.py:451, in Future.result(self, timeout)
    449     raise CancelledError()
    450 elif self._state == FINISHED:
--> 451     return self.__get_result()
    453 self._condition.wait(timeout)
    455 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
    401 if self._exception:
    402     try:
--> 403         raise self._exception
    404     finally:
    405         # Break a reference cycle with the exception in self._exception
    406         self = None

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/tenacity/__init__.py:382, in Retrying.__call__(self, fn, *args, **kwargs)
    380 if isinstance(do, DoAttempt):
    381     try:
--> 382         result = fn(*args, **kwargs)
    383     except BaseException:  # noqa: B902
    384         retry_state.set_exception(sys.exc_info())  # type: ignore[arg-type]

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/llama_index/embeddings/openai/base.py:180, in get_embeddings(client, list_of_text, engine, **kwargs)
    176 assert len(list_of_text) <= 2048, "The batch size should not be larger than 2048."
    178 list_of_text = [text.replace("\n", " ") for text in list_of_text]
--> 180 data = client.embeddings.create(input=list_of_text, model=engine, **kwargs).data
    181 return [d.embedding for d in data]

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/resources/embeddings.py:113, in Embeddings.create(self, input, model, dimensions, encoding_format, user, extra_headers, extra_query, extra_body, timeout)
    107         embedding.embedding = np.frombuffer(  # type: ignore[no-untyped-call]
    108             base64.b64decode(data), dtype="float32"
    109         ).tolist()
    111     return obj
--> 113 return self._post(
    114     "/embeddings",
    115     body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams),
    116     options=make_request_options(
    117         extra_headers=extra_headers,
    118         extra_query=extra_query,
    119         extra_body=extra_body,
    120         timeout=timeout,
    121         post_parser=parser,
    122     ),
    123     cast_to=CreateEmbeddingResponse,
    124 )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/_base_client.py:1200, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
   1186 def post(
   1187     self,
   1188     path: str,
   (...)
   1195     stream_cls: type[_StreamT] | None = None,
   1196 ) -> ResponseT | _StreamT:
   1197     opts = FinalRequestOptions.construct(
   1198         method="post", url=path, json_data=body, files=to_httpx_files(files), **options
   1199     )
-> 1200     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/_base_client.py:889, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
    880 def request(
    881     self,
    882     cast_to: Type[ResponseT],
   (...)
    887     stream_cls: type[_StreamT] | None = None,
    888 ) -> ResponseT | _StreamT:
--> 889     return self._request(
    890         cast_to=cast_to,
    891         options=options,
    892         stream=stream,
    893         stream_cls=stream_cls,
    894         remaining_retries=remaining_retries,
    895     )

File /opt/homebrew/Caskroom/miniforge/base/envs/pytorch_env/lib/python3.10/site-packages/openai/_base_client.py:980, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
    977         err.response.read()
    979     log.debug("Re-raising status error")
--> 980     raise self._make_status_error_from_response(err.response) from None
    982 return self._process_response(
    983     cast_to=cast_to,
    984     options=options,
   (...)
    987     stream_cls=stream_cls,
    988 )

NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

[dosubot[bot]]

🤖

Based on the information you've provided, it seems like the NotFoundError with error code 404 is being raised when the Llama library is trying to create a KnowledgeGraphIndex from documents. This error typically indicates that a requested resource could not be found on the server.

Given that the error is propagated through several methods, including get_embeddings in llama_index/embeddings/openai/base.py, Embeddings.create in openai/resources/embeddings.py, and SyncAPIClient.post in openai/_base_client.py, it's possible that the issue lies in the interaction with the OpenAI API.

Here are a few things you could check:

1. API Key and Base URL: Ensure that the OpenAI API key and base URL you're using are correct. The SyncAPIClient uses these to make requests to the OpenAI API. If the API key or base URL is incorrect, it could lead to a NotFoundError.

2. API Version: Check if the OpenAI API version you're using is supported. If you're using an unsupported version, it could lead to a NotFoundError.

3. YouTube URLs: As mentioned in the context, the YouTube URLs you're using might not be valid or might not be in the correct format. The _parse_video_id function in the YoutubeLoader class parses the YouTube URL and returns the video ID if the URL is valid. If the URL is not valid, it returns None, which could lead to a NotFoundError.

4. Transcripts Availability: The YouTube videos you're trying to load might not have transcripts available. The load function in the YoutubeLoader class tries to load the transcripts of the YouTube video. If the transcripts are not available, it could lead to a NotFoundError.

If you've checked all of these and the issue still persists, it would be helpful to have more information about the exact error message you're seeing. This could help in pinpointing the exact cause of the issue.

As for the role of the SyncAPIClient.request method in the openai/_base_client.py file, I'm afraid I don't have enough information to provide a detailed explanation. However, typically, such a method in a client class would be responsible for making synchronous HTTP requests to a specified API endpoint, handling any errors that occur during the request, and returning the response.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• libs/community/langchain_community/document_loaders/youtube.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[AnjaliKeshri]

Have you got any ans?