GithubFileLoader API Error #17453

PrashantDixit0 · 2024-02-13T09:48:58Z

Checked other resources

I added a very descriptive title to this issue.
I searched the LangChain documentation with the integrated search.
I used the GitHub search to find a similar question and didn't find it.
I am sure that this is a bug in LangChain rather than my code.

Example Code

from langchain.document_loaders import GithubFileLoader
loader = GithubFileLoader(
repo="langchain-ai/langchain", # the repo name
access_token="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",
github_api_url="https://api.github.com",
file_filter=lambda file_path: file_path.endswith(
".md"
), # load all markdowns files.
)
documents = loader.load()
print(documents)

Error Message and Stack Trace (if applicable)

No response

Description

raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.github.com/api/v3/repos/langchain-ai/langchain/git/trees/master?recursive=1

System Info

System Information

OS: Linux
OS Version: #17~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jan 16 14:32:32 UTC 2
Python Version: 3.11.4 (main, Jul 5 2023, 13:45:01) [GCC 11.2.0]

Package Information

langchain_core: 0.1.22
langchain: 0.1.6
langchain_community: 0.0.19
langsmith: 0.0.87

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

dosubot · 2024-02-13T09:49:40Z

🤖

Hey there @PrashantDixit0, fancy seeing you here again! Hope this message finds you coding well 😉

Based on the error message you provided, it seems like the URL being used to access the GitHub API is incorrect. The error message indicates a 404 error for the URL https://api.github.com/api/v3/repos/langchain-ai/langchain/git/trees/master?recursive=1.

The correct base URL for the GitHub API is https://api.github.com, not https://api.github.com/api/v3. The /api/v3 in the URL is redundant and likely causing the 404 error.

Please try updating the github_api_url parameter in your GithubFileLoader instance to https://api.github.com:

loader = GithubFileLoader(
    repo="langchain-ai/langchain",  # the repo name
    access_token="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",
    github_api_url="https://api.github.com",
    file_filter=lambda file_path: file_path.endswith(
        ".md"
    ),  # load all markdowns files.
)

If you're still encountering issues after making this change, it would be helpful to see the full traceback of the error. This can provide more context about what's causing the error and where in the code it's occurring.

Also, please ensure that your GitHub personal access token is correct and has the necessary permissions to access the repository. If the token is incorrect or doesn't have the necessary permissions, this could also cause a 404 error.

Lastly, please verify that the repository langchain-ai/langchain exists and that you're using the correct branch name. If the repository or branch doesn't exist, this would also result in a 404 error.

For more information about the GithubFileLoader class and how it interacts with the GitHub API, you can refer to the GithubFileLoader source code and the unit tests for GithubFileLoader.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

PrashantDixit0 · 2024-02-13T09:59:04Z

I am already updating github_api_url as https://api.github.com/, but still getting this error

PrashantDixit0 · 2024-02-13T10:04:57Z

@shufanhao @baskaryan I am following this documentation page as a reference and using the same code snippet given, but still getting same error

brk21 · 2024-02-26T21:40:53Z

loader = GithubFileLoader(
    repo="langchain-ai/langchain",  # the repo name
    access_token=os.environ["GITHUB_ACCESS_TOKEN"],
    github_api_url="https://api.github.com",
    file_filter=lambda file_path: file_path.endswith(
        ".md"
    ),  # load all markdowns files.
)

docs = loader.load()

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[7], line 13
      2 g = Github(auth=auth)
      4 loader = GithubFileLoader(
      5     repo="langchain-ai/langchain",  # the repo name
      6     access_token=os.environ["GITHUB_ACCESS_TOKEN"],
   (...)
     10     ),  # load all markdowns files.
     11 )
---> 13 docs = loader.load()

File /opt/conda/envs/analyst-copilot/lib/python3.11/site-packages/langchain_community/document_loaders/github.py:246, in GithubFileLoader.load(self)
    243 def load(self) -> List[Document]:
    244     documents = []
--> 246     files = self.get_file_paths()
    247     for file in files:
    248         content = self.get_file_content_by_path(file["path"])

File /opt/conda/envs/analyst-copilot/lib/python3.11/site-packages/langchain_community/document_loaders/github.py:218, in GithubFileLoader.get_file_paths(self)
    213 base_url = (
    214     f"{self.github_api_url}/api/v3/repos/{self.repo}/git/trees/"
    215     f"{self.branch}?recursive=1"
    216 )
    217 response = requests.get(base_url, headers=self.headers)
--> 218 response.raise_for_status()
    219 all_files = response.json()["tree"]
    220 """ one element in all_files
    221 {
    222     'path': '.github', 
   (...)
    227 }
    228 """

File /opt/conda/envs/analyst-copilot/lib/python3.11/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1016     http_error_msg = (
   1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   1018     )
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 404 Client Error: Not Found for url: https://api.github.com/api/v3/repos/langchain-ai/langchain/git/trees/main?recursive=1

timkitch · 2024-03-01T16:05:30Z

Same here. I see in the code for GithubFileLoader that it's incorrectly hardcoding the URL /api/v3 prefix as:

base_url = (
            f"{self.github_api_url}/api/v3/repos/{self.repo}/git/trees/"
            f"{self.branch}?recursive=1"
        )

So, not possible to override that part. It's broken.

Description- - Changed the GitHub endpoint as existing was not working and giving 404 not found error - Also the existing function was failing if file_filter is not passed as the tree api return all paths including directory as well, and when get_file_content was iterating over these path, the function was failing for directory as the api was returning list of files inside the directory, so added a condition to ignore the paths if it a directory - Fixes this issue - #17453 Co-authored-by: Radhika Bansal <Radhika.Bansal@veritas.com>

…i#17622) Description- - Changed the GitHub endpoint as existing was not working and giving 404 not found error - Also the existing function was failing if file_filter is not passed as the tree api return all paths including directory as well, and when get_file_content was iterating over these path, the function was failing for directory as the api was returning list of files inside the directory, so added a condition to ignore the paths if it a directory - Fixes this issue - langchain-ai#17453 Co-authored-by: Radhika Bansal <Radhika.Bansal@veritas.com>

shawnesquivel · 2024-04-05T20:21:24Z

Having the same issue.

shufanhao · 2024-04-06T13:45:02Z

Same here. I see in the code for GithubFileLoader that it's incorrectly hardcoding the URL /api/v3 prefix as:
base_url = (
            f"{self.github_api_url}/api/v3/repos/{self.repo}/git/trees/"
            f"{self.branch}?recursive=1"
        )
So, not possible to override that part. It's broken.

did you use the latest code ?

timkitch · 2024-04-06T14:05:57Z

Same here. I see in the code for GithubFileLoader that it's incorrectly hardcoding the URL /api/v3 prefix as:
base_url = (
            f"{self.github_api_url}/api/v3/repos/{self.repo}/git/trees/"
            f"{self.branch}?recursive=1"
        )
So, not possible to override that part. It's broken.
did you use the latest code ?

I haven't yet tried with the latest. But, I just reviewed the recent updates to that code and it appears to me that it should now work.

shawnesquivel · 2024-04-06T18:27:05Z

I can confirm it's working with this requirements.txt file.

langchain==0.1.14
orjson==3.9.15 # needed for langchain > 0.1.7 https://github.com/langchain-ai/langchain/issues/19719
openai==1.14.2
python-dotenv==1.0.0
langchain-community==0.0.30
langchain-openai==0.0.4
langchain-text-splitters

Bluthunder · 2024-04-08T10:38:10Z

Can some one confirm if the above issue is resolved, i can still reproduce same error with
lanchain version -0.1.14
open ai version - 1.16.2

shufanhao · 2024-04-14T02:41:13Z

Can some one confirm if the above issue is resolved, i can still reproduce same error with
lanchain version -0.1.14
open ai version - 1.16.2

@Bluthunder make sure your langchain-community also is updated to latest or updated to langchain-community==0.0.30

gjuoun · 2024-04-15T19:22:10Z

The error still exists, because GithubFileLoader does not handle HTTP 404 errors, in my case
HTTPError: 404 Client Error: Not Found for url: ...

Bluthunder · 2024-04-16T05:38:29Z

@shufanhao thanks.

My error was resolved by adding branch name when instantiating Loader such as below

loader = GithubFileLoader(
repo=repo_name,
access_token=ACCESS_TOKEN,
github_api_url="https://api.github.com",
branch=branch_name,
file_filter=lambda file_path: file_path.startswith(directory)

dosubot bot added Ɑ: doc loader Related to document loader module (not documentation) 🤖:bug Related to a bug, vulnerability, unexpected error with an existing feature labels Feb 13, 2024

RadhikaBansal97 mentioned this issue Feb 16, 2024

community: Change github endpoint in GithubLoader #17622

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GithubFileLoader API Error #17453

GithubFileLoader API Error #17453

PrashantDixit0 commented Feb 13, 2024

dosubot bot commented Feb 13, 2024 •

edited

About Dosu

PrashantDixit0 commented Feb 13, 2024 •

edited

PrashantDixit0 commented Feb 13, 2024 •

edited

brk21 commented Feb 26, 2024 •

edited

timkitch commented Mar 1, 2024

shawnesquivel commented Apr 5, 2024

shufanhao commented Apr 6, 2024

timkitch commented Apr 6, 2024

shawnesquivel commented Apr 6, 2024

Bluthunder commented Apr 8, 2024 •

edited

shufanhao commented Apr 14, 2024

gjuoun commented Apr 15, 2024

Bluthunder commented Apr 16, 2024

GithubFileLoader API Error #17453

GithubFileLoader API Error #17453

Comments

PrashantDixit0 commented Feb 13, 2024

Checked other resources

Example Code

Error Message and Stack Trace (if applicable)

Description

System Info

System Information

Package Information

Packages not installed (Not Necessarily a Problem)

dosubot bot commented Feb 13, 2024 • edited

Sources

About Dosu

PrashantDixit0 commented Feb 13, 2024 • edited

PrashantDixit0 commented Feb 13, 2024 • edited

brk21 commented Feb 26, 2024 • edited

timkitch commented Mar 1, 2024

shawnesquivel commented Apr 5, 2024

shufanhao commented Apr 6, 2024

timkitch commented Apr 6, 2024

shawnesquivel commented Apr 6, 2024

Bluthunder commented Apr 8, 2024 • edited

shufanhao commented Apr 14, 2024

gjuoun commented Apr 15, 2024

Bluthunder commented Apr 16, 2024

dosubot bot commented Feb 13, 2024 •

edited

PrashantDixit0 commented Feb 13, 2024 •

edited

PrashantDixit0 commented Feb 13, 2024 •

edited

brk21 commented Feb 26, 2024 •

edited

Bluthunder commented Apr 8, 2024 •

edited