Hitting a Wall with API Rate Limit in GithubRepoLoader

[l4time]

While trying to process a directory in our GithubRepoLoader, I stumbled upon a '403 API rate limit exceeded' error. It appears we're hitting GitHub's limit for unauthenticated requests.

It is preventing us from fetching the repository files we need. And as the error message suggests, we might have better luck with authenticated requests. GitHub's REST API documentation.

Here's the error log to give more context:
Error: Failed to process directory: , Error: Unable to fetch repository files: 403 {"message":"API rate limit exceeded for x.x.x.x. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting";} at GithubRepoLoader.handleError (/app/node_modules/langchain/dist/document_loaders/web/github.cjs:180:23) at GithubRepoLoader.processDirectory (/app/node_modules/langchain/dist/document_loaders/web/github.cjs:153:18) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GithubRepoLoader.load (/app/node_modules/langchain/dist/document_loaders/web/github.cjs:108:9) at async githubQueueController (/app/queue/controllers/github.controller.js:15:18) at async Queue.queueHandler (/app/queue/index.js:49:25)

[n4ze3m]

Yeah, rate limiting is a significant issue. I will check the GitHub docs this weekend and come up with a solution.

[n4ze3m]

Hey, sorry for the late update, but I just released v0.0.20. It includes a custom GitHub loader that clones the repository locally instead of using the GitHub API. Additionally, it supports private repositories by allowing you to provide an access token, similar to the GitHub API. Please let me know if you encounter any issues