feat: Parallelize the GithubRepoLoader #2224
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
the
auto:improvement
![dosubot[bot]](https://avatars.githubusercontent.com/in/324583?s=60&v=4){kind=small}
| @@ -0,0 +1,53 @@ | |||
| import { jest, test } from "@jest/globals"; | |||
There was a problem hiding this comment.
This comment is flagging a new HTTP request using the fetch function in the code, and it is recommended for maintainers to review this change.
yroc92
force-pushed
the
feature/parallelize-github-loader
branch
3 times, most recently
from
edcda8c
to
21d433c
Compare
yroc92
force-pushed
the
feature/parallelize-github-loader
branch
from
21d433c
to
c8caf3d
Compare
|
Looks nice! Just a bit worried about rate limits/the change in behavior - see comments |
jacoblee93
added
question
jacoblee93
added
lgtm
|
Hey @yroc92, added max concurrency support with an internal utility we have. Thank you, will get this merged tonight! |
|
Thanks again! |
|
@jacoblee93 good to know about the concurrency limit support! It’s definitely easy to hit rate limits if you don’t filter out enough files with gitignore. But parallelizing the ops won’t impact the rate limit much because the limits are calls per hour. Using an API key makes the limits much higher. |
|
Ah gotcha. Well, I defaulted to 2 but you can always set it to |
Instead of synchronously fetching file content and directory info, this change leverages promises and
Promise.allto parallelize the file retrieval whenrecursive = trueforGithubRepoLoader.Fixes #2196
Twitter: https://twitter.com/yroc92