In embed_docsite, we download the docs source from github repos before processing and uploading to pinecone.
In github_utils right now, we download these files one at a time by downloading the index and then fetching each file.
I'd be willing to bet a shiny pound that git clone --depth 1 will be easier, faster, and more robust.
I've love to swap out the implementation here on a slow day. And run a quick test on the time-to-download before and after the change
In embed_docsite, we download the docs source from github repos before processing and uploading to pinecone.
In
github_utilsright now, we download these files one at a time by downloading the index and then fetching each file.I'd be willing to bet a shiny pound that
git clone --depth 1will be easier, faster, and more robust.I've love to swap out the implementation here on a slow day. And run a quick test on the time-to-download before and after the change