Certain search results aren't retuned by the code search API

[gbouv]

Select Topic Area

Question

Body

I want to search for files with a certain name across public repositories on Github. The file name I'm looking for is kurtosis.yml.

I went through the search API documentation (which is great BTW), and ended up with this simple query:

curl -XGET 'https://api.github.com/search/code?q=filename%3Akurtosis.yml&per_page=100&page=1' \
    --header 'Authorization: Bearer <MY_TOKEN>'

For the token, I created a "Fined grained token" which has only access to public repositories. I just want to search for public repos anyways, so that should be enough.

It currently returns 62 results. At first I was happy with it. But after taking a closer look, I realized some repositories are missing. AFAICT all the following repos are public, and contains a kurtosis.yml file:

• https://github.com/kurtosis-tech/nginx-package
• https://github.com/adschwartz/flink-package
• https://github.com/bharath-123/eth2-package

Can you help me understand why those repos do not appear in my search result? Is there something I'm missing which somehow excludes them from the search? I compared the settings of the ones I own with some others that are returned by the search, and i could not see any difference.
Any help would be appreciated here.

[look]

Hi @gbouv, for all these examples it is because the repository was not indexed (I checked this by searching within the repository and so they should be added to the index soon). We haven't indexed every public repository so unfortunately the results will not be complete.

[gbouv]

Hey @look ! Thanks for your reply, and sorry for the delay coming back to you, somehow I missed the notification.
That's interesting, and do you have an idea of what the index frequency currently is? I.e. if I create a brand new repo today (which matches my search criteria above), when approximatively should I expect it to show up in the results list?

[look]

It depends! Not all repositories are indexed on creation, but if someone issues a search, we add it (if possible, some repos can't be indexed due to limitations).

[gbouv]

Got it, thanks for the clarification :)