How the token mechanism works for the CrawProjectJob

[s-dimaria]

I have a question about how the crawl works. I read in the code that there is an option to enter multiple PATs to make the crawl more efficient.
I wanted to ask if the PATs are all from the same user or are they generated?

[dabico]

Hi @s-dimaria

I've migrated your inquiry to discussions since this is a more fitting place for questions about the platform and its inner workings.

Yes, you can use multiple access tokens, configured either in the application.properties or in your custom docker-compose.override.yml file during deployment. Detailed instructions on how to do so can be found in the README. The system for token management is rather simple: it uses an access token until the rate limits for it are depleted, after which it switches to the next specified token in the list. Said list of tokens is iterated in a looping manner, with the last token looping back to the first. However, if the token manager exhausts all options, it will simply busy-wait until the current token's rate limits refresh. These notions are central to your question: should the tokens all come from a single user? The GitHub API documentation states that all tokens from a user share the same rate limit pool. For this reason, using multiple access tokens associated with the same user is no better than using just one token from said user.

On a side note, our platform currently uses only one token, and even then it rarely (if ever) enters a waiting state. The token switching was a feature that was seen as a necessity in the platform's infancy. Back then, we relied solely on GitHub's REST API and hitting the rate limit was quite common. This year, we migrated to GraphQL, greatly diminishing the average number of requests that the platform's crawler makes to the APIs, allowing it to mine virtually unimpeded. In other words, the switch to GraphQL has diminished the need for token switching, and as such the feature is a hold-over from the project's past. I doubt that it will be removed (I might rework it at some point to allow user-defined limits to token usage instead of completely exhausting rate limits), but at the same time, I doubt that this feature is of much use to anyone.

I hope this answers your questions. If you have any further inquiries, feel free to continue the discussion.

[s-dimaria]

Thank you for your response. It made me understand it better.

I have one more question, regarding the crawl of repositories I noticed that it is done by splitting into languages, how come this choice?

And how is the mechanism for updating a group of repositories already retrieved?

[dabico]

As for the first question, the choice to segment mining by language was made to keep search intervals manageable. GitHub's Search APIs are limited to 1000 results. To stay under this limit, we mine each language individually, and segment searches into smaller fragments by the last update date. For example, if we searched for Swift repositories that were last updated between 2022 and 2023 and got more than 1000 results, we would repeat this request with two queries, each targeting a separate half of 2022. We repeat this procedure for as long as the intervals contain more than the specified number of results. If we did not separate mining by language and instead mined all repositories, we run the risk of encountering search intervals exceeding the result limit that can not be further split.

As for the second question, the mining of repositories is performed by the last update date of a project. This ensures that repositories that were updated after being crawled are picked up by the crawler again. For the crawler to know what information it possesses so far, we maintain crawl progress information for each language. This is essentially a mapping between a language and a timestamp, which signifies that we have mined the information of all projects that were updated up to said timestamp. Crawling for each language is commenced from these checkpoints, and they are updated whenever an interval of repositories has been mined to completion. We also have some minor optimizations during crawling to determine if the repository was updated since it was last encountered. While the Search API does not contain all the information we desire, it does contain just enough for us to decide whether fetching additional data is needed or not.

I hope this clears things up.