[Bug][gitextractor] subtask Clone Git Repo ended unexpectedly

[realhuseyn]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

My pipeline results in errors. One of them is related to clone GitHub repositories. My pipeline includes 60+ repositories. At the beginning of the pipeline, tasks finish successfully, but after 15 or 20th tasks, all other tasks finish with git clone errors.

time="2024-08-26 13:18:26" level=info msg=" [pipeline service] [pipeline #119] [task #7340] start executing task: 7340"
time="2024-08-26 13:18:26" level=info msg=" [pipeline service] [pipeline #119] [task #7340] start plugin"
time="2024-08-26 13:18:26" level=info msg=" [pipeline service] [pipeline #119] [task #7340] [gitextractor.PrepareTaskData] UseGoGit: false"
time="2024-08-26 13:18:26" level=info msg=" [pipeline service] [pipeline #119] [task #7340] [gitextractor.PrepareTaskData] SkipCommitStat: false"
time="2024-08-26 13:18:26" level=info msg=" [pipeline service] [pipeline #119] [task #7340] [gitextractor.PrepareTaskData] SkipCommitFiles: true"
time="2024-08-26 13:18:26" level=info msg=" [pipeline service] [pipeline #119] [task #7340] total step: 4"
time="2024-08-26 13:18:26" level=info msg=" [pipeline service] [pipeline #119] [task #7340] executing subtask Clone Git Repo"
time="2024-08-26 13:18:27" level=error msg=" [pipeline service] [pipeline #119] [task #7340] subtask Clone Git Repo ended unexpectedly\n\tWraps: (2) git cmd [git clone https://git:*************************************@github.com/Pl/adapter--.git /tmp/gitextractor3060880397 --depth=1 --bare] in failed: Cloning into bare repository '/tmp/gitextractor3060880397'...\n\t | remote: Invalid username or password.\n\t | fatal: Authentication failed for 'https://github.com/P***l/adapter-**-**.git/'\n\tError types: (1) *hintdetail.withDetail (2) *errors.errorString"

What do you expect to happen

The pipeline uses one GitHub connection. Why are some of repositories cloning successfully and all other repositories not cloning with "Invalid username or password" error?

How to reproduce

Rerun pipeline

Anything else

No response

Version

v1.0.1-beta7

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

I found a similar issue that might be relevant to your problem. The issue titled "[Bug][Git Extractor] Git Extractor task failed in a GitHub blueprint" was closed and the suggested solution was to change the Docker base image ^[1].

The "Invalid username or password" error in the gitextractor subtask of your pipeline is likely caused by the permission of your tokens. Specifically, it could be due to using an unsupported authentication method or a token without the necessary permissions for the entities you want to collect ^[2].

To resolve this issue, ensure that you are using a supported authentication method and that your token has the necessary permissions for the repositories you are trying to clone. Additionally, if you are dealing with a private GitLab server with a self-signed certificate, you might need to add your root CA to the devlake container to avoid certificate verification issues ^[2].

If the issue persists, consider checking the rate limits and throttling mechanisms that could affect the cloning of multiple repositories in a single pipeline run. DevLake uses a dynamic rate limit at around 12,000 requests/hour to collect GitLab data, and tokens under the same IP address share this rate limit ^[3].

To continue talking to Dosu, mention @dosu.

[realhuseyn]

I found a similar issue that might be relevant to your problem. The issue titled "[Bug][Git Extractor] Git Extractor task failed in a GitHub blueprint" was closed and the suggested solution was to change the Docker base image [1].

The "Invalid username or password" error in the gitextractor subtask of your pipeline is likely caused by the permission of your tokens. Specifically, it could be due to using an unsupported authentication method or a token without the necessary permissions for the entities you want to collect [2].

To resolve this issue, ensure that you are using a supported authentication method and that your token has the necessary permissions for the repositories you are trying to clone. Additionally, if you are dealing with a private GitLab server with a self-signed certificate, you might need to add your root CA to the devlake container to avoid certificate verification issues [2].

If the issue persists, consider checking the rate limits and throttling mechanisms that could affect the cloning of multiple repositories in a single pipeline run. DevLake uses a dynamic rate limit at around 12,000 requests/hour to collect GitLab data, and tokens under the same IP address share this rate limit [3].

To continue talking to Dosu, mention @dosu.

No. It is not similar. My repositories are private. I have already deployed the latest version of devlake. Also, in my pipeline, some repositories clone successfully, others can't clone. It is GitHub Enterprise Cloud, not on-premise

[klesh]

1. Have you tried running it again? If so, did you encounter the same outcome? Specifically, were some tasks successful initially while other failed with "Authentication error"?
2. Did those repositories belong to the same connection?
3. How many token did you set for the connection (or connection)?

[KyriosGN0]

we are also running into this, when all pipelines ran at the same time it happened but when i gave them different schedules it worked fine, using a private github org with github app auth

[realhuseyn]

1. Have you tried running it again? If so, did you encounter the same outcome? Specifically, were some tasks successful initially while other failed with "Authentication error"?
2. Did those repositories belong to the same connection?
3. How many token did you set for the connection (or connection)?

1. Yes, I tried repeatedly. Usually, it begins in the same stage. At first, it is working normally
2. Yes. I have one GitHub connection
3. I use one GitHub app token.

UPDATE: I changed the rate limit manually, for example, I set 10,000 requests per hour, and the pipeline begins with the error after stage 38. Before, it began after stages 20, 21, or 22. I think that it exceeds rate limits, therefore some stages end with errors. But, I don't see any errors about it at logs

[klesh]

It’s quite likely you're encountering this error due to the rate limiting. We introduced it to give users the ability to reduce request frequency and avoid 429 errors. 😅

I recommend trying the approach shared by @KyriosGN0, as unfortunately, there’s not much else we can do to prevent this kind of issue(likely the data source or some point in the network closes the connection).

[KyriosGN0]

@klesh does it make sense to introduce a re-try in this plug-in? (Or in general to give plugins the ability to rerun a failed task automatically?)

[klesh]

@KyriosGN0
auto retry when cloning: maybe, but it is unlikely to help when there are too many clones.
rerun: there is a button on the right of the task already?

[realhuseyn]

It’s quite likely you're encountering this error due to the rate limiting. We introduced it to give users the ability to reduce request frequency and avoid 429 errors. 😅

I recommend trying the approach shared by @KyriosGN0, as unfortunately, there’s not much else we can do to prevent this kind of issue(likely the data source or some point in the network closes the connection).

If the application returns an error about the rate limit, it would be clearer for users. Also, We should be able to change the duration of the pipeline. For example, if the pipeline works slowly, it won't exceed the rate limit.

[klesh]

To clarify, there are 2 things here:

1. Ratelimit affect API request, setting it to a high value might cause the data source plugin, e.g. Github to fail, normally it returns 429 errors, we can provide a friendly message for them, I think it is reasonable.
2. Gitextractor doesn't issue any API request, so it won't be affected by ratelimit, and there will be no clear indicator that it was ratelmited. In fact, it was my guess that the clone failed due to high concurrent cloning, I don't know it was ratelimited or some routers in the network just drop the connection.