Fix corrupted bare Git repository recovery in DAG bundles

[dheerajturaga]

When using git DAG bundles, corrupted bare repositories can cause all tasks
landing on a host to fail with InvalidGitRepositoryError. This adds retry
logic that detects corrupted bare repositories, cleans them up, and attempts
to re-clone them once before failing.

Changes:

• Add InvalidGitRepositoryError handling in _clone_bare_repo_if_required()
• Implement cleanup and retry logic with shutil.rmtree()
• Add comprehensive tests for both successful retry and retry failure scenarios
• Ensure all existing tests continue to pass

[prdai]

Not required, but just a thought, we could use tenacity, here instead of a custom retry loop. It might make the retry/backoff logic easier to read/maintain. For example:

@retry(stop=stop_after_attempt(3), wait=wait_exponential(), reraise=True)
def clone_bare_repo(url, path, env=None):
    return Repo.clone_from(url, path, bare=True, env=env)

[dheerajturaga]

cc: @jedcunningham @ephraimbuddy @kaxil @jscheffl

When using gitbundles with edge workers (EdgeExectutor) it could happen that the git connection could be unstable causing git clone/ git bare clone to fail. If the bare clone is broken, all subsequent tasks on the worker fail unless you manually ssh onto the machine to tidy up the bare repo. This is an attempt to self heal

Im hoping to get this in the next wave of provider release. cc: @eladkal

[dheerajturaga]

Not required, but just a thought, we could use tenacity, here instead of a custom retry loop. It might make the retry/backoff logic easier to read/maintain. For example:

@retry(stop=stop_after_attempt(3), wait=wait_exponential(), reraise=True)
def clone_bare_repo(url, path, env=None):
    return Repo.clone_from(url, path, bare=True, env=env)

good idea!

[potiuk]

Nice!

[dheerajturaga]

@potiuk this looks like an unrelated failure?

[jscheffl]

For me this looks good. I would have preferred to catch other AirflowExceptions directly in the PR and not reverting these - breaking change doe not apply in my eyes also it is no Dag or user code touching these exceptions.... Now some other PR needs to clean this... but anyway LGTM.

[potiuk]

LGTM. @ephraimbuddy ?