bug: Gitingest fetches entire repo for tagged subdirectories

[jpotw]

Hi! I've noticed some unexpected behavior with gitingest when trying to fetch a subdirectory from a specific tag. It seems to grab the whole repository instead of just the subdirectory, which works fine for branches.

Expected:

When I use a URL like this (pointing to a subdirectory within a tag):

gitingest should only fetch the files in that subdirectory, just like it does for branches.

Observed:

gitingest downloads a ton of files (hitting the max file limit on big repos like PyTorch) and seems to ignore the subdirectory part of the tag URL. It's pulling the entire repository for that tag.

Steps to Reproduce:

1. Tag (Fails): Run:

   gitingest https://github.com/pytorch/pytorch/tree/v2.4.1/torch/distributed/elastic/agent/server

   (Note: Actually this command will time out without removing --recurse-submodules – see Why Use --recurse-submodules in clone_repo? It slows down cloning large repos #195).

2. Observe (Tag): You'll see output like this, indicating it's processing the whole repository:

   Maximum file limit (10000) reached
   ... (repeated many times) ...
   Analysis complete! Output written to: digest.txt
   
   Summary:
   Repository: pytorch/pytorch
   Files analyzed: 10000  # Should be much smaller!
   
   Estimated tokens: 16.8M
   ...
   

3. Branch (Works): Now try the same subdirectory, but on the main branch:

   gitingest https://github.com/pytorch/pytorch/tree/main/torch/distributed/elastic/agent/server

4. Observe (Branch): You'll see the correct output:

   Analysis complete! Output written to: digest.txt
   
   Summary:
   Repository: pytorch/pytorch
   Files analyzed: 4  # Correct!
   Subpath: /torch/distributed/elastic/agent/server
   
   Estimated tokens: 12.3k
   ...
   

Comparison (Branch Behavior - Working):

Just to confirm, this works perfectly for subdirectories on branches (both main and others):

• Main: gitingest .../tree/main/... (Correct output: 4 files)
• Other Branch: gitingest .../tree/gh/qqaatw/26/orig/... (Correct output: 4 files)

I've included the full commands and expected output in the original description, but the key difference is the Files analyzed count.

It seems like gitingest handles tagged subdirectories differently than branch subdirectories, leading to unexpected behavior and hitting the file limit.

I'd be happy to help investigate and potentially submit a PR if you can confirm this is a bug! Let me know what you think.

Thanks!

[filipchristiansen]

Thanks for creating this issue! Work is already in progress:

https://github.com/cyclotruc/gitingest/blob/d16cbd3055948c818175e27b80d677a942fac70d/src/gitingest/repository_clone.py#L84-L85

[jpotw]

Thanks for opening this issue! I appreciate the heads-up. It looks like some related work is already underway here:

gitingest/src/gitingest/repository_clone.py

Lines 84–85 in d16cbd3:

if partial_clone:  
    clone_cmd += ["--filter=blob:none", "--sparse"]  

Hi, thanks for your response! I’m not entirely sure if this addresses the specific issue I raised, so I’d love to clarify things a bit.

To recap the two issues I submitted:

In #195, I noted that cloning large repos (like PyTorch) within 60 seconds wasn’t feasible. I found that removing --recurse-submodules from this line resolved the issue. Out of curiosity, is there a specific reason the team typically includes this flag? It seems particularly inefficient when dealing with large third-party submodules.

In #196(the current page), I pointed out that tags (e.g., cyclotruc/gitingest/tree/v0.1.3) and their subdirectories aren’t being recognized properly (instead the full repository in main branch is being fetched), unlike commits or branches. I’ve identified the root cause of this and am actively working on a fix for this now.

Issue #195 feels like a quick fix, while I’m currently digging into #196.

Could you assign both issues to me so I can take ownership of them?

Thanks so much!

[filipchristiansen]

Hi @jpotw. I read this again and now understand what you were pointing out. I will look into it. Do you have idea what the issue might be?

[jpotw]

Hi @filipchristiansen. Regarding your question, it's been months since I last looked at this codebase, so my memory might be a bit hazy, but I recall the existing logic only identified branches and commits, without specific handling for tags.

I noticed a similar approach to add tag handling here right above your comment, but it might not be integrated into the current codebase yet.

I'll check the repository's main branch first to see if this issue is persisting.

[jpotw]

I've just run the main branch of this repository and confirmed that the issue persists. I believe I can implement a fix for this, similar to the approach in #206 but with minimal code changes.

Features I am planning to add:

• CLI Tag Options: Adding --tag and -t flags for specifying tags. To prevent confusion, the process will abort if both branch and tag names are provided.
• Tag Fetching: The system will fetch repository tags in addition to branches and commits. In cases of identical branch and tag names, branch names will take priority. (This prioritization logic can be revisited in future updates if necessary.)

I can get started on this once you confirm this plan.