[Waiting Review] Improvements in Tfs branches support #480

Merged
merged 50 commits into from Jul 10, 2014

9 participants

@pmiossec
git-tfs member

This pull request bring bing improvements in TFS branch support:

  • auto initialization of a tfs branch, if needed, and fetching needed changesets when encountering a merge changeset to be able to create the merge commit with its 2 parents (default behavior). See image below...
  • tfs branch renaming
  • baseless merge (when the merge changeset was a merge of a branch without the same base as the trunk branch)
  • add a --ignore-branches option to ignore merge changeset and process it as a normal changeset (for those that can't manage branches due to a messy tfs repository).

git-tfs_branches
If I clone tfs/default (git tfs clone https://tfs.codeplex.com:443/tfs/TFS16 $/vtccds/trunk) and it encounter a merge changeset, it create and fetch automatically the branch tfs/b1 to be able to create the merge commit.

@pmiossec pmiossec was assigned Nov 19, 2013
@KindDragon

What missing in this PR? When you plan merge it?
I really want this feature.

@KindDragon

Maybe support only simple renaming branch at start and baseless merge?

@pmiossec
git-tfs member

I have worked on that yesterday after quite some long time without having time to do it :(
I have rebased my work onto master and corrected the 5 unit tests that failed since a long time...

What I need before merging this pull request is a good code review, doing some test on different scenarii (I have only tested with --with-branches option and need to be sure to not have broken something when you don't use it) and some (successful) tests by other user.

Did you have tested it successfully yourself? Does that solve cases where released git-tfs failed?

@KindDragon

Now I try to merge this branch with my to check it out.

@pmiossec
git-tfs member

Maybe support only simple renaming branch at start and baseless merge?

I'm not sure I want to do that. It's perhaps what I should have done in the beginning but things are too intricate and I'm not sure I will be able to merge them one by one without lot of works! :(

@pmiossec
git-tfs member

@KindDragon I successfully manage the last point I want to be solved before trying to merge the PR.
Need some more work to do it well, some history cleaning, perhaps some code review and we will be ok...

@KindDragon

👍 👍 👍

@pmiossec
git-tfs member

@spraints @sc68cal @KindDragon
It's no more a work in progress! It's time to review \o/
I will try to say more about this PR tomorrow when my laptop battery will not be empty anymore...

@KindDragon KindDragon and 1 other commented on an outdated diff Feb 7, 2014
GitTfs/Commands/Clone.cs
@@ -64,6 +65,9 @@ public int Run(string tfsUrl, string tfsRepositoryPath, string gitRepositoryPath
VerifyTfsPathToClone(tfsRepositoryPath);
+ if (withBranches && initBranch != null)
+ fetch.IgnoreBranches = false;
+
if (retVal == 0) fetch.Run(withBranches);

I think branch --init --all should work almost identical as clone --with-branches.
Maybe we shouldn't call fetch if withBranches == true?

@pmiossec
git-tfs member
pmiossec added a note Feb 8, 2014

hmm... Need some explanation... ;)
branch --init --all and clone --with-branches couldn't work as identical as because the branch command could only be used in an already initialized repository.
Except from that, with this PR, they do the same thing! \o/
you could do:

  • init + branch --init --all
  • clone + branch --init --all
  • clone --with-branches and you will end up with the same repo!

Maybe we shouldn't call fetch if withBranches == true?

No, must keep the way it works. clone is init + fetch. If we remove fetch what is remaining? And I don't really understand why you want to condition fetch call...

           if (withBranches && initBranch != null)
               fetch.IgnoreBranches = false;

That's just a security to prevent use of --with-branches and --ignore-branches that are 2 options incompatibles.
--ignore-branches is an option to keep the way git-tfs actually works for those where the TFS repository is a mess and can't support branches.

withBranches is an option to the fetch command to tell it to do special things to support branches when fetching. So it's OK...

To conclude, all is OK, like that! I have tested and approved!! ;)

No, must keep the way it works. clone is init + fetch. If we remove fetch what is remaining? And I don't really understand why you want to condition fetch call...

Clone call after Fetch - InitBranch.Run() if withBranch = true. I thought it will fetch all branches?

@pmiossec
git-tfs member
pmiossec added a note Feb 8, 2014

Oh! I understand! you're right, we can remove the call to the fetch because after we call initBranch.Run() now that the pull request manage branch initialization "on the fly".
But I'm not sure I will remove this call. Because if you've got a problem in the initBranch call, there is more chance that you've got your main branch fetched... I don't know....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@KindDragon KindDragon and 1 other commented on an outdated diff Feb 7, 2014
GitTfs.VsCommon/TfsHelper.PostVs2010.Common.cs
- return rootChangesetInParentBranch.ChangesetId;
+ AddNewRootBranch(rootBranches, new RootBranch(rootChangesetInParentBranch.ChangesetId, tfsPathBranchToCreate));
+ Trace.WriteLineIf(renameFromBranch != null, "Found original branch '" + renameFromBranch + "' (renamed in branch '" + tfsPathBranchToCreate + "')");
+ if (renameFromBranch != null)
+ GetRootChangesetForBranch(rootBranches, renameFromBranch);

I think better pass changeset id when renaming occur to GetRootChangesetForBranch, so this method pass it to TrackMerges to improve performance.

@pmiossec
git-tfs member
pmiossec added a note Feb 8, 2014

I don't see what you want...
If you could do it, I will be pleased.

I tried to implement it, but it seems TrackMerge method does not allow limit to which changeset it will scan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pmiossec
git-tfs member

I will try to say more about this PR tomorrow when my laptop battery will not be empty anymore...

I have updated the PR description with what this PR is doing...

@pmiossec
git-tfs member

GetRootChangesetForBranch produce stack overflow when UPPER_CASE_BRANCH renamed to lower_case_branch

I have included your 2 commits in the PR. Is the problem solved by the remove of the 2 .ToLower()? Or does that still need to be investigated?

@KindDragon

I cannot reproduce bug anymore

@KindDragon

@pmiossec How to clone a single branch with all parent branches and branches that have been merged into this branch?

@pmiossec
git-tfs member
@theyeag

I'm testing this build and it looks like the code is causing a loop.

My command: git-tfs.exe clone --with-branches http://tfs:port/tfs/project $/project/rootBranch NewProjectName

Output

Branches to Initialize successively :
-$/project/serviceBranch (6580)
=> Working on TFS branch : $/project/branch/
Fetching from dependent TFS remote Dev...
Fetching from dependent TFS remote branch/branch1...
etc...
Branches to Initialize successively :
-$/project/serviceBranch (6580)
=> Working on TFS branch : $/project/branch/
Fetching from dependent TFS remote Dev...
Fetching from dependent TFS remote branch/branch1...
etc...

This was looping for about a 1 1/2 days with both tests.

Is there anything I can do to help debug this?

@pmiossec
git-tfs member

@theyeag your log is a little light ;) but I think (and I hope) I know where your problem come from...

I will update this pull request with a correction but the bad news is that you end in a loop because git-tfs was unable to initialized a branch. I think that it was unable to find the root changeset of the branch :(

@theyeag

Ha, I was trying to get it down to the bare minimum. I hate diving through useless information so I try to save others where I can :). I have a guess to what may have caused this.
I made some baseless merges and deletions to reparent a branch from Main -> Dev -> Branch to Main -> Dev -> Branch -> SubBranch.

Let me know if I can provide any logs, tests, or examples. I'm digging into the code but my C# is a little rusty. It may be a bit before I'm much use.

@pmiossec
git-tfs member

@theyeag here is a new version. I hope it will solve the problem of the loop and better know why one of your branch couldn't be initialized...

I made some baseless merges and deletions to reparent a branch

Basless merge should be supported and deletion too (but perhaps not all cases possible)...

@spraints spraints and 1 other commented on an outdated diff Feb 24, 2014
GitTfs/Core/TfsChangeset.cs
@@ -33,13 +33,24 @@ public LogEntry Apply(string lastCommit, IGitTreeModifier treeBuilder, ITfsWorks
var resolver = new PathResolver(Summary.Remote, initialTree);
var sieve = new ChangeSieve(_changeset, resolver);
workspace.Get(_changeset.ChangesetId, sieve.GetChangesToFetch());
- foreach (var change in sieve.GetChangesToApply())
+ var changes = sieve.GetChangesToApply().ToList();
+ if (!IsRenameChangeset(changes, workspace.Remote.TfsRepositoryPath))
@spraints
git-tfs member

This seems like a really broad exclusion. I read it as "skip any changeset that has a deleted item outside of the directory I'm working in." I'm concerned that a changeset with changes like this:

  • Edit $/Project/FileICareAbout.cs
  • Delete $/OtherProject/FileIDoNotCareAbout.cs

will get skipped.

Also, I'd really prefer that logic like IsRenameChangeset lives in ChangeSieve so that it can be covered more easily by unit tests.

@pmiossec
git-tfs member

....and then prevent all the delete when the changeset is a renamed changeset!

I will have a look at how to solve that in a better way....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@spraints spraints commented on an outdated diff Feb 24, 2014
GitTfsTest/Commands/InitBranchTest.cs
newBranchRemote.Expect(r => r.RemoteRef).Return("refs/remote/tfs/" + GIT_BRANCH_TO_INIT).Repeat.Once();
newBranchRemote.Expect(r => r.Fetch(Arg<bool>.Is.Anything)).Return(new GitTfsRemote.FetchResult(){IsSuccess = true}).Repeat.Once();
newBranchRemote.MaxCommitHash = "sha1AfterFetch";
- gitRepository.Expect(x => x.CreateBranch("refs/remote/tfs/" + GIT_BRANCH_TO_INIT, "sha1BeforeFetch")).Return(true).Repeat.Once();
- gitRepository.Expect(x => x.CreateBranch("refs/heads/" + GIT_BRANCH_TO_INIT, "sha1AfterFetch")).Return(true).Repeat.Once();
+ //gitRepository.Expect(x => x.CreateBranch("refs/remote/tfs/" + GIT_BRANCH_TO_INIT, "sha1BeforeFetch")).Return(true).Repeat.Once();
+ //gitRepository.Expect(x => x.CreateBranch("refs/heads/" + GIT_BRANCH_TO_INIT, "sha1AfterFetch")).Return(true).Repeat.Once();
@spraints
git-tfs member

Please remove commented-out code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@spraints spraints and 1 other commented on an outdated diff Feb 24, 2014
GitTfs/Util/ChangeSieve.cs
@@ -77,6 +77,13 @@ public IEnumerable<ApplicableChange> GetChangesToApply()
compartments.Updated.Add(ApplicableChange.Update(change.GitPath, change.Info.Mode));
}
}
+ if (change.Change.Item.ItemType == TfsItemType.Folder
+ && change.GitPath == string.Empty
+ && change.Change.ChangeType.IncludesOneOf(TfsChangeType.Delete))
+ {
+ compartments.Deleted.Add(ApplicableChange.Delete(change.GitPath));
+ }
+
@spraints
git-tfs member

I don't understand why this is here? Why do we care about deleting folders outside of the git directory?

Can you add a description of what this is doing and/or tests to ChangeSieveTests that illustrate this case?

@pmiossec
git-tfs member

This code (and its other part in TfsChangeset.cs) is here is to detect and prevent the deletion of all the files of the changeset of the current branch which occurs when deleting or renaming (which is a delete + a creation) a branch.
The complete delete of all files make no sense with git and introduce some noise so I preferred to prevent it!
Here is the only way I found to prevent it. It appears more as a workaround as it was one month ago before the refactoring of this part of git-tfs :(

In fact, I detect the delete of the main folder because in this case change.GitPath return string.Empty and I add a special 'delete change' that permit to detect in TfsChangeset.cs that the changest is a rename changeset.........

@pmiossec
git-tfs member

I have found a better way to do all that. Thanks for challenging me ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@spraints
git-tfs member

@pmiossec - ❤️ This looks like a pretty awesome improvement!
@KindDragon and @theyeag - thanks for your work on this!

Other than the inline comments, can you

@KindDragon

@pmiossec Clone trunk crash for me in that line KindDragon@494b56e

@pmiossec
git-tfs member

@KindDragon I have cherry-picked your commit but if you end here, I think there is a case that is not well managed by the PR :(

And I am interested by knowing what is this case. Delete branch? something else?

@KindDragon

I will try reproduce this issue again

@KindDragon
       trunk
         |
         +-+
         | BGFX
         | ^
         | |
         | +-------+
         | |  DVLP_BRNCH2
         | |       ^
         | |       |
         | |       +
         | |  DVLP_BRNCH1
         | +-------+
         | |
         +-+
         |
    first commit

My issue occur when git-tfs try fetch branch DVLP_BRNCH2, but branch DVLP_BRNCH1 has not been fetched. FindTfsRemoteOfChangeset should somehow return all not fetched parent branches.

@pmiossec
git-tfs member

@KindDragon I have done something to manage the case. Could you try what I've done?

@KindDragon

@pmiossec Now it return error

branch DVLP_BRNCH1
error: branch not found. Verify that all the folders have been converted to branches (or something else :().
@KindDragon

This commit fix issue KindDragon@2d24756

@pmiossec
git-tfs member

@KindDragon thanks for debugging that (and all the other things ;))

@KindDragon

@pmiossec Another fix :-) for some branch with complex merge history KindDragon@a3ea5f8
I'm not fully understand how TrackMerges works, maybe that new condition in GetMergeInfo could be implemented differently. I'm not sure that GetRootChangesetForBranch for VS2008 works in same way as for Post.VS2010

@KindDragon

Slowest clone part now is fetching specific branch version in Sep.Git.Tfs.Core.TfsChangeset.Apply #522

@pmiossec
git-tfs member

@KindDragon I have rebased my work on master.

I have rebased all your work into my branch. You could find your work in this branch https://github.com/pmiossec/git-tfs/tree/KindDragon_optimized and verify that I have made a good work by comparing to the branch https://github.com/pmiossec/git-tfs/tree/allchanges_to_verify_integrity

Either if I am wrong, there is no changes missing... There is just one more ( a try catch that you remove during a merge). I don't know if it was intended!

I have also found 2 regressions in your work :(

  • The first one is 'badly' corrected by the last commit in the branch. You test if the repository directory exist to know if the git repository is intitialized which is not always tree. I always create the directory first and clone after. And it won't work :(
  • The second one lay in the commit ParseEntries() called only when really needed which prevent the job done in the commit pmiossec@dfdc50c

Another fix :-) for some branch with complex merge history KindDragon@a3ea5f8

I can't cherry-pick this commit because it is based on others of your previous. I will see what I could do...

I'm not fully understand how TrackMerges works, maybe that new condition in GetMergeInfo could be implemented differently.

I don't understand all what tfs api call do too :(

@pmiossec
git-tfs member

@KindDragon I have also include in master some of your commits (that I could have tested).

@KindDragon

@pmiossec Thank you. I will check it later

My current problem with exception new NotImplementedException("TODO: Manage renamed branch by integrating IniTBranch.InitBranchSupportingRename() here")
I think we can remove this exception from FindTfsRemoteOfChangeset, because we call now FindMergedRemoteAndFetch if sha not found or GetRootChangesetForBranch can also return all root branches and remove call FindMergedRemoteAndFetch.

@pmiossec
git-tfs member

@KindDragon oups! I missed this case. Can you test the quick fix I made. I hope it will do it...

@KindDragon

@pmiossec It seems changes have helped, but I can not try clone from scratch soon.

@KindDragon

The first one is 'badly' corrected by the last commit in the branch. You test if the repository directory exist to know if the git repository is intitialized which is not always tree. I always create the directory first and clone after. And it won't work :(

Thank you for fix

The second one lay in the commit ParseEntries() called only when really needed which prevent the job done in the commit pmiossec@dfdc50c

I fix in commit KindDragon@cb14242

@KindDragon KindDragon and 1 other commented on an outdated diff Mar 4, 2014
GitTfs.VsCommon/TfsHelper.PostVs2010.Common.cs
if (merge == null)
{
- throw new GitTfsException("An unexpected error occured when trying to find the root changeset.\nFailed to find root changeset for " + tfsPathBranchToCreate + " branch in " + tfsPathParentBranch + " branch");
+ merge = merges.FirstOrDefault(m => m.SourceChangeType.HasFlag(ChangeType.Rename)
+ || m.SourceChangeType.HasFlag(ChangeType.SourceRename));

Code does not work in my case:

tfsPathParentBranch = trunk
tfsPathBranchToCreate = $/TFS/branches/custom
merges: {`Delete, SourceRename` C100 `$/TFS/branches/dvlp` Source `Merge` C100 `$/TFS/branches/custom`}

This should fix problem KindDragon@ffb97fb

@pmiossec
git-tfs member
pmiossec added a note Mar 4, 2014

For me it cause an end loop on my test repository :( :
git tfs clone https://tfs.codeplex.com:443/tfs/TFS16 $/vtccds/trunk . --with-branches --username=vtccds_cp@snd --password=vtccds

I rewrote commit, can you try to cherry pick it again

@pmiossec
git-tfs member
pmiossec added a note Mar 4, 2014

A lot better ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@pmiossec pmiossec referenced this pull request Mar 12, 2014
Merged

Small improvements #530

@KindDragon

Fetch big branch profiling:
fetch

@pmiossec
git-tfs member

@KindDragon I have seen that you have rebased your work on master. I have done the same thing on the same commit to be able to synchronize our developments. I really want we do it because otherwise it's a little difficult for me to follow all your developments...

I have rebased all your commits onto my dev branch rename_branch. Here is the result. I have compared to your working branch and there is no differences (except 1 or 2 very unimportant details --some function move during conflicts resolution--). Can you take 5 minutes to verify that (with GitExtensions ;) ) and hard reset your branch to this last commit to be synchronized. It will easy a lot future developments... I hope you understand...

@pmiossec
git-tfs member

I have worked a little more on that to go further. With a rebase -i, I managed to take all the commits I wanted and include them in this pull request. The last 3 commits I didn't take should, for me, be part of 2 others later PR...

We should also fix the 17 unit tests broken. 16 broken by commit "Fetch checks that branch is initialized" that also prevent cloning :( (I have removed the commit from this PR) and 1 was broken by pmiossec@2868d70 !

In the previous message, I ask you if you could reset your dev branch to optimized branch I tell you. In fact, you MUST because you did a bad merge when rebasing and you lost some code :( In 'changeSieve.cs' file...

I have also find another regression that I must investigate (but later because I have no more battery), but we could not clone by giving the folder '.' if we are in an empty directory (crash!).

Otherwise, I successfully did a good clone ;)

@KindDragon

@KindDragon I have seen that you have rebased your work on master. I have done the same thing on the same commit to be able to synchronize our developments. I really want we do it because otherwise it's a little difficult for me to follow all your developments...

Cool! I did not know you watch too for my commits

16 broken by commit "Fetch checks that branch is initialized" that also prevent cloning :( (I have removed the commit from this PR)

Yes, I found this too yesterday. It turns out Clone command uses fetch command. Maybe InitBranch more suitable for Clone now?

@KindDragon

Optimized branch was my internal branch, with some stuff that I want in feature move to separate pull request.

I will create branch rename_branch based on your rebased rename_branch branch with commits that dependents on your pull request.

@Dahko

Tried the current state of pmiossec:rename_branch with my difficut case of fetching history of the moved folder (http://stackoverflow.com/questions/22524750/preserve-history-of-moved-tfs-folders-in-git-tfs)
The result was git-tfs fetched only the "new" commits, and twice, and in reverse order. The resulting history looks like:
oldest_but_one_new_commit (HEAD)
....
last_new_commit
oldest_new_commit
....
last_new_commit

So, completely screwed. I can provide more info if you requested.

@KindDragon

Could you clone using latest build from this branch with --debug option and send us output?

@PeteW

Philippe - I have ugly TFS repo :(
This is a good test for the code :) and using your wonderful changes I was successful.
It takes a few fixes! Let me tell you...

1: Bug in Sep.Git.Tfs.Core.GitTfsRemote::FindOrInitTfsRemoteOfChangeset(..). Finding the remote repository for a parent changeset doesnt always work right. GetPathInGitRepo fails when I have 2 branches [$/Pete-3.1] and [$/Pete-3.1.1] because both start with "$/Pete-3.1" they will both be returned as matches. If the wrong repo is returned, searching for a specific commit will fail. This causes infinite loop. By adding slash at the end or something you can fix infinite loop problem

2: This is not pretty. Some joker somehow changed TFS branch heirarchy. Branch X used to be child of Branch Y, but is now child of Z. This crashes the code in TfsHelper.PostVs2010.Common::GetRootChangeSetForBranch. How do you detect this and correct it? I'm still trying to figure that out because TFS api (VersionControl.QueryRootBranchObjects(RecursionType.Full)) gives you false information and cannot be trusted. I fixed this for myself by altering the AllTfsBranches dictionary with a custom correction and I was OK. I still trying to figure out a more generic way to cover this case, there seems to be no good way to find the root changeset for a branch. Perhaps a command-line option for override branch heriarchy definition is something you would consider.

@pmiossec
git-tfs member

@PeteW I'm glad for you ;)

For your problems, you are the best placed to solve them and test them (a thing that you seem to have done successfully!). So could you do pull requests, at least for the first case. I will be happy to merge it in this pull request!

For the 2nd problem, it will be difficult to solve if TFS give us bad information :( Perhaps accept a input file where user specify branch parents and root changeset Id...

Feel free to do pull request even for not finished developments for us to be able to see what you do and let us talk about that. Just prefix your pull request with '[WIP]' and we will not merge it ;)

@PeteW

Pull request sent to your rename_branch. It seems youve made many changes in the past few days I hope I merged it correctly but either way I think you will see what I am trying to do

@thuannguy

I got the latest code at git://github.com/git-tfs/git-tfs.git to local and ran:
d:\Migration\GitTfs-trunk>git-tfs.exe clone -d "https://..." "$/.../Dev" "/Repo/..." --export --with-branches
I got the error below. Could you please tell me how to resolve it?

Try fetching changesets...
info: refs/remotes/tfs/default: Getting changesets from 313 to current ...
Looking for changeset 313 in git repository...
=> Commit not found!
Looking for changeset 313 in git repository...
=> Commit not found!
Changesets fetched!
Cleaning...
git command: Starting process: git gc
git stderr: error: failed to run repack
git command time: [00:00:00.3123424] gc
Sep.Git.Tfs.Core.GitCommandException: Command exited with error code: 255
at Sep.Git.Tfs.Core.GitHelpers.Close(Process process) in D:\Freelancer\GitSou
rce\git-tfs\GitTfs\Core\GitHelpers.cs:line 212
at Sep.Git.Tfs.Core.GitHelpers.<>c__DisplayClass8.b__7() i
n D:\Freelancer\GitSource\git-tfs\GitTfs\Core\GitHelpers.cs:line 65
at Sep.Git.Tfs.Core.GitHelpers.Time(String[] command, Action action) in D:\Fr
eelancer\GitSource\git-tfs\GitTfs\Core\GitHelpers.cs:line 187
at Sep.Git.Tfs.Core.GitHelpers.CommandOutputPipe(Action1 handleOutput, Strin
g[] command) in D:\Freelancer\GitSource\git-tfs\GitTfs\Core\GitHelpers.cs:line 6
0
at Sep.Git.Tfs.Core.GitHelpers.CommandNoisy(String[] command) in D:\Freelance
r\GitSource\git-tfs\GitTfs\Core\GitHelpers.cs:line 52
at Sep.Git.Tfs.Core.GitRepository.GarbageCollect(Boolean auto, String additio
nalMessage) in D:\Freelancer\GitSource\git-tfs\GitTfs\Core\GitRepository.cs:line
557
Warning:
git gc` failed!
warning: Some Tfs branches could not have been initialized:

  • $/[...]/Released/4.1
  • $/[...]/Dev

Please report this case to the git-tfs developers! (report here : https://github
.com/git-tfs/git-tfs/issues/461 )
git command: Starting process: git merge refs/remotes/tfs/default
git command time: [00:00:02.1707887] merge refs/remotes/tfs/default

@PeteW

According to my experiences this means it is looking for changeset 313 but it is not finding it because it is looking in the wrong branch. This happened for me when I had TFS branches that were proper substrings of each other such as $/Pete/3.1 and $/Pete/3.1.1 this confused the system and I sent a fix for that. If that is not your case then I dont know

@thuannguy

Hi PeteW,
Thank you for your comment. I checked my source code. Changeset 313 belongs to a Main branch which doesn't appear on the log above. Let me explain my situation here. The hierarchy used to be:
Main --- Dev
--- Release3.0
--- Release4.1

Before migrating to GIT, I decided to cut the Main branch off because we haven't used it in reality. I dropped Release3.0 branch as well because it is no longer in use. I used the reparent functionality to change the hierarchy to:

Dev -- Release 4.1
Main -- Release 3.0 (there is no relationship between Main and Dev anymore)

I guess it is the reason that I got "Looking for changeset 313 in git repository... => Commit not found!" But is that error the root cause of the "git gc" exception? I hope some experts can give me a hint about how to fix this error :)

@PeteW

It is not because of the git gc. Git gc seems to cause errors for me as well but it is because some files are still locked by the process when trying to invoke the garbage cleaner. Git gc errors are harmless in my experience. The fact that you re-parented branches is more than likely the issue. Some joker did the same thing in my TFS repo and it ended up confusing the clone functionality. I cannot say for certain that we have the same issue because I was receiving a different error. I worked around the error by modifying the source to override the branch heirarchy that TFS reports.

@thuannguy

Thanks PeteW 👍 I decided to dodge this problem by changing the requirement. After all, I need history of the dev branch only. I migrated that branch alone and brought the other branches to GIT manually. Problem solved!

@KindDragon

@pmiossec I think baseless merge should be as cherry pick commit (with only one parent), because it's often used to merge only some changesets.

It would be correct to call VersionControlServer.GetMergeCandidates for merged changeset and add it as parent only if no candidates returned. What do you think?

@KindDragon

@pmiossec BTW, I think we can try use git-replace to add parent to old commit.

@spraints
git-tfs member

This branch currently clocks in at

27 changed files with 899 additions and 326 deletions

Honestly, I kind of dread getting notifications about it, and I tend to avoid looking at it. I assume there are stable parts of this that we could merge to master. Can you split it up into smaller chunks that we can review and merge more quickly?

@pmiossec
git-tfs member

@pmiossec I think baseless merge should be as cherry pick commit (with only one parent), because it's often used to merge only some changesets.

Not sure to see the gain of doing that. Perhaps we should do like a cherrypick if there is only one changeset (to have a cleaner history) otherwise do a normal baseless merge (because we want to keep the interesting history of changesets).

But it's something that we won't do in that PR which become quite big (like @spraints said)...

@pmiossec BTW, I think we can try use git-replace to add parent to old commit.

Very interesting! I didn't know that command. It's really a thing that we should do! But again, I think it should not be in this PR.

I will open a PR where we will put all the things that should be done after this PR is merged...

@pmiossec pmiossec referenced this pull request Apr 21, 2014
Open

Improvements post #480 #587

0 of 2 tasks complete
@pmiossec
git-tfs member

I assume there are stable parts of this that we could merge to master. Can you split it up into smaller chunks that we can review and merge more quickly?

I have created #586 that could be merged without problem. Except from that, that's only a whole piece that should be merged in one time :(
When I begun that work I thought it would have been shorter... and more quickly merged!
But now that we have merged the master branch in our dev branch (because it was error prone to rebase) it will be much more difficult to do another thing that merge all the branch.

My plan is to release a last version of git-tfs before merging this development...

Retrospectively, my big mistake was to make this needed refactoring ( move initialization of a new branch in GitTfsRemote : pmiossec@5180e88 ) in this dev branch instead of in master before basing all my work on that. In my defense, once I saw it was needed, it was too late...

I will have a look if something could be done but I think there is no chances :(

@pmiossec
git-tfs member

It took quite some time but I have rebased all the work done into master (removing some commits already merged into master) and now it's ready to move forward!!! All should be on good shape...

@irontoby

So I'm trying to follow along with this discussion but I must admit that TFS's branch handling has me very confounded and I can't make any sense of it. I built your latest code from this PR and tried to run it against $/Project/Main which is our "default"/trunk branch. That branch in turn has a branch $/Project/QA/QA and that branch has a branch $/Project/Dev/Dev (don't ask why they nested them like that...)

So I got your latest code and ran the following:

$ git tfs clone http://hpsjvltfs01:8080/tfs/DefaultCollection $/Project/Main --workspace="c:\ws"

Initialized empty Git repository in C:/dir/Main/.git/
Fetching from TFS remote 'default'...
C152361 = e3f215c435411304031cb1360e8034e37049c7fc
C152362 = 62e41086f052be3c62ded22a397b1ff44397ddce
error: a problem occured when trying to clone the repository. Try to solve the p
roblem described below.
In any case, after, try to continue using command `git tfs fetch`

An unexpected error occured when trying to find the root changeset.
Failed to find root changeset for $/Project/Dev/Dev branch in $/Project/QA/QA branch

So is there anything I can do in this case, or am I pretty much hosed in trying to get a reasonable clone into git?

Just as a personal note, thanks so much for all your work on this, I'm especially sympathetic because 6 years ago I faced many the same problems reverse-engineering a different MS VCS, writing a tool to convert from VSS to SVN :)

@pmiossec
git-tfs member

So is there anything I can do in this case

First, can you give more log by add the option -d to the command launched? Put it in a gist if it's too long!

Second, yes, you can try to find what's wrong inGetRootChangesetForBranch()

or am I pretty much hosed in trying to get a reasonable clone into git?

At the moment, I think so :( You should be in a case not supported by git-tfs. The only one I know for the moment is where a branch has been "re-parented". It's parent has been changed. but there could be others...

I have an idea to support these cases but I want this PR merged before doing another big development. My idea is to let the user describe in a file the branches relations with also the id of the root changeset and git-tfs will follow these data when he do not succeed to find it himself. With that, I hope we will be able to support every cases even if it's not the best solution....

@pmiossec
git-tfs member

First, can you give more log by add the option -d to the command launched?

Put it in a gist if it's too long!

@irontoby

@pmiossec no I certainly understand wanting to get this merged! I was running from command line so didn't have a chance to debug into GetRootChangesetForBranch() but I'll do that next.

Here is the gist with my -d output, I cut out all the file names since I don't think my company would want it posted publicly but if that helps for some reason I could send you a private gist.
https://gist.github.com/irontoby/2060d6768e73129609e4

@PeteW

@irontoby I dealt with re-parented branch in TFS. This is a nasty problem because the TFS API gives you misleading information. My march 21st comment (part 2) discusses how I hacked around this by hard-coding the dictionary values to the correct settings. See Sep.Git.Tfs.VsCommon.TfsHelperVs2010Base::AllTfsBranches_get method. override the dictionary AFTER it has been fetched by TFS but BEFORE it is returned. Sorry I wish I had more elegant solution but as pmoissec explained it is not easy fix.

@pmiossec
git-tfs member

@irontoby I have made a quick development here #615
When the root changeset id is not found by git-tfs, it is asked to the user. You will have to have a look in your tfs history to find the good changeset id but that's the only thing that I can do to solve your problem quickly.

@PeteW
@irontoby

@pmiossec okay I'll test it out when I get a chance, many thanks. It's my employer's TFS repo so being able to test it out depends on getting some free moments at work. I wasn't expecting you to solve my problem quickly just wanted to give you extra data to work with :)

The changes you've made here have already helped me out, because now I can clone our "hotfix" TFS branches into a new repo (which previously git-tfs would complain it couldn't find the root for). So even though it still doesn't "tie them together" as a proper merge commit, I can at least add them as a new git remote and cherry-pick in what I need.

Thanks!

@spraints
git-tfs member

I don't have time to review this right now. @sc68cal or @KindDragon - I'll take a 👍 from either of you in place of my review.

pmiossec and others added some commits Sep 10, 2013
@pmiossec pmiossec Find the REAL root changeset when there is rename changeset(s) and ge…
…t the list of successive branches...

and manage case where using flag '--nofetch' but with renamed branches
f245095
@pmiossec pmiossec Add debug datas f93dbc9
@pmiossec pmiossec Manage rename tfs branch the good way 46e5c6b
@pmiossec pmiossec Retrieve also deleted tfs branches if needed 250286d
@pmiossec pmiossec Don't create git branch when renamed branch! 27ebc9b
@pmiossec pmiossec Delete remotes corresponding to renamed branches 29df90f
@pmiossec pmiossec Manage and don't stop initializing branches if some fails (and report…
… at the end of the process)
7cc7851
@pmiossec pmiossec Finding root changeset: better finding of renaming
- Protection against find itself as parent branch
- Support of renaming of renaming
- keep the last "renamed" merge is important when no root found (with what is supposed to be the root branch)

+ catch problem when finding root changesets
189b08f
@pmiossec pmiossec Correct UnitTests 74661c4
@pmiossec pmiossec Move InitBranch() in the GitTfsRemote (to be able to init one from th…
…ere!)
4a62b6b
@pmiossec pmiossec write what should be done... c68bd41
@pmiossec pmiossec being able to fetch until a changeset is reached f6e9cf4
@pmiossec pmiossec Create on the fly the missing remote (with fake Sha1 for the moment) 2be7917
@pmiossec pmiossec Fetch just changesets needed (until merge changeset) f15debb
@pmiossec pmiossec Find the sha of the root changeset of the tfs branch to create 0d5ed20
@pmiossec pmiossec better way to find if changesets fetched e114350
@pmiossec pmiossec Get the TfsRemotes if branch was already initialized f759c4d
@pmiossec pmiossec Introduce a config value GitTfsConstants.AutoInitBranches
...to know if we should auto init branch when merge is encountered
 and the branch not actualy initialized
56fc7fd
@pmiossec pmiossec Add support of baseless merge
Git-tfs now support to merge a root branch (which is not branched from an already initialized branch) in another branch.
The root branch is now initialize on the fly and the merge commit created...
f805917
@pmiossec pmiossec Create local branch only when it don't already existing (to prevent c…
…rash)...
95bde55
@pmiossec pmiossec Add an option "-ignore-branches" to ignore merged branch when fetchin…
…g in merge changeset

It correspond to the old way git-tfs worked!
971abcb
@pmiossec pmiossec disable branch support in `subtree`command 9ca63cd
@KindDragon KindDragon Logging improved 3d5119e
@KindDragon KindDragon FetchWithMerge method refactored fb6e78c
@pmiossec pmiossec Correct Unit Test f594f98
@pmiossec pmiossec Add unit test to test auto-init of branches and `--ignore-branches` o…
…ption
6f4d149
@pmiossec pmiossec preventing crash when init branch is no more needed
...and should correct end loop when fail to find root changeset
cf9dfd4
@pmiossec pmiossec refactoring / cleaning 88dfce4
@pmiossec pmiossec refacto InitBranchSupportingRename acc8b11
@KindDragon KindDragon FindMergedRemoteAndFetch crash fixed 7f4ce2d
@pmiossec pmiossec FindTfsRemoteOfChangeset : Add message when not finding commit 8ca3abc
@pmiossec pmiossec try to intialize the parent branch when commit not found... ff819a2
@pmiossec pmiossec Doc & release note 7b3d082
@KindDragon KindDragon Parent branch fetching fixed 0895882
@pmiossec pmiossec Managing branch renaming in FindTfsRemoteOfChangeset() b1f7910
@KindDragon KindDragon Renamed branch handling improved 591323c
@KindDragon KindDragon Minor changes in GetMergeInfo() e62cd02
@KindDragon KindDragon VersionControlException exception handling improved 6a716a0
@KindDragon KindDragon ParseEntries() called only when really needed 9b4c538
@KindDragon KindDragon RootBranch DebbuggerDisplay attribute added 8ed59ef
@KindDragon KindDragon Support resume interrupted clone in debug mode e96a09d
@KindDragon KindDragon Check parent branch history only before current branch changeset 74abd9e
@KindDragon KindDragon Apply() moved after ProcessMergeChangeset() 00ae6d4
@KindDragon KindDragon FetchWithMerge trace improved fe7c3d6
@KindDragon KindDragon InitBranch support fetching parent branch e693572
@KindDragon KindDragon Option added to ignore not initialized branches on fetch 5c30b7e
@KindDragon KindDragon OmittedParentBranch set when IgnoreBranches == true 129e648
@KindDragon KindDragon `branch init` support initializing by specifying git branch or git re…
…mote
6b6dcde
@KindDragon KindDragon Test `ShouldDoNothingBecauseRemoteAlreadyExisting` fixed e5b24d6
@pmiossec pmiossec Ask user when don't find root changeset a702198
@pmiossec
git-tfs member

I plan to merge that very soon (one or two days) and continue building on top except if someone yell against it...

It was waiting too long!

cc @spraints @KindDragon

@pmiossec pmiossec merged commit ee86eed into git-tfs:master Jul 10, 2014

1 check passed

Details continuous-integration/travis-ci The Travis CI build passed
@pmiossec pmiossec referenced this pull request Jul 10, 2014
Closed

Fix missing merge parent #628

@KindDragon

@pmiossec I think we should change git-tfs pull default behavior for old repositories, it shouldn't initialize new branches automatically or it's break current git-tfs workflow in my company.

Maybe made ignore-not-init-branches = true for old repositories

@pmiossec pmiossec deleted the pmiossec:rename_branch branch Jul 13, 2014
@pmiossec
git-tfs member

Perhaps you're right. I'll think about that. How to change that. Perhaps we will have to change the state of the internal parameter...

@KindDragon

Or maybe merge options ignore-branches and ignore-not-init-branches to one with 3 states:
1. Fetch all merged branches (default for new repositories)
2. Fetch only initialized merged branches (default for old repositories)
3. Don't fetch merged branches

@spraints
git-tfs member

I plan to merge that very soon (one or two days) and continue building on top except if someone yell against it...

It was waiting too long!

🙇 Thanks for your patience, I think this is a good time to have merged this.

@StephenHurn

Unfortunately I have been having some trouble using this feature.

I have a convoluted branch/merge heirarchy on the TFS server that I am attempting to pull from.

Project A branches to a subfolder of Project B and H
Project B branches to a subfolder of Project C, D and H
Project C branches to a subfolder of E, F, G and H
Project F branches to project H

I want Project F and project H

Unfortunately I don't seem to be able to clone F, with H as a branch without also having git-tfs attempt to clone A, B and C. The --ignore-branches switch does not seem to work with --with-branches.

My goal: I want to clone a TFS repository and one of the branches that originated from this repository.
What was happening: I got hit by the rename bug and could not initialize the new branch.
What is happening now: I can now initialize the new branch, but not without also initializing every other branch that this new branch has ever had a merge changeset from.

There could be a couple of solutions to this problem:

1) git tfs branch could implement an --ignore-branches option
2) git tfs clone could allow the user to specify which branches to initialize during the clone (--with-branch=VALUE)

Unless I'm missing something?

@StephenHurn

Ok it turns out that I was missing something.

I achieved my goal by doing the following:
1) Cloning using the --ignore-branches option.
2) run the command "git config git-tfs.ignore-not-init-branches "true""
3) Init the branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment