WIP use TreeDefinition instead of index to make trees

[kgybels]

No description provided.

[sc68cal]

[sc68cal]

[KindDragon]

👍

[KindDragon]

👍

[KindDragon]

   Clone in this PR works several times faster for me :+1: :+1: :+1: 

My mistake, a little faster

[KindDragon]

   Clone in this PR works several times faster for me :+1: :+1: :+1: 

My mistake, a little faster

[pmiossec]

I think you should let the previous value (and add git to your path). Everyone doesn't install git in the "program files" folder....

[pmiossec]

I think you should let the previous value (and add git to your path). Everyone doesn't install git in the "program files" folder....

[pmiossec]

I think you should let the previous value (and add git to your path). Everyone doesn't install git in the "program files" folder....

[sc68cal]

@KindDragon Awesome - do you have a comparison in runtime between this PR and using the temporary index? It would be a useful stat to see how much faster we get from not shelling out to the Git binary.

[sc68cal]

@KindDragon Awesome - do you have a comparison in runtime between this PR and using the temporary index? It would be a useful stat to see how much faster we get from not shelling out to the Git binary.

[pmiossec]

I have made some tests with my codeplex test repository (with some interesting cases inside) and I end up with exactly the same sha1. So, that's good...

So, after some little cleaning, I am good for merging.

PS :

• I can't see (and measure) improvements because I am constraint by the network.
• Some tests failed on Travis build but not on my computer. I hope Travis will pass in the next build or we will have to have a look why before merging...

[pmiossec]

I have made some tests with my codeplex test repository (with some interesting cases inside) and I end up with exactly the same sha1. So, that's good...

So, after some little cleaning, I am good for merging.

PS :

• I can't see (and measure) improvements because I am constraint by the network.
• Some tests failed on Travis build but not on my computer. I hope Travis will pass in the next build or we will have to have a look why before merging...

[sc68cal]

Going to kick travis again - that's how badly I want this to get merged 😄

[sc68cal]

Going to kick travis again - that's how badly I want this to get merged 😄

[sc68cal]

Ah. I know why. It's because he hard coded in that path to Git, and Travis runs the build and tests on Mono+Linux.

@kgybels please fix that and we should be good to go!

---- System.ComponentModel.Win32Exception : ApplicationName='c:\program files (x86)\git\bin\git.exe', CommandLine='log --no-color --pretty=medium HEAD --', CurrentDirectory=''

[sc68cal]

Ah. I know why. It's because he hard coded in that path to Git, and Travis runs the build and tests on Mono+Linux.

@kgybels please fix that and we should be good to go!

---- System.ComponentModel.Win32Exception : ApplicationName='c:\program files (x86)\git\bin\git.exe', CommandLine='log --no-color --pretty=medium HEAD --', CurrentDirectory=''

[sc68cal]

@pmiossec Would it be useful to investigate automating a smoke test, that does a clone from that codeplex test repository and makes sure the SHA ends up being the right one?

[sc68cal]

@pmiossec Would it be useful to investigate automating a smoke test, that does a clone from that codeplex test repository and makes sure the SHA ends up being the right one?

[pmiossec]

@sc68cal It could be done but this repository is the one I use for development and there is 2 cases not supported by the current version of git-tfs and only supported by my work in progress. The current version of git-tfs can't clone the tfs repository :(

[pmiossec]

@sc68cal It could be done but this repository is the one I use for development and there is 2 cases not supported by the current version of git-tfs and only supported by my work in progress. The current version of git-tfs can't clone the tfs repository :(

[KindDragon]

@KindDragon Awesome - do you have a comparison in runtime between this PR and using the temporary index? It would be a useful stat to see how much faster we get from not shelling out to the Git binary.

My mistake, new version a little faster.
But new code consume a lot more memory. 1,5Gb after 2 hours (old code around 0,5Gb). Maybe we have some memory leaks.

[KindDragon]

@KindDragon Awesome - do you have a comparison in runtime between this PR and using the temporary index? It would be a useful stat to see how much faster we get from not shelling out to the Git binary.

My mistake, new version a little faster.
But new code consume a lot more memory. 1,5Gb after 2 hours (old code around 0,5Gb). Maybe we have some memory leaks.

[KindDragon]

Main problem with GitRepository.ParseEntries not solved

[KindDragon]

Main problem with GitRepository.ParseEntries not solved

[kgybels]

@KindDragon

Main problem with GitRepository.ParseEntries not solved

We need a case-insensitive TreeDefinition to be able to get rid of GitRepository.ParseEntries.
See comment on libgit2/libgit2sharp#587 for more information.

[kgybels]

@KindDragon

Main problem with GitRepository.ParseEntries not solved

We need a case-insensitive TreeDefinition to be able to get rid of GitRepository.ParseEntries.
See comment on libgit2/libgit2sharp#587 for more information.

[kgybels]

VS2013 insists on adding BOMs to all my new code files and changed project files. Is it a problem to leave them in?

[kgybels]

VS2013 insists on adding BOMs to all my new code files and changed project files. Is it a problem to leave them in?

[spraints]

🎸

VS2013 insists on adding BOMs to all my new code files and changed project files. Is it a problem to leave them in?

BOMs should be fine.

Main problem with GitRepository.ParseEntries not solved

We need a case-insensitive TreeDefinition to be able to get rid of GitRepository.ParseEntries.
See comment on libgit2/libgit2sharp#587 for more information.

I wonder if we should replace initialTree with something that can do the same thing (i.e. same interface), but make it lazy and based off of a LibGit2Sharp Tree?

[spraints]

🎸

VS2013 insists on adding BOMs to all my new code files and changed project files. Is it a problem to leave them in?

BOMs should be fine.

Main problem with GitRepository.ParseEntries not solved

We need a case-insensitive TreeDefinition to be able to get rid of GitRepository.ParseEntries.
See comment on libgit2/libgit2sharp#587 for more information.

I wonder if we should replace initialTree with something that can do the same thing (i.e. same interface), but make it lazy and based off of a LibGit2Sharp Tree?

[spraints]

... and then I look at the code, and see that you've started doing this. Doing the case-insensitive tree manipulation should be doable. Instead of _treeDefinition.Add(path, file, Mode.ToFileMode(mode));, we could do something like this (make-believe code with make-believe APIs):

  var parts = path.Split('/');
  var tree = _treeDefinition.Tree;
  while(parts.Length > 1) {
    var subdir = tree.Entries.SingleOrDefault(e => e.Name.ToLower() == parts[0].ToLower());
    if(subdir == null) {
      subdir = new Tree();
      tree.Add(parts[0], subdir);
    }
    tree = subdir;
  }
  tree.Add(parts[0], file, mode);

[spraints]

... and then I look at the code, and see that you've started doing this. Doing the case-insensitive tree manipulation should be doable. Instead of _treeDefinition.Add(path, file, Mode.ToFileMode(mode));, we could do something like this (make-believe code with make-believe APIs):

  var parts = path.Split('/');
  var tree = _treeDefinition.Tree;
  while(parts.Length > 1) {
    var subdir = tree.Entries.SingleOrDefault(e => e.Name.ToLower() == parts[0].ToLower());
    if(subdir == null) {
      subdir = new Tree();
      tree.Add(parts[0], subdir);
    }
    tree = subdir;
  }
  tree.Add(parts[0], file, mode);

[kgybels]

@spraints

I wonder if we should replace initialTree with something that can do the same thing (i.e. same interface), but make it lazy and based off of a LibGit2Sharp Tree?

That is what I did in my last 2 commits, except I used TreeDefinition instead of Tree.

I also think we can consolidate IGitTreeBuilder and IGitTreeInformation. The information of the tree with some changes already applied to it, should be sufficient, I don't think we need the tree exactly like it is "initially" (the parent tree), while applying changes. The responsibility of preserving file mode when a change is an update can also be moved to IGitTreeBuilder entirely. To summarize, with IGitTreeBuilder working case-insensitive and handling the file mode on its own, IGitTreeInformation becomes redundant.

[kgybels]

@spraints

I wonder if we should replace initialTree with something that can do the same thing (i.e. same interface), but make it lazy and based off of a LibGit2Sharp Tree?

That is what I did in my last 2 commits, except I used TreeDefinition instead of Tree.

I also think we can consolidate IGitTreeBuilder and IGitTreeInformation. The information of the tree with some changes already applied to it, should be sufficient, I don't think we need the tree exactly like it is "initially" (the parent tree), while applying changes. The responsibility of preserving file mode when a change is an update can also be moved to IGitTreeBuilder entirely. To summarize, with IGitTreeBuilder working case-insensitive and handling the file mode on its own, IGitTreeInformation becomes redundant.

[kgybels]

@spraints

... and then I look at the code, and see that you've started doing this. Doing the case-insensitive tree manipulation should be doable. Instead of _treeDefinition.Add(path, file, Mode.ToFileMode(mode));

I made TreeDefinition itself handle the case-insensitive part (like this). Still need to think about how to properly expose both use cases, though.

[kgybels]

@spraints

... and then I look at the code, and see that you've started doing this. Doing the case-insensitive tree manipulation should be doable. Instead of _treeDefinition.Add(path, file, Mode.ToFileMode(mode));

I made TreeDefinition itself handle the case-insensitive part (like this). Still need to think about how to properly expose both use cases, though.

[sc68cal]

Looks great @kgybels ! Keep up the great work!

[sc68cal]

Looks great @kgybels ! Keep up the great work!

[kgybels]

@KindDragon Would you be so kind as to rerun your performance benchmark with the latest version of this PR? It would also be nice to know how big the repository that you are testing against is? I am using a repository with about 18000 files and the improvement is not that impressive, but I hope improvement scales with the size of the repository.

[kgybels]

@KindDragon Would you be so kind as to rerun your performance benchmark with the latest version of this PR? It would also be nice to know how big the repository that you are testing against is? I am using a repository with about 18000 files and the improvement is not that impressive, but I hope improvement scales with the size of the repository.

[KindDragon]

Clone approximately 10-20% faster

We can also try to do something with StructureMap.Container.GetInstance(12.16%)

[KindDragon]

Clone approximately 10-20% faster

We can also try to do something with StructureMap.Container.GetInstance(12.16%)

[spraints]

#519 had major conflicts with this. I spent some time with the two branches and git-imerge and I think I have a good merge of the two (spraints/git-tfs@1f75d48).

We need to get libgit2sharp back to the libgit2 repo before merging this. @kgybels - can you work on a PR based on the libgit2sharp version this branch is using (https://github.com/kgybels/libgit2sharp/compare/experimental)?

[spraints]

#519 had major conflicts with this. I spent some time with the two branches and git-imerge and I think I have a good merge of the two (spraints/git-tfs@1f75d48).

We need to get libgit2sharp back to the libgit2 repo before merging this. @kgybels - can you work on a PR based on the libgit2sharp version this branch is using (https://github.com/kgybels/libgit2sharp/compare/experimental)?

[kgybels]

@spraints It's not sure yet that this is something that LibGit2Sharp wants to support (see libgit2/libgit2sharp#587 and libgit2/libgit2sharp#607 for discussion). However, I'm willing to make the PR for it, but if we want any chance of getting it in, it needs to be decent, preferably with unit tests.

The first commit is something we could already put on master, even without the case-insensitive TreeDefinition. It already gets rid of calling git update-index and git write-tree for creating the tree. Once we have a version of LibGit2Sharp with the case-insenstive TreeDefinition, I'll rebase the rest of the work on git-tfs's master.

@KindDragon I toke a quick look at the code where that GetInstance is used. It seems like it is a bad idea to use that in a hotspot like that. However, it seems trivial to rework that code a bit to get rid of it, if only I had more time!

[kgybels]

@spraints It's not sure yet that this is something that LibGit2Sharp wants to support (see libgit2/libgit2sharp#587 and libgit2/libgit2sharp#607 for discussion). However, I'm willing to make the PR for it, but if we want any chance of getting it in, it needs to be decent, preferably with unit tests.

The first commit is something we could already put on master, even without the case-insensitive TreeDefinition. It already gets rid of calling git update-index and git write-tree for creating the tree. Once we have a version of LibGit2Sharp with the case-insenstive TreeDefinition, I'll rebase the rest of the work on git-tfs's master.

@KindDragon I toke a quick look at the code where that GetInstance is used. It seems like it is a bad idea to use that in a hotspot like that. However, it seems trivial to rework that code a bit to get rid of it, if only I had more time!

[spraints]

@spraints It's not sure yet that this is something that LibGit2Sharp wants to support (see libgit2/libgit2sharp#587 and libgit2/libgit2sharp#607 for discussion). However, I'm willing to make the PR for it, but if we want any chance of getting it in, it needs to be decent, preferably with unit tests.

I just read up on that, and I'd be fine with making this solely a concern in git-tfs. IGitTreeDefinition and IGitTreeBuilder both provide places where we can do the necessary work.

The first commit is something we could already put on master, even without the case-insensitive TreeDefinition. It already gets rid of calling git update-index and git write-tree for creating the tree. Once we have a version of LibGit2Sharp with the case-insenstive TreeDefinition, I'll rebase the rest of the work on git-tfs's master.

#529 has just the first commit (and some minor cleanup). Also, merge commits aren't a big deal to me. Feel free to use spraints/git-tfs@1f75d48 as a starting point for working on the case-sensitivity stuff.

[spraints]

@spraints It's not sure yet that this is something that LibGit2Sharp wants to support (see libgit2/libgit2sharp#587 and libgit2/libgit2sharp#607 for discussion). However, I'm willing to make the PR for it, but if we want any chance of getting it in, it needs to be decent, preferably with unit tests.

I just read up on that, and I'd be fine with making this solely a concern in git-tfs. IGitTreeDefinition and IGitTreeBuilder both provide places where we can do the necessary work.

The first commit is something we could already put on master, even without the case-insensitive TreeDefinition. It already gets rid of calling git update-index and git write-tree for creating the tree. Once we have a version of LibGit2Sharp with the case-insenstive TreeDefinition, I'll rebase the rest of the work on git-tfs's master.

#529 has just the first commit (and some minor cleanup). Also, merge commits aren't a big deal to me. Feel free to use spraints/git-tfs@1f75d48 as a starting point for working on the case-sensitivity stuff.

[pmiossec]

Seems merged with #529 (no?)

[pmiossec]

Seems merged with #529 (no?)