Git subtree evaluation for Pharo

Setup of the environment

The main setup of the evaluation will be a ''parent'' repository that will contain a ''child'' repository as subtree. For most of the questions (e.g., commit, push, pull) it is enough having both of them as a git repository in your github user. Pull requests will require some extra work: for example, with @tesonep we created a fork of the repositories in each his own github account to emulate the real use case where people will want to change things in Pharo.

1. Create a parent repository (e.g., pharo)

# in https://github.com/guillep/Subtree_experiment_parent
git clone git@github.com:guillep/Subtree_experiment_parent.git

2. Create a child repository (e.g., glamour)

# in https://github.com/guillep/Subtree_experiment_child
git clone git@github.com:guillep/Subtree_experiment_child.git

3. Install child as subtree of parent

$ git subtree add --prefix=child git@github.com:guillep/Subtree_experiment_child.git master                                                     git fetch git@github.com:guillep/Subtree_experiment_child.git master
warning: no common commits
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From github.com:guillep/Subtree_experiment_child
 * branch            master     -> FETCH_HEAD
Added dir 'child'

This will load the child into parent, and create a couple of commits: first it is the initial commit imported from the child repo into ours; second is the commit that merges or includes the child into our parent repository.

Check for resulting repository: https://github.com/guillep/Subtree_experiment_parent/tree/0ceaea21e3e21354df5f6d63a07333f6576af4e6
https://github.com/guillep/Subtree_experiment_parent/commits/0ceaea21e3e21354df5f6d63a07333f6576af4e6

Q1: How to publish (commit & push) changes?

Scenario 1.1: Change parent but not child

This simple (the simplest) scenario just changes the parent without modifying the child project.

#in parent directory
touch newfile
git add newfile
git commit -m "test"
git push

Committing and pushing from parent works as usual. No extra work is required. The relationship between parent and child still works ok.

Scenario 1.2: change child but not parent. Try add and commit from parent

This scenario checks whether add and commit operations work recursively or not in subtrees. For that we create and add a file in the child subtree:

$ cd child
$ touch newfile2

Going to the parent and checking git status will show that the parent knows exactly what files changed in the child.

$ cd ..
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	child/newfile2

nothing added to commit but untracked files present (use "git add" to track)

git add works from the parent

$ git add child/newfile2

Checking the status again to see that git will still see correctly changes in child.

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   child/newfile2

Finally, we can push the change from the parent.

$ git push
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 373 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To git@github.com:guillep/Subtree_experiment_parent.git
   05a9248..d12a264  master -> master

This push will effectively change the child project inside parent but not the original child project, and we will be able to see it in e.g., github's UI.

• https://github.com/guillep/Subtree_experiment_parent/tree/d12a26477983f28ff1cee148c5b437bf933f7ddb/child
• https://github.com/guillep/Subtree_experiment_child

This shows that child behaves as a normal subfolder. Operations add, commit and push are recursive in subtrees.

Q1 General Conclusion

Subtrees work as normal directories for operations such as add, commit, pull and push. These operations will work recursively on subdirectories as in any normal git repository. A parent evolves together with its subtrees.

Q2: How to get (clone) a repository?

Scenario 2.1: clone latest version

This scenario evaluates what steps are to be done when cloning a repository with submodules. For this, we clone in a separate directory the repository, initialize the submodules, and then we check the status of the submodule.

git clone git@github.com:guillep/Submodule_experiment_parent.git parent2
git submodule init
git submodule update

cd Submodule_experiment_child
git status

This shows that:

• the required commit was loaded as demanded
• however, the submodule is in a "detached head mode". This means that it is just the commit but there is no branch used. This means that if any work is meant to do in such submodule, a branch should be created or checked out, otherwise, there is a risk of losing all commits.

Scenario 2.2: clone specific (older version) should clone correct child

This scenario checks if the link between parent and child really is versioned. For this we take and old version of the parent repository that should point to an old version of the child repository, and we check that what is fetched in the child is the correct commit.

git clone ... or git init, fetch, pull...
git reset --hard 100c67d15ca037680044e30f70b0fc75cd6f5143

git submodule init
git submodule update

Again, here the loaded commit is the correct one, and we have the same dettached head problem as in scenario 2.

Scenario 2.3: clone parent whose child commit has not been pushed!

This scenario checks what happens when we clone a repository that has inconsistent child submodules. That is, submodules whose commits were not pushed.

#commit but not push child
touch newfiletopull
git add newfiletopull
git commit -m "newfiletopull"
    
cd ..
git add Submodule_experiment_child
git commit -m "updated submodule that is not yet pushed"
git push

Cloning the project in a separate directory will work, but the submodule initialization will fail.

#clone in new folder
git clone git@github.com:guillep/Submodule_experiment_parent.git parentBadChild
cd parentBadChild
git submodule init
git submodule update #=> cannot checkout!

Q2 General conclusions

Getting/Fetching/Cloning a repository with submodules requires some extra actions:

• submodule init and update
• also this fetches a specific commit and is not aware of branches so as they are they are not editable. Branches should be checked out for each submodule.

Q3: How to update (pull/merge) changes?

Scenario 3.1: change parent from a separate location and pull parent

This scenario checks whether pulling a parent with changes creates some conflict.

#in parent in a second cloned repository
touch newfiletopull
git add newfiletopull
git commit -m "newfiletopull"
git push

#in parent in first repository
git pull

Pulling a parent project just works as a normal project and shows no problems.

Scenario 3.2: change child from a separate location, commit and push parent and then pull parent

This scenario checks what happens when we change a child submodule and we pull that change from parent in a separate directory.

#in child in second cloned repository
touch newfiletopull
git add newfiletopull
git commit -m "newfiletopull"
git push
cd .. #go to parent
git add Submodule_experiment_child
git commit -m "updated submodule"
git push

#in parent in second cloned repository
git pull
git status #=> detects that parent depends on child commit that has to be pulled
git submodule update

This shows that pulling will not update childs directly. For that, we require to do a submodule update. However, this may make us lose local changes in the submodule.

Scenario 3.3: change child independently of parent and then pull in parent

This scenario checks the co-evolution of projects. A submodule can be cloned independently of the parent, be changed and have new commits. What happens in the parent if a submodule has changes.

#in child in second cloned repository
touch newfiletopull
git add newfiletopull
git commit -m "newfiletopull"
git push

We will see that doing a git submodule update will not change the child submodule, because it will load a specific commmit: the specific commit that we used when we committed the parent.

#in parent in second cloned repository
git status #=> does not realize changes in child. This is normal
git pull #=> does nothing
git submodule update

If we want to update the child, we need to go into it and do a pull from the correct repository/branch where the updates are. And then, we need to update the parent to use the new child's commit.

# go to child and pull
cd Submodule_experiment_child
git pull

#need to update parent
cd ..
git add Submodule_experiment_child
git commit -m "update submodule"
git push

Q3 General Conclusions

Updating submodules should be done separately. Also, we should be careful if we have local changes: we need to merge them instead of simply doing a git submodule update.

Q4: How to branch?

Scenario 4.1: create branch in parent

This scenario checks what happens in the child if we create a branch in the parent.

#in parent
git checkout -b test
cd Submodule_experiment_child
git branch #=> still in master. Normal

As submodules evolve separately, branches in them also are manage separately. Branches created in the parent are not seen in the childs and vice-versa. Moreover, as we saw before parents only know childs specific commits but not what branch was used during development.

Scenario 4.2: create branch in child

This scenario checks what happens in the parent when a branch is created in the child.

cd Submodule_experiment_child
git checkout -b test
    
# go to parent
cd ..
git status #=> no changes!

We see that the parent does not know if the child created branches or not. Links between submodules are commit based, no branch based.

Q5: How to do pull requests?

We evaluated in this case a single scenario were in a second forked repository:

• a change was made in child
• parent was updated to point to new commit of child
• then we issued two pull requests: one from child and one from parent.

Q5 Conclusions

Merging these pull requests (or trying to) independently showed the following:

• Making a pull request on the parent will not "drag over" changes in childs. Actually it will just drag "pointers to the childs" but when doing git submodules update it will fail, as if the submodules commit were not pushed. This happens because they were pushed but in the repository of the one that made the pull request, but the repository that is pulling does not have any knowledge of where it is. It is the same problem as in scenario 1.4.

• On the contrary, accepting first the pull requests in the childs will make the parent pull request succeed.

Some options we where thinking that could solve these problems:

• Option 1: issue pull request for child and parents. Then the parent pull request will fail until the childs are integrated. Actually it does not fail, but it cannot build, because submodules cannot be updated, because commits cannot be found.

• Option 2: issue a pull request for parent only and build some infrastructure to support merging childs. Then at validation and integration time, two things should be done:

  • first should take from the pull request the list of changed submodules
  • then a merge should be done from the remote repository into each submodule. How to match each submodule should be done by convention as we do not have any extra information in the pull request.
  • once all submodules are merged, validation can take place, or a new integration can take place by committing and pushing first all submodules, then parent