H-uru and OpenUru.org repositories are incompatible

[cwalther]

Following the discussion in issue #21, I did some experiments and came to the conclusion that Hg-Git is a reliable way (possibly the only one) of losslessly moving commits back and forth between Mercurial and Git, i.e. without losing commit identity and getting duplicates. So far, so good.

So I set out, as an exercise, to merge my cursors branch from my clone of the OpenUru.org Mercurial repository into a clone of the H-uru Git repository – only to find that it doesn’t work. It doesn’t work because the two repositories share no history, they are completely independent from a version control point of view. Their first 7 revisions should be the same, but they aren’t. The difference is that the OpenUru.org one has a .hgignore file and the H-uru one doesn’t. This means that pulling from one into the other will pull in “duplicates” of these revisions, and branches based on one set can’t be merged with branches based on the other set because there is no common ancestor (more precisely, they conflict on every change to either side because the first common ancestor is the empty repository).

This is quite an unfortunate situation, I think. In a way, it means that what I always feared would happen sooner or later, that the H-uru Plasma repository would eventually turn into a permanent fork (in the traditional sense) without any hope of ever being able to share code with upstream again, has already happened from the outset.

The only way I can think of of fixing this is pulling (the original version of) these 7 revisions in again and rebasing everything else on them. I have succeeded in that at https://github.com/cwalther/Plasma (I think). However, an upstream rebase is quite an inconvenience to everyone who already has clones of the repository (with unmerged changes) and should not be done lightly.

Does anyone have a better idea?

[dpogue]

Given that there are 11 forks, and countless clones already, I don't think "rewriting history" in the git sense is really an option.

There's an alternative that requires a bit more effort, but should achieve the same result. If someone clones your rebased repository, adds H-uru/Plasma as a remote and pulls its master into a branch, they could use hg-git to pull patches from OU and cherrypick them onto the H-uru/Plasma branch. Not ideal, but better (IMO) than a forced rebase, which will cause conflicts for everyone.

[branan]

As Paradox said, cherry-picking (or hg-export) is probably saner. We've been making changes to fairly core sections of the code, and things are going to stop merging cleanly really quickly.

That being said, I really encourage you to look at basing your work on our fork - we've got several bug fixes, a couple updated features, plus support for Visual Studio 2008/2010 express which will make your work more accessible to other developers.

Along with a management/dev team that's actively committing and working with the code, and a well-defined route to get your commits merged into master.

[cwalther]

Paradox, I don’t see how what you’re describing would “achieve the same result” in terms of permanently fixing anything. Rewriting history would be a one-time annoyance, while the cherry-picking you describe would have to be done every time someone wants to port a change in all future.

Obviously having compatible origins will not suddenly make it easy to port changes between diverged branches, but at least it would be one barrier less. Cherry-picking is inevitable at some point, but I fear the current state encourages doing it outside of version control by generating and applying patches or the like, losing commit metadata.

But anyway, I respect your decision. I just want the issue to be known.

As for basing my work on the H-uru fork, I would like to do that, and probably will, but as long as I have to assume that doing so will reduce my chances of getting my contributions into MOULa, I’m in a dilemma.

[branan]

So you know, Cyan has so far not pulled anything directly from OpenURU - they've only integrated patches sent to them by developers. In fact, because of the GPL that's probably the only way they can integrate patches (otherwise their entire internal codebase would become GPL licensed), unless OpenURU adds a copyright-assignment or GPL-waiver requirement before pushing code to them.

The only cleanups that should be needed to get simple patches from us to Cyan should be converting spaces back to tabs and fixing the file paths. Both of those can certainly be scripted. I've got a couple of patches I want to get exported to Cyan, so I'll get a tool to convert patches for that written up and get it into the repo. No exact ETA on that, I have way too much on my plate. Hopefully before the weekend

[cwalther]

I am reopening this in light of recent developments and non-developments on the corresponding OpenUru.org forum thread. (Read that for more detail.)

Based on Branan’s input, I have devised a solution that not only bridges the OpenUru.org and H-uru repository roots (by putting both roots into the same repository), but also facilitates porting of changesets across the folder moves and whitespace changes done in the H-uru branch, where Git’s automatic file copy/move detection fails (in my experience), by taking avantage of Mercurial’s explicit copy/move tracking.

Putting this into effect would consist of pulling the changesets marked in green here into the H-uru repository and basing further development on the merge at the top:

• This includes the first 7 revisions of the OpenUru.org repository that only differ by the inclusion of an empty .hgignore file from their counterparts at the root of the H-uru repo. In other words, it introduces a second root into the revision graph. This looks a bit ugly, but has the advantage that any branches based on the OpenUru.org repository can now simply be pulled (pushed) in using Hg-Git.
• The merge "Merge ou-huru-bridge into master" introduces no file changes, the file tree on master is exactly the same before and after it (it is a merge of two identical trees). Existing development should therefore be unaffected by it.

Thanks to the "Replaying the file moves" revision, which contains explicit Mercurial rename tracking information, any branch branching off one of the initial OpenUru.org revisions can be merged into master using Mercurial without causing tree conflicts – changes to moved files will automatically be redirected to the moved files. (I have not been able to achieve this using Git – as mentioned, it seems that the whitespace changes are too much for its rename detection.) There will usually still be content conflicts due to the whitespace changes, but these can be resolved more easily than tree conflicts.

The result can look like this, for example:

Backporting of changes from the H-uru branch onto the OpenUru.org root should also become easier since they can now be cherry-picked inside the same repository instead of having to work with patches or the like and risking to lose commit metadata.

Of note, anyone who wants to move the "Merge ou-huru-bridge into master" revision from Git to Mercurial needs to use a version of Hg-Git that includes schacon/hg-git@709b284 – older versions have a bug that will subtly mangle such a merge, with the effect that the explicit file move tracking when trying to merge an OU-based branch will not work. There is no necessity for that though, as people working in Mercurial can directly pull that revision from the Mercurial repository that I will provide. All other operations (including moving the merge from Mercurial to Git) are safe using the released version of Hg-Git.

What is the opinion of you guys on this? Would you accept the pollution of your repository with some ugliness in return for better interoperability with OpenUru.org clones? Or would anyone of you even have a better solution?

If it is decided to go forward with this, I can provide it in the form of a GitHub pull request, so work required on your part should be minimal.

[branan]

This looks sane to me. Good job figuring out how to make the move work correctly. If you want to push that to github and open a pull request, I'll give it a more thorough review.

[cwalther]

Sure. The version that the pictures above are from, and that is still current as of right now, is now at cwalther/Plasma. Creation of this is scripted, and my plan is to recreate it shortly before it is (potentially) finally pulled, so that we’re based on the latest state of master and don’t create too much spaghetti history.

Here’s the script, for reference:

hg clone https://Foundry.OpenUru.org/hg/CWE CWE-hg
cd CWE-hg
hg up -r e1a9b3a06ebf
hg branch ou-huru-bridge
hg mv MOULOpenSourceClientPlugin/Plasma20/Docs MOULOpenSourceClientPlugin/Plasma20/Sources .
cd Sources/Plasma/FeatureLib/pfConsole/
mkdir ../pfConsoleCore
hg mv pfConsoleCmd.cpp pfConsoleCmd.h pfConsoleContext.cpp pfConsoleContext.h pfConsoleEngine.cpp pfConsoleEngine.h ../pfConsoleCore
cd ../../../..
cd Sources/Plasma/PubUtilLib/plAudio/
hg mv plOGGCodec.cpp plOGGCodec.h plWavFile.cpp plWavFile.h ../plAudioCore
cd ../../../..
hg ci -m "Replaying the file moves of the H-uru master branch."
cd ..

git clone https://github.com/H-uru/Plasma.git CWE-git
cd CWE-hg
hg bookmark -f -r ou-huru-bridge ou-huru-bridge
hg push -r ou-huru-bridge ../CWE-git
cd ../CWE-git
git checkout ou-huru-bridge
git branch temp ou-huru-bridge
git reset --hard master
git reset --soft temp
git branch -D temp
git commit -m "All other changes of the H-uru master branch, squashed into a single commit to avoid cluttering up the revision graph. Check the other leg of the merge for the actual history." --author "H-uru Team <none@none>"
cd ..

cd CWE-hg
hg pull -r master ../CWE-git
hg pull -r ou-huru-bridge ../CWE-git
hg up --clean -r master
hg merge -r ou-huru-bridge
hg ci -m "Merge ou-huru-bridge into master."
hg push -r master ../CWE-git
cd ..

(Differences to the latest script posted on OpenUru.org are: - Include the “MOULa build 1.902” revision on the OU side since its contents are also part of the H-uru side already; - Added --author "H-uru Team <none@none>". The merge is done in Mercurial, so the fixed version of Hg-Git is not required.)

I didn’t open a pull request, because I don’t actually want you to pull it into H-uru/Plasma just yet. This is only the opinion gathering phase. Before it is finally pulled, I want to give a shout to rarified, who is apparently still researching alternative solutions. If he is about to implement something completely different on OpenUru.org, we may want to coordinate with him to avoid having to make an even bigger mess in the future. For review, you can just pull everything from my fork (starting from master), it does not contain any unrelated revisions.

[cwalther]

• And here’s the same thing in Mercurial: https://bitbucket.org/cwalther/cwe-ou-huru-bridge
• I wonder, is there anything I can replace H-uru Team <none@none> with to make GitHub recognize the author as the H-uru organization? Do organizations have e-mail addresses? (Can organizations even be commit authors?)

[dpogue]

You can try using hurutesting [at] gmail [dot] com as the email address. I'm not entirely sure what the results will be...

[cwalther]

The result is cwalther/huruauthortest@8e17046 … Works! So that’ll go into my script – thanks! Assuming you guys are OK with it. It’s not really an important point but I thought it was nice to make it clear that I’m not taking credit for the code changes in that commit. (And actually maybe I should apply it to the file move commit too.)

[cwalther]

Closed by pull request #41. Thanks!