Collaborating with jj over Git?

[colemickens]

I'm trying to understand the expected workflow for users that are using jj and using the git store.

1. Does one lose functionality by using the git backing store rather than (presumably a new native jj specific store with optimizations/etc?)?

2. After I've made some history with jj, how do I reflect that back into git?

I couldn't really figure it out so I started poking at jj git push but it seems mostly to not do anything. Do I need to manually go to the git store and git reset --hard <rev> where <rev> is the git commit hash corresponding to a jj revision?

Is it expected to be able to have jj git push do this automatically ever, or am I approaching the entire workflow wrong?

[martinvonz]

First of all, thanks for trying it and providing feedback!

Does one lose functionality by using the git backing store rather than (presumably a new native jj specific store with optimizations/etc?)?

There is very little functional difference. The native backend is currently very naive and slow. I use the git backend not only because I need to so I can push to this repo here on GitHub, but because also because it's just a much better backend so far, mostly thanks to the packfile format. The only real drawback is issue #27, but the workaround is trivial (just try the failed command again -- there's no risk of corruption or anything) and I rarely run into it anyway.

In the longer term, I hope that the clearer transactional semantics will make it possible for a native jj backend to do things that are hard to do with the git backend. For example, I hope it should be possible to create a packfile-equivalent per operation and never have loose objects. I believe Git has some tricks for doing that specifically on fetch.

After I've made some history with jj, how do I reflect that back into git? Do I need to manually go to the git store and git reset --hard <rev> where <rev> is the git commit hash corresponding to a jj revision?

Good question! The commits are already in the git object store, but you currently have to manually update the branches in the underlying git repo. If you created the jj repo from an external git repo via jj init --git-store=<path>, then you just go to that path and update the branches. If you created the jj repo via jj git clone or jj init --git, then you'll find the git repo in .jj/store/git, so you can do e.g. git --git-dir=.jj/store/git branch <branch name> <some hash>.

There's already jj git import for importing updated refs from the underlying git into the jj repo. The plan is to create jj git export for updating refs in the underlying git repo based on changed in the jj repo. I was just thinking earlier today whether I should implement that :) Since I personally almost never use any git commands in the underlying git repo (which is internal in my case, i.e. in .jj/store/git/), it's just not been a priority. I'm happy to start working on that on the weekend if it'd help you. It should be pretty simple.

I couldn't really figure it out so I started poking at jj git push but it seems mostly to not do anything. Is it expected to be able to have jj git push do this automatically ever, or am I approaching the entire workflow wrong?

jj git push is for pushing to a git remote. That's what I use for pushing commits to this repo, for example.

[martinvonz]

By the way, the Git comparison has a table of commands that may be useful (though it doesn't answer the questions you had here).

[colemickens]

I'm happy to start working on that on the weekend if it'd help you. It should be pretty simple.

Definitely not necessary, this comment mostly clears up my questions.

I had one more question fall out of this, but answered experimentally -- how does this all work with git pointers, GC, etc? And it seems the answer really lies in jj git push which is doing the "magic" I was wondering about.

1.I created ~/code/JJEXP/git as a git repo (seems to have picked master as its branch name)
2. Used it as the backing for ~/code/JJEXP/jj1
3. Then set /tmp/git as a remote for the git repo.
4. Create a new branch main in jj and added some ... revs... (by doing jj close after an edit a few times)
5. jj git push to see what happens.

Interestingly:

• there is NO main branch in the ~/code/JJEXP/git repo
• however, in /tmp/git we now have main branch pointing to the expected commit/rev

This really helped me understand the full answer to how collaboration takes place. I couldn't understand how you were pushing loose commits around, but now I see that jj git push is indeed doing some of that git branch manipulation for me. (I guess maybe I'm curious what jj export would do that jj git push isn't already doing, just doing it in bulk?) (edit: maybe jj export would reflect those branches directly in the backing git repo? can't say I really feel like I "need" that with any urgency either)

If I got something wrong or there's important callouts here I'm happy to hear them. Otherwise I'm probably "unblocked" to keep tinkering. Thanks for the detailed answers and working on this!

[martinvonz]

I had one more question fall out of this, but answered experimentally -- how does this all work with git pointers, GC, etc?

The answer to the question about GC specifically is that whenever jj's git backend creates a commit, it also writes a git ref under refs/jj/keep/ just to prevent GC.

And it seems the answer really lies in jj git push which is doing the "magic" I was wondering about.

jj git push will push any new commits to the remote ("origin" by default) along with their associated branch pointers. It assumes that the branch has the same name on the remote as it has locally. The push happens from the underlying git repo (using libgit2). When that push happens, the git repo will get a refs/remotes/origin/main ref (in your case). Then the command will import refs from the git repo as if you had run jj git import. Importing the refs/remotes/origin/main ref results in the branch in jj getting updated (though that would be a no-op in your case).

• there is NO main branch in the ~/code/JJEXP/git repo

Correct, but there would have been a main remote-tracking branch, i.e. the refs/remotes/origin/main noted above.

• however, in /tmp/git we now have main branch pointing to the expected commit/rev

Yes.

This really helped me understand the full answer to how collaboration takes place. I couldn't understand how you were pushing loose commits around, but now I see that jj git push is indeed doing some of that git branch manipulation for me. (I guess maybe I'm curious what jj export would do that jj git push isn't already doing, just doing it in bulk?) (edit: maybe jj export would reflect those branches directly in the backing git repo? can't say I really feel like I "need" that with any urgency either)

Yes, exactly. The idea is that jj git export would reflect the branches directly in the git repo without updating any remote.

If I got something wrong or there's important callouts here I'm happy to hear them.

See above :)

Otherwise I'm probably "unblocked" to keep tinkering. Thanks for the detailed answers and working on this!

No problem. Thanks for tinkering with it, and let me know if anything's still unclear.

[martinvonz]

I have been thinking about what would make it easier for users to adopt jj. I suspect the biggest reasons are the fact that you need to run jj commands in a separate working copy from where you run git commands, and the lack of the jj git export command described above (happy to hear whether you agree, @colemickens, since you have at least experimented a bit with jj).

I think it would be really nice if one could use jj and git commands in the same workspace. It's actually already easy to set that up by running jj init --git-store=. in an existing Git repo. However, it stops being easy very soon after that :P For example, if you check out a different commit using either git or jj, the other side will not know about that and you'll just see an unexpected diff when you run jj diff or git diff, respectively.

I think it should be possible to have a mode where jj commands first import changes from the underlying Git repo, then make any changes (e.g. rebase commits), and finally export changes back to the underlying Git repo. There are several things that would make that import-change-export cycle lossy, but I hope it can still be useful. The most important piece of information to import and export is surely the checked-out commit (i.e. HEAD in the Git repo). Having at least that updated means that you can interleave at least git status, git log, git blame, etc with jj commands.

I'll probably prioritize working on that idea next, switching away from working on support for multiple working copies (#13).

[kuchta]

@martinvonz, I absolutely love that idea 💘. git two-way interop is single most important thing for me to start using jj on daily basis, because of dependance on other git tools. Why is there even the need to import git stuff to jj, if it is supposed to use git as a (backend) store? This is probably most confusing thing about jj and I believe that I'm not alone. I understand that jj probably have a broader data model, but couldn't that data model be persisted in a git compatible way? Because if not, than it's probably better to say we have a distinct data store and sync must be done explicitly through import/export (or probably even more intuitively by using jj git fetch/pull/push defaulting to local git repo when in git workspace) commands than implicitly but partially. Until jj becomes widely used thing I would be using it mostly as better thought out interface for git repositories and as such calling explicit import/export seems like too much for a better cli.

[ilyagr]

You are in luck: this has long been implemented, see the colocated repo docs.

As a historical note, I don't know whether I'd have ended up using jj before this was implemented.