Add git export / import connector

[blaggacao]

I'd like to store my markdown documents in a true git backend for the following reasons:

• Auditing
• Standard revision tooling
• Possibility to integrate a lightweight PR document approval process

[ccoenen]

I used to have a simple database export to a git repository, that's fairly simple to do. It was based on this pull request, but was never really ready for prime time:

hackmdio/codimd#303

[SISheogorath]

I have to say it once more: Git is not for real-time collaboration and therefore not a proper backend for CodiMD. Yes, exports sounds like an idea, but a permanent backend using Git, is not achievable in any useful way.

Why?

Let's say we have a sentence "The sky is blue" and now, 2 people work on this document. A adds "Today" at the beginning, and the other adds "like the sea" at the end.

How are we supposed to represent this in git? Making a commit for each of them? That would result in hundreds of commits within a few minutes for 2 or 3 people working on a document, and none of them would make sense. Also what to put there as a commit message?

Or should we record the changes of each user for 5 minutes and then make a commit? That doesn't work. Git would break in a merge conflict here, which is not properly resolvable. Well, might by using some force towards the final document we already recorded in CodiMD? Maybe, but I foresee a lot of trouble.

So far, I consider real-time collaboration using OT and git diffs as not workable together. Exports and imports, yes. As real backend for the document storage, no.

Not to mention the problem of "how do organize this repository-wise?". One repository per user? One repository per document? One repository per instance? Or do we introduce collections first, which are repositories which then hold the documents? Are they read-only or do we let other people push things there? How do we merge those pushes with the real-time editing?

Git is simply not made for real-time collaboration. And CodiMD is exactly that in first place: A real-time-collaboration tool for markdown.

Sorry, to break a bit with your hopes for that. Of course I'm willing to listen, when you found proper solutions for the mentioned problems :)

[almereyda]

Let's rename this issue to Add git backup backend or Add git export then?

I can already export pads as GitLab snippets, but each time a new one. Why not extend this support to allow for reusing snippets (in GitLab, they have no underlying git repository, as in opposition to GitHub), or whole repositories where we might want to keep collections of pads?

We should be able to choose the repository in which we export our pads each time, and also the filename, with the option to remember the last setting.

Both GitHub's and GitLab's APIs allow for repository discovery, file creation and commits through the REST API.

[SISheogorath]

That sounds quite reasonable :) I would really like to see this happening and consider it quite realistic.

@almereyda would you be interested in working on that? And if so, how can we enable you to do so? Do you need I information about where to find the code, guidance on the actual implementation or anything else?

[almereyda]

That would sound lovely to take me on a tour for such an entry level issue. I guess I start from https://github.com/codimd/server/tree/master/docs/dev and walk myself through the code.

For the start, I would begin with GitLab to embody that function on instances that already authenticate against it.
Secondly, one could think of linking any GitLab instance per pad/user/instance.

This could be taken as an example to implement the same for GitHub enabled applications later on.

[blaggacao]

@almereyda to make it a proper repository, - in theory - a hierarchical organization ("folder") structure would be needed. It probably fits in with the ongoing discussion about tagging and other metadata. Else mapping it out flat might be an option...

[SISheogorath]

@blaggacao I think we talked here about cloning data from your CodiMD instance, but having a button to export your note to an existing repository in GitLab or GitHub.

The problem with a direct git export from CodiMD, as mentioned above is the organisation of the notes. We currently expect that we can decide on a note-to-note bases who has access to a note and who doesn't. This makes it impossible to organize a collection of notes as git repository without violating access control.

@almereyda For now, I would recommend to go the easy way and just improve the gitlab export function, which is currently have for snippets to work for repositories as well :)

You should start here:

https://github.com/codimd/server/blob/6462968e84e8d92292dd23764a9e558d7800147d/public/js/index.js#L965-L997

That's the current (horrible) snippet code. There is also some code in the actions section:

https://github.com/codimd/server/blob/95aa0ee47b1927c45e81322dd64bed9c308eb40c/lib/response.js#L536-L580

Please note that this code will be moved by #213.

If there is more you need, feel free to ask :)

[nsheff]

I have found this feature on hackmd.io and it works well:

[SISheogorath]

Well, HackMD.io is closed source, so there is no benefit of seeing it there 🤷🏻

[goerz]

You could do it like Overleaf: Create commits from online edits in the web interface only when someone pulls git. That way, people can collaborate asynchronously with git, or synchronously via the web interface

[ErikMichelson]

I'm not too firm with the git-integration in overleaf.
How should HedgeDoc identify that someone pulls the git repo? This would require HedgeDoc to act as a git server itself or am I missing something?

[nsheff]

This would require HedgeDoc to act as a git server itself

No. With overleaf, everything is manual. So the user has an interface and clicks either "push updates from Overleaf to Github" or "pull updates into Overleaf from GitHub". All hedgedoc would need to do is have git CLI installed to it can pull.

If you want it to auto-update like you suggest, another option is that you can set up webhooks, and have GitHub ping a hedgedoc endpoint when there's a push. But I'm not sure that's the best approach, as the hedgedoc file may have changed independently.

[goerz]

Oh, I'm not talking about Overleaf's integration with Github (never tried that). What I mean is the git support that's built into the paid version of Overleaf, where every Overleaf project can be cloned directly from a https://git.overleaf.com/<uuid> URL

[goerz]

And yes, it would require hedgedoc itself to act as a git server