Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mirror remote repository feature #301

Closed
gitblit opened this issue Aug 12, 2015 · 69 comments
Closed

Mirror remote repository feature #301

gitblit opened this issue Aug 12, 2015 · 69 comments

Comments

@gitblit
Copy link
Owner

@gitblit gitblit commented Aug 12, 2015

Originally reported on Google Code with ID 5

Might be nice to have a feature to clone a remote repository.
Not sure about how best to handle subsequent fetches.

Reported by James.Moger on 2011-06-30 11:57:47

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

I have a plan for this now.  I will implement a "mirror repository" feature which allows
you to clone a remote repository and optionally set an auto-pull frequency (never,
15mins...1 week).  Not sure if RefSpec should be controllable or if it should be hard-wired
to basically everything.  The repository will have the usual controls (freeze, access
restriction, etc) and it will be up to the user to ensure that the mirror is not really
committed to... or if it is, the commits must be on a separate branch.  This feature
would be most useful for teams tracking a 3rd party dependency.

Reported by James.Moger on 2011-07-27 12:48:13

  • Status changed: Accepted
  • Labels added: Milestone-0.6.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Pushed this back to the next release.

Reported by James.Moger on 2011-09-26 20:39:06

  • Labels added: Milestone-0.7.0
  • Labels removed: Milestone-0.6.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Reported by James.Moger on 2011-10-12 11:56:17

  • Labels added: Milestone-0.8.0
  • Labels removed: Milestone-0.7.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Reported by James.Moger on 2011-12-15 15:05:33

  • Labels removed: Milestone-0.8.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Reported by James.Moger on 2012-02-06 13:18:22

  • Labels added: Milestone-0.9.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Issue 348 has been merged into this issue.

Reported by James.Moger on 2012-02-06 15:45:59

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Moving to next release

Reported by James.Moger on 2012-03-28 00:03:34

  • Labels added: Milestone-1.0.0
  • Labels removed: Milestone-0.9.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Reported by James.Moger on 2012-12-04 14:26:48

  • Status changed: New

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Reported by James.Moger on 2012-12-07 14:28:26

  • Labels removed: Milestone-1.0.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Hi, is this feature implemented?

Reported by lgoral81 on 2013-07-17 08:22:53

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Nope.

Reported by James.Moger on 2013-07-17 11:43:05

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Are you referring to the federation client ? 
The last snapshot doesn't crash , but 1.3.0 only pulls the first 
state of the repository . Subsequent calls don't pull the latest changes 
Is that related ? 

Reported by tamirtw on 2013-07-23 08:38:25

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

This feature is a little bit like the federation client.  Github offers a mirror feature
- this would be exactly like that.  Clone & periodically fetch a specific repo from
any source.

Federation client doesn't (properly) pull.  I wasn't aware of that, please report this
as a separate issue and attach any relevant console messages.

Reported by James.Moger on 2013-07-23 11:51:46

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

This feature would really make gitblit my all in one tool. I prefer having a full local
clone of any library I use. This also involves using git-svn sometimes. I vote for
this feature!

Reported by robindegen on 2013-11-07 12:26:14

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Yeah, I could use this myself; I just need to _do_ it.  I can tell you, however, that
git-svn would not be on the menu.  Gitblit is a pure Java solution and git-svn isn't
implemented by JGit.  Even if I wanted to implement my own git-svn I believe SVNKit's
license is incompatible with the Apache license.  Maybe not - I can't recall the details
but it seemed like an issue.

Reported by James.Moger on 2013-11-07 12:48:23

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

When you get around to implementing this feature, I could have a look at implementing
this for SVN. I have no need to push back to svn repo's, I just want to have a copy
of them when I work with them.

Perhaps a 2 pass system, where a background script just clones and maintains a normal
git-svn repository (or even git-hg), and then Gitblit mirrors from that. That's how
I currently have things set up on my dev server, with a cronjob.

Reported by robindegen on 2013-11-07 12:51:40

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Hi James
¡To_do_it!, yeah... an idea for what it help.... :)
You should dial the repository as a mirror, not accept push (it may not solve conflicts),
only fetch from origin and then pull (auto merge) with the branch head (default of
origin - no possibility to change).
An periodicity/interval should be scheduled.

P.S 
This I have done with a linux script and crontab hourly - don't know if you remember
I told you -.

Thk

Reported by eguervos on 2013-11-07 21:18:12

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

I pushed an implementation of a mirror executor to master.  There is no ui for initially
creating the mirror so you'll have to git clone --mirror {yada}.  SSH probably won't
work.  Credentials also probably won't work... but who knows.  It runs periodically
for all identified mirrors.

I suppose that I'll do a "clone repo" page at some point, but it's not high on my list
right now.

Reported by James.Moger on 2013-11-13 23:02:38

  • Status changed: Started

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Live mirror here: http://next-gitblit.rhcloud.com/summary/jgit.git

Reported by James.Moger on 2013-11-13 23:03:24

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

I added some more mirror repos and move the jgit mirror so the above link is now invalid.

Reported by James.Moger on 2013-11-14 18:03:45

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Thank you James! I'll have a look this weekend, or monday when i'm back at the office
:) 

Reported by robindegen on 2013-11-14 18:11:01

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

I know bug trackers aren't supposed to be used for the "me too" and "+1" comments but
with this item I just couldn't resist ;-) So here goes: Thanks James! Really cool and
useful feature!

Reported by kermitthefragger on 2013-11-15 08:55:18

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Just to clarify a few things...
1) Will it be possible to say which branches to mirror? I don't think this is actually
a necessary feature but thought I'd ask anyway.

2) I assume that you'll be able to have your own branches in a mirrored repository?
So the remote branches/tags/commits, etc. will all be present in the mirror along with
any branches created specifically in the local mirror.

3) Will there be an option for "Just in time" syncing with the remote I.e. The mirror
does a "pull" from the remote on a pull from or push to the mirror? You could have
a fairly long sync interval to catch the times when the local mirror is quiet but would
be up-to-date whenever any operations occur.

4) Will I be able to convert existing repositories into mirrors? Will there be a UI
element such as a "Convert to Mirror" button and URL field to do this or will I just
run...

$ git remote add --mirror=fetch origin <url>

...in the appropriate repositories and GitBlit will pick them up as mirrors (after
a restart)?

-----
5) Will this lead to a bi-directional mirror? Essentially commits pushed to the local
(mirror) repository would be pushed to the remote, the push to the local would fail
if the push to remote fails, etc.

Cheers,
Alex

Reported by alex.lewis001 on 2013-11-15 14:27:16

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

alex:
1. A true git mirror always includes all branches, no need to specify. It's a full
100% copy of the server.

2. I wouldn't recommend that. I'd recommend starting a new clone from your mirror,
and push to that. Leave the mirror as-is.

3. This is not really what mirroring is intended for in git.

4. Why would you want to mirror your own repositories?

I suggest you do some reading into git clone --mirror :)

Reported by robindegen on 2013-11-15 14:29:52

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

robindegen:
I suppose that's the crux of the questions, is this essentially a "git clone --mirror"
or something above and beyond that?

So the use case I'm thinking of is (I'll use jgit as an example)...
* I'm working on a team that's adding a feature to jgit.
* We "mirror" jgit to a central GitBlit instance.
* We created a branch for our work in the GitBlit jgit mirror for the teams work.
* We'd like the master (and possibly any/all of the public jgit branches) kept up-to-date
automatically.
* We can browse the latest work on jgit from out GitBlit instance, more easily merge
changes into our feature-branch, etc. all done via GitBlit rather than needing to interact
manually with the remote jgit repo.

I wasn't suggesting I mirror my own repositories. Like you, I like to have a clone
of a 3rd party library in my GitBlit instance. Until now I've just cloned the remote
repository to my machine, added my Gitblit instance as a remote and then pushed to
GitBlit. To keep my GitBlit instance up to date I've subsequently (on my local machine)
pulled from origin and pushed to GitBlit. My GitBlit copy may also include branches
that I've created for my own work. So I was wondering whether I'd be able to convert
this type of repo in GitBlit into a "mirrored" one and in doing so it would replace
the manual "pull from origin and push to GitBlit phase" and do that automatically for
me on the GitBlit server.

I'm also trying to avoid having direct Git command line access on the GitBlit server,
setting cron jobs, etc. I.e. consolidating configuration in GitBlit so it's easier
to manage, backup, etc. in a corporate environment. 

Reported by alex.lewis001 on 2013-11-15 14:59:19

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

"Gitblit need make a pull (git pull <remote> - as the master branch as not changed)"

Basically, true.  If a fork's tracking branches become special (currently they are
not) then after fetching from the origin (refs/remotes/origin/xxx), the tracking branches
(refs/heads/xxx) would forced to match their origin counterparts (refs/remotes/origin/xxx).

Since Gitblit forks are bare repositories, you can not "pull".  "pull" requires a working
directory to resolve merge failures - which a bare repo does not have.  And we won't
care about merge failures - because we are not merging - we are tracking the origin.
 The git equivalent would be to fetch, as you say, and then use the update-ref plumbing
command to change the sha-1 that the tracking branches (refs/heads/xxx) point to.

"Furthermore, I can have as many copies of a mirror as branches have, in order to follow
the work of a team, this will eliminates the need to constantly check out between branches."

I don't understand this point.

Reported by James.Moger on 2013-11-15 19:36:37

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

"Furthermore, I can have as many copies of a mirror as branches have, in order to follow
the work of a team, this will eliminates the need to constantly check out between branches."

Is the reason of why having the repository with a name in order to differentiate it
from other mirror of the same but different branches showing.

Reported by eguervos on 2013-11-15 20:04:27

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Essentially a mirror is a working copy, in fact Gitblit with that repo should behave
the same way, the difference lies in we can clone a mirror and it's useful, that can
only be achieved if gitblit doing pull master branch.
The 1st clone (mirror) is set by the HEAD, the fetch does not move, if you clone this
repo find that what you have cloned is dated when cloned (created the mirror) as the
fetch put the commits to the work area and not in the working directory, and this is
important to know, if you want to clone a mirror and working with it.

Reported by eguervos on 2013-11-15 20:14:53

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

It sounds like you are using multiple Gitblit instances (one per developer?) on repos
with working copies.  This will only be for server-side forks in a centralized environment.

Reported by James.Moger on 2013-11-15 20:25:08

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

or team
...on repos with working copies. <- yes
...This will only be for server-side forks in a centralized environment. <- or mirror
with automatic scheduled updates

Reported by eguervos on 2013-11-15 20:29:11

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Mirror that behave like working copies, preventing pull, but allowing be cloned.

Reported by eguervos on 2013-11-15 20:30:59

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

When you show log of a repository that is working copy that have been made various fetch,
in reality what you see is the work area overlying from the working directory, the
current working directory is where the HEAD sign.
If you make a clone of this, that drag to your clone is where the HEAD points not the
working area, that may be much higher, when you do a "show log" of this clon, is when
you realize you don't have reality  the work done to date, but the one on the date
you created the mirror.

Reported by eguervos on 2013-11-15 21:07:23

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Now, it's clear.

By definition, a "git mirror" is a *bare* repository with a refspec of "refs/*:refs/*"
(and a mirror=true flag).  It does not have a working copy (checkout) and can't be
treated as a normal checkout.  Every time I have used the word mirror in this discussion,
that is what I have meant.  I'm not inventing an alternate definition of "mirror",
just using standard "git clone --mirror".

A Gitblit fork is also a bare repository and can not be treated as a checkout.  Anything
else is a repository clone with a working copy.  

Despite the fact that you can use Gitblit with working copies, as you do... I'm not
going to touch non-bare repositories with this feature.  Gitblit is meant to be a centralized
git server.  It does have some support for client-side use, but for now I'm not going
to consider that use.

Reported by James.Moger on 2013-11-15 21:26:04

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Mirror works fine, with what you've done so far, it is enough (so far).

Thk so much

Reported by eguervos on 2013-11-16 09:04:46

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

+1 Me too! ;)

No seriously, this sounds like good stuff. We, too, are currently using "git clone
--mirror" with subsequent "git remote update" in a cronjob. It would be perfect to
have all of that in the Gitblit UI, but having Gitblit replace the cronjob would already
be cool.

Here is another thought from how we use this. We mirror not only to have a remote site's
repos locally, but also use the mirrors to implement business continuity. That is to
say that if the remote site goes down, we switch the mirror repos on the mirroring
site to be the active master that developers can push to who have been using the remote
repo so far. Once the remote site is operational again and the repos hosted there can
be used for pushes again (i.e. have been synced back) we switch the local repos back
to mirror mode.
Of course, it would be great to be able to do that with Gitblit support. =)

Reported by f.zschocke on 2013-11-18 11:27:24

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

I have a snapshot build running from a5c69b6b7 on our server. I will let you know if
we run into any issues with mirrors.

Reported by robindegen on 2013-11-18 11:52:01

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

@Florian: That was part of the intention of the Federation feature but I don't think
it ever quite got there.

Reported by James.Moger on 2013-11-18 12:46:25

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

At first i thought my repo did not show up, but i had to clear cache. Seems to work
nicely now. I will check in a few days if it actually synched properly.

Reported by robindegen on 2013-11-18 15:17:02

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

@James: My mirrors do not appear to be updating. It identifies them as a mirror properly
(showing the icon), but I mirror cloned for example the linux kernel 2 days ago, and
it now just shows last commit 2 days ago, even though there have been recent comments
the last 2 days.

Is there anything else i need to set up in the config or something? Currently i'm running
git commit a5c69b6b7...

Reported by robindegen on 2013-11-19 12:34:29

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Sounds like the mirror executor is not running; it's disabled by default.
Did you enable it?

git.enableMirroring=true

Changing this requires a restart.

Reported by James.Moger on 2013-11-19 14:27:30

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Thanks, I'll ty that when in the next maintenance tonight.

Reported by robindegen on 2013-11-19 14:29:07

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Well, what would be the difference between the federation and the mirror feature then?
The difference I can see is that you don't need the remote using Gitblit for the mirroring
to work. Which is what I am looking for since unfortunately not all of our sites are
using this brilliant tool.

Reported by f.zschocke on 2013-11-19 16:26:27

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

One difference is that the federation feature was written 2 years ago and the mirror
feature was written last week.  :)

Mirror is a focused subset of federation suitable for use with generic http/https/git
servers.
Federation was written to continuously replicate one gitblit to another: repos, users,
settings, etc.  If new repos are added to Gitblit A, and Gitblit B pulls from Gitblit
A - Gitblit B automatically clones Gitblit A's new repositories.

The Mirror is only about updating refs for a specific repository that you have manually
configured as a mirror.  You could use the Mirror feature to pull from another Gitblit.

Federation can also be used in other clever ways since it supports repository subsets
and various permission levels.

I guess you could summarize as:
Federation = Backup/replication
Mirror = Tracking 3rd party repo

Reported by James.Moger on 2013-11-19 17:06:00

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Well, I would have liked to use Gitblit at every site in order to enable federation,
to implement the scenario described above. I it is out of the scope for Mirroring,
reading your explanation.

Reported by f.zschocke on 2013-11-19 17:27:00

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Turning it on seems to have worked fine. My mirrors are properly being updated now.

Reported by robindegen on 2013-11-19 18:31:02

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Reported by James.Moger on 2013-11-29 02:29:12

  • Labels added: Milestone-1.4.0

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

I'm simplifying this issue.  Gitblit now has a mirroring mechanism and it's been working
well for months.  Separate issues can be opened for enhancing use of the mirroring
mechanism (e.g. create mirror from web ui).

Reported by James.Moger on 2014-02-21 16:29:54

  • Status changed: Queued

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

1.4.0 released.

Reported by James.Moger on 2014-03-09 18:06:21

  • Status changed: Done

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Hello git experts ,

I have tried like below 


git clone --mirror git@github.com/numerics.git:29418/repo/tools/
git remote update
git push --mirror git@localhost:29418/numerics.git:29418/repo/tools/

with above I can get all refs and branches  but  if I will do/create local branch and
I will do git push few fixes to my local branch & to master branch in git@localhost:29418/numerics.git:29418/repo/tools/


then how I will merge or add the latest changes from git@github.com/numerics.git:29418/repo/tools/
to my local repository git@localhost:29418/numerics.git:29418/repo/tools/

Do I need to create like 

[remote "origin"]
    url = git@github.com/numerics.git:29418/repo/tools/
    mirror = true
[remote "mirrors"]
    url = git@localhost:29418/numerics.git:29418/repo/tools/

or git push to mirror and rename to again origin 

Reported by mailtodeepu on 2014-10-21 04:17:17

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

Hi, you need make a fork of this repo-branch to push in. Don't push o work
with mirror , mirror it's read only o to fork. Mybe can federate? I don't
remenber if this is it.
El 21/10/2014 06:17, <gitblit@googlecode.com> escribió:

Reported by eguervos on 2014-10-21 08:15:05

@gitblit
Copy link
Owner Author

@gitblit gitblit commented Aug 12, 2015

eguervos is right.  You need to think of a git mirror as an actual mirror.  It reflects,
it does not alter/add.

In the scenario you are describing you have 3 repos:

  [Github]       [Gitblit]
      |              |
      |              |
      +----[Clone]---+

Gitblit does not have any special features for working with working copy clones other
than viewing them.

In this 3-repository scenario you would use command-line git from your clone to pull
from Github and push to Gitblit.

The mirror feature allows you to keep a bare clone in Gitblit of your Github (or any
other source) repository.  This is a one-way, perfect-copy scheduled operation.

  [Github]------>[Gitblit]

One perhaps overlooked feature of Gitblit mirrors is that if your source repository
is a Gitblit-hosted repository which uses the BranchTicketService (for example https://dev.gitblit.com/summary/gitblit.git),
a Gitblit mirror of this repository will automatically include the complete ticket
information.  It will be read-only - because it's a mirror - but everything will be
there.

Reported by James.Moger on 2014-10-21 12:14:10

@flaix flaix added this to the 1.4.0 milestone Dec 13, 2016
@flaix flaix added this to the 1.4.0 milestone Dec 13, 2016
@flaix flaix closed this as completed Dec 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants