push denied after deleting files with bfg

[nick-youngblut]

After deleting select files from my repo with bfg, I'm denied when using git push.

The commands that I used (repo name changed to protect the innocent):

git clone --mirror /path/to/my/repo.git
bfg -D file1 repo.git
bfg -D file2 repo.git
bfg -D file3 repo.git
cd repo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push

The output from 'git push'

Counting objects: 857, done.
Delta compression using up to 40 threads.
Compressing objects: 100% (280/280), done.
Writing objects: 100% (817/817), 6.25 MiB, done.
Total 817 (delta 513), reused 803 (delta 503)
remote: error: denying non-fast-forward refs/heads/develop (you should pull first)
remote: error: denying non-fast-forward refs/heads/feature/align_breakpoints (you should pull first)
remote: error: denying non-fast-forward refs/heads/master (you should pull first)
To /var/git/sharchaea/genome_align/
! [remote rejected] develop -> develop (non-fast-forward)
! [remote rejected] feature/align_breakpoints -> feature/align_breakpoints (non-fast-forward)
! [remote rejected] master -> master (non-fast-forward)
error: failed to push some refs to '/var/git/sharchaea/genome_align/'

Any idea what's going on?
Thanks

[rtyley]

Can you tell me who's hosting the remote Git server? GitHub, BitBucket, perhaps an internal corporate service?

I think it's quite likely the remote server has been configured to reject non-fast-forward updates - meaning that it's disallowing history-rewriting! Does the remote have receive.denyNonFastForwards true set in it's config?

[nick-youngblut]

Sorry for not specifying that in my initial comment. I was trying to push to GitHub, so I'm guessing that history-rewriting is not the issue.

[rtyley]

Are you sure you cloned with --mirror? Could you repeat the steps just to be sure?

http://rtyley.github.io/bfg-repo-cleaner/#usage

[nick-youngblut]

I repeated twice before I posted the original comment. The commands in my original comment pasted from my history (with file names changed).

[orthodoc]

I was unable to do a git push after deleting a file. I got the following error:

fatal: remote error: 
  You can't push to git://github.com/orthodoc/my.git
  Use https://github.com/orthodoc/my.git

I did use the --mirror flag when cloning

[rtyley]

You cloned using a read-only url. This is why you couldn't push your
updated history back.

https://help.github.com/articles/which-remote-url-should-i-use
On 10 May 2014 23:31, "Biswajit Dutta Baruah" notifications@github.com
wrote:

I was unable to do a git push after deleting a file. I got the following
error:

fatal: remote error:
You can't push to git://github.com/orthodoc/my.git
Use https://github.com/orthodoc/my.git

I did use the --mirror flag when cloning

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-42756358
.

[orthodoc]

Does that mean that I have to clone like this:

git clone --mirror https://github.com/orthodoc/my.git

[rtyley]

Yup - use either that or the SSH url. The generic git protocol you're using
is not passing any authentication credentials, so GitHub is obviously not
going to accept pushes to your repo!
On 10 May 2014 23:44, "Biswajit Dutta Baruah" notifications@github.com
wrote:

Does that mean that I have to clone like this:

git clone --mirror https://github.com/orthodoc/my.git

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-42756652
.

[orthodoc]

I did that. Successfully pushed the changes. Some refs failed to go through. Probably Issue #36 that you have already explained. I checked history of my repo and the its squeaky clean. bfg made my day. I can't thank you enough.

[nishatafshan]

Hi,
I am facing similar issue. I have cloned using ssh url.
git clone --mirror ssh://git@server/repo.git

java -jar bfg-1.12.8.jar --strip-blobs-bigger-than 100M repo.git
cd repo.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive

Above steps cleans my repo.git from 6.4G to 1.2G. Later I try to push.
git push

But I get below error:-
Counting objects: 217307, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (80946/80946), done.
Writing objects: 100% (214361/214361), 666.42 MiB | 1.15 MiB/s, done.
Total 214361 (delta 118358), reused 209184 (delta 114750)
remote: You are attempting to update refs that are reserved for Stash's pull request
remote: functionality. Stash manages these refs automatically, and they may not be
remote: updated by users.
remote:
remote: Rejected refs:
remote: refs/pull-requests/190/from
remote: refs/pull-requests/247/from
remote: refs/pull-requests/247/merge
To ssh://git@server/repo.git
! [remote rejected] Branch -> Branch (pre-receive hook declined)
...
error: failed to push some refs to 'ssh://git@server/repo.git'

How should i fix it. Should I try with https url?

[javabrett]

The error message is fairly clear - Stash won't let you push refs under "pull-requests". There's a few things you could do, including renaming that branch in your local, making it non-tracking, deleting it etc. I don't know what the impact on clean-up will be in Stash if it is retaining those refs - maybe there is a clean-up process for those.

[nishatafshan]

@javabrett You are correct. I had to get rid of the pull requests. I got them merged / declined and then started with the bfg clean up steps. There is another way to get rid of pull request from mirror clone - http://christoph.ruegg.name/blog/git-howto-mirror-a-github-repository-without-pull-refs.html

[nishatafshan]

@javabrett @rtyley Would you have any ideas on how to resolve below issue.

Post cleanup when I try push I am getting below error:-
Counting objects: 218648, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (81662/81662), done.
Writing objects: 100% (215818/215818), 666.89 MiB | 1.07 MiB/s, done.
Total 215818 (delta 119117), reused 210562 (delta 115381)
remote: Communication breakdown with Stash.
To ssh://repo.git
! [remote rejected] Branch ->Branch (pre-receive hook declined)
! [remote rejected] CUC-112 -> CUC-112 (pre-receive hook declined)
...
error: failed to push some refs to 'ssh://repo.git'

[javabrett]

It looks like a hook, pre-receive hook, is declining the push, but may be doing-so silently. You need to establish why that hook is rejecting the pushes to those refs, which may involve making the hook output more-verbose. Perhaps those refs are locked/protected by Stash.

[nishatafshan]

@javabrett there are no explicit hooks set in stash for the repo. How do I find pull more details to judge the cause?

[javabrett]

I recommend:

• web search of that error message and go through the list of possible causes
• checking the Stash server logs
• contacting Atlassian support

[akaguny]

in work with github i see the same errors (clean with -rt flag):

 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'git@github.com:***/***.git'

[trvinh]

in work with github i see the same errors (clean with -rt flag):

 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'git@github.com:***/***.git'

Hi @akaguny , what did you mean by "clean with -rt flag"?

I am also trying to clean up my repository following the manual and then pushing it back into github using these commands:

>git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
>git reflog expire --expire=now --all
>git gc --prune=now --aggressive
>git push

Then I got the same errors as @akaguny:

Counting objects: 7641, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3731/3731), done.
Writing objects: 100% (7641/7641), 56.76 MiB | 16.46 MiB/s, done.
Total 7641 (delta 3812), reused 7641 (delta 3812)
remote: Resolving deltas: 100% (3812/3812), done.
To https://github.com/abc/???.git
 + bef1e57...12792c1 bioconductor -> bioconductor (forced update)
 + 2504b81...ecb033d gh-pages -> gh-pages (forced update)
 + 4365d3b...43a7c82 master -> master (forced update)
 + 2eac6e7...346947d v0.1.0 -> v0.1.0 (forced update)
 + 4b2eca8...e522f0d v0.1.1-beta -> v0.1.1-beta (forced update)
 + d61480b...51034ad v0.2.0 -> v0.2.0 (forced update)
 + e5e5a97...e8b889d v0.2.1 -> v0.2.1 (forced update)
 + 29bca00...4709246 v0.3.0 -> v0.3.0 (forced update)
 + a53becf...ff9fa7b v0.3.1 -> v0.3.1 (forced update)
 + 135edb8...f734108 v0.3.2 -> v0.3.2 (forced update)
 ! [remote rejected] refs/pull/12/head -> refs/pull/12/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/17/head -> refs/pull/17/head (deny updating a hidden ref)
...
 ! [remote rejected] refs/pull/95/head -> refs/pull/95/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/abc/???.git'

Does anyone know the solution?
Many thanks!!

[javabrett]

If you need to delete Pull Requests (and their hidden branches) permanently (because they contain history you want to delete), you have to contact GitHub Support with that request.

[trvinh]

thanks so much @javabrett. I will try to contact them.

[RWOverdijk]

Isn't this just simply because you rewrite history? After running these commands you have a completely different history (in the eyes of github for example). I had to force push because of it, but it makes sense to me.

[javabrett]

@RWOverdijk did you rewrite a repo cloned from GitHub, including pull-requests, and try and push that rewritten history back, to GitHub? That's what a lot of the issues in this issue relate to, and the topic of my comment.

The refs/pull/ namespace in GitHub is read-only, and will retain old, pre-rewrite history, and block you from pushing new history.

[RWOverdijk]

I did do that, but using push --force made github yield to my bidding.

[javabrett]

Interesting. Perhaps GitHub opened this up? To confirm beyond doubt you'd need to:

• have a GitHub repo with Pull Requests with history that will be rewritten
• clone the repo in a way that ensures those PR branches are cloned
• note head commit on some PR
• run BFG and check that the PR commit hash changed
• force push back to GItHub
• check that the PR on GitHub has the new history, and that old history can't be referenced through the PR(s).

In my recollection, PRs were always read-only, so this would fail without contacting GitHub support and asking for PRs to be dropped.

[RWOverdijk]

Ah, no I don't do this through a PR. I force push to master. I may have missed that part 😅

[harrylincoln]

Because I had never pushed to the origin before, you need to set it with the push, so...

git push -u origin master

More: refs issue

[jerrylogansquare]

I had this issue with bitbucket, I went into the repo settings and turned this on:

Allow rewriting branch history

whalla

[Wind010]

After removing the pull request references in my local mirror, I was able to push up to remote:
https://stackoverflow.com/questions/47776357/git-push-mirror-without-pull-refs

Luckily it was a new repository and only one PR.

[hpsaturn]

With Gitlab I have the same issue, but the repo is cleaned, some branches and pull requests are lost. In Gitlab is imposible or they don't have a setting like "Allow rewriting branch history". Well it works, but you need keep in mine this with bfg.

[cuococarlos]

With Gitlab I have the same issue, but the repo is cleaned, some branches and pull requests are lost. In Gitlab is imposible or they don't have a setting like "Allow rewriting branch history". Well it works, but you need keep in mine this with bfg.

Hi, I have a same problem with Gitlab. I don't quite understand how Gitlab has been able to fix it. Any help would be appreciated. Sorry for my basic english.

[markfaine]

@cuococarlos Did you ever figure this out? I'm having the same issue with Gitlab.