New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sometimes deploy to gh-pages fails #973

Closed
pjbull opened this Issue Jun 23, 2016 · 12 comments

Comments

Projects
None yet
4 participants
@pjbull
Contributor

pjbull commented Jun 23, 2016

Encountered while fixing #966. Originally added --force in #967, but it makes more sense to do some more research on the root cause.

Possibly:

  • GH token permissions
  • Same GH token used on multiple machines(?)
  • Some git call is failing silently

Will investigate repro steps and update here.

@pjbull

This comment has been minimized.

Contributor

pjbull commented Jun 26, 2016

Hm. There's a bit of a 🐰 hole here. For context, this is using the token approach from #632 to deploy updates using a CI server.

Looks like the root cause is in these lines of code.

Here's where I see the problem:

  • Run mkdocs gh-deploy --clean --remote-name https://{TOKEN}@github.com/owner/repo.git
  • When we call git rev-list with a URL as the remote, we would normally see the error fatal: Invalid object name 'https'.
  • However, try_rebase returns True on line 76 if rev-list fails, so we just never do the rebase.
  • Now when we try to git push we see an error (which before #967 would fail silently):
To https://{TOKEN}@github.com/owner/repo.git
! [rejected]        gh-pages -> gh-pages (fetch first)
error: failed to push some refs to 'https://{TOKEN}@github.com/owner/repo.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Some possible fixes:

  • Add --force to the push command (cheap, well understood, but rebase still fails so the git history on gh-pages gets real long).
  • If remote name starts with 'http', run git remote set-url origin {REMOTE-URL} before the rebase. This should allow the git commands that can't hand a URL as a remote to function properly. This doesn't address the fact that the rebase fails.
  • Go for a simpler git model that will always destroy the gh-pages branch for real. E.g., check if branch gh-pages exists (git branch -r). If so, delete it locally (git branch -D gh-pages) and remotely (git push origin :gh-pages). This could replace the try_rebase call in ghp_import and we could continue with run_import knowing the gh-pages branch is fresh.

Maybe I'm missing a better option, if so happy to do some more investigation if y'all have suggestions.

@waylan

This comment has been minimized.

Member

waylan commented Jun 26, 2016

@pjbull thanks for doing all the work on this. I agree using force is probably the best way to address this. However, as I understand it, this is not a problem for all users (perhaps only for those doing automated updates from CI servers), so the use of force should be an option. In fact, in all the examples you point to, the user is explicitly including the --force flag in their script.

That being the case, my inclination is to add a --force flag (off by default) to the gh-deploy command which then gets passed along. Then you (and others) can simply include that flag in your automation script: mkdocs gh-deploy --force ....

The only hesitation I have is that we might want to explore merging recent updates to ghp_import which are missing from our vendored copy. If we are going to do that, I think that should happen first. And perhaps we should offer to pass our local modifications to ghp_import back upstream. Maybe that project could then be better at both being a lib and a command line tool. If so, we would no longer need to vendor it, which makes for simpler maintenance.

@waylan

This comment has been minimized.

Member

waylan commented Jun 26, 2016

By the way, a possible workaround might be to use git-subtree instead of our gh-deploy command. It can copy (and preserve) the history of a subdirectory to a separate branch. It will only copy the commits which effected the specified sub-directory. Additionally, any commits which included changes in both the specified subdirectory and other parts of your source only include the changes to the subdirectory. I did a full write-up on how to use it with GitHub Pages a while back.

Just be aware that if you have been using our gh-deploy command, you will need to delete the existing gh-pages branch both locally and remotely before using git-subtree the first time.

Personally, I prefer the approach of git-subtree and would like to use it under-the-hood for gh-deploy. However, it requires installation of a non-standard git extension which does not have a clean install path on all systems. The existing solution is just easier to get new users up-to-speed on.

@pjbull

This comment has been minimized.

Contributor

pjbull commented Jun 27, 2016

👌 I'll start with --force as an option, since that seems easiest to implement and for existing users. I'll take a look at pulling the upstream changes to the vendored copy.

Just to get a couple more notes about other options in one place:
Looks like subtree is now in git, so no need for a separate install (though we would take a git>=1.7.11 dependency).

Another option, which seems pretty clean comes from this comment:

  • Keep site in .gitignore
  • On deploy, git clone gh-pages into the site directory
  • Clean and build the site
  • cd into the directory, git add, git commit and git push
@waylan

This comment has been minimized.

Member

waylan commented Jun 27, 2016

Looks like subtree is now in git...

Yes, I know, but as that page states:

... it's in the "contrib" subtree for now, so it's not installed by default.

Which is the same as not being included in git on most systems. So, yes, a separate install is still necessary.

@waylan waylan added the Bug label Aug 10, 2016

@waylan waylan added this to the 0.16 milestone Aug 10, 2016

waylan added a commit to waylan/mkdocs that referenced this issue Aug 11, 2016

Add --force flag to gh-deploy command.
This replicates the recent addition to the ghp_import.py tool (see
davisp/ghp-import@49cfe6e).
The default is to not force, but if needed (perhaps when deploying
from a CI server), the --force flag can be included.

Fixes mkdocs#973.
@waylan

This comment has been minimized.

Member

waylan commented Aug 11, 2016

@pjbull I just created a PR (#1023) which adds a --force flag. Does this resolve your problem? As far I can tell, it is working, but I wasn't having a problem which needed the force to begin with.

waylan added a commit to waylan/mkdocs that referenced this issue Aug 11, 2016

Add --force flag to gh-deploy command.
This replicates the recent addition to the ghp_import.py tool (see
davisp/ghp-import@49cfe6e).
The default is to not force, but if needed (perhaps when deploying
from a CI server), the --force flag can be included.

Fixes mkdocs#973.
@pjbull

This comment has been minimized.

Contributor

pjbull commented Aug 11, 2016

Tested and it works beautifully. Thanks @waylan! Confirmed repro w/o --force flag and then success with --force.

(Also, was hoping to get a PR to you all, but it was a busy summer... I'll pick off another issue to help out on.)

@waylan

This comment has been minimized.

Member

waylan commented Aug 11, 2016

Thanks for the quick feedback. Once the CI tests finish, I'll merge the PR.

@waylan waylan closed this in #1023 Aug 11, 2016

@gregglind

This comment has been minimized.

gregglind commented Aug 17, 2016

(what is the 'while we wait for 0.16' solution here?)

@waylan

This comment has been minimized.

Member

waylan commented Aug 17, 2016

I don't know that there is one. However, I expect that 0.16 will be released very soon (see #907).

@azbarcea

This comment has been minimized.

azbarcea commented Aug 26, 2018

The way it seems to work for me, without --force flag is:

git branch -D gh-pages
git checkout gh-pages
git pull --rebase
git checkout master
mkdocs build --clean
mkdocs gh-deploy

Not sure what objects are messed up, because I see the git pull --rebase doesn't actual merges anything. --force flag will definitely not work for those whose history is not over-writable. 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment