Skip to content

Sometimes deploy to gh-pages fails #973

Closed

Description

@pjbull

Encountered while fixing #966. Originally added --force in #967, but it makes more sense to do some more research on the root cause.

Possibly:

  • GH token permissions
  • Same GH token used on multiple machines(?)
  • Some git call is failing silently

Will investigate repro steps and update here.

Activity

  1. pjbull commented on Jun 26, 2016

    @pjbull
    ContributorAuthor

    Hm. There's a bit of a 🐰 hole here. For context, this is using the token approach from #632 to deploy updates using a CI server.

    Looks like the root cause is in these lines of code.

    Here's where I see the problem:

    • Run mkdocs gh-deploy --clean --remote-name https://{TOKEN}@github.com/owner/repo.git
    • When we call git rev-list with a URL as the remote, we would normally see the error fatal: Invalid object name 'https'.
    • However, try_rebase returns True on line 76 if rev-list fails, so we just never do the rebase.
    • Now when we try to git push we see an error (which before Error on failure to git push as part of gh-deploy #967 would fail silently):
    To https://{TOKEN}@github.com/owner/repo.git
    ! [rejected]        gh-pages -> gh-pages (fetch first)
    error: failed to push some refs to 'https://{TOKEN}@github.com/owner/repo.git'
    hint: Updates were rejected because the remote contains work that you do
    hint: not have locally. This is usually caused by another repository pushing
    hint: to the same ref. You may want to first integrate the remote changes
    hint: (e.g., 'git pull ...') before pushing again.
    hint: See the 'Note about fast-forwards' in 'git push --help' for details.
    

    Some possible fixes:

    • Add --force to the push command (cheap, well understood, but rebase still fails so the git history on gh-pages gets real long).
    • If remote name starts with 'http', run git remote set-url origin {REMOTE-URL} before the rebase. This should allow the git commands that can't hand a URL as a remote to function properly. This doesn't address the fact that the rebase fails.
    • Go for a simpler git model that will always destroy the gh-pages branch for real. E.g., check if branch gh-pages exists (git branch -r). If so, delete it locally (git branch -D gh-pages) and remotely (git push origin :gh-pages). This could replace the try_rebase call in ghp_import and we could continue with run_import knowing the gh-pages branch is fresh.

    Maybe I'm missing a better option, if so happy to do some more investigation if y'all have suggestions.

  2. waylan commented on Jun 26, 2016

    @waylan
    Contributor

    @pjbull thanks for doing all the work on this. I agree using force is probably the best way to address this. However, as I understand it, this is not a problem for all users (perhaps only for those doing automated updates from CI servers), so the use of force should be an option. In fact, in all the examples you point to, the user is explicitly including the --force flag in their script.

    That being the case, my inclination is to add a --force flag (off by default) to the gh-deploy command which then gets passed along. Then you (and others) can simply include that flag in your automation script: mkdocs gh-deploy --force ....

    The only hesitation I have is that we might want to explore merging recent updates to ghp_import which are missing from our vendored copy. If we are going to do that, I think that should happen first. And perhaps we should offer to pass our local modifications to ghp_import back upstream. Maybe that project could then be better at both being a lib and a command line tool. If so, we would no longer need to vendor it, which makes for simpler maintenance.

  3. waylan commented on Jun 26, 2016

    @waylan
    Contributor

    By the way, a possible workaround might be to use git-subtree instead of our gh-deploy command. It can copy (and preserve) the history of a subdirectory to a separate branch. It will only copy the commits which effected the specified sub-directory. Additionally, any commits which included changes in both the specified subdirectory and other parts of your source only include the changes to the subdirectory. I did a full write-up on how to use it with GitHub Pages a while back.

    Just be aware that if you have been using our gh-deploy command, you will need to delete the existing gh-pages branch both locally and remotely before using git-subtree the first time.

    Personally, I prefer the approach of git-subtree and would like to use it under-the-hood for gh-deploy. However, it requires installation of a non-standard git extension which does not have a clean install path on all systems. The existing solution is just easier to get new users up-to-speed on.

  4. pjbull commented on Jun 27, 2016

    @pjbull
    ContributorAuthor

    👌 I'll start with --force as an option, since that seems easiest to implement and for existing users. I'll take a look at pulling the upstream changes to the vendored copy.

    Just to get a couple more notes about other options in one place:
    Looks like subtree is now in git, so no need for a separate install (though we would take a git>=1.7.11 dependency).

    Another option, which seems pretty clean comes from this comment:

    • Keep site in .gitignore
    • On deploy, git clone gh-pages into the site directory
    • Clean and build the site
    • cd into the directory, git add, git commit and git push
  5. waylan commented on Jun 27, 2016

    @waylan
    Contributor

    Looks like subtree is now in git...

    Yes, I know, but as that page states:

    ... it's in the "contrib" subtree for now, so it's not installed by default.

    Which is the same as not being included in git on most systems. So, yes, a separate install is still necessary.

  6. added this to the 0.16 milestone on Aug 10, 2016
  7. added a commit that references this issue on Aug 11, 2016
    b3d78f2
  8. waylan commented on Aug 11, 2016

    @waylan
    Contributor

    @pjbull I just created a PR (#1023) which adds a --force flag. Does this resolve your problem? As far I can tell, it is working, but I wasn't having a problem which needed the force to begin with.

  9. added a commit that references this issue on Aug 11, 2016
    b623323
  10. pjbull commented on Aug 11, 2016

    @pjbull
    ContributorAuthor

    Tested and it works beautifully. Thanks @waylan! Confirmed repro w/o --force flag and then success with --force.

    (Also, was hoping to get a PR to you all, but it was a busy summer... I'll pick off another issue to help out on.)

  11. waylan commented on Aug 11, 2016

    @waylan
    Contributor

    Thanks for the quick feedback. Once the CI tests finish, I'll merge the PR.

  12. gregglind commented on Aug 17, 2016

    @gregglind

    (what is the 'while we wait for 0.16' solution here?)

  13. waylan commented on Aug 17, 2016

    @waylan
    Contributor

    I don't know that there is one. However, I expect that 0.16 will be released very soon (see #907).

  14. azbarcea commented on Aug 26, 2018

    @azbarcea

    The way it seems to work for me, without --force flag is:

    git branch -D gh-pages
    git checkout gh-pages
    git pull --rebase
    git checkout master
    mkdocs build --clean
    mkdocs gh-deploy
    

    Not sure what objects are messed up, because I see the git pull --rebase doesn't actual merges anything. --force flag will definitely not work for those whose history is not over-writable. 🤔

  15. e-orz commented on Apr 10, 2019

    @e-orz

    Each gh-deploy command creates another commit on the local gh-pages branch. If for some reason the local gh-pages branch and the remove have been diverged (someone else pushed docs, for example), you won't be able to push without force.
    to fix it:

    1. git branch -D gh-pages
    2. git checkout -t upstream/gh-pages (change upstream to your actual remote)
    3. git checkout master

    Than your local gh-pages and the remote one are the same. Now you can gh-deploy that will create another commit on top of the current commit and will work without force.

    The actual "bug" in here is that gh-deploy doesn't pull the latest from the remote.

  16. added a commit that references this issue on Feb 11, 2020
  17. added a commit that references this issue on Jun 30, 2021
  18. added a commit that references this issue on Nov 19, 2022
  19. added a commit that references this issue on Oct 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions