New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while executing gh-deploy #722

Closed
ThomasArdal opened this Issue Sep 8, 2015 · 34 comments

Comments

Projects
None yet
9 participants
@ThomasArdal

ThomasArdal commented Sep 8, 2015

I've successfully deployed a MkDocs site using the gh-deploy command. When I try to deploy some additional changes to my master branch, I get the following error:

c:\docs>mkdocs gh-deploy --clean
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: c:\docs\site
INFO    -  Copying 'c:\docs\site' to 'gh-pages' branch and pushing to GitHub.
Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\Python34\Scripts\mkdocs.exe\__main__.py", line 9, in <module>
  File "C:\Python34\lib\site-packages\click\core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python34\lib\site-packages\click\core.py", line 644, in main
    rv = self.invoke(ctx)
  File "C:\Python34\lib\site-packages\click\core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python34\lib\site-packages\click\core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python34\lib\site-packages\click\core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "C:\Python34\lib\site-packages\mkdocs\cli.py", line 186, in gh_deploy_command
    gh_deploy.gh_deploy(config, message=message)
  File "C:\Python34\lib\site-packages\mkdocs\gh_deploy.py", line 69, in gh_deploy
    remote_branch)
  File "C:\Python34\lib\site-packages\mkdocs\utils\ghp_import.py", line 163, in ghp_import
    if not try_rebase(remote, branch):
  File "C:\Python34\lib\site-packages\mkdocs\utils\ghp_import.py", line 78, in try_rebase
    if sp.call(cmd) != 0:
  File "C:\Python34\lib\subprocess.py", line 537, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Python34\lib\subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "C:\Python34\lib\subprocess.py", line 1086, in _execute_child
    args = list2cmdline(args)
  File "C:\Python34\lib\subprocess.py", line 663, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: 'str' does not support the buffer interface
@d0ugal

This comment has been minimized.

Member

d0ugal commented Sep 8, 2015

So, this sounds like we might have hit an issue related to #703 - I think we are passing unicode to an API that expects a bytestring?

@d0ugal d0ugal added the Bug label Sep 8, 2015

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Sep 8, 2015

That sounds weird. All of my md files are in UTF-8 and it worked the first time i published the site to gh-pages.

@facelessuser

This comment has been minimized.

Contributor

facelessuser commented Sep 8, 2015

@ThomasArdal, I don't think he was referring to the content of your md files, but the encoding of the arguments getting passed from command line or encoding between methods internally.

@d0ugal

This comment has been minimized.

Member

d0ugal commented Sep 8, 2015

Sorry, I was too vague. I am lagging after travelling and was trying to be too lazy 😀

I can't account for why this worked and then didn't. Other than maybe it always works for the first deploy and the issue is to do with the code that is different in subsequent deploys?

The cause of the problem does seem to be internal and not related to your files. Given you are on Windows I suspect it will be related to that.

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Sep 9, 2015

@d0ugal Anyway I can debug what's going on here? Or maybe use some other mkdocs commands to eliminate stuff.

@waylan

This comment has been minimized.

Member

waylan commented Sep 9, 2015

@ThomasArdal looking at the traceback, the error is definitely within the gh-deploy code (therefore other commands won't exhibit the problem). The code builds up a git command as a string, then calls a subprocess passing in that command. The encoding issue appears to be with that command string. I have two things you can check:

  1. What settings, if any, do you have set in your config file which would be used by the deploy script? And what are they set to?
  2. I noticed that mkdocs/utils/ghp_import.py contains from __future__ import unicode_literals. Could you try commenting out that line (line 18)? You will need to reinstall the altered code.

I suspect the improperly encoded string is coming from one of those two places. In the first case, a config setting or in the second case, a default value defined in the code.

@ramramps

This comment has been minimized.

ramramps commented Sep 9, 2015

I tried commenting the line from future import unicode_literals and installed the code. still the same error. Isn't this a showstopper? can someone help to clear this?

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Sep 10, 2015

I guess a workaround is to build the site, make a copy of the output, switch to the gh-pages branch, paste the content and push the changes. But being able to do this with a single command would be way more sweet.

@d0ugal

This comment has been minimized.

Member

d0ugal commented Sep 10, 2015

I spent a bit of time looking into this, the problem I have is that I am not able to reproduce it. We don't have anyone working on MkDocs that uses windows, so tracking these things down can be tricky.

@waylan

This comment has been minimized.

Member

waylan commented Sep 10, 2015

I was able to spend a little bit of time on a Windows box yesterday and was not able to reproduce it there either. Although, that box only had Python 2.7 installed. Afterward, I noticed that the error was reported in Python 3.4...

Which is weird. If the error is in Python 3 then I expect that it would not be related to unicode_literals as that changes the behavior of Python 2 to match Python 3 (mostly). As I understand it, it should have no effect on how Python 3 behaves. I am now thinking that perhaps there is a Windows specific bug in the gh-deploy code (perhaps the upstream package which MkDocs has vendored). But without a way to reproduce it, I'm at a loss of what the bug could be.

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Sep 10, 2015

I tried downgrading to Python 2.7.10 and it turns out that the bug is not there anymore. Success.

I'm able to re-create the bug on a new repository with Python 3.4.x. First time I run a gh-deploy, everything works fine. The second time I run mkdocs gh-deploy --clean I get the same error as previously mentioned.

With Python 2.7.10, I still get a warning though. Seems to be related to the previous bug:

C:\Python27\lib\site-packages\mkdocs\cli.py:86: Warning: Click detected the use of the unicode_literals future import. This is heavily discouraged because it can introduce subtle bugs in your code. You should instead use explicit u"" literals for your unicode strings. For more information see http://click.pocoo.org/python3/
@common_options

@d0ugal

This comment has been minimized.

Member

d0ugal commented Sep 10, 2015

Great, good to know 2.7 works. So I think that suggests we have a Python 3 bug on windows.

That warning should be safe to ignore, it will be disabled in the next release.

@waylan

This comment has been minimized.

Member

waylan commented Sep 10, 2015

Yes, that warning was addressed in #703. However, as you state, the bug doesn't exist in Python 2.7 (where unicode_literals matters) so I think that is a red herring in the case. Just to be sure though, try commenting out/removing the from __future__ import unicode_literals line from each of the following files (when using Python 3.4--I expect it will break things in Python 2.7)):

  • mkdocs.__main__.py (previously mkdocs/cli.py)
  • mkdocs/commands/gh_deploy.py
  • mkdocs/utils/ghp_import.py

I would be surprised if that fixes the issue, but it would be helpful to know either way so we can focus on the actual problem.

@Bruno17

This comment has been minimized.

Bruno17 commented Jan 9, 2016

had exactly that same issue with python 3.5
installed python 2.7.11 and reinstalled mkdocs and the issue has gone

@waylan

This comment has been minimized.

Member

waylan commented Jan 11, 2016

I jut took another look at this. Again, not being able to reproduce it, its difficult to narrow down. In any event, I found this, which looks somewhat related. Interestingly, a comment points to a change in Python 3.2 which caused previously working code to break. While that was regarding gzip, and this is regarding subprocess, its entirely possible that a similar change in 3.5 is causing the problem.

What I know for sure is that something being passed to subprocess is a Unicode string and needs to be a bytestring (but apparently only in Python >= 3.5). Unfortunately, without the ability to replicate it (no access to Python 3 on Windows), I have no way of determining what that is.

If you'd like this fixed, I'd recommend trying to force the pieces passed to the ghp_import tool to bytestring and see if the problem goes away (probably something in mkdocs/commands/gh_deploy.py). Narrow it down to the one string that is causing the problem and I expect we'll have a solution.

@waylan

This comment has been minimized.

Member

waylan commented Jan 12, 2016

To be more precise, I expect one or more of the variables passed in on L68-69 (probably remote_name & remote_branch) needs to be a bytestring.

@thebigmunch

This comment has been minimized.

thebigmunch commented Mar 1, 2016

I believe the problem actually lies with p.communicate() (on Python 3) returning bytes unless the Popen object has universal_newlines=True. The rest of the vars are all str type as expected.

@d0ugal

This comment has been minimized.

Member

d0ugal commented Mar 2, 2016

Interesting. It does use universal_newlines on one of the Popen calls. We can add it to the others. https://github.com/mkdocs/mkdocs/blob/master/mkdocs/utils/ghp_import.py#L144

@waylan

This comment has been minimized.

Member

waylan commented Mar 31, 2016

I pushed a fix to #881. However, as I have no way to test it, we'll need someone who has the problem to apply the patch and verify the problem is fixed before accepting it.

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Apr 1, 2016

@waylan I can test this next week if you'd like?

@waylan

This comment has been minimized.

Member

waylan commented Apr 6, 2016

@ThomasArdal any success with your tests?

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Apr 7, 2016

@waylan Looking at it now.

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Apr 7, 2016

@waylan Doesn't seem like I can checkout the pull request. I've tried checkout out your clone, but I'm not sure what to do to test it. I'm not a Python expert, but would love to help testing this. Could you maybe build a package or show me how?

@waylan

This comment has been minimized.

Member

waylan commented Apr 7, 2016

This command should download and install it for you:

pip install https://github.com/waylan/mkdocs/archive/722.zip
@ThomasArdal

This comment has been minimized.

ThomasArdal commented Apr 7, 2016

@waylan When trying to install, I get the following error:

c:\Python34>pip install https://github.com/waylan/mkdocs/archive/722.zip
You are using pip version 7.0.1, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting https://github.com/waylan/mkdocs/archive/722.zip
  Downloading https://github.com/waylan/mkdocs/archive/722.zip
     - 1.9MB 422kB/s
Exception:
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\pip\basecommand.py", line 223, in main
    status = self.run(options, args)
  File "C:\Python27\lib\site-packages\pip\commands\install.py", line 280, in run

    requirement_set.prepare_files(finder)
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 317, in prepare_
files
    functools.partial(self._prepare_file, finder))
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 304, in _walk_re
q_to_install
    more_reqs = handler(req_to_install)
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 469, in _prepare
_file
    session=self.session)
  File "C:\Python27\lib\site-packages\pip\download.py", line 825, in unpack_url
    session,
  File "C:\Python27\lib\site-packages\pip\download.py", line 677, in unpack_http
_url
    unpack_file(from_path, location, content_type, link)
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 644, in unpac
k_file
    flatten=not filename.endswith('.whl')
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 531, in unzip
_file
    leading = has_leading_dir(zip.namelist()) and flatten
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 246, in has_l
eading_dir
    prefix, rest = split_leading_dir(path)
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 230, in split
_leading_dir
    path = str(path)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 49:
 ordinal not in range(128)
@waylan

This comment has been minimized.

Member

waylan commented Apr 7, 2016

Did you notice the second and third lines:

You are using pip version 7.0.1, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

See if that makes a difference.

Or, if you already have a git checkout, you could just run python setup.py install from that directory.

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Apr 7, 2016

I've done the following, but still didn't manage to test this:

  1. Clone your fork
  2. Checkout branch 722
  3. python setup.py sdist
  4. pip install file://path_to_mkdocs_clone/dist/mkdocs-0.15.3.zip
  5. Headed to my mkdocs projects
  6. mkdocs gh-deploy --clean

Now I get the following error:

c:\docs>mkdocs gh-deploy --clean
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: c:\docs
Traceback (most recent call last):
  File "c:\Python34\Scripts\mkdocs-script.py", line 9, in <module>
    load_entry_point('mkdocs==0.15.3', 'console_scripts', 'mkdocs')()
  File "C:\Python34\lib\site-packages\click\core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python34\lib\site-packages\click\core.py", line 644, in main
    rv = self.invoke(ctx)
  File "C:\Python34\lib\site-packages\click\core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python34\lib\site-packages\click\core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python34\lib\site-packages\click\core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "C:\Python34\lib\site-packages\mkdocs\__main__.py", line 194, in gh_deplo
y_command
    gh_deploy.gh_deploy(config, message=message)
  File "C:\Python34\lib\site-packages\mkdocs\commands\gh_deploy.py", line 54, in
 gh_deploy
    if not _is_cwd_git_repo():
  File "C:\Python34\lib\site-packages\mkdocs\commands\gh_deploy.py", line 16, in
 _is_cwd_git_repo
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "C:\Python34\lib\subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "C:\Python34\lib\subprocess.py", line 1112, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
@waylan

This comment has been minimized.

Member

waylan commented Apr 21, 2016

Hmm, I was just able to test this on Windows 7 with Python 3.5 and could not reproduce the reported problem. I used the following commands to ensure a clean/fresh start (using my personal fork of MkDocs and the MkDocs documentation):

git push origin --delete gh-pages
git branch -D gh-pages

mkdocs gh-deploy --clean

# edit docs/index.md ...

mkdocs gh-deploy --clean

Everything ran fine without applying the changes in #881. Note that I first ensured any pg-pages branches were deleted to real-world-test the first deploy, then after making a change, real-world-test a second deploy.

@waylan

This comment has been minimized.

Member

waylan commented Apr 21, 2016

Okay, I feel dumb now. I was using Python2 in my last comment. Once I forced Python3 (py -3 -m mkdocs ghdeploy --clean) I get the error. And the fix in #881 does not fix the problem completely.

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Apr 21, 2016

@waylan Oh no

waylan added a commit to waylan/mkdocs that referenced this issue Apr 21, 2016

@waylan

This comment has been minimized.

Member

waylan commented Apr 21, 2016

Okay, once i was able to replicate the problem myself, the fix was easy. After I figured it out, then I found davisp/ghp-import#52 which is (mostly) doing the same thing. Once the CI tests pass, I'll commit #904.

@waylan waylan closed this in #904 Apr 21, 2016

@ThomasArdal

This comment has been minimized.

ThomasArdal commented Apr 21, 2016

👍

@sillevl

This comment has been minimized.

sillevl commented Apr 27, 2016

I had the same problem, and I can confirm the bug is solved in the current master (https://github.com/mkdocs/mkdocs/tree/5713289658701b3ab80e5975f9825955403eecfb)

👍 Thanks for solving this. I lost several hours trying to figure out what was wrong.

@JSSU

This comment has been minimized.

JSSU commented Aug 13, 2016

Thanks all! I am use python 2.7 instead of 3.5 and it finally works with my windows 7. Really appreciate you all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment