Error while executing gh-deploy

[ThomasArdal]

I've successfully deployed a MkDocs site using the gh-deploy command. When I try to deploy some additional changes to my master branch, I get the following error:

c:\docs>mkdocs gh-deploy --clean
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: c:\docs\site
INFO    -  Copying 'c:\docs\site' to 'gh-pages' branch and pushing to GitHub.
Traceback (most recent call last):
  File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python34\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\Python34\Scripts\mkdocs.exe\__main__.py", line 9, in <module>
  File "C:\Python34\lib\site-packages\click\core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python34\lib\site-packages\click\core.py", line 644, in main
    rv = self.invoke(ctx)
  File "C:\Python34\lib\site-packages\click\core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python34\lib\site-packages\click\core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python34\lib\site-packages\click\core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "C:\Python34\lib\site-packages\mkdocs\cli.py", line 186, in gh_deploy_command
    gh_deploy.gh_deploy(config, message=message)
  File "C:\Python34\lib\site-packages\mkdocs\gh_deploy.py", line 69, in gh_deploy
    remote_branch)
  File "C:\Python34\lib\site-packages\mkdocs\utils\ghp_import.py", line 163, in ghp_import
    if not try_rebase(remote, branch):
  File "C:\Python34\lib\site-packages\mkdocs\utils\ghp_import.py", line 78, in try_rebase
    if sp.call(cmd) != 0:
  File "C:\Python34\lib\subprocess.py", line 537, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Python34\lib\subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "C:\Python34\lib\subprocess.py", line 1086, in _execute_child
    args = list2cmdline(args)
  File "C:\Python34\lib\subprocess.py", line 663, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: 'str' does not support the buffer interface

[d0ugal]

So, this sounds like we might have hit an issue related to #703 - I think we are passing unicode to an API that expects a bytestring?

[ThomasArdal]

That sounds weird. All of my md files are in UTF-8 and it worked the first time i published the site to gh-pages.

[facelessuser]

@ThomasArdal, I don't think he was referring to the content of your md files, but the encoding of the arguments getting passed from command line or encoding between methods internally.

[d0ugal]

Sorry, I was too vague. I am lagging after travelling and was trying to be too lazy 😀

I can't account for why this worked and then didn't. Other than maybe it always works for the first deploy and the issue is to do with the code that is different in subsequent deploys?

The cause of the problem does seem to be internal and not related to your files. Given you are on Windows I suspect it will be related to that.

[ThomasArdal]

@d0ugal Anyway I can debug what's going on here? Or maybe use some other mkdocs commands to eliminate stuff.

[waylan]

@ThomasArdal looking at the traceback, the error is definitely within the gh-deploy code (therefore other commands won't exhibit the problem). The code builds up a git command as a string, then calls a subprocess passing in that command. The encoding issue appears to be with that command string. I have two things you can check:

1. What settings, if any, do you have set in your config file which would be used by the deploy script? And what are they set to?
2. I noticed that mkdocs/utils/ghp_import.py contains from __future__ import unicode_literals. Could you try commenting out that line (line 18)? You will need to reinstall the altered code.

I suspect the improperly encoded string is coming from one of those two places. In the first case, a config setting or in the second case, a default value defined in the code.

[ramramps]

I tried commenting the line from future import unicode_literals and installed the code. still the same error. Isn't this a showstopper? can someone help to clear this?

[ThomasArdal]

I guess a workaround is to build the site, make a copy of the output, switch to the gh-pages branch, paste the content and push the changes. But being able to do this with a single command would be way more sweet.

[d0ugal]

I spent a bit of time looking into this, the problem I have is that I am not able to reproduce it. We don't have anyone working on MkDocs that uses windows, so tracking these things down can be tricky.

[waylan]

I was able to spend a little bit of time on a Windows box yesterday and was not able to reproduce it there either. Although, that box only had Python 2.7 installed. Afterward, I noticed that the error was reported in Python 3.4...

Which is weird. If the error is in Python 3 then I expect that it would not be related to unicode_literals as that changes the behavior of Python 2 to match Python 3 (mostly). As I understand it, it should have no effect on how Python 3 behaves. I am now thinking that perhaps there is a Windows specific bug in the gh-deploy code (perhaps the upstream package which MkDocs has vendored). But without a way to reproduce it, I'm at a loss of what the bug could be.

[ThomasArdal]

I tried downgrading to Python 2.7.10 and it turns out that the bug is not there anymore. Success.

I'm able to re-create the bug on a new repository with Python 3.4.x. First time I run a gh-deploy, everything works fine. The second time I run mkdocs gh-deploy --clean I get the same error as previously mentioned.

With Python 2.7.10, I still get a warning though. Seems to be related to the previous bug:

C:\Python27\lib\site-packages\mkdocs\cli.py:86: Warning: Click detected the use of the unicode_literals future import. This is heavily discouraged because it can introduce subtle bugs in your code. You should instead use explicit u"" literals for your unicode strings. For more information see http://click.pocoo.org/python3/
@common_options

[d0ugal]

Great, good to know 2.7 works. So I think that suggests we have a Python 3 bug on windows.

That warning should be safe to ignore, it will be disabled in the next release.

[waylan]

Yes, that warning was addressed in #703. However, as you state, the bug doesn't exist in Python 2.7 (where unicode_literals matters) so I think that is a red herring in the case. Just to be sure though, try commenting out/removing the from __future__ import unicode_literals line from each of the following files (when using Python 3.4--I expect it will break things in Python 2.7)):

• mkdocs.__main__.py (previously mkdocs/cli.py)
• mkdocs/commands/gh_deploy.py
• mkdocs/utils/ghp_import.py

I would be surprised if that fixes the issue, but it would be helpful to know either way so we can focus on the actual problem.

[Bruno17]

had exactly that same issue with python 3.5
installed python 2.7.11 and reinstalled mkdocs and the issue has gone

[waylan]

I jut took another look at this. Again, not being able to reproduce it, its difficult to narrow down. In any event, I found this, which looks somewhat related. Interestingly, a comment points to a change in Python 3.2 which caused previously working code to break. While that was regarding gzip, and this is regarding subprocess, its entirely possible that a similar change in 3.5 is causing the problem.

What I know for sure is that something being passed to subprocess is a Unicode string and needs to be a bytestring (but apparently only in Python >= 3.5). Unfortunately, without the ability to replicate it (no access to Python 3 on Windows), I have no way of determining what that is.

If you'd like this fixed, I'd recommend trying to force the pieces passed to the ghp_import tool to bytestring and see if the problem goes away (probably something in mkdocs/commands/gh_deploy.py). Narrow it down to the one string that is causing the problem and I expect we'll have a solution.

[waylan]

To be more precise, I expect one or more of the variables passed in on L68-69 (probably remote_name & remote_branch) needs to be a bytestring.

[thebigmunch]

I believe the problem actually lies with p.communicate() (on Python 3) returning bytes unless the Popen object has universal_newlines=True. The rest of the vars are all str type as expected.

[d0ugal]

Interesting. It does use universal_newlines on one of the Popen calls. We can add it to the others. https://github.com/mkdocs/mkdocs/blob/master/mkdocs/utils/ghp_import.py#L144

[waylan]

I pushed a fix to #881. However, as I have no way to test it, we'll need someone who has the problem to apply the patch and verify the problem is fixed before accepting it.

[ThomasArdal]

@waylan I can test this next week if you'd like?

[waylan]

@ThomasArdal any success with your tests?

[ThomasArdal]

@waylan Looking at it now.

[ThomasArdal]

@waylan Doesn't seem like I can checkout the pull request. I've tried checkout out your clone, but I'm not sure what to do to test it. I'm not a Python expert, but would love to help testing this. Could you maybe build a package or show me how?

[waylan]

This command should download and install it for you:

pip install https://github.com/waylan/mkdocs/archive/722.zip

[ThomasArdal]

@waylan When trying to install, I get the following error:

c:\Python34>pip install https://github.com/waylan/mkdocs/archive/722.zip
You are using pip version 7.0.1, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting https://github.com/waylan/mkdocs/archive/722.zip
  Downloading https://github.com/waylan/mkdocs/archive/722.zip
     - 1.9MB 422kB/s
Exception:
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\pip\basecommand.py", line 223, in main
    status = self.run(options, args)
  File "C:\Python27\lib\site-packages\pip\commands\install.py", line 280, in run

    requirement_set.prepare_files(finder)
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 317, in prepare_
files
    functools.partial(self._prepare_file, finder))
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 304, in _walk_re
q_to_install
    more_reqs = handler(req_to_install)
  File "C:\Python27\lib\site-packages\pip\req\req_set.py", line 469, in _prepare
_file
    session=self.session)
  File "C:\Python27\lib\site-packages\pip\download.py", line 825, in unpack_url
    session,
  File "C:\Python27\lib\site-packages\pip\download.py", line 677, in unpack_http
_url
    unpack_file(from_path, location, content_type, link)
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 644, in unpac
k_file
    flatten=not filename.endswith('.whl')
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 531, in unzip
_file
    leading = has_leading_dir(zip.namelist()) and flatten
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 246, in has_l
eading_dir
    prefix, rest = split_leading_dir(path)
  File "C:\Python27\lib\site-packages\pip\utils\__init__.py", line 230, in split
_leading_dir
    path = str(path)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 49:
 ordinal not in range(128)

[waylan]

Did you notice the second and third lines:

You are using pip version 7.0.1, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

See if that makes a difference.

Or, if you already have a git checkout, you could just run python setup.py install from that directory.

[ThomasArdal]

I've done the following, but still didn't manage to test this:

1. Clone your fork
2. Checkout branch 722
3. python setup.py sdist
4. pip install file://path_to_mkdocs_clone/dist/mkdocs-0.15.3.zip
5. Headed to my mkdocs projects
6. mkdocs gh-deploy --clean

Now I get the following error:

c:\docs>mkdocs gh-deploy --clean
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: c:\docs
Traceback (most recent call last):
  File "c:\Python34\Scripts\mkdocs-script.py", line 9, in <module>
    load_entry_point('mkdocs==0.15.3', 'console_scripts', 'mkdocs')()
  File "C:\Python34\lib\site-packages\click\core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python34\lib\site-packages\click\core.py", line 644, in main
    rv = self.invoke(ctx)
  File "C:\Python34\lib\site-packages\click\core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python34\lib\site-packages\click\core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python34\lib\site-packages\click\core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "C:\Python34\lib\site-packages\mkdocs\__main__.py", line 194, in gh_deplo
y_command
    gh_deploy.gh_deploy(config, message=message)
  File "C:\Python34\lib\site-packages\mkdocs\commands\gh_deploy.py", line 54, in
 gh_deploy
    if not _is_cwd_git_repo():
  File "C:\Python34\lib\site-packages\mkdocs\commands\gh_deploy.py", line 16, in
 _is_cwd_git_repo
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "C:\Python34\lib\subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "C:\Python34\lib\subprocess.py", line 1112, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

[waylan]

Hmm, I was just able to test this on Windows 7 with Python 3.5 and could not reproduce the reported problem. I used the following commands to ensure a clean/fresh start (using my personal fork of MkDocs and the MkDocs documentation):

git push origin --delete gh-pages
git branch -D gh-pages

mkdocs gh-deploy --clean

# edit docs/index.md ...

mkdocs gh-deploy --clean

Everything ran fine without applying the changes in #881. Note that I first ensured any pg-pages branches were deleted to real-world-test the first deploy, then after making a change, real-world-test a second deploy.

[waylan]

Okay, I feel dumb now. I was using Python2 in my last comment. Once I forced Python3 (py -3 -m mkdocs ghdeploy --clean) I get the error. And the fix in #881 does not fix the problem completely.

[ThomasArdal]

@waylan Oh no

[waylan]

Okay, once i was able to replicate the problem myself, the fix was easy. After I figured it out, then I found c-w/ghp-import#52 which is (mostly) doing the same thing. Once the CI tests pass, I'll commit #904.

[ThomasArdal]

👍

[sillevl]

I had the same problem, and I can confirm the bug is solved in the current master (https://github.com/mkdocs/mkdocs/tree/5713289658701b3ab80e5975f9825955403eecfb)

👍 Thanks for solving this. I lost several hours trying to figure out what was wrong.

[JSSU]

Thanks all! I am use python 2.7 instead of 3.5 and it finally works with my windows 7. Really appreciate you all!