Installed black via vim-plug on macOS; starting vim throws ModuleNotFoundError: No module named 'black'

[erikr]

Describe the bug
I get ModuleNotFoundError: No module named 'black' when I run vim, and thus cannot use black in vim on macOS:

$ vim
Error detected while processing /Users/erik/.vim/plugged/black/plugin/black.vim:
line  194:
Traceback (most recent call last):
  File "<string>", line 87, in <module>
ModuleNotFoundError: No module named 'black'
Press ENTER or type command to continue

Surely I am not the only macOS + vim user who wants to use black!

To Reproduce

1. Install vim-plug (link).

2. Add Plug 'psf/black' next to the other plugins in ~/.vimrc.

3. Install via vim followed by :PlugInstall, or via vim -c PlugInstall:

   

4. Run vim from terminal.

Expected behavior
I expect to not get that error, for vim to load, and for me to be able to use black as described in the README.

Environment (please complete the following information):

• Version: whichever is pulled by the default vim-plug installation instructions. Is it not master?
• OS and Python version: macOS 10.15.4, Python 3.7.6 managed by Anaconda
• Vim: 8.2 (installed via Homebrew)

Additional context
@akselsjogren suggested upgrading black used by the vim-plugin to the latest from master:

cd ~/.vim
black/bin/pip install --upgrade git+https://github.com/psf/black.git

However I do not have pip in bin and thus cannot try this:

$ tree ~/.vim/black
.
├── bin
│   ├── python
│   ├── python3
│   └── vim
├── include
├── lib
│   └── python3.8
└── pyvenv.cfg

Also, see #1293 for further discussion and other things I've tried to fix.

[dlukes]

This is the error with which the Black virtualenv setup fails:

Please wait, one time setup for Black.
Creating a virtualenv in /Users/david/.vim/black...
(this path can be customized in .vimrc by setting g:black_virtualenv)
Error detected while processing /Users/david/.vim/plugged/black/plugin/black.vim:
line  195:
Traceback (most recent call last):
  File "<string>", line 87, in <module>
  File "<string>", line 70, in _initialize_black_env
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 391, in create
    builder.create(env_dir)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 68, in create
    self._setup_pip(context)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/venv/__init__.py", line 289, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/david/.vim/black/bin/vim', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Note the last line, the command starts with '/Users/david/.vim/black/bin/vim' instead of '/Users/david/.vim/black/bin/python3'.

I've done a bit of spelunking and I think it's ultimately a bug in newer versions of Vim. I've proposed a workaround in #1380. Can you please try it out and report back whether it was of any help, @erikr?

[dlukes]

I think it's ultimately a bug in newer versions of Vim

More precisely: It turns out it's actually a bug in Vim for which black.vim already has a workaround, which however stopped working because of changes in newer Python versions. Cf. #1380 (comment)

[erikr]

@dlukes Fantastic detective work!

I edited my ~/.vim/plugged/black/plugin/black.vim to match the changes you made in https://github.com/psf/black/pull/1380/files, but still get this error when I launch vim:

Error detected while processing /Users/erik/.vim/plugged/black/plugin/black.vim:
line  196:
Traceback (most recent call last):
  File "<string>", line 89, in <module>
ModuleNotFoundError: No module named 'black'
Press ENTER or type command to continue

Is there something else I need to change? Thanks!

[dlukes]

After editing black.vim, you need to wipe the virtualenv and restart Vim, so that black.vim can re-attempt to create it and hopefully not bail out in the middle of the operation this time :)

[erikr]

Thanks, progress!

$ rm -rf ~/.vim/black
$ vim
Please wait, one time setup for Black.
Creating a virtualenv in /Users/erik/.vim/black...
(this path can be customized in .vimrc by setting g:black_virtualenv)
Installing Black with pip...WARNING: You are using pip version 19.2.3, however version 20.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

DONE! You are all set, thanks for waiting ✨ 🍰 ✨
Pro-tip: to upgrade Black in the future, use the :BlackUpgrade command and restart Vim.
Press ENTER or type command to continue

Installation took a long time (2 minutes), but it solved this particular error.

I think your PR #1380 should be linked to this issue to automate closure.

Now there is a new problem:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 94, in Black
  File "<string>", line 130, in get_configs
AttributeError: module 'black' has no attribute 'find_pyproject_toml'

I thought this error was addressed by #1273. Suggested next move?

[dlukes]

This is the error resulting from incompatible versions of black.vim (the Vim plugin) and black.py (the Python library) we were previously discussing in #1293 -- you have the master version of black.vim, which however installs the latest release (i.e. code that's older than master) of black.py. Since the latest release of black.vim doesn't work with recent versions of Vim, your only option is to update black.py (so that both black.vim and black.py are the same bleeding edge -- master -- version). As per the instructions you provided in the OP:

cd ~/.vim
black/bin/pip install --upgrade git+https://github.com/psf/black.git

[erikr]

Success! Thanks for your patience and clear explanations! I was confused about the difference between release and master.

Should I close this issue or wait for your PR to close it?

[ichard26]

@erikr Waiting for PR is recommended. Unless this issue is unrelated to that pull request.

[dlukes]

I was confused about the difference between release and master.

Sorry for not making that clear enough earlier :) The trouble basically is that Vim plugin managers default to installing from the master branch on GitHub, which for Black happens to be the bleeding edge development version of black.vim, but black.vim then installs black.py from PyPI using pip, which defaults to installing the latest stable release. Sometimes, these different versions might happen to work together, but they also might be incompatible, which is currently the case.

So you need to override the defaults either for the plugin so that both are the latest stable release (that's what the new install instructions recommend), or for the Python library so that both are the latest development version from the master branch on GitHub (that's what running pip install ... git+https://... does).

With recent versions of Vim, only the second option works, because the latest stable release of black.vim is incompatible with them. Note that this also means that when you update the black.vim plugin, you shouldn't upgrade black.py with :BlackUpgrade, but with that pip command instead.

You might want to check back at some point in the future whether a new stable release of Black has been made with support for recent versions of Vim, and if so, consider switching away from the bleeding edge, as it's safer / more convenient (e.g. because you don't need to mess with pip on the command line).

FYI, while tweaking the PR, I also ran into this fun little problem you might encounter with Vim + Python. If you're not too heavily invested in Vim-specific features, maybe it would be worth it to try out Neovim. Admittedly, I've also had my fair share of Python-related troubleshooting with Neovim, but at least it works with Black's latest stable release, and the issues were generally less weird than "sys.executable is vim instead of python3" or "pre-compiled wheels don't work" :) Just make sure that the Python you're using has the pynvim module installed (you can troubleshoot that by running :checkhealth provider from inside Neovim).

[davidjb]

rm -rf ~/.vim/black was literally all I needed - thanks @erikr. Starting Vim again saw the virtualenv created; I was coming from a very old version of the plugin.

[erikr]

@dlukes I now have a new error that is introduced after

cd ~/.vim
black/bin/pip install --upgrade git+https://github.com/psf/black.git

(which still addresses the cannot find pyproject.toml bug)

Now, when I call Black in vim when editing any non-empty .py file, I get a new error:

'<=' not supported between instances of 'int' and 'str'

I reproduced this on Ubuntu 20.04 and macOS 10.15.5, both in Python 3.8.3, Vim 8.1 with +Python3 support, black installed via vim-plug, etc.

Have you seen this? I am not sure how to troubleshoot it so any input is appreciated!

[dlukes]

not sure how to troubleshoot it

Try running ~/.vim/black/bin/black on the same file from the command line. If you get the same error, then it's a bug in black itself, not in its Vim integration.

In that case, it would be great if you could reduce your Python source file to the smallest possible example that triggers the error and file an issue.

That's one of the joys of living at the bleeding edge -- you get to spend more time reporting bugs.

You can try installing progressively older versions of black by appending @master~1, @master~2 etc. to the URL in the pip command (e.g. ~/.vim/black/bin/pip install --upgrade 'git+https://github.com/psf/black.git@master~1'). If you're lucky, the bug has been introduced recently and you'll reach a version which works for you pretty quickly. No guarantees though.

If your Vim config is not too complicated, it might also be worth it to switch to Neovim, as I suggested previously, so that you can simply use stable black (Plug 'psf/black', { 'branch': 'stable' }). Here's how you install Neovim, and here's a config migration guide (for compatible configs, it's as easy as sourcing your .vimrc from Neovim's init.vim).

[erikr]

Try running ~/.vim/black/bin/black on the same file from the command line. If you get the same error, then it's a bug in black itself, not in its Vim integration.

black runs fine itself.

I identified the problem is with pyproject.toml and opened a new issue #1496.

Thanks for the prompt and helpful reply!

[dlukes]

Great, well done!

[xrisk]

I had this problem when I used the stable copy of black.vim. Installing the HEAD copy fixed the bug.

Edit: never mind, now I have the find_pyproject_toml bug 🤦

[Tianyu00]

Just to provide my solution... :

According to my understanding, the black plugin is installed in folder '.vim/plugged/black' (I am using vim-plug)
When freshly installed, '.vim/plugged/black/plugin/black.vim' will create a virtual environment at '.vim/black' (default path) where python3, pip3, and black will be installed
So the message basically says the black failed to be installed in this virtual environment.
So my solution is just to manually create a virtual environment at location '.vim/black' and install black.
So I go to '.vim/black', enter python and do:

import venv
venv.create('.', with_pip=True)

exit python and do bin/pip3 install black

Then the problem seems to be solved. Feel free correct me if something is wrong.

[Tianyu00]

Also, you can add 'print(virtualenv_site_packages)' on line 80 in file '.vim/plugged/black/plugin/black.vim' to see where vim? is looking for the black package and the what version the python should be in the virtual environment.

[davidjb]

Had the same issue again after updating Vim via homebrew (Vim is now built with Python 3.8 vs 3.7 previously) causing the the same ModuleNotFoundError error -- presumably since the venv no longer functions with a now-missing version of Python. I wrote this for my vim-plug configuration:

function! UpdateBlack(info)
  if a:info.status != 'unchanged' || a:info.force
    silent !rm -rf ~/.vim/black
    redraw!
    BlackUpgrade
  endif
endfunction
Plug 'psf/black', { 'do': function('UpdateBlack') }

Now the venv gets recreated if I :PlugUpdate or I can call :PlugUpdate! black to force it. It's somewhat of a hack; :BlackUpgrade (re-)creates the environment if it doesn't exist and we force that situation with an rm.