Dependency conflict due to myst-parser~=0.13.5

[rossbar]

Describe the bug

The dependency on myst-parser~=0.13.5 in setup.cfg causes a cascade of dependencies that eventually forces the markdown-it-py version to be pinned to the 0.6 series, which is incompatible with other commonly used tools, e.g. jupytext.

To Reproduce

In a clean environment:

pip install --upgrade myst-nb
pip install jupytext

Which gives:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
myst-parser 0.13.7 requires markdown-it-py~=0.6.2, but you have markdown-it-py 1.1.0 which is incompatible.

It required a bit of manual dependency tracing to figure out that the problem originated here.

Expected behavior

This has already been fixed in master, so I guess the question is: is there a release imminent?

Environment

• Python Version: 3.9.5
• Package versions or output of jupyter-book --version: myst-nb 0.12.3
• Operating System: Linux

Additional context

Note that the current version pinning scheme has caused problems before: #289. Not that I have any suggestion for how it can be improved, just wanted to raise awareness!

[chrisjsewell]

Heya, yeh so indeed possibly there is some way to "help" pip give better feedback 🤷

If you would do e.g. pip install myst-nb jupytext, then pip would install the (older) version of jupytext that uses markdown-it-py 0.6. It is just that pip isn't so good at handling "consecutive" installs.

This is a known thing though, given that I was the one that upgraded jupytext to markdown-it-py v1 lol. You can already use myst-nb==0.13.0a1 and yes we will be releasing myst-nb 0.13 "proper" shortly 🤞

[rossbar]

If you would do e.g. pip install myst-nb jupytext, then pip would install the (older) version of jupytext that uses markdown-it-py 0.6. It is just that pip isn't so good at handling "consecutive" installs.

Unfortunately this can result in something even nastier - there are some situations (we hit them in CI) where this sends pip into a dependency-resolving frenzy that eventually results in timeouts, so this situation is particularly bad for setting up environments programatically.

For a minimal reproducer: clone https://github.com/numpy/numpy-tutorials/ and attempt to install all dependencies simultaneously, e.g. pip install -r requirements.txt -r site/requirements.txt

[astrojuanlu]

It also bothers pip-tools, although this is a bug on their side jazzband/pip-tools#1277

(.venv) juanlu@arion2:~/Projects/RTD/nbs$ cat requirements.in                                                                                            
myst-nb                                                                                                                                                  
(.venv) juanlu@arion2:~/Projects/RTD/nbs$ pip-compile                                                                                                    
Could not find a version that matches markdown-it-py~=0.6.2,~=1.0 (from myst-parser==0.13.7->myst-nb==0.12.3->-r requirements.in (line 1))               
Tried: 0.1.0, 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.5, 0.4.6, 0.4.7, 0.4.7, 0.4.8,
 0.4.8, 0.4.9, 0.4.9, 0.5.0, 0.5.0, 0.5.1, 0.5.1, 0.5.2, 0.5.2, 0.5.3, 0.5.3, 0.5.4, 0.5.4, 0.5.5, 0.5.5, 0.5.6, 0.5.6, 0.5.7, 0.5.7, 0.5.8, 0.5.8, 0.6.0
, 0.6.0, 0.6.1, 0.6.1, 0.6.2, 0.6.2, 1.0.0, 1.0.0, 1.1.0, 1.1.0                                                                                          
Skipped pre-versions: 1.0.0b1, 1.0.0b1, 1.0.0b2, 1.0.0b2, 1.0.0b3, 1.0.0b3                                                                               
There are incompatible versions in the resolved dependencies:               
  markdown-it-py~=1.0 (from mdit-py-plugins==0.2.8->myst-parser==0.13.7->myst-nb==0.12.3->-r requirements.in (line 1))                                   
  markdown-it-py~=0.6.2 (from myst-parser==0.13.7->myst-nb==0.12.3->-r requirements.in (line 1))

(the correct solution is using mdit-py-plugins==0.2.6, which is what pip does thanks to the new dependency resolver)

[chrisjsewell]

Yeh this is already covered in jupyter-book/jupyter-book#1351

[rossbar]

A quick update: this is causing issues again with pip v21.2.1's dependency resolver. See pypa/pip#10201