Don't allow using third-party lexers #168

mroberge · 2020-06-20T04:32:19Z

Describe the bug
PyPI returned an error, "400 The description failed to render for 'text/x-rst'."
Yet my package passes twine check dist/*

Expected behavior
In the past, my README.rst file uploaded just fine. You can see it here https://pypi.org/project/HydroFunctions/

To Reproduce
The package I am trying to upload is located in https://github.com/mroberge/hydrofunctions

the new README.rst file does not upload
the old version 0.1.7 README.rst does upload
I was able to upload hydrofunctions 0.2.0 after I created a simple pypi_readme.rst as my new readme in setup.py

python setup.py sdist bdist_wheel
This produces "hydrofunctions-0.2.2.tar.gz" and "hydrofunctions-0.2.0-py3-none-any-whl".

(myactive_env) C:\mycomputer\PyDev\src\hydrofunctions>twine check dist/*
Checking dist\hydrofunctions-0.2.0-py3-none-any.whl: PASSED
Checking dist\hydrofunctions-0.2.0.tar.gz: PASSED

Everything seems fine and has passed so far.

twine upload dist/*

Tells me it will upload to pypi.org/legacy/
asks for username and password
uploads the wheel file, indicates 100%
Gives an HTTPError: 400 Client Error: The description failed to render for 'text/x-rst'. full error below.

Things I've Tried

specifying long_description_content_type="text/x-rst", AND leaving it out to rely on the default (same results)
deleting the .whl file and only uploading the .tar.gz file. (same results)
simplifying my README.rst:
- not combining it with docs/history.rst in setup.py
- removing a link at the end of README.rst
- removing an unusual unicode character: ⇨
I reverted back to an earlier state that I was able to upload (version 0.1.7) using git checkout SHA.
- I built the distributions, renamed the files "hydrofunctions-0.1.7.1.tar.gz", checked them, and uploaded them successfully.
- ???
- The new README.rst file has a number of new code directives and a link at the bottom of the page, but everything displays fine in Github, readthedocs, and with Sphinx. It also displays in http://rst.ninjs.org/ without errors.

My final solution, which worked, was to create a simplified readme file, 'pypi_readme.rst' for upload as my long_description.

Somehow my .rst works for Sphinx, readthedocs, github, and ninjs.org, but not on PyPI.

My Platform

twine version: 3.1.1
wheel version: 0.34.2
setuptools version: 47.3.0
Python version: 3.7.1
Windows 10

Similar Issues

Additional context
Full Error message:

C:\mycomputer\PyDev\src\hydrofunctions>twine upload dist/* --verbose

Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: mroberge
Enter your password:
Uploading hydrofunctions-0.2.0-py3-none-any.whl
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 346k/346k [00:01<00:00, 317kB/s]
Content received from server:
<html>
 <head>
  <title>400 The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information.</title>
 </head>
 <body>
  <h1>400 The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information.</h1>
  The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>
The description failed to render for &#x27;text/x-rst&#x27;. See https://pypi.org/help/#description-content-type for more information.


 </body>
</html>
HTTPError: 400 Client Error: The description failed to render for 'text/x-rst'. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/

The text was updated successfully, but these errors were encountered:

I filed an issue at the pypa warehouse repo: https://github.com/pypa/warehouse/issues/8144

di · 2020-06-22T15:56:12Z

Can you attach the distributions you're trying to upload here, or put them somewhere I can take a look at them?

mroberge · 2020-06-22T19:03:59Z

I went back to my earlier commit that failed to upload and re-built the dist.

I just figured out that you can see the "long_description" that gets generated by python setup.py sdist by looking in the dist/PKG-INFO file. This has a field, "description", which contains the rst text that PyPI uses as the homepage for each package. The text is padded with 8 spaces at the start of every line, but it is possible to remove those spaces and render it.

When I tried this, it passed both http://rst.ninjs.org/ and the distribution passed twine check dist/*

I've attached my sdist. Github won't take my wheel, but if it helps I can share it from a Google Drive or something.

Thank you!!!
hydrofunctions-0.2.0.tar.gz

mroberge · 2020-06-22T19:13:19Z

There should be some way for users to see the rst file that is generated by the long_description in setup.py
There should be some way for users to see what errors occur when PyPI renders the long_description .rst
Why doesn't twine check check the rst used by PyPI? or maybe it does, but it uses a different rendering engine?
Even with PyPI checking and rejecting malformed rst, many projects have broken links and other rendering problems

==>Maybe PyPI shouldn't reject files due to rst errors. It seems like a good idea at first, but it doesn't fix the problem of projects with messed up homepages, and it creates new problems.

di · 2020-06-22T19:13:34Z

Here's what I get when I run twine check on that distribution:

$ twine check hydrofunctions-0.2.0.tar.gz
Checking hydrofunctions-0.2.0.tar.gz: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 57: Warning: Cannot analyze code. No Pygments lexer found for "ipython".

mroberge · 2020-06-22T19:33:36Z

I just created a new environment with Python 3.7, hydrofunctions, and twine. When I ran twine check on the distribution that I attached above, I didn't get that error.

Do you have any ideas for how I can reproduce this?

The error comes from a change I made in how I display example code in the rst file. The code that works uses the double colon syntax with the greater than signs,

like this::
>>> python code example here

The file that doesn't work uses this notation:

        .. code-block:: ipython
        
            In  [3]: herring.ok
            Out [3]: True

I appreciate your help!

di · 2020-06-22T19:47:50Z

What version of twine do you have? twine --version

mroberge · 2020-06-22T19:49:53Z

Twine 3.1.1

di · 2020-06-22T19:49:58Z

There should be some way for users to see the rst file that is generated by the long_description in setup.py

The "file that is generated" is just whatever you pass as the description field in your setup.py, there's no additional manipulation that happens. The 8 spaces are just due to how setuptools serializes all the metadata.

There should be some way for users to see what errors occur when PyPI renders the long_description .rst

This is what twine check should be doing

Why doesn't twine check check the rst used by PyPI? or maybe it does, but it uses a different rendering engine?

I think this is the source of our issues 🙂

Even with PyPI checking and rejecting malformed rst, many projects have broken links and other rendering problems

Not sure what the suggestion is here, sorry!

di · 2020-06-22T19:51:50Z

OK, how about pygments and docutils?

$ pip show pygments

$ pip show docutils

mroberge · 2020-06-22T19:56:15Z

The environment that I used to look at the distribution I sent you has pygments 2.6.1 & docutils 0.16.
When I filed this Issue I uploaded from an environment that I use for development that has pygments 2.6.1 & docutils 0.14

di · 2020-06-22T20:13:11Z

Hmm, it appears that IPython installs its own lexers:

https://github.com/ipython/ipython/blob/a2685d78f2403f84e6cb915ae11a4f6033ccc5f6/setup.py#L237-L241

Running twine check shouldn't take these into consideration. Moving this to pypa/readme_renderer as a bug.

mroberge · 2020-06-23T01:34:56Z

I figured out why my package failed on PyPI but not when I ran twine check locally.

My package requires Jupyter, which installs it's own lexer, apparently. If I run twine from the same environment that I build my distribution from, twine will pass my rst files. But if I run twine from a bare environment that hasn't installed my package, then twine will fail my distribution.

This will reproduce the issue::

(base) C:\mytest> conda create -n pypi-test python=3.7
(base) C:\mytest> activate pypi-test

(pypi-test) C:\mytest> pip install twine
(pypi-test) C:\mytest> twine check hydrofunctions-0.2.0.tar.gz
Checking hydrofunctions-0.2.0.tar.gz: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 57: Warning: Cannot analyze code. No Pygments lexer found for "ipython".

So, sure enough, I had updated my README.rst file from the last functioning version, to show code examples that look like they were run in Jupyter notebooks:

.. code-block:: ipython

    In  [3]: herring.ok
    Out [3]: True

miketheman · 2022-07-19T18:42:45Z

Took a look at this today, and it appears that passing ipython to code-block doesn't break the experience any longer, rather does not highlight at all, providing a regular <pre> block.

Using twine version 4.0.1, readme_renderer 35.0, I tested the sdist attached, as well as hydrofunctions 0.2.3 (latest release) and everything works with no warnings.

I'm going to assume this was fixed sometime in the past two years and recommend closure.

di · 2022-07-19T18:58:11Z

Closing for now but happy to revisit if this is still reproducible.

mroberge · 2022-07-19T19:23:30Z

Thank you for looking into this!
As you say, ipython code-blocks don't get highlighted at all on pypi, but that is okay. The important thing is that there are no errors!

In previous versions, this example would error, but now emits a code block. If the end user includes a syntax that Pygments does not implement yet, or makes a mistake, the resulting HTML will be valid and succeed, but may not be as pretty as other code blocks. Resolves: pypa#116 Refs: pypa#168 Signed-off-by: Mike Fiedler <miketheman@gmail.com>

mroberge referenced this issue in mroberge/hydrofunctions Jun 20, 2020

Create a simple readme for pypi

e99257c

I filed an issue at the pypa warehouse repo: https://github.com/pypa/warehouse/issues/8144

di transferred this issue from pypi/warehouse Jun 22, 2020

di changed the title ~~"The description failed to render for 'text/x-rst'" BUT package passed twine check dist/*~~ Don't allow using third-party lexers Jun 22, 2020

di added the bug label Jun 22, 2020

di closed this as completed Jul 19, 2022

miketheman mentioned this issue Aug 21, 2022

test: confirm that unknown lexers emit code blocks #256

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't allow using third-party lexers #168

Don't allow using third-party lexers #168

mroberge commented Jun 20, 2020

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020 •

edited

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020

mroberge commented Jun 23, 2020 •

edited

miketheman commented Jul 19, 2022

di commented Jul 19, 2022

mroberge commented Jul 19, 2022

Don't allow using third-party lexers #168

Don't allow using third-party lexers #168

Comments

mroberge commented Jun 20, 2020

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020 • edited

di commented Jun 22, 2020

mroberge commented Jun 22, 2020

di commented Jun 22, 2020

mroberge commented Jun 23, 2020 • edited

miketheman commented Jul 19, 2022

di commented Jul 19, 2022

mroberge commented Jul 19, 2022

di commented Jun 22, 2020 •

edited

mroberge commented Jun 23, 2020 •

edited