New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add markdown support #869

Closed
nlhkabu opened this Issue Dec 29, 2015 · 47 comments

Comments

Projects
None yet
@nlhkabu
Member

nlhkabu commented Dec 29, 2015

I've been working styling the information panel (issue #801).

One thing that I've noticed is that a number of package long descriptions are not coming through with the correct formatting:

screenshot from 2015-12-29 17-40-04
screenshot from 2015-12-29 17-40-15

The first is clearly markdown, the second, I am not sure, but I have noticed a number of other packages using the same syntax.

Obviously, it would be great to render these out to HTML so the new styles will work.

@nlhkabu nlhkabu added the help needed label Dec 29, 2015

@nlhkabu nlhkabu added this to the Become PyPI milestone Dec 29, 2015

@nchammas

This comment has been minimized.

nchammas commented Dec 29, 2015

The first is Markdown (as you pointed out) and the second is reStructuredText.

The current PyPI does not render long descriptions in Markdown. For example, here is the project from your first screenshot on the current PyPI.

There is a related discussion over on the PyPI repo that you should probably look at. It seems there is some stuff that needs to be figured out in order for PyPI to be able to support multiple formats for the long description. Currently, only RST is supported.

Here is the project from your second screenshot on the current PyPI. The long description renders correctly since it's in RST.

@nlhkabu

This comment has been minimized.

Member

nlhkabu commented Dec 29, 2015

Thanks @nchammas - looks like I walked into a minefield on that one :P

So from that thread I take it that Markdown support is unlikely in the short term.

On the RST side - do we have plans to support this (ping @dstufft)? I assume we will want to do this to provide parity with the current site?

@dstufft

This comment has been minimized.

Member

dstufft commented Dec 29, 2015

Hmm, we should be using the exact same code to render descriptions as PyPI is. Sounds like a bug of some sort.

@dstufft

This comment has been minimized.

Member

dstufft commented Dec 30, 2015

Looking at this again, it appears RST rendering is working the same as it does in the legacy site. I think the confusion comes from the fact that our test data comes from TestPyPI not from PyPI so you need to compare to https://testpypi.python.org/pypi/django-excel-response3/ which also has broken display. My guess is they fixed some broken RST and didn't update TestPyPI with the fix.

@dstufft

This comment has been minimized.

Member

dstufft commented Dec 30, 2015

I'm going to drop this from "Become PyPI". We don't currently support Markdown and actually doing so properly is more effort than it's worth to try and get it into our initial launch.

@dstufft dstufft removed this from the Become PyPI milestone Dec 30, 2015

@nlhkabu

This comment has been minimized.

Member

nlhkabu commented Dec 30, 2015

Ok, this makes sense... thanks for the follow up @dstufft

@boxed

This comment has been minimized.

boxed commented Aug 18, 2016

I've recently had a lot of problems getting the rST to work on pypi. Some problems were fixed by running the check command in setup.py, but one error I had didn't show up there: I had the string 'foo__' which is a reference in rST. This error didn't show up when running the check command so I had to do a binary search of the description to find it :P

Clearly some work on improving the error checking on this would be nice, but isn't strictly a part of warehouse although warehouse suffers from it.

@hadim

This comment has been minimized.

hadim commented Jan 29, 2017

Could you elaborate a bit more about why it's a lot of effort to support Markdown ?

A lot of pypi packages are hosted on github where they have a Markdown README file. Maintaining an alternative README file in .rst for pypi seems to be a lot of work for nothing... That is why most of developers (me included) don't bother converting the description field and let it as Markdown.

It follows that a lot of packages does not have a nicely formated page on pypi.

As the future platform for Python packages, don't you think it's worth trying to put some effort in it ?

@hadim

This comment has been minimized.

hadim commented Jan 29, 2017

I just see that some work is undergoing there pypa/readme_renderer#3

@hadim

This comment has been minimized.

hadim commented Jan 29, 2017

Please see pypa/python-packaging-user-guide#258 for a more complete overview.

@Lucas-C

This comment has been minimized.

Lucas-C commented Jun 30, 2017

By reading this thread, it seems to me this feature is pending on the completion of the PR mentioned by @hadim ?
pypa/readme_renderer#3
as this README rendereding lib is used by warehouse: https://github.com/pypa/warehouse/blob/master/requirements/main.in#L37

Is that correct ?

@alonisser

This comment has been minimized.

alonisser commented Jul 1, 2017

+1 fo markdown.. I would like to help with coding if supporting markdown is an option

@9999years

This comment has been minimized.

9999years commented Jul 8, 2017

I’ll also help implement support for Markdown or other languages! I think one of Github’s strengths is strong markup-language support. (Github currently supports nine different languages!)

I think it would make sense to implement the documentation as a subset of HTML rather than any one markup language; one of RST’s big problems is that there’s no support for nested inline styles, such as code links or bold italics, and we could avoid the inflexibilities of any particular markup language by supporting several or a superset of all of them.

@pradyunsg

This comment has been minimized.

Member

pradyunsg commented Jul 8, 2017

pypa/readme_renderer#51 implements Markdown Support and making it easier to add support for more languages in the future (I guess?).

It still needs to be reviewed AFAICT.

@gemerden

This comment has been minimized.

gemerden commented Jul 23, 2017

Ehm, while understanding that feature requests can lead to discussion which can lead to better solutions, i am a bit lost on why this is complicated, especially compared to the value it adds. Seems that find a way to indicate what format to render and render it that way does not have to take 3 years.

I have seen loads of suggestions for solutions (e.g. add long_description_format to setup) that might for some reason or other not be perfect, but all are better than the current state.

My personal experience:

  • oh, pypi does not render markdown,
  • hm, google, maybe there is an automatic converter,
  • read about pypandoc,
  • install pypandoc,
  • change an run setup.py,
  • re-upload to pypi,
  • check doc, hm, mostly ok, but conversion is not perfect,
  • change markdown,
  • 2 or 3 versions further: all good (i think, getting tired of rereading the readme).

And i am pretty sure that pypandoc is the easiest way to go and this took me about 3 hours for a simple project. Multiply by number of uploaders from github ...

Anyway, bit ranty, sorry about that, is there an eta yet?

@dstufft

This comment has been minimized.

Member

dstufft commented Jul 23, 2017

It's Open Source Software, the ETA is that it'll get done whenever someone has to motivation to do it. Someone took the effort to go through and get agreement on the mechanism to indicate what format to render things in, now someone needs to go through and add support for that to setuptools, twine, and Warehouse.

@ghost

This comment has been minimized.

ghost commented Aug 10, 2017

@di

This comment has been minimized.

Member

di commented Mar 16, 2018

With #3296 merged and a pre-release of twine available, it is now technically possible to publish descriptions on PyPI using Markdown. 🎉

For example, see https://pypi.org/p/markdown-description-example/

Here's a step-by-step guide with everything necessary to make this work for your new or existing project: https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi

I'll leave this issue open until @brainwane makes the official twine release is made on Monday. Please feel free to comment here if you try this and are having issues.

@brainwane

This comment has been minimized.

Member

brainwane commented Mar 16, 2018

I'd love for people to start preparing PRs for documentation improvements for the sample project used in the official package distribution guide, and for that guide itself.

@brainwane

This comment has been minimized.

Member

brainwane commented Mar 16, 2018

Also, per Dustin's guide, if you're testing this, make a source distribution (sdist), not a wheel. Wheel support may depend on pypa/wheel#231 being resolved.

@sfdye

This comment has been minimized.

sfdye commented Mar 17, 2018

@di Will this work now if we deploy to pypi from travis?
https://github.com/PyGithub/PyGithub/blob/master/.travis.yml

@9999years

This comment has been minimized.

9999years commented Mar 17, 2018

I’ve created a PR for the sample project (pypa/sampleproject#66) and I’m working on one for the guide as well. For everyone’s reference (from @di’s blog post) twine>=1.11.0 and setuptools>=38.6.0 are required for Markdown support.

@brainwane

This comment has been minimized.

Member

brainwane commented Mar 17, 2018

Thanks for your PR, @9999years! Versions: pkginfo 1.4.2, Twine 1.11.0 (right now 1.11.0rc1 will work), setuptools 38.6.0, and I'm not entirely clear regarding wheel since wheel support does not work yet as of wheel 0.30.0. Dustin's blog post mentions some of those. I am not sure about Python versions (distutils being part of the Python standard library). Twine 1.11.0 requires the relevant version of pkginfo.

There's an open issue for the packaging docs.

@sfdye I see you're creating both a wheel and a source distribution but it looks like you only upload an sdist so I think you'll be fine, but of course I defer to @di here. (I'm experimenting with Markdown support with this test project and Markdown support works if I upload only an sdist, but if I upload a wheel, or an sdist and a wheel, then Markdown support does not yet work.)

Those of you who want to use GitHub-flavored Markdown (beyond what's in the CommonMark specification), please go over to pypa/packaging-problems#126 to find out what's necessary, again, in a few different codebases.

@9999years

This comment has been minimized.

9999years commented Mar 18, 2018

Okay, the guide PR is at pypa/python-packaging-user-guide#457.

@ncoghlan

This comment has been minimized.

Member

ncoghlan commented Mar 18, 2018

For the time being, we don't have any plans for distutils itself to support metadata versions beyond 1.1 - the guidance for a number of years has been to use setuptools instead (regardless of Python version), and this is just one more reason to continue that recommendation.

However, I think the existence of the standard library distutils API implementation is a genuine problem given that we don't really want people to be using it, so I filed pypa/packaging-problems#127 to discuss how we might manage the process of making setuptools the reference implementation of the traditional distutils API.

@di

This comment has been minimized.

Member

di commented Mar 19, 2018

and I'm not entirely clear regarding wheel since wheel support does not work yet as of wheel 0.30.0

Yes, per pypa/wheel#231 we will need to instruct users wanting to build wheels to use an as-of-yet-unreleased version of wheel (working on this now).

@brainwane

This comment has been minimized.

Member

brainwane commented Mar 19, 2018

Twine 1.11.0 is now out and includes some fixes that'll stop users from trying stuff that won't work anymore.

We're leaving this issue open till @di's pypa/wheel#232 is merged & released, so users can upload Markdown READMEs in their wheels.

(You can thank Dustin in person at PyCon in May, where he's speaking and sprinting.)

@brainwane

This comment has been minimized.

Member

brainwane commented Apr 2, 2018

@di

This comment has been minimized.

Member

di commented Apr 2, 2018

Yep! Thanks everyone for your patience w/ this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment